Python Generators & Iterators

Generators vs. Lists — the Core Distinction

A list computes and stores every value in memory upfront. A generator yields values lazily, one at a time, only when asked — using O(1) memory regardless of how many values it will eventually produce.

squares_list = [x**2 for x in range(1_000_000)]   # computes and stores all 1M values NOW
squares_gen  = (x**2 for x in range(1_000_000))   # computes NOTHING yet — just a generator object

next(squares_gen)   # 0  — computed only now, on demand
next(squares_gen)   # 1

This is the single reason to reach for a generator over a list comprehension: when you only need to iterate once, and don't need random access or the full collection materialized — a generator trades some flexibility for genuinely bounded memory use, regardless of how large the logical sequence is (even infinite sequences are representable as generators).

`yield` and How It Actually Works

yield turns an ordinary-looking function into a generator function. Calling it doesn't run the body at all — it returns a generator object. Each call to next() resumes execution from exactly where it last paused, runs until the next yield, and pauses again — all local variables and the current execution point are preserved between calls.

def countdown(n):
    while n > 0:
        yield n
        n -= 1

for x in countdown(3):
    print(x)   # 3, 2, 1

When a generator function finally runs off the end (no more yields reached), it automatically raises StopIteration — which is exactly what a for loop is built to catch and treat as "iteration complete," so you never see this exception directly in normal usage.

Why this is more than "return but pausable": writing an equivalent class-based iterator manually requires implementing __iter__ and __next__ yourself, and manually tracking state across calls with instance variables (self.n, etc.). A generator function gets all of that — automatic __iter__/__next__, and automatic state preservation of local variables — for free, just by using yield in an otherwise normal-looking function body.

Generator Expressions

The same lazy-evaluation idea, written as a compact expression (parentheses instead of list comprehension's square brackets) for when the generator is consumed immediately by an enclosing function:

sum(i*i for i in range(10))                      # 285 — sum of squares, no intermediate list built
sum(x*y for x, y in zip(xvec, yvec))              # dot product, same idea
unique_words = set(word for line in page for word in line.split())

More memory-friendly than the equivalent list comprehension specifically because no intermediate list is ever materialized — sum() pulls one value at a time from the generator expression and discards it immediately after adding it to the running total.

Context Managers — a Related Use of the Same Mechanism

A context manager guarantees setup (__enter__) and cleanup (__exit__) run even if an exception occurs inside the with block — commonly built two ways:

# Class-based: explicit __enter__/__exit__
class ManagedFile:
    def __init__(self, path):
        self.path = path
    def __enter__(self):
        self.file = open(self.path)
        return self.file
    def __exit__(self, exc_type, exc_val, exc_tb):
        self.file.close()
        return False   # False = don't suppress the exception; let it propagate

with ManagedFile("data.txt") as f:
    data = f.read()

# Generator-based: simpler, using contextlib
from contextlib import contextmanager

@contextmanager
def managed_file(path):
    f = open(path)
    try:
        yield f          # everything before yield = __enter__; everything after = __exit__
    finally:
        f.close()

The generator-based version is a direct application of the same pause/resume mechanism as yield above — code before yield runs as setup, the yielded value is what as f binds to, and code after yield (in the finally block, ensuring it runs even on exception) is the cleanup. This is frequently a cleaner, less boilerplate-heavy way to write a context manager than a full class with explicit dunder methods, for simple setup/teardown cases.

Interview Drills

Why would you choose a generator expression over a list comprehension when summing a large computed sequence? — The generator never materializes the full sequence in memory — sum() consumes one value at a time and discards it — while a list comprehension builds and holds the entire list before summing; for a large or unbounded sequence, this is the difference between O(1) and O(n) memory.
What actually happens when a generator function runs past its last yield with no more values to produce? — It raises StopIteration automatically; a for loop (or anything using the iterator protocol) catches this internally and simply ends the loop — you don't see the exception in normal for-loop usage.
How does the @contextmanager-decorated generator function know what counts as setup vs. cleanup? — Everything executed before the yield statement is treated as __enter__ logic (its yielded value becomes what as x binds to); everything after yield (typically in a finally block) is treated as __exit__ logic, run even if the with block's body raised an exception.

Cross-links