Python Iterators and Generators: How Lazy Evaluation Changes the Way Programs Work

Opening snapshot: you have a text file with ten million log lines and one minute to produce an aggregated value. Do you create a giant list, or do you process lines as they arrive?

Experiment series: the same task, three approaches

I’ll use one concrete task throughout: read integers from a source, filter those that are odd, square them, then sum the first 1,000 results. We’ll start with a list, move to a built-in iterator approach, and finish with a generator function. This progressive experiment makes the trade-offs visible.

Experiment 1 — Eager lists (the straightforward, memory-heavy way)

First, the common beginner attempt using a list comprehension:

with open('numbers.txt') as f:
    nums = [int(line) for line in f]

odd_squares = [n * n for n in nums if n % 2]
result = sum(odd_squares[:1000])
print(result)

What happens here:

• The file is read line-by-line when the comprehension runs, but because we wrap into a list, every integer is stored in memory in nums.
• We then build a second list, odd_squares, which duplicates storage during processing.
• If numbers.txt contains millions of entries, memory usage spikes. The code is easy to reason about, but it trades memory for simplicity.

This approach is perfectly fine for small data sets, but it breaks when your dataset doesn’t fit comfortably in RAM.

Experiment 2 — Iterators and lazy built-ins

Next, avoid the intermediate lists and rely on built-in iterators. Python’s file objects are already iterators over lines; functions like map() and filter() also return iterator-like objects in Python 3.

def parse_ints(file_path):
    with open(file_path) as f:
        for line in f:
            yield int(line)

ints = parse_ints('numbers.txt')  # ints is an iterator
odd = filter(lambda n: n % 2, ints)  # lazy filter object
squares = map(lambda n: n * n, odd)  # lazy map object
# take first 1000 and sum
from itertools import islice
result = sum(islice(squares, 1000))
print(result)

Notes:

• No list is materialized. Values are produced only as needed — this is lazy evaluation.
• islice stops after 1,000 values, so you never process the entire file if you don’t need to.
• The built-ins map and filter return iterator objects; see Python’s iter() and next() docs for the iterator protocol.

Experiment 3 — Generator functions with yield

A generator function uses yield to produce values lazily. You get the same memory benefits but often with clearer logic for sequential transformations.

def odd_square_stream(file_path):
    with open(file_path) as f:
        for line in f:
            n = int(line)
            if n % 2:
                yield n * n

# Use the generator directly
stream = odd_square_stream('numbers.txt')
from itertools import islice
result = sum(islice(stream, 1000))
print(result)

Key points about yield:

• A function that contains yield returns a generator object. The generator implements the iterator protocol.
• On each call to next(), execution resumes until the next yield. See the language reference for yield expressions: Python docs: yield expressions.

State diagram (described and shown)

Prose state diagram: Imagine a flow of tokens. The source (file) hands a token to an Iterable. When we call iter() (explicitly or implicitly in a loop), we get an Iterator that holds execution state: which line was last read, any local variables, and where to continue. Calling next() advances the iterator-producing mechanism, returns a value, and preserves the execution point for the next call. When the source is exhausted, next() raises StopIteration, and the iterator is done.

Iterable / Generator

Iterator (stateful)

iter() next()

value

The SVG above represents the same flow you’d visualize mentally: source → iterable → iterator → values. The iterator holds the execution frame for generator functions and keeps track of progress for file objects and other iterables.

Decision matrix: when to choose which approach

Practical decisions and patterns

Here are practical heuristics you can use while coding:

• If the dataset is small and you need random access or multiple passes, a list is fine.
• If you only need sequential access or can short-circuit (e.g., first N items), prefer iterators or generators.
• Prefer generator functions when the transformation logic is sequential and has multiple statements — they read like a step-by-step recipe and are easier to debug than chained map/filter.
• When composing many small transforms, use generator expressions or the itertools helpers to keep the pipeline lazy and efficient.

See practical examples using list comprehensions, for-loops, and how functions shape control flow in Python functions. If this is part of a larger project, remember environment-specific differences: check your virtual environment so dependencies and Python version are consistent.

Debugging lazy pipelines: common pitfalls and fixes

Lazy evaluation changes the point at which errors appear. These are common surprises and how to fix them:

Problem: “It worked once, then the generator is empty”

Generators and iterators are single-pass. After consuming them, they are exhausted. Example:

g = (n*n for n in range(10))
print(sum(g))   # consumes all
print(next(g))  # StopIteration here

Fix: Either recreate the iterator/generator when you need a fresh pass, or collect results into a list if you genuinely need multiple passes.

Problem: Silent StopIteration bubbling from a nested generator

When using yield from or composing multiple generators, unhandled StopIteration can terminate your pipeline earlier than you expect. Use tests and explicit checks, or make sure your generators yield exactly what downstream consumers expect. Refer to the iterator protocol: iter() and next().

Problem: Heavy computation in generator holds resources

If your generator opens resources (a DB cursor or file) and yields in the middle, ensure proper cleanup. The simplest pattern is to manage resources with context managers inside the generator or to create a wrapper that yields values and closes resources on exit.

def resource_stream(path):
    with open(path) as f:
        for line in f:
            yield process(line)
# file is closed when generator completes or is garbage-collected

Measuring performance: quick checklist

• Use timeit or a small benchmark harness to compare approaches on realistic data.
• Measure both peak memory and runtime — sometimes lazy pipelines are slightly slower but dramatically reduce memory.
• Avoid converting to list just to peek at contents; use itertools.islice or itertools.tee (with caution) instead.

Visual reasoning: why lazy evaluation feels different

Lazy pipelines shift when and where work happens. Instead of building up state up-front, each consumer-driven call to next() pulls a value through the pipeline. Visualize it as back-pressure: the consumer asks for one item, the generator produces one item, and resources are only used long enough to produce that item. This reduces peak memory and often simplifies streaming logic.

FAQ

State, ownership, and the cost of pausing

The most useful mental model is not “a generator saves memory.” It is “a generator owns a position in a conversation.” The caller requests a value; the generator runs until it can provide one; then control returns to the caller. That makes the boundary explicit, but it also means the caller owns the pace. A fast producer can be held back by a slow consumer, while exceptions may appear only when a later value is requested.

That state has consequences for API design. A function that returns a list promises a completed collection. A function that returns a generator promises a process that can still fail, pause, or be exhausted. Name the distinction in the function’s documentation and tests. If callers need to retry, decide whether retrying means creating a new generator or replaying the original source. A file, socket, or database cursor may not be replayable without a new resource.

def batches(items, size):
    batch = []
    for item in items:
        batch.append(item)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch

This batching generator illustrates a subtle point: yielded values can remain alive after the generator moves on because the consumer still references them. Lazy production reduces intermediate work, but it does not guarantee low memory if the consumer stores every yielded result.

A reliable test for a lazy function

Test both timing and behavior. Create the generator and assert that the source has not yet been consumed; then request one item and assert the first observable effect; finally exhaust it and verify the stopping behavior. This catches accidental eager work and makes the one-pass contract visible. For a broader discussion of project-owned environments and reproducibility, connect this experiment with Vandutz’s guide to virtual environments.

Sources used for this explanation

• Python language reference — yield expressions
• Python built-in: iter()
• Python built-in: next()

Alex Carter

Alex Carter is the editorial name behind Vandutz Academy, a programming blog for beginners. Alex reviews and tests the examples and explanations published on the site, with a focus on making Python, JavaScript, web development, and developer tools easier to understand.