Python generators do one piece at a time

You need the total of three workshop fees. A list works perfectly. Later, the same job receives millions of records, and someone suggests a generator. The temptation is to replace brackets with parentheses and announce that memory is now solved. A better first step is to watch when the work happens.

A list holds its collected values. A generator produces values as a consumer requests them. It can pause between results and resume from the same place. Python's yield statement creates this behaviour in a generator function. Python tutorial on generators

This lesson assumes you can read a Python function and a for loop. You can follow the printed results without setting up Python; running the snippets is optional.

Make the timing visible

Run this small example:

def fees():
    for amount in (100, 200, 300):
        print("Preparing", amount)
        yield amount

stream = fees()
print("Created")
print(next(stream))
print(sum(stream))
print(list(stream))

The output is:

Created
Preparing 100
100
Preparing 200
Preparing 300
500
[]

Calling fees creates the generator; the print inside its body has not run yet. The first next call requests one value. The sum consumes the two remaining values, so its result is 500 rather than 600. The final list is empty because the stream is exhausted.

This is a useful warning for data work. If you preview a few records by consuming an iterator, a later calculation starts after those records. A preview is not necessarily a harmless peek. Decide whether to create a fresh stream, deliberately include the previewed values, or store a small sample separately.

Keep the whole chain lazy

Here is a complete example using a tiny in-memory text file:

from io import StringIO

def read_fees(lines):
    for line in lines:
        if line.strip():
            yield int(line.strip())

with StringIO("100\n\n200\n300\n") as source:
    total = sum(read_fees(source))

print(total)  # 600

The function handles one line at a time, ignores blank lines, converts each nonblank line, and yields the number. The consumer adds the values without building a list of all converted fees. The in-memory source here is intentionally tiny; for an actual file, open it inside a with block and consume the generator while that file is open.

Avoid returning a generator that depends on a file you have already closed. The delayed work will happen later, when the resource may no longer be available. Also decide how bad input should be handled. The text “unknown” causes int to raise an error when that line is consumed. Silently treating it as zero would change the meaning of the total.

Notice where memory can return

A generator does not make every later operation memory-light. Calling list on the entire stream collects all its results. Sorting an arbitrary full stream also requires retaining data or using an explicit external-sorting strategy. Keeping every processed record in another collection defeats the purpose of streaming.

Iterator helpers can support piece-by-piece processing. For example, itertools.islice can take a limited number of values, but it still consumes them from the underlying iterator. Some helpers buffer data internally, so read their behaviour rather than assuming every iterator has constant memory use. Python itertools documentation

The benefit depends on what the source, transformation, and consumer each retain. Our example avoids a list of converted fees; it does not make a claim about every possible file reader or business calculation.

Try a prediction before running

Create a new fees generator. Call next on it twice, then call sum. What is the sum, and what would another next call do?

The sum is 300 because only the final fee remains. Another next call raises StopIteration unless a default value is supplied. To calculate all three fees again, create a new generator rather than reusing the exhausted one.

Use generators when your task naturally moves forward through items. Use a list when you genuinely need repeated access, indexing, or a small reusable collection. Understanding the trade-off is more useful than treating either choice as automatically better.