Warming up the neural circuits...
By the end of this chapter you will:
Most real data processing tasks are not "compute on 20 rows" problems. They are "process millions of records without crashing memory" problems.
Generators solve this by producing values on demand. Instead of building a full list in RAM, they stream one item at a time through your pipeline.
def countdown(n: int):
while n > 0:
yield n
n -= 1
gen = countdown(3)
print(next(gen)) # 3
print(next(gen)) # 2
print(next(gen)) # 1yield pauses function execution and preserves local . The next next() resumes where it stopped.
numbers = range(1_000_000)
eager = [n * 2 for n in numbers] # big list in memory
lazy = (n * 2 for n in numbers) # generator, computed on demand
print(next(
from pathlib import Path
def read_lines(path: Path):
with path.open("r", encoding="utf-8") as f:
for line in f:
yield line.strip()
import tracemalloc
tracemalloc.start()
data = (n * 2 for n in range(500_000))
_ = sum(data)
current, peak = tracemalloc.get_traced_memory()
print(f"current={
def accumulator():
total = 0
while True:
value = yield total
if value is None:
break
total += value
gen = accumulator()
print(next(gen)) # prime generator -> 0
print
Keep pipeline stages pure and narrow: one stage reads, one stage parses, one stage filters, one stage writes. This keeps testing and debugging straightforward.
| Situation | Prefer |
|---|---|
| Need random access / repeated traversal | List |
| Need stream processing over large inputs | Generator pipeline |
| Need immediate materialized snapshot | list(generator) at explicit boundary |
| Need composition of stages | Multiple small generator functions |
| Mistake | Symptom | Fix |
|---|---|---|
| Consuming generator twice | Second pass empty | Recreate generator from source |
| Mixing side effects in every stage | Hard debugging and flaky tests | Keep transforms pure; isolate side-effect sink |
| Turning stream into list too early | Peak memory spikes | Delay materialization to final boundary |
Forgetting to prime before send() | TypeError: can't send non-None value to a just-started generator | Call next(gen) once before first send(value) |
| Assuming generator expression runs immediately | Missing logs/output until iteration | Remember execution starts on iteration only |
Square values from 1 to nGenerator yielding squares lazilyrange(1, 20)2, 4, 6read -> parse -> filter generators for a simple log stream.Lines like INFO:user login and ERROR:db timeoutYield only ERROR linesProcess 200000 transformed numbersGenerator path shows lower peak memoryPipeline feeding both CSV export and top-10 dashboard previewExplain explicit boundaries for materializationBeginner:
"What is the difference between a list comprehension and generator expression?"
List comprehensions are eager and allocate full results immediately. Generator expressions are lazy and compute one value at a time.
"Why can a generator be memory-efficient?"
Because it stores execution state, not the entire output collection.
Senior:
"How would you design a robust streaming pipeline?"
Use small composable generator stages, explicit error handling boundaries, and a single side-effect sink (write/store/log).
"When would you avoid generators even for large data?"
When downstream consumers need random access, multiple passes, or global operations requiring full materialization.
yield preserves execution state and enables incremental computation.Need streaming? -> generator
Need repeatable random access? -> list
Need memory proof? -> tracemalloc
Need multi-stage flow? -> read -> parse -> filter -> sinkWhat is the key behavior of yield in a generator function?
Prefer generator expressions in large read-transform-write workflows.
Each stage consumes and yields lazily, so peak memory stays low.
Use this before and after refactors to validate memory claims.
For most production code, yield and yield from are enough. Use send/throw only when the complexity is justified.