A generator function is defined like a normal function but uses yield instead of return. Calling it returns a generator iterator — an object that produces values one at a time when iterated. The function body is suspended at each yield and resumes from that exact point on the next call to next().
Functions that produce values one at a time
What you'll learn: What yield does (pauses the function and returns a value), the difference between a generator function and a regular function, and why generators use dramatically less memory for large sequences.
How to read this tab: After reading, write a generator that produces Fibonacci numbers infinitely. Use itertools.islice to take the first 20. This single exercise teaches yield, infinite sequences, and islice together.
A regular function runs to completion and returns one value. A generator function pauses at each yield, returns that value, and waits — resuming exactly where it left off when you ask for the next value. Think of it as a vending machine: it dispenses one item at a time, on request, rather than dumping its entire inventory at once.
One yield turns any function into a generator function. Calling it no longer executes the body — it returns a generator object. Nothing in the body runs until you call next() or iterate. This lazy evaluation is both the most useful and the most confusing aspect: add a print() at the top of a generator function, call it, and see that nothing prints. Call next(), and it prints.
What makes a generator
Any function containing a yield statement is a generator function. Calling it does not execute the body — it returns a generator iterator object. The body only runs when you iterate the generator (with next() or a for loop).
def count_up(start, stop):
"""A generator that counts from start to stop."""
n = start
while n <= stop:
yield n # pause here, return n, resume next call
n += 1
# Calling count_up does NOT run the body yet
gen = count_up(1, 5)
print(type(gen)) # <class 'generator'>
# Each call to next() runs until the next yield
print(next(gen)) # 1
print(next(gen)) # 2
print(next(gen)) # 3
# A for loop calls next() automatically until StopIteration
for value in count_up(1, 5):
print(value, end=" ") # 1 2 3 4 5
# The generator version of range — for illustration
def my_range(n):
i = 0
while i < n:
yield i
i += 1The comparison that makes generators click: [x*x for x in range(1_000_000)] allocates ~8MB. (x*x for x in range(1_000_000)) allocates 104 bytes — always, regardless of how many items. Both produce the same results when iterated. For large data, this difference is the difference between a program that works and one that crashes with MemoryError.
Why generators matter — memory
A list comprehension builds the entire list in memory. A generator expression produces values one at a time. For large or infinite sequences, this difference is enormous.
import sys
# List — all million numbers in memory at once
big_list = [x * x for x in range(1_000_000)]
print(sys.getsizeof(big_list)) # ~8 MB
# Generator — one number at a time, constant memory
big_gen = (x * x for x in range(1_000_000))
print(sys.getsizeof(big_gen)) # 104 bytes — always
# Same computation, radically different memory use
print(sum(big_list)) # 333332833333500000
print(sum(big_gen)) # 333332833333500000 — identical result
# Infinite generator — impossible as a list
def natural_numbers():
n = 1
while True: # runs forever — but yields one at a time
yield n
n += 1
# Safe to use with islice
from itertools import islice
first_10 = list(islice(natural_numbers(), 10))
print(first_10) # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]✅ Beginner tab complete
- I know that calling a generator function returns a generator object — it does NOT execute the body
- I know each call to next() runs until the next yield, then pauses
- I understand why generators use O(1) memory regardless of sequence length
- I know generators are single-use — once exhausted, they yield nothing on re-iteration
Pipelines, yield from, and generator composition
What you'll learn: Generator pipelines for memory-efficient data processing. yield from for delegating to another iterable. How to build composable, lazy data transformation chains.
How to read this tab: Build a pipeline: read_lines → filter_empty → parse → process → output. Measure memory use on a large file before and after converting it to a pipeline. The numbers make the concept visceral.
Pipelines are the generator killer feature. Chain generator functions together: read_lines | filter | parse | transform. Each function yields one item at a time to the next stage. Memory use stays constant even for multi-gigabyte inputs. This is how professional Python processes large files — not by loading them into memory.
Generator pipelines
Because generators are lazy, you can chain them together into a processing pipeline where data flows through each stage on demand — no intermediate lists, constant memory regardless of input size.
def read_lines(filename):
"""Generator: yield one line at a time from a file."""
with open(filename, encoding="utf-8") as f:
for line in f:
yield line.rstrip()
def filter_non_empty(lines):
"""Generator: skip blank lines."""
for line in lines:
if line:
yield line
def parse_csv_line(lines):
"""Generator: split each line into a list."""
for line in lines:
yield line.split(",")
# Chain them — nothing runs until we iterate
lines = read_lines("data.csv")
non_empty = filter_non_empty(lines)
rows = parse_csv_line(non_empty)
for row in rows:
print(row) # processes one line at a time, constant memory
# Even with a 10GB file — memory use stays tinyyield from is more than a loop shorthand. It also passes send() values and throw() exceptions through to the inner generator, enables the return value to propagate out, and is the foundation of Python's async/await machinery. Always prefer yield from over a for/yield loop when delegating to another generator.
yield from and generator delegation
yield from (PEP 380) delegates to another iterable or generator, yielding all its values. It also passes send() values and exceptions through to the inner generator, enabling clean composition of generators.
# yield from — delegate to another iterable
def chain_iterables(*iterables):
for it in iterables:
yield from it # yield each item from it in turn
result = list(chain_iterables([1, 2], [3, 4], [5, 6]))
print(result) # [1, 2, 3, 4, 5, 6]
# Flatten nested lists of arbitrary depth
def flatten(nested):
for item in nested:
if isinstance(item, list):
yield from flatten(item) # recursive delegation
else:
yield item
print(list(flatten([1, [2, [3, 4]], [5, 6]])))
# [1, 2, 3, 4, 5, 6]return value inside a generator function raises StopIteration(value) — the value becomes the exception's value attribute, not a regular return value. This is how yield from receives the final value from a delegated generator.✅ Intermediate tab complete
- I can build a multi-stage generator pipeline where each stage is a generator function
- I know yield from delegates to another iterable (or generator) completely
- I can use yield from for recursive flattening of nested structures
- I know return inside a generator raises StopIteration(value)
Generator internals, send(), and the coroutine connection
What you'll learn: How CPython stores a generator's execution state in a frame object on the heap. The send() method for bidirectional communication. How asyncio coroutines are built on top of generators (PEP 342, PEP 380, PEP 492).
How to read this tab: Inspect gi_frame, gi_frame.f_lasti, and gi_code on a generator. Then experiment with send() — prime with next(), then send values in. This reveals the full coroutine protocol generators enable.
Generator internals: frame suspension and the gi_ attributes
When a generator is created, Python allocates a frame object for the generator function — the same structure used for regular function calls, but stored on the heap rather than the stack. The generator iterator holds a reference to this frame. When next() is called, CPython resumes execution in that frame from the last yield point. The frame stores the instruction pointer (f_lasti) and all local variables.
def my_gen():
yield 1
yield 2
g = my_gen()
# Generator attributes (CPython implementation)
print(g.gi_frame) # the suspended frame object
print(g.gi_frame.f_lasti) # -1 before first next()
next(g)
print(g.gi_frame.f_lasti) # byte offset of last executed instruction
print(g.gi_code) # the code object (shared across all instances)
print(g.gi_running) # True only while executing inside next()
# send() — inject a value into the generator at the yield point
def accumulator():
total = 0
while True:
value = yield total # yield sends total OUT, receives value IN
total += value
acc = accumulator()
next(acc) # prime the generator (advance to first yield)
print(acc.send(10)) # 10
print(acc.send(20)) # 30
print(acc.send(5)) # 35PEP 342 (Python 2.5) added send(), throw(), and close() to turn generators into coroutines. PEP 380 (Python 3.3) added yield from for delegation. PEP 492 (Python 3.5) introduced async def/await as native syntax for coroutines — asyncio's coroutines are implemented on top of generators at the CPython level.
✅ Expert tab complete
- I can explain what gi_frame, gi_code, and gi_running contain
- I can write a generator that uses yield as an expression to receive sent values
- I understand that asyncio coroutines are generators under the hood in CPython