Learn the concept
An iterator produces values one at a time and is consumed as you advance it. A generator function uses yield to produce an iterator without materializing every result at once. This helps process large inputs incrementally.
A consumed generator cannot simply be reused from the beginning. Create a new generator when you need another pass. Streaming also changes error timing: a failure may occur during iteration rather than when the generator was created.
Batching controls memory and downstream request size. Handle the final partial batch, empty input, and invalid batch size. A generator reduces the need to hold all output in memory, but any operation that converts it to a list or sorts the entire stream can reintroduce that cost.
Run and inspect
def batches(items, size):
if size < 1:
raise ValueError("Size must be positive")
batch = []
for item in items:
batch.append(item)
if len(batch) == size:
yield batch
batch = []
if batch:
yield batch
assert list(batches(range(5), 2)) == [[0, 1], [2, 3], [4]]
assert list(batches([], 2)) == []
Your exercise
Use the generator to process ten thousand synthetic document IDs in batches without constructing a second complete list.
Check your understanding
The last partial batch is retained, invalid sizes fail, and you can explain when the generator’s work actually runs.