Learn the concept
An iterator produces values one at a time and is consumed as you advance it. A generator function uses yield to produce an iterator without materializing every result at once. This helps process large inputs incrementally.
A consumed generator cannot simply be reused from the beginning. Create a new generator when you need another pass. Streaming also changes error timing: a failure may occur during iteration rather than when the generator was created.
Batching controls memory and downstream request size. Handle the final partial batch, empty input, and invalid batch size. A generator reduces the need to hold all output in memory, but any operation that converts it to a list or sorts the entire stream can reintroduce that cost.
Run and inspect
def batches(items, size):
if size < 1:
raise ValueError("Size must be positive")
batch = []
for item in items:
batch.append(item)
if len(batch) == size:
yield batch
batch = []
if batch:
yield batch
assert list(batches(range(5), 2)) == [[0, 1], [2, 3], [4]]
assert list(batches([], 2)) == []
Your exercise
Use the generator to process ten thousand synthetic document IDs in batches without constructing a second complete list.
Check your understanding
The last partial batch is retained, invalid sizes fail, and you can explain when the generator’s work actually runs.
Guided workshop: process input without materializing everything
A generator produces one item at a time. Its function body starts when you request the first item, and execution pauses at each yield. It is useful when reading a file larger than memory, but downstream code can accidentally undo the benefit by collecting everything into a list.
# academy-check
def batches(items, size):
if size <= 0:
raise ValueError("positive size required")
batch = []
for item in items:
batch.append(item)
if len(batch) == size:
yield batch
batch = []
if batch:
yield batch
assert list(batches(range(5), 2)) == [[0, 1], [2, 3], [4]]
assert list(batches([], 2)) == []
Rebinding batch after yielding creates a fresh list for the next batch. Clearing the same list with batch.clear() would also clear a list the consumer still holds. The final if batch emits a partial batch and avoids an empty extra batch when the input length is divisible by the size.
Exercise: create a generator that takes document records and yields only those with nonempty text. Keep a rejected counter outside the generator by returning rejection records as explicit outcomes, or put counting in the consuming loop. Avoid assuming a generator has a usable len().
Worked trace: for input texts "alpha", "", and "beta", the consumer sees two accepted records and one rejection if you represent every outcome. A filter that drops empty strings silently cannot produce an accurate audit report unless it separately counts them.
Quiz: when does batches([], 0) raise? When iteration starts, since the function contains yield. Why not reuse the same generator for two reports? A generator is consumed; recreate it or persist the outcomes. What memory is bounded here? The batch buffer, not the input if the caller already constructed a huge list, and not the final results if the caller uses list().