Learn the concept
Calling an async function creates a coroutine object. Its work runs when awaited or scheduled. Async I/O allows other tasks to progress while one waits; it does not make CPU-bound work automatically parallel.
Use bounded concurrency to avoid overwhelming a downstream service. A semaphore can limit simultaneous operations within one process. Shared limits across several processes require additional coordination. Define deadlines and propagate cancellation rather than swallowing it as an ordinary recoverable error.
Separate waiting time, execution time, and total task time when measuring performance. A task can spend most of its budget in a queue before the actual model request begins. The example uses a fake I/O wait and needs no network service.
Run and inspect
import asyncio
async def run():
limit = asyncio.Semaphore(2)
async def work(value):
async with limit:
await asyncio.sleep(0)
return value * 2
return await asyncio.gather(*(work(n) for n in range(4)))
assert asyncio.run(run()) == [0, 2, 4, 6]
Your exercise
Instrument a shared counter and prove that no more than two tasks run concurrently. Add a failing task and document your cancellation policy.
Check your understanding
The limit is demonstrated by observed counts, not inferred from total runtime alone.