Assignment
Run two local workers over a durable queue of ten synthetic tasks, including one that waits for approval and one that exceeds its budget.
Build sequence
- Implement task claim, heartbeat or lease, cancellation, and terminal states.
- Enforce a shared per-user concurrency limit.
- Restart a worker while a task is paused and resume with its recorded version.
- Prepare a demo, trace walkthrough, and recovery runbook for the final portfolio.
Acceptance checks
- A task cannot execute concurrently under two valid claims.
- Budget and cancellation stops are visible to the caller.
- The runbook explains recovery for pending, failed, and uncertain-effect states.
Evidence to keep
Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.
Guided workshop: claim work with a version check
A queue decouples request acceptance from long-running execution. The caller receives a task ID and can inspect status. Workers claim tasks, record progress, and publish terminal states. A crash requires lease expiry or another explicit recovery mechanism.
# academy-check
import sqlite3
with sqlite3.connect(":memory:") as db:
db.execute("CREATE TABLE jobs (id TEXT PRIMARY KEY, state TEXT, version INTEGER)")
db.execute("INSERT INTO jobs VALUES ('j1', 'queued', 0)")
def claim(expected):
with db:
cursor = db.execute("UPDATE jobs SET state='running', version=version+1 WHERE id='j1' AND state='queued' AND version=?", (expected,))
return cursor.rowcount == 1
assert claim(0)
assert not claim(0)
The conditional update prevents two successful claims of the same queued version. A production worker needs a claim owner, lease expiry, heartbeat, and fencing token if expired workers might continue running. A lease timeout alone does not stop an old process from making a remote call.
Build steps: add queued, running, waiting-for-approval, completed, failed, and cancelled states. Store remaining steps and cost budget. Check cancellation before each new expensive operation. Use the recovery lab's operation IDs so retrying a task does not duplicate already-applied effects.
Exercise: run two workers against a temporary file-backed SQLite database and prove that only one claims the initial task version. Kill the claimant, expire its lease under a controlled clock, and let another worker resume. Deliver a late result from the first worker and reject it by claim version.
Worked observability: expose queue depth, age of the oldest queued job, completion and failure counts, retry count, and budget exhaustion. Logs should connect task ID, attempt ID, and operation ID. A single average latency chart will not explain whether a task waited in a queue or stalled inside a tool.
Assessment: demonstrate quotas, cancellation, worker crash, duplicate delivery, and unavailable storage. Declare a dead-letter or manual-review policy after a bounded number of attempts. The runbook should explain how to inspect a stuck task and resume it without granting an unlimited budget or repeating a customer-visible action.