Lab: recover an interrupted agent task

Assignment

Build a three-step sandbox workflow with SQLite checkpoints and a fake remote service that records operation IDs.

Build sequence

1. Persist states and use version-checked transitions.
2. Inject a crash before and after each remote operation.
3. Start two recovery workers for the same task.
4. Add cancellation and document which completed effects cannot be undone.

Acceptance checks

• Recovery does not repeat a completed side effect.
• Only one worker claims the next transition.
• The final state explains partial effects and any required compensation.

Evidence to keep

Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.

Guided workshop: recover a committed local operation

A durable task needs a stable identity and persisted transitions. A crash can occur before a write, after a write, or after the write but before the caller receives the result. Retrying safely requires knowing what actually committed.

Copy code# academy-check
import sqlite3

def apply(db, operation_id, text):
    with db:
        existing = db.execute("SELECT text FROM updates WHERE operation_id=?", (operation_id,)).fetchone()
        if existing:
            if existing[0] != text:
                raise ValueError("operation payload changed")
            return "already_applied"
        db.execute("INSERT INTO updates VALUES (?, ?)", (operation_id, text))
    return "applied"

with sqlite3.connect(":memory:") as db:
    db.execute("CREATE TABLE updates (operation_id TEXT PRIMARY KEY, text TEXT NOT NULL)")
    assert apply(db, "op-1", "Draft answer") == "applied"
    assert apply(db, "op-1", "Draft answer") == "already_applied"
    assert db.execute("SELECT count(*) FROM updates").fetchone()[0] == 1

This single-process example uses a database uniqueness constraint and a transaction. A concurrent implementation must handle competing inserts and reconcile the existing payload; a read-before-insert check alone is not a lock. For a mutation and an operation ledger in the same database, commit them together.

Crash exercise: use a file-backed database, apply one operation, terminate the process, and run it again with the same ID. It should return already_applied. Then retry the same ID with changed text; it must reject the changed payload. Persist the task checkpoint so recovery knows which operation ID to reuse.

Remote-side-effect case: if a ticket API commits an update and the worker crashes before recording success locally, the local ledger cannot prove whether the remote update occurred. Supply the operation ID as an idempotency key if the API supports it, or query and reconcile remote state. Do not claim exactly-once behavior across systems without an end-to-end mechanism.

Checkpoint: draw states for pending, running, waiting for approval, applied, failed, and cancelled. Include version checks on transitions. Replaying a checkpoint should reuse recorded decisions and operation IDs where appropriate, rather than asking the model to invent a new mutation. Demonstrate failures at each boundary and describe the recovery choice.