Lab: compare fixed, routed, and reflective flows

Assignment

Prepare twelve tasks: direct lookup, ambiguous lookup, missing policy, and conflicting evidence. Use identical underlying documents.

Build sequence

1. Implement a fixed workflow and a router with an explicit unknown route.
2. Add one reflection step that receives a claim-support check.
3. Validate any proposed search or tool action before execution.
4. Compare quality and calls used, including cases where reflection makes the result worse.

Acceptance checks

• Unrecognized routes lead to clarification or a safe stop.
• Every revision has a recorded trigger and a bounded budget.
• The report identifies whether reflection improves actual evidence support.

Evidence to keep

Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.

Guided workshop: compare architectures on one task

Use the same 12 support cases for three designs: a fixed search-and-draft flow, a router choosing billing or access search, and a draft-review-revise loop. Keep source documents, output schema, and maximum tool-call budget constant. Otherwise a more expensive architecture may appear better simply because it used more attempts.

A plan is a list of proposed steps with explicit dependencies. Check it before execution:

Copy code# academy-check

def validate_plan(steps):
    known = set()
    for step in steps:
        if step["id"] in known or step["tool"] not in {"search", "draft", "review"}:
            raise ValueError("invalid step")
        if not set(step["after"]).issubset(known):
            raise ValueError("dependency must precede step")
        known.add(step["id"])
    return True

assert validate_plan([
    {"id": "s", "tool": "search", "after": []},
    {"id": "d", "tool": "draft", "after": ["s"]},
])

This validator deliberately accepts only plans in dependency order. It rejects forward references and therefore cycles, but it does not prove that the plan solves the task. The runtime still validates tool arguments, authority, and remaining budget at each step.

Worked reflection rubric: the reviewer checks whether every claim is supported, every citation exists, and the requested question is answered. It returns structured defects such as unsupported_claim with a claim index. Permit at most one revision in this exercise. A vague instruction to “think harder until correct” has no termination or measurable acceptance condition.

Router failure: a mixed billing-and-access ticket cannot be safely forced into one route merely because the model must return a label. Add a review or multi-source route and test ambiguous inputs. Unknown routes must lead to a safe stop, not dynamic execution.

Deliverable: record correctness, supported citations, average and worst-case calls, latency, and failure categories for each design. Explain whether any quality gain survives a matched-budget comparison. ReAct-style action-observation loops, plan-and-execute, and evaluator-optimizer patterns are design choices; none is automatically the right architecture for every task.