Build an HTTP API that answers questions from a small collection of public documents. A web or Android client is optional. Use documents you can redistribute and keep model credentials on the server.
Scope and architecture
Start with 10–20 documents about one product. Store a stable document ID, source URL, and text for each document. Split text into passages, retrieve relevant passages for a question, and ask the model to answer using only that evidence. Return an answer, citations, and an explicit unsupported-answer state.
Request: {"question":"How do I reset my workspace?"}. Response: {"answer":"…","sources":[{"id":"guide-3","url":"…"}],"supported":true}. Validate this structure before returning it. A schema-valid response can still be factually wrong; evaluate the answer separately.
Build sequence
- Write a basic retrieval baseline and inspect the top passages for five questions.
- Add answer generation and preserve source identifiers end to end. Never accept invented citation IDs.
- Create 30 evaluation questions: 15 supported, 10 unsupported, and 5 ambiguous. Record expected evidence before tuning prompts.
- Add authentication, request limits, timeouts, and safe error responses. Avoid logging private question text by default.
- Deploy an endpoint and record model/version, retrieval settings, latency, and approximate usage cost.
Acceptance checks
- Every returned source exists in the supplied evidence and actually supports its associated claim.
- Unsupported questions trigger an explicit fallback rather than a fabricated answer.
- Empty input, a model timeout, and an unavailable retrieval service return controlled responses.
- Retrieved text containing instructions to ignore the user cannot change the application’s permissions or reveal credentials.
- Report supported-answer accuracy and unsupported-question handling separately. Choose a target before the final run, record all failures, and explain tradeoffs against the retrieval-only baseline.
Deliverables
Submit a repository with setup instructions, document provenance, a sample request, an evaluation table, and a short deployment runbook. The runbook should describe configuration, secret management, troubleshooting, and how to disable the endpoint. Explain one change that improved quality and one that increased cost without enough benefit.
This is an independent project specification. The linked guided reference teaches the request path; your own deployment and evaluation evidence is still required.
Guided workshop: assemble and defend your release
Use Build a knowledge assistant end to end as a reference for the connected request path. Implement your own corpus, evaluation cases, and deployment choices. The reference includes a fixture-only HTTP service and an optional real-provider adapter; neither substitutes for your capstone evidence.
Organize the repository into application code, fixtures, tests, evaluation results, and operational documents. A reviewer should find one setup command, one deterministic test command, one optional live-evaluation command, and a clearly labelled deployment procedure. Live commands must state their credential and cost requirements.
| Milestone | Build from course material | Evidence required before continuing |
|---|---|---|
| 1. Import | Python CLI + incremental index | Reimport, update, deletion, malformed-record tests |
| 2. Retrieve | Lexical baseline + measured alternative | Frozen queries, expected evidence, recall and latency |
| 3. Answer | Provider adapter + output validator | Supported, unsupported, refused, and malformed cases |
| 4. Protect | Verified principal + scoped retrieval | Two-tenant negative tests and cache-isolation checks |
| 5. Evaluate | Scorecard + held-out dataset | Versioned quality, failures, cost, and decision |
| 6. Operate | Deployment manifest + runbook | Health, dependency outage, disable, restore, rollback |
Worked integration decision: keep the canonical document ID and version from ingestion through retrieval, citations, and the evaluation report. If one layer invents a new unrelated ID, tracing a wrong answer back to its source becomes difficult. Preserve that identity across adapters rather than repairing it at the UI.
Independent challenge: replace one policy after release, then prove the next answer uses the new version and an old cached answer is not returned. Interrupt the replacement midway and show that users see a consistent active version. Explain the tradeoff between freshness and availability.
Assessment answers: a schema-valid answer can still be ungrounded; a permitted document can still contain a malicious instruction; a passing offline suite does not measure a real model; a container image does not back up its data. Your demonstration must address all four distinctions.
Score each milestone 0 for absent, 1 for a happy-path demonstration, or 2 for reproducible success and failure evidence. Require at least 10 of 12 points, with 2 each for protection and operations, under this course rubric. Any unauthorized disclosure or unreconciled duplicate mutation blocks completion regardless of total score. A peer should reproduce the result without access to your editor session.