Assignment
Create a 20-document collection and 15 questions: five exact-code queries, five paraphrases, and five unsupported requests. Label supporting passages first.
Build sequence
- Implement lexical retrieval and a dense or supplied-vector baseline.
- Compare top-three evidence recall and inspect failures by query type.
- Combine candidate rankings and apply a small reranking experiment.
- Generate answers only from authorized evidence and validate citation IDs.
Acceptance checks
- Report candidate recall separately from answer correctness.
- Unsupported questions do not receive confident invented answers.
- The hybrid strategy is justified with case-level evidence and measured extra latency.
Evidence to keep
Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.
Guided workshop: build and measure a search baseline
Retrieval selects candidate evidence; generation uses it to answer. Start with an interpretable lexical baseline so you can inspect failures before adding embeddings, vector storage, hybrid retrieval, and a reranker.
# academy-check
import re
def tokens(text):
return set(re.findall(r"\w+", text.casefold()))
passages = {
"reset": "Reset password from account settings",
"invoice": "Download invoice from billing settings",
"contact": "Contact support for an unsupported request",
}
def retrieve(question, limit=2):
query = tokens(question)
ranked = sorted(((len(query & tokens(text)), key) for key, text in passages.items()),
key=lambda row: (-row[0], row[1]))
return [key for score, key in ranked[:limit] if score > 0]
assert retrieve("reset password")[0] == "reset"
assert retrieve("astronomy") == []
This is token overlap, not semantic search. It may miss “I cannot sign in” even when the password passage helps, and common words can dominate. Record such failures before adding a learned representation. To understand semantic ranking mechanics, compare fixed toy vectors with the cosine function from the math module; then replace those toy vectors with actual embeddings for a measured experiment.
Exercise: create 20 questions with expected evidence IDs. Compute recall at 2 as the proportion of required evidence IDs present in the top two results, averaged under a stated convention. Include questions with no supporting evidence and score those separately. Evaluate lexical, embedding, and combined rankings against the same frozen cases.
Hybrid walkthrough: reciprocal rank fusion assigns a document a score such as 1/(60 + lexical_rank) + 1/(60 + vector_rank) for rankings in which it appears. Ranks start at 1; a missing ranking contributes zero. It combines positions rather than assuming raw lexical and cosine scores are calibrated to the same scale. A reranker then scores a smaller candidate set using the query and passage together, adding latency and cost.
Answer stage: supply retrieved IDs alongside passages and require output citations to reference only that set. Check that the cited passage supports the claim, not merely that the ID exists. For no evidence, return an explicit unsupported state. Retrieved text remains untrusted even if it contains instructions.
Acceptance: keep per-question rankings, quality, latency, and cost; explain where each strategy fails. Choose the simplest method meeting your declared target. Reusing the final test cases repeatedly to tune retrieval turns them into development data.