What you will learn
Sparse and dense retrieval, Hybrid search, Reranking, Recall at k, Grounding.
Engineering the capability
Lexical search is useful for exact product names, error codes, and identifiers. Dense retrieval can match related phrasing when words differ. Hybrid retrieval combines candidate sets, but their raw scores are not necessarily comparable. Rank fusion is one way to combine orderings without assuming identical score scales.
A reranker examines a smaller candidate set more carefully. It can improve ordering but cannot recover a relevant document that the candidate stage never found. Evaluate candidate recall before spending time on answer prompts or reranking.
Keep retrieval evaluation separate from generation. Recall at k measures whether expected evidence is present in the first k results. Answer evaluation checks whether the generated claims are supported. Add access filtering and freshness constraints before evidence enters the model. An impressive similarity score never grants permission to read a document.
Combine ranked candidates
def fuse_rankings(rankings, offset=60):
if offset <= 0:
raise ValueError("Offset must be positive")
scores = {}
for ranking in rankings:
seen = set()
rank = 0
for document_id in ranking:
if document_id in seen:
continue
seen.add(document_id)
rank += 1
scores[document_id] = scores.get(document_id, 0) + 1 / (offset + rank)
return sorted(scores, key=lambda item: (-scores[item], item))
assert fuse_rankings([["a", "b"], ["b", "c"]])[0] == "b"
assert fuse_rankings([["a", "a"], ["b"]]) == ["a", "b"]
The offset controls how strongly rank differences influence the fused score; tune it only on development data. Apply permission filters before fusion. These scores rank evidence candidates and are not probabilities that an answer is correct.
Worked case
A question includes error E417. Dense search retrieves general connection troubleshooting but misses the page containing that exact code. Lexical retrieval finds it immediately. Merge candidates from both methods, then rerank. If all candidates are irrelevant, a reranker still returns an ordered list; the application needs an insufficient-evidence behavior.
Put it into practice
Continue with the next lab: compare retrieval strategies. Build the artifact, record the failure cases, and explain the tradeoff before moving on.