Lab: build an incremental document index

Assignment

Prepare six local documents with headings, one table, one duplicate, and two versions of a policy. Include a deletion event in a second ingestion run.

Build sequence

1. Create a manifest containing source ID, version, hash, access scope, and parse status.
2. Compare fixed-size and heading-aware chunks on five questions.
3. Re-run unchanged ingestion and prove it does not duplicate records.
4. Update a policy and remove a document; inspect the resulting searchable chunks.

Acceptance checks

• Every chunk can be traced to a source location and version.
• Deleted or superseded content is excluded from current-policy queries.
• Parse failures are visible in the manifest rather than silently indexing empty text.

Evidence to keep

Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.

Guided workshop: make updates and deletions observable

A retrievable passage needs stable identity, a document version, source provenance, tenant scope, and text. Reindexing should replace a document's previous passages rather than append another indistinguishable copy. Perform visibility changes atomically when using persistent storage.

Copy code# academy-check
import hashlib

index = {}
def replace_document(tenant, document_id, text):
    version = hashlib.sha256(text.encode()).hexdigest()[:16]
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks = [{"id": f"{document_id}:{version}:{n}", "text": p, "version": version}
              for n, p in enumerate(paragraphs)]
    index[(tenant, document_id)] = chunks
    return chunks

first = replace_document("a", "guide", "Old policy\n\nContact support")
second = replace_document("a", "guide", "New policy")
assert len(index[("a", "guide")]) == 1
assert first[0]["version"] != second[0]["version"]
del index[("a", "guide")]
assert ("a", "guide") not in index

This in-memory replacement illustrates identity and visibility, not durability. For a database-backed index, write a new version to staging, verify all chunks and embeddings, then switch the active-version pointer in one transaction. A partially completed embedding job should not expose half an old document and half a new one.

Exercise: extend your Python importer with a content digest, provenance URL, and deleted state. Import the same content twice and skip unnecessary embedding work. Change one paragraph, then verify that queries only return the current active version. Delete the document and ensure both retrieval results and answer caches stop exposing it.

Worked failure: if embedding fails after chunk 2 of 5, leave the old active version available and record the new version as failed. A retry can resume or rebuild staging. Do not advertise the new version as ready because its metadata row exists.

Chunking review: start with paragraphs and inspect passages that are too long or depend on a preceding heading. Add a size limit and controlled overlap if needed. Preserve source offsets or page identifiers for citations. Measure retrieval on questions spanning chunk boundaries instead of choosing chunk size by intuition alone.

Checkpoint: submit a before/after table of active versions, document counts, chunk counts, and deletion checks. Verify tenant-scoped IDs, empty files, duplicate IDs, malformed text, and interrupted imports. A deletion receipt needs to identify which stores and caches were cleared and any remaining retention window.