What you will learn
Parsing, Chunking, Provenance, Deduplication, Incremental updates.
Engineering the capability
Retrieval quality begins before the search call. Parse documents into text with source IDs, section boundaries, version information, and access scope. Preserve headings that explain a paragraph and units that explain a table. A parser that merges two columns can create statements that were never present in the source.
Choose chunks according to document structure and retrieval needs. Overlap may preserve boundary context but also increases duplicates. Record the parser and chunking version so you can reproduce an index and compare strategies.
Treat ingestion as a repeatable pipeline. Stable document IDs and content hashes help identify changes; chunk identifiers must account for content or document version. Plan updates and deletions explicitly. Removing a document from the source repository is not enough if old chunks remain searchable. Keep an ingestion manifest with counts, errors, and rejected files.
Worked case
Policy v1 says refunds take 14 days; v2 says 7 days. Appending v2 without retiring v1 creates contradictory retrieval. Store both for audit if needed, but make the current version filter explicit for ordinary answers. Historical questions can use a separately authorized version-aware query.
Put it into practice
Continue with the next lab: build an incremental document index. Build the artifact, record the failure cases, and explain the tradeoff before moving on.