Lab: profile prompt size and answer behavior

Assignment

Use a local tokenizer if available, or a deterministic token-count fixture for budget logic. Model-quality experiments can use a configured model separately.

Build sequence

1. Implement budget accounting with reserved output and an explicit truncation policy.
2. Create short, long, and multilingual inputs; keep character and token counts separate.
3. Place the same evidence near the beginning and end of long contexts and record answer support.
4. Record time to first output and total response time independently.

Acceptance checks

• Budget checks reject requests that cannot fit mandatory instructions and the response reserve.
• The report distinguishes measured model results from simulated timing or token fixtures.
• A larger context is recommended only when its measured benefit justifies latency and cost.

Evidence to keep

Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.

Guided workshop: budget context and observe attention limits

A tokenizer maps text to model-specific token IDs. The context budget covers input and generated output under the provider's rules. Characters divided by four is, at best, a rough English estimate; it is not a validation rule for arbitrary languages, code, or images.

Begin with a numerical budget independent of any SDK:

Copy code# academy-check
def evidence_budget(window, instructions, question, history, output, reserve):
    remaining = window - instructions - question - history - output - reserve
    if remaining < 0:
        raise ValueError("request exceeds context budget before evidence")
    return remaining

assert evidence_budget(8000, 600, 200, 1200, 1000, 500) == 4500

The 4,500 remaining tokens are available for retrieved evidence under this exercise's accounting assumptions. The reserve handles message framing and estimation uncertainty. Confirm the actual accounting contract for the selected model; do not assume every provider treats these terms identically.

Experiment: choose ten questions with known source passages. Run three prompt variants with the same model and decoding settings: minimal supporting context, supporting context plus irrelevant passages, and context near your permitted budget. Keep the supporting passage near the beginning in half the runs and near the end in the others. Record model identifier, prompt hash, input and output usage, latency, and correctness.

Worked interpretation: if adding irrelevant material increases input usage and decreases supported-answer accuracy, a larger prompt is hurting this workload. It does not prove that the model always ignores long contexts. Repeat across questions and separate unsupported answers from truncation or transport failure.

Explain the intuition of attention: tokens build context-dependent representations by weighting information from other permitted positions. That mechanism does not guarantee that a model follows the intended source hierarchy, retains every fact, or treats an embedded instruction as harmless data. Application boundaries must enforce permissions separately.

Checkpoint: implement a preflight rejection when the reserved output plus required input cannot fit. Never silently truncate a source ID away from its passage. Your report must distinguish measured provider token counts from estimates and preserve the exact prompt versions used. A local fake can test budget arithmetic; the answer-behavior experiment requires actual model runs and must remain labelled pending until performed.