What you will learn
Tokens, Attention, Context windows, Sampling, KV cache.
Engineering the capability
Tokenization converts text into model-specific units. A word can occupy multiple tokens, and different languages can have different token costs. Budget input, retrieved evidence, tool schemas, and output together. Character count is useful for an application limit but is not a reliable token count.
In a decoder language model, causal attention lets a position use preceding context. Training adjusts weights; inference uses those weights to generate new tokens. Generation usually has a prompt-processing phase and then incremental decoding. A key-value cache reuses attention intermediates during decoding; it consumes memory and does not give the model durable user memory.
Temperature changes the distribution used for sampling. Lower temperature can reduce variation but does not guarantee truth or identical results across systems. Context capacity also does not guarantee that every fact in a long prompt will influence the answer correctly. Test evidence placement and irrelevant material on your own task.
Enforce a request budget
def evidence_budget(context_limit, instructions, question, output_reserve, margin):
values = (context_limit, instructions, question, output_reserve, margin)
if any(value < 0 for value in values):
raise ValueError("Token counts must be nonnegative")
remaining = context_limit - instructions - question - output_reserve - margin
if remaining < 0:
raise ValueError("Mandatory request content does not fit")
return remaining
assert evidence_budget(8000, 700, 300, 1000, 200) == 5800
try:
evidence_budget(100, 70, 30, 20, 10)
except ValueError:
pass
else:
raise AssertionError("An over-budget request was accepted")
Supply counts from the selected model’s tokenizer and account for any additional protocol overhead. This function budgets space; it does not truncate content or ensure that the model uses all evidence effectively.
Worked case
Suppose a model allows a combined 8,000-token budget. Reserve 1,000 for the response, 700 for application and tool instructions, and 300 for the question. That leaves at most 6,000 for evidence and history, before a safety margin. Sending 8,000 tokens of documents leaves no planned space for the answer.
Put it into practice
Continue with the next lab: profile prompt size and answer behavior. Build the artifact, record the failure cases, and explain the tradeoff before moving on.