Assignment
Create 60 synthetic tickets, including 12 urgent examples and 10 near-duplicate pairs. Keep a separate group identifier for related records.
Build sequence
- Write a keyword classifier and a majority-class baseline.
- Split by group and document why a naive random split leaks information.
- Compute the confusion matrix, precision, and recall without using the held-out cases to tune keywords.
- Inspect five mistakes and propose a new feature or data collection change.
Acceptance checks
- No group appears in both development and test data.
- Metric calculations match hand-counted true positives, false positives, and false negatives.
- The report states the sample size, baseline comparison, and cost of missed urgent tickets.
Evidence to keep
Submit the working artifact or decision document described above, the inputs used, and the observed results for every acceptance check. Include one failed attempt and the change you made after investigating it. Label fixture-based outcomes separately from live-system measurements. These artifacts become part of your final portfolio review.
Guided workshop: compare with a baseline you can explain
Create six synthetic tickets with labels fixed before writing a classifier. A keyword rule provides a transparent starting point. A future model must justify its cost and complexity against this baseline on held-out data.
# academy-check
cases = [
("invoice missing", "billing"), ("charged twice", "billing"),
("password reset", "access"), ("cannot log in", "access"),
("thanks", "review"), ("invoice and password issue", "review"),
]
def classify(text):
billing = any(word in text for word in ("invoice", "charged"))
access = any(word in text for word in ("password", "log in"))
if billing == access:
return "review"
return "billing" if billing else "access"
predictions = [classify(text) for text, _ in cases]
assert predictions == [label for _, label in cases]
The final ticket matches both categories and is sent to review. That is a policy decision about ambiguity. It is different from forcing a label and calling the result confident. Six handpicked examples passing is a smoke test, not evidence of broad performance.
Independent exercise: collect or author 30 additional synthetic tickets before changing the rule. Include spelling variants, negation, unrelated messages, multiple intents, and vocabulary missing from the keywords. Split them into development and held-out groups. Preserve conversation groups when splitting.
Worked error analysis: “I was not charged; I cannot access my account” contains charged but does not describe a billing problem. The rule fails because matching a token ignores meaning and negation. Adding one exception may improve a development example while harming another. Write the failure category before choosing a fix.
Compare accuracy, per-category precision and recall, and review rate. A system that routes everything to review may avoid incorrect automatic assignments while offering little automation. Record the human workload as well as correctness. For a rare urgent category, overall accuracy can hide unacceptable misses.
Assessment: propose a learned classifier or embedding-based approach only after listing the baseline's failure modes. Explain what training labels, data split, and inference dependencies it needs. The portfolio artifact should contain frozen cases, predictions, a confusion table, and a reasoned decision to retain or replace the baseline. Do not present fixture performance as measured customer performance.