A neural network composes parameterized transformations and nonlinear activations. Training uses gradients to adjust parameters. More capacity can represent more complex relationships, but it can also fit noise or memorize examples.
Generalization means performing well on relevant unseen inputs. Split data before fitting learned preprocessing, and keep related records together when they could leak information across splits. A validation set supports development decisions; a protected test set supports a later assessment.
Overfitting appears when training performance improves while held-out behavior worsens. Address it through data quality, appropriate model capacity, regularization, and evaluation design. A larger network does not repair a mislabeled target or an unrealistic test set.
Exercise: design train, validation, and test splits for customer tickets containing duplicates and time-dependent policies. Explain why a random row split may be misleading.
Check: no near-duplicate group crosses the boundary, and the evaluation reflects the intended deployment setting.
Guided workshop: learn a parameter from data
A training procedure adjusts parameters to reduce a loss on examples. Inference uses the learned parameters on a new input. To see this without a library, fit prediction = weight * x to three examples. The squared-error derivative with respect to the weight is 2*x*(prediction-y).
# academy-check
examples = [(1.0, 2.0), (2.0, 4.0), (3.0, 6.0)]
weight = 0.0
for _ in range(100):
gradient = sum(2*x*(weight*x-y) for x, y in examples) / len(examples)
weight -= 0.05 * gradient
assert abs(weight - 2.0) < 0.001
assert abs(weight * 4 - 8) < 0.01
print(round(weight, 3))
The output approaches 2.0. Each update moves against the gradient. A step size that is too large can diverge; a very small one may converge slowly. This toy relationship has no noise and passes through the origin. Real observations may require an intercept, additional features, and careful error analysis.
Exercise: add one corrupted training pair, (3, 30), and observe the fitted parameter and the error on the original relationship. Then put examples into training, validation, and held-out test groups. Use validation for choosing a setting and open the test set only after fixing your choice.
Worked leakage case: tickets from one conversation share phrasing. Randomly splitting individual messages puts nearly identical material in both training and test sets. Split by conversation, customer, or time when that reflects the deployment question. Do not compute normalization statistics across the test set before training.
A classifier predicts a category; a regressor predicts a numeric value. A neural network composes parameterized transformations and nonlinearities, trained using gradients through those transformations. An embedding is a learned representation, not a lookup table that automatically knows your current private data. Supervised learning uses target labels; unsupervised methods find structure without those target labels. Neither replaces a deployment evaluation.
Assessment: explain why low training loss is insufficient, why increasing the number of layers does not fix leaked data, and why an always-majority-class baseline is useful. Save the toy optimizer, a loss trace, your split policy, and one failure analysis before moving to pretrained models.