A loss function measures how a prediction differs from its target. Training adjusts parameters to reduce loss on training examples. A gradient describes how a small parameter change affects the loss locally; an optimizer uses that information to choose an update.
For a prediction w times x and squared loss (w times x minus y) squared, the derivative with respect to w is 2x(wx minus y). A learning rate scales the update. Too large a step can overshoot; too small a step can make progress slow.
Low training loss alone does not show that a model generalizes. Keep validation behavior separate and compare against a baseline. Training data quality and target definition often matter more than adding model complexity.
Exercise: implement ten gradient updates for a one-parameter linear model on small synthetic data. Record loss after each step and compare two learning rates.
Check: you can derive the update, observe when it improves or diverges, and avoid claiming test performance from training loss.