Backpropagation

Backprop is the chain rule on a graph. Forward: compute each node. Backward: each node receives \(\partial\mathcal{L}/\partial\text{output}\) and multiplies by its local derivative to send \(\partial\mathcal{L}/\partial\text{input}\) to its parents. Weights get \(\partial\mathcal{L}/\partial W\) and take a step \(W\leftarrow W-\eta\,\partial\mathcal{L}/\partial W\).

Do a two-layer scalar example on paper before you trust autograd. If the loss is NaN, you probably exploded a multiply or took \(\log 0\). If nothing learns, the learning rate or the data scale is wrong — not “the math of backprop.” Pair with the neural-nets lesson. This sheet is not a reprint of the appendix.