Training a Neural Network

A network with one hidden layer learns a function from examples \((x_i, y_i)\). Each step of gradient descent moves every weight and bias by \(-\eta\,\partial \mathcal{L} / \partial \theta\). Press Train and watch the curve fit the points, and the loss go down.

Target Points Noise

Hidden neurons Activation Learning rate η

Speed show each neuron

The network

Edge colour: sign of the weight (blue \(+\), red \(-\)); thickness: its size. Green: the parameter that just moved.

One gradient step, parameter by parameter

The data and the network's prediction \(\hat y = f_\theta(x)\)

examples \((x_i, y_i)\) prediction \(f_\theta(x)\) one neuron \(v_j\,\sigma(w_j x + b_j)\): the prediction is their sum, plus \(c\)

The loss \(\mathcal{L}(\theta_k)\) along training

Logarithmic scale: each grid line is 10 times smaller than the one above.

step \(k\) = 0 loss \(\mathcal{L}(\theta_k)\) = gradient size \(\|\nabla \mathcal{L}\|\) = – parameters: