Incremental (Stochastic) Gradient Descent

The stochastic gradient descent algorithm is a simple variation on gradient descent: instead of adding up the gradients over all examples, we update weights incrementally.
That is, change the weights after each example.
For each training example $d$ in $D$ we compute the gradient $\nabla E_{d} [\overset{⇀}{w}]$ and then $\overset{⇀}{w} \leftarrow \overset{⇀}{w} - η \nabla E_{d} [\overset{⇀}{w}]$
We can view this as going down one of the different landscapes (one for each $d$ ) defined as $E_{d} [\overset{⇀}{w}] \equiv \frac{1}{2} (t_{d} - o_{d})^{2}$ at each step.