Gradient Descent Landscape

• Gradient $$\nabla E\left[\stackrel{\rightharpoonup }{w}\right]\equiv [\frac{\partial E}{\partial w_0},\frac{\partial E}{\partial w_1},\cdot \; \cdot \; \cdot \frac{\partial E}{\partial w_n}]$$ Training rule: $$\Delta \stackrel{\rightharpoonup }{w}=-\eta \nabla E\left[\stackrel{\rightharpoonup }{w}\right]$$ i.e., $$\Delta w_i=-\eta \frac{\partial E}{\partial w_i}$$
• The gradient specifies the direction that produces the steepest increase in $E$ (so, negate it).

José M. Vidal .

12 of 33