B is "the target function is invariant to small rotations
of the character in the image". How do we express this
mathematically?
Define a transformation $s(\alpha, x)$ which rotates $x$ by $\alpha$ degrees. Then say
\[ \frac{\partial f(s(\alpha,x_i))}{\partial \alpha} = 0 \]
Then incorporate these into the error function. How?
Add an additional term to the error function
\[ E = \sum_i \left[ (f(x_{i}) - \hat{f}(x_{i}))^{2} + \mu_{i} \sum_{j}
\left(\frac{\partial f(s_j(\alpha, x_i))}{\partial \alpha} - \frac{\partial \hat{f}(s_j(\alpha, x_i))}{\partial{\alpha}} \right)_{(\alpha=0)}^{2} \right] \]
then modify the gradient descent rule to use this. How?