Here are my solutions to exercise 6.

The Backpropagation Algorithm

Part 1 - Backpropagation with a Single Modified Neuron

Question

Suppose we modify a single neuron in a feedforward network so that the output from the neuron is given by $f({\sum }_j{w}_j{x}_j+b)$, where $f$ is some function other than the sigmoid. How should we modify the backpropagation algorithm in this case?

Solution

We first would have to calculate the derviate for the function $f$ since it is needed for the backpropagation output error vector ${\delta }^L$ and ${\delta }^l$. But other than that the neural network does not need any tweaking. You may think that it needs tweaking because ${\delta }_j^l$ is dependent on $f$ but we have defined ${\delta }_j^l=\frac{\partial C}{\partial z_j^l}$ and ${\delta }_j^l\ne \frac{\partial C}{\partial a_j^l}$. We did this because it makes our lives easier for this particular case!

From Michael Nielsen:

You might wonder why the demon is changing the weighted input $z_j^l$. Surely it'd be more natural to imagine the demon changing the output activation $a_j^l$, with the result that we'd be using $\frac{\partial C}{\partial a_j^l}$ as our measure of error. In fact, if you do this things work out quite similarly to the discussion below. But it turns out to make the presentation of backpropagation a little more algebraically complicated. So we'll stick with ${\delta }_j^l=\frac{\partial C}{\partial z_j^l}$ as our measure of error.

Part 2 - Backpropagation with Linear Neurons

Question

Suppose we replace the usual non-linear $\sigma$ function with $\sigma (z)=z$ throughout the network. Rewrite the backpropagation algorithm for this case.

Solution

Since $\sigma (z)=z$ then ${\sigma }^{\prime }(z)=1$, so it follows that ${\delta }^L={\nabla }_{a}C\circ {\sigma }^{\prime }(z^L)={\nabla }_{a}C$. Also, ${\delta }^l=((w^{l+1}{)}^T{\delta }^{l+1})\circ {\sigma }^{\prime }(z^l)=(w^{l+1}{)}^T{\delta }^{l+1}$.