Back to overview
Matrix Product
Neural Net Problems - Exercise 4
March 26, 2020

Here are my solutions to exercise 4.

Implementing Our Network to Classify Digits

Part 1


Write out a=σ(wa+b)a'=\sigma(wa+b) in component form, and verify that it gives the same result as the rule, 11+exp(jwjxjb)\frac{1}{1+\exp(- \sum_j w_j x_j - b)}, for computing the output of a sigmoid neuron.


Let it be stated that I have not yet taken linear algebra at college, so I have very limited experience with it.

Let us say that layer 2 has 22 nodes and layer 1 has 33 nodes.

The weights from layer 1 to layer 2 can be expressed as the following (wjiw_{ji}, where jj is the neuron in the second layer and ii is the neuron in the first layer):

w=[w11w12w13w21w22w23]a=[a1a2a3]b=[b1b2]w = \begin{bmatrix} w_{11} & w_{12} & w_{13}\\ w_{21} & w_{22} & w_{23} \end{bmatrix} a = \begin{bmatrix} a_1\\ a_2\\ a_3 \end{bmatrix} b = \begin{bmatrix} b_1\\ b_2 \end{bmatrix}
wa=[w11a1+w12a2+w13a3w21a1+w22a2+w23a3]wa+b=[(w11a1+w12a2+w13a3)+b1(w21a1+w22a2+w23a3)+b2]a=σ(wa+b)=[σ((w11a1+w12a2+w13a3)+b1)σ((w21a1+w22a2+w23a3)+b2)]\begin{gathered} wa = \begin{bmatrix} w_{11} a_1 + w_{12} a_2 + w_{13} a_3\\ w_{21} a_1 + w_{22} a_2 + w_{23} a_3 \end{bmatrix}\\ wa + b = \begin{bmatrix} (w_{11} a_1 + w_{12} a_2 + w_{13} a_3) + b_1\\ (w_{21} a_1 + w_{22} a_2 + w_{23} a_3) + b_2 \end{bmatrix}\\ a' = \sigma(wa + b) = \begin{bmatrix} \sigma((w_{11} a_1 + w_{12} a_2 + w_{13} a_3) + b_1)\\ \sigma((w_{21} a_1 + w_{22} a_2 + w_{23} a_3) + b_2) \end{bmatrix} \end{gathered}

This is exactly the same as 11+exp(jwjxjb)\frac{1}{1+\exp(- \sum_j w_j x_j - b)} but computed for both neurons at once with matrices!

Say we wanted to compute the output of the first sigmoid neuron in layer 2.

a1=σ(jwjaj+b)=σ((w1a1+w2a2+w3a3)+b)a_1' = \sigma(\sum_j w_j a_j + b) = \sigma((w_1 a_1 + w_2 a_2 + w_3 a_3) + b)

Also, I wanted to note that I found this website called the ml cheatsheet and it has been really useful in describing the mathematic concepts.

The header image was taken from Khan Academy.