Neural Net Problems - Exercise 2

Here are my solutions to exercise 2.

The Architecture of Neural Networks

Question

There is a way of determining the bitwise representation of a digit by adding an extra layer to the three-layer network above. The extra layer converts the output from the previous layer into a binary representation, as illustrated in the figure below. Find a set of weights and biases for the new output layer. Assume that the first $3$ layers of neurons are such that the correct output in the third layer (i.e., the old output layer) has activation at least $0.99$ , and incorrect outputs have activation less than $0.01$ .

Solution

This is converting a base-ten value to base-two value.

I first listed out all the possible numbers.

(9)_{10} (8)_{10} (7)_{10} (6)_{10} (5)_{10} (4)_{10} (3)_{10} (2)_{10} (1)_{10} (0)_{10} = (1001)_{2} = (1000)_{2} = (0111)_{2} = (0110)_{2} = (0101)_{2} = (0100)_{2} = (0011)_{2} = (0010)_{2} = (0001)_{2} = (0000)_{2}

Node One

I noticed that the first thing I could do it solve for the first binary digit. The first binary digit tells us whether or not the number is odd or even. I found that the bias is unnecessary since the dot product will produce either $1$ or $0$ . The weights for the first node are as follows:

I am going to denote the weight due to number in the old output layer, $j$ , $w_{j}$

w_{9} w_{8} w_{7} w_{6} w_{5} w_{4} w_{3} w_{2} w_{1} w_{0} = 1 = 0 = 1 = 0 = 1 = 0 = 1 = 0 = 1 = 0

So if the value is odd the $w \cdot x = 1$ and if even $w \cdot x = 0$ .

Node Two

For node two it should only be activated if the numbers are ${2, 3, 6, 7}$

w_{9} w_{8} w_{7} w_{6} w_{5} w_{4} w_{3} w_{2} w_{1} w_{0} = 0 = 0 = 1 = 1 = 0 = 0 = 1 = 1 = 0 = 0

Node Three

For node three it should only be activated if the numbers are ${4, 5, 6, 7}$

w_{9} w_{8} w_{7} w_{6} w_{5} w_{4} w_{3} w_{2} w_{1} w_{0} = 0 = 0 = 1 = 1 = 1 = 1 = 0 = 0 = 0 = 0

Node Four

For node four it should only be activated if the numbers are ${8, 9}$

w_{9} w_{8} w_{7} w_{6} w_{5} w_{4} w_{3} w_{2} w_{1} w_{0} = 1 = 1 = 0 = 0 = 0 = 0 = 0 = 0 = 0 = 0

Again, there are no biases for any of the nodes.