Questions about the Backpropogation Algorithm
- by Colemangrill
I have a few questions concerning backpropogation. I'm trying to learn the fundamentals behind neural network theory and wanted to start small, building a simple XOR classifier. I've read a lot of articles and skimmed multiple textbooks - but I can't seem to teach this thing the pattern for XOR.
Firstly, I am unclear about the learning model for backpropogation. Here is some pseudo-code to represent how I am trying to train the network. [Lets assume my network is setup properly (ie: multiple inputs connect to a hidden layer connect to an output layer and all wired up properly)].
SET guess = getNetworkOutput() // Note this is using a sigmoid activation function.
SET error = desiredOutput - guess
SET delta = learningConstant * error * sigmoidDerivative(guess)
For Each Node in inputNodes
For Each Weight in inputNodes[n]
inputNodes[n].weight[j] += delta;
// At this point, I am assuming the first layer has been trained.
// Then I recurse a similar function over the hidden layer and output layer.
// The prime difference being that it further divi's up the adjustment delta.
I realize this is probably not enough to go off of, and I will gladly expound on any part of my implementation. Using the above algorithm, my neural network does get trained, kind of. But not properly. The output is always
XOR 1 1 [smallest number]
XOR 0 0 [largest number]
XOR 1 0 [medium number]
XOR 0 1 [medium number]
I can never train the [1,1] [0,0] to be the same value.
If you have any suggestions, additional resources, articles, blogs, etc for me to look at I am very interested in learning more about this topic. Thank you for your assistance, I appreciate it greatly!
