5 replies to this topic

[Lariyn]

Hello I am receiving network outputs that are close to always being the same value, but does vary by like 0.001, regardless of network input. If you follow the link I posed a link to download my source code, and posted my training set, and training algorithm.

Here is the link -
http://forum.codecal...html#post262625


I am new to neural networks and i hope to explore them more but I have experienced the same problem in my first two neural network projects.

Edited by Lariyn, 12 July 2010 - 03:53 AM.

[zoranh]

Hi,

I've done the quick inspection of your code. Deltas seem to be calculated correctly, but weight updates seem wrong:

pcNeuralNetwork->ReturnNeuron( i, j ).SetWeight( k, pcNeuralNetwork->ReturnNeuron( i, j ).ReturnWeight( k ) + ( dbLearningRate * pcNeuralNetwork->ReturnNeuron( i - 1, k ).ReturnOutput() * pcNeuralNetwork->ReturnNeuron( i, j ).ReturnDelta() ) );

Why do you add learning rate multiplied by delta? You should be subtracting the value, and that's the point of backpropagation. The algorithm calculates delta as a measure of error made by the node. Then this error is multiplied by learning rate and subtracted from current weight, so that next error should become smaller than it was - this is because sigmoid function is strictly growing, so you should just walk down the error gradient to get to the point where error is around zero.

Please tell if this was the problem. Otherwise, please correct me!

And a couple of advices when programming nerual networks.

• Try training the network with different starting weights, including different ranges of weights. It is hard to tell right from wrong in this area, just keep experimenting and you'll get some sense of it.
  
• Try different algorithms in parallel to backpropagation. For example, simulated annealing has much higher educational value than backpropagation - use it often to gain feeling of how neural networks actually work.
  
• Try different implementations of backpropagation algorithm - there are exotic implementations that are very well grounded in automation theory, etc. As much texts you read in the field, you'll perform better when teaching networks with any algorithm.

[Lariyn]

I greatly appreciate your response. I did not spend much time going through the training algorithm so I am not the least bit surprised that the mathematics are not correct. You sound right on the money when you talk about experimenting, as I am new I need all the experience I can get too. And I will fix the learning algorithm first off, then look into different implementations of backpropagation, and different algorithms as well. I will post back tomorrow with the results my good sir.

[Lariyn]

Works perfectly now. Neural Network has been train for XOR. Even was able to train it to do some multiplication with two numbers between 0.0 - 1.0. After I fixed everything the momentum was still causing problems, commented it out. Ill figure that part out next. anyways late.

[Lariyn]

Here is the code I managed to thankfully complete the training after i fixed momentum.

Artificial Neural Network

[opwuaioc]

I won't pretend to be competent enough to really read this thoroughly, but thanks for sharing!