Introduction
A two year old child is able to do it, but it takes a cluster of no less than 16.000 computers to perform the task of recognizing a cat in a Youtube video. Somehow, our regular algorithms are simply not sophisticated enough to help us with this apparently easy task. The human brain however is very good at solving problems which involve a tremendous amount of input (video data) and recognizing a pattern (like a cat) in it. Besides that, it is able to learn to recognize cats by looking at other videos or images of cats.
In this article, I’ll try to explain what a neural network is and what they can be used for without going into a lot of mathematical details. Hopefully, it’s just enough to get you started creating your own cool stuff like the examples I linked to at the bottom of this article.
Artificial neural networks
The answers to difficult problems are often found in nature. To solve problems similar to the recognize-a-cat problem, scientists are studying the inner workings of our brains for many years now. They found out that the brain contains a huge collection of so called neurons which are wired together in a network structure. The anatomy of a single neuron looks like this:
Dendrites are the sensors of a neuron. They receive inputs from other neurons that will be processed in the soma. The soma then outputs some value through the axon. Because I’m not a biologist I don’t have a clue what really happens, but I’m sure there’s some pretty complex stuff going on in neural networks (NN).
The computer model that is based on this network is called an artificial neural network (ANN). It is not a resemblance of a real neural network, but it does use the same terminology.
Simple optical character recognition (OCR) in javascript
For this article I prepared a demo project written in javascript that makes use of an ANN. Some of the code I already wrote years ago (in Vb.Net), but I translated it to javascript & HTML5 because that makes it easier to display this demo on the web. To go to the demo page directly, click here. To view the source code of this page on Github, click here.
The demo project lets you draw a character (A, B or C) and displays it’s best guess:
You can train the neural network with new characters or improve the recognition of existing characters using the Train methods as explained on the demo page.
The simplest form of an ANN is a so called feed forward neural network. The neurons in such a network are organized in layers. A neural network can have many layers, depending on the requirements. In my OCR example, there are only two layers: input and output. The input layer is fed with the users drawing on the canvas. I downscaled the drawing to a 15 x 15 grid, so I now have 225 input values: a pixel is colored (1) or not (0). The network now calculates which character has the best match with the drawing.
The output neurons (only 3 at this point) receive a zero or one from the input layer on their dendrites (the arrows in the image). Each arrow has a corresponding weight value which is used to determine how important the zero or one actually is. The weights can be changed by training the network, which I’ll show you later on. Every neuron generates an output value by simply adding up all the input values multiplied by their weight. That’s it. No complicated math involved! The output neuron with the highest output value is the winner, because in this example it indicates the character that has the best match with the drawing.
The following image zooms in on the inner working of a single neuron:
The left part of this image represents the sum of all the dendrite values, multiplied by their weights. The right part represents the so called transformation function. This is, in my case, a function that will squeeze any input value to a value between 0 and 1: sigmoid function. Doing this makes sure that each next neuron in the network can determine whether it will ‘activate’ or not. Some ANN’s are programmed to pass on the value (activate) only when it’s value is higher than some threshold, e.g. 0.5. This is not applicable in the OCR demo project.
Here’s all the code of the Neuron class in Brain.js:
//a neuron is the calculation unit and is responsible for generating an output value.
var Neuron = (function () {
//neuron constructor. They have names for easy retrieval.
function Neuron(name) {
this.Name = name;
this.Dendrites = [];
this.AxonValue = 0.5;
}
//generate an output value based on the input values multiplied by the corresponding weights.
//the output value is always between 0 and 1 because of the sigmoid function (http://en.wikipedia.org/wiki/Sigmoid_function)
Neuron.prototype.Think = function () {
var sum = 0;
if (this.Dendrites.length > 0) {
for (var i = 0; i < this.Dendrites.length; i++) {
sum += this.Dendrites[i].SourceNeuron.AxonValue * this.Dendrites[i].Weight;
}
//apply sigmoid function to transform the sum to a value between 0 and 1
//AxonValue is just a sexy name.
this.AxonValue = 1 / (1 + Math.exp(-sum));
}
};
return Neuron;
}
)();
To kick off the networkon, all we have to do is call the Think() method of each Layer.
//make each layer in the network 'think' (generate output values)
Brain.prototype.Think = function () {
for (var i = 0; i < this.Layers.length; i++) {
this.Layers[i].Think();
}
};
Each layer then calls the Neuron’s Think() method.
//make all neurons in the layer generate an output value
Layer.prototype.Think = function () {
for (var i = 0; i < this.Neurons.length; i++) {
this.Neurons[i].Think();
}
};
In my humble opinion, the fun thing of this approach is that there’s isn’t any code involved which has actually has anything to do with character recognition!
Training the network
The only thing left to explain is how the weights on the dendrites get their values. This refers to training the network which is a crucial part of an ANN: it’s ability to learn. Just like a human brain would have, the ANN first needs to have a good understanding of what is considered for example an “A” character. Therefore, the user needs to draw a few good A’s on the canvas first. In a training cycle, the network network will run as usual, but instead of showing the output to the user it calculates the difference (delta) between the output value and the input value. The delta or “error” value indicates how ‘wrong’ the dendrite’s weight is. To correct the weight, it will be increased by the product of the error value, the input value and a constant learning rate. The learning is only there to make sure that a training cycle doesn’t have too much impact at a time. In javascript, this looks like this:
//train an output neuron with some inputdata. The inputdata is considered a good example for the output neuron.
Brain.prototype.Train = function (inputData, outputNeuron) {
//no layers, no glory
if (this.Layers.length === 0) {
return;
}
//fill the first layer with input data to feed the network
var inputLayer = this.Layers[0];
for (var i = 0; i < inputData.length; i++) {
inputLayer.Neurons[i].AxonValue = inputData[i];
}
//generate output for the given inputs
this.Think();
//adjust weights using the delta
//the generated output is compared to the training input: the drawing in this case.
//the subtraction is the error which will be corrected by adjusting the weight.
var delta = 0;
var learningRate = 0.01;
for (var i = 0; i < outputNeuron.Dendrites.length; i++) {
var dendrite = outputNeuron.Dendrites[i];
delta = parseFloat(Math.max(inputData[i], 0) - outputNeuron.AxonValue);
dendrite.Weight += parseFloat(Math.max(inputData[i], 0) * delta * learningRate);
}
}
Cool examples
You’ve made it to the end of this crash course in ANN’s. Now you’ve seen all the important aspects of ANN’s that you need to get started building your own mind blowing software ;-). Please take a look at the links below to see some nice examples of neural networks.
The development branch 