Neural Network (Part 2) : The Neuron

The Neuron

The main purpose of the Neuron is to store the values that flow through the neural network. They also do a bit of processing through the use of an Activation function. The activation function is quite an important feature of the neural network. It essentially enables the neuron to make decisions (yes/no and greyzone) based on the values provided to them. The other feature of activation functions is the ability to map values of infinite size to a range of 0 to 1. We will be using the Sigmoid function.

The above pictures can be found at Wikipedia

The neuron accumulates signals from one or more connections, adds a bias value, and then sends this value through the activation function to produce a neuron output. Each neuron output will be within the range of 0 to 1 (due to the activation function).

So lets take a look at  a single neuron with 2 connections (+ bias) feeding into it.

As seen below.

connEntry                                                       weight                                             

Neuron.Connection1.connEntry = cEn1           Neuron.Connection1.weight = cW1

Neuron.Connection2.connEntry = cEn2           Neuron.Connection2.weight = cW2

Neuron.Bias.connEntry = bEn = 1 (always)                Neuron.Bias.weight = bW

connExit                                             

Neuron.Connection1.connExit = cEx1

Neuron.Connection2.connExit = cEx2

Neuron.Bias.connExit = bEx

You need to multiply the connEntry value by the weight to get the connExit value.

Connection #1:                      

         cEx1 = cW1  x   cEn1

Connection # 2:                      

          cEx2 = cW2  x  cEn2

Bias:                                       

                      bEx = bW x 1

As you can see from the Bias calculation, the Bias.connEntry (bEn) value is always 1, which means that the Bias.connExit (bEx) value is always equal to the Bias.weight (bW) value.

The Neuron Input Value is equal to the sum of all the connExit values (cEx1 and cEx2 in this case) plus the bias (bEx), and can be represented by the following formula.

Neuron Input Value:                                                                                                    

Neuron Input Value = cEx1 + cEx2 + bEx      

 or

Neuron Input Value = [cW1 x cEn1] + [cW2 x cEn2] + [bW x 1]

Now that we have the Neuron Input Value, we can put this through the Activation Function to get the Neuron Output Value. This can be represented by the following formula.

Neuron Output Value:                                                                                                  

Neuron Output Value = 1 / (1 + EXP(-1 x Neuron Input Value))

We will use this Neuron Output value as an input to the next layer, therefore it automatically becomes a connEntry for one of the neuron connections in the next layer.

Now that we know what a neuron does, let us create a neuron class. See the code below.

processing code Neuron Class

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

/* ------------------------------------------------------------------------

   A neuron does all the calculations to convert an input to an output

-------------------------------------------------------------------------  */



class Neuron{

  Connection[] connections={};

  float bias;

  float neuronInputValue;

  float neuronOutputValue;

  float deltaError;

  

  //The default constructor for a Neuron

  Neuron(){

  }

  

  //The typical constructor of a Neuron - with random Bias and Connection weights

  Neuron(int numOfConnections){

    randomiseBias();

    for(int i=0; i<numOfConnections; i++){

      Connection conn = new Connection();

      addConnection(conn);

    }

  }

  

  //Function to add a Connection to this neuron

  void addConnection(Connection conn){

      connections = (Connection[]) append(connections, conn);

  }

  

  //Function to return the number of connections associated with this neuron.

  int getConnectionCount(){

      return connections.length;

  }

  

  //Function to set the bias of this Neuron

  void setBias(float tempBias){

    bias = tempBias;

  } 

  

  //Function to randomise the bias of this Neuron

  void randomiseBias(){

    setBias(random(1));

  }

  

  //Function to convert the neuronInputValue to an neuronOutputValue

  //Make sure that the number of connEntryValues matches the number of connections

  

  float getNeuronOutput(float[] connEntryValues){

    if(connEntryValues.length!=getConnectionCount()){

      println("Neuron Error: getNeuronOutput() : Wrong number of connEntryValues");

      exit();

    } 

    

    neuronInputValue=0;

    

    //First SUM all of the weighted connection values (connExit) attached to this neuron. 

    for(int i=0; i<getConnectionCount(); i++){

      neuronInputValue+=connections[i].calcConnExit(connEntryValues[i]);

    }

    

    //Add the bias to the Neuron's inputValue

    neuronInputValue+=bias;

    

    //Send the inputValue through the activation function to produce the Neuron's outputValue

    neuronOutputValue=Activation(neuronInputValue);

    

    //Return the outputValue

    return neuronOutputValue;

  } 

  

  //Sigmoid Activation function

  float Activation(float x){

    float activatedValue = 1 / (1 + exp(-1 * x));

    return activatedValue;

  }

  

}

code formatter

When you create a new Neuron, you have the option to create the basic neuron shell with the Neuron() constructor, however, you are more likely to call the 2nd constructor, which allows you to set the number of input connections attached to this neuron.

For example, if I create a neuron using New Neuron(4), then this will automatically

• attach and associate four connections with this neuron


• randomise the weight of each connection


• randomise the bias of this neuron

These are the functions within the Neuron Class:

• addConnection():  adds a new connection to this neuron


• getConnectionCount(): returns the number of connections associated with this neuron


• setBias(): sets the Bias of the neuron to a specific value.


• randomiseBias() : randomises the Bias of this neuron


• getNeuronOutput() : values are fed through the neuron's connections to create the neuronInputValue, which is then sent through the Sigmoid Activation function to create the neuronOutputValue.


• Activation() :  is the function used by the getNeuronOutput() function to generate the neuronOutputValue.

Up next:  Neural Network (Part 3)  : The Layer

To go back to the table of contents click here