\hypertarget{NeuralNet_Acknowledgements}{}\subsection{Acknowledgements}\label{NeuralNet_Acknowledgements}
All code in this neural net package was written by David Bailey of the University of Manchester.\hypertarget{NeuralNet_AssumedKnowledge}{}\subsection{Assumed knowledge}\label{NeuralNet_AssumedKnowledge}

\begin{DoxyItemize}
\item Standard Template Library (S\-T\-L) vectors.
\end{DoxyItemize}\hypertarget{NeuralNet_Remarks}{}\subsection{Remarks}\label{NeuralNet_Remarks}

\begin{DoxyItemize}
\item All neural net classes are in the namespace {\ttfamily nnet}.
\item This is by no means a complete guide to every feature available, at present at least.
\end{DoxyItemize}\hypertarget{NeuralNet_BasicPrinciplesOfAnArtificialNeuralNetwork}{}\subsection{Basic principles of an artificial neural network}\label{NeuralNet_BasicPrinciplesOfAnArtificialNeuralNetwork}
This is a very basic introduction to the principles of a neural network (geared specifically at the way this package works). If you have any experience with neural networks you can safely skip this section.\par
 \par
 Neurons are created to accept an arbitrary number of inputs, and based on these provide a single output value. The output is given by the neurons $ threshold function $, which can be any given function of the neurons $ activation value $ (see the \hyperlink{NeuralNet_NeuronDescriptions}{Neuron Descriptions} for the functions actually provided with this package).\par
 The activation value is given by multiplying each input by a pre calculated $ weight $ depending on how important that input is, and summing these results. Each neuron can also be given a bias, depending on how important that neuron is to the network, but more on that later.\par
 Calculating these weights is the important part, and is what differentiates a well performing network from a bad one. This process is known as $ training $, and is performed by a training algorithm (see \hyperlink{NeuralNet_TrainingTheNetwork}{Training the network} for the algorithms provided here). Basically, you provide the training algorithm with a set of data that you know the answers to (the result you would want the network to give you), and it changes the weights to give the best possible results for all the elements in the data set.\par
 \par
 As a basic example, imagine a network composed of a single neuron that tells you if a food is bad for you or not. Say it is set up with three inputs, fibre content, fat content and colour. For simplicity, lets give the neuron a linear threshold function, so just a function that multiplies the activation value by a set constant, say $ k $. The output of the network would be \[ output=f(activation value)=k\times (fibrecontent\times weight_{fibre}+fatcontent\times weight_{fat}+colour\times weight_{colour}) \] The network is useless until the values of the weights are adjusted so that they give an accurate output. To do this, a large database of foods is required where the properties of colour, fibre and fat content are known, as well as some reliable value as to how healthy the food is. The training algorithm then modifies the weights to try and get the best match of the output to the expected value for each food in the database. When somebody comes along with a new food, its properties can be put into the network and a (hopefully) reliable value as to how healthy it is pops out the other end.\par
 Ideally, once trained, the weight given to colour will be zero since that is completely irrelevant (ignoring artificial colourings). However, if the training sample has just a few blue foods, which just happen to be bad for you, then the training algorithm will wrongly ascribe a high weight.\-to the colour input. Also, if the training sample foods have pretty similar fat and fibre contents, but are radically differently healthy (say, maybe due to salt content), then the training algorithm will probably be unable to make any sense of the sample, and give useless weights. This emphasises the need to select a large and varied training sample (as well as setting up the network with meaningful inputs in the first place).\par
 \par
 Realistically, a network will be composed of many neurons so that all 'cross effects' between the inputs are taken into account (where a weighting for one input needs to depend on other inputs as well). Here, the network would be built up with layers of neurons where the input for each neuron in a layer is the output from each neuron in the layer before. The final layer would have just one neuron, so that you get just one output for the network.\hypertarget{NeuralNet_CreatingAndTraining}{}\subsection{Creating and training a new neural net}\label{NeuralNet_CreatingAndTraining}
The method used to create a new network varies slightly depending on the algorithm used to train it. Sections \hyperlink{NeuralNet_BuildingTheNeuronLayers}{Building the neuron layers} and \hyperlink{NeuralNet_CreatingTheNetwork}{Creating the network} describe how to setup the network ready for training, which is common to all training algorithms. The {\ttfamily Batch\-Back\-Propagation\-Algorithm}, {\ttfamily Back\-Propagation\-C\-G\-Algorithm} and {\ttfamily Genetic\-Algorithm} algorithms require the training data to be pre-\/stored in a {\ttfamily nnet\-::\-Neural\-Net\-Data\-Set} class (section \hyperlink{NeuralNet_BuildingTheTrainingSample}{Building the training sample}), and will train themselves over the whole data set. {\ttfamily Back\-Propagation\-Algorithm} on the other hand performs one training step at a time to provide more control over each training step.\par
 Descriptions of the algorithms are given in \hyperlink{NeuralNet_TrainingTheNetwork}{Training the network}.\hypertarget{NeuralNet_BuildingTheNeuronLayers}{}\subsubsection{Building the neuron layers}\label{NeuralNet_BuildingTheNeuronLayers}
Only simple nets can be built, where each neuron takes the outputs of all of the neurons in the previous layer as its inputs. Details about the neurons behaviour are given in \hyperlink{NeuralNet_NeuronDescriptions}{Neuron Descriptions}.\par
 There are two methods, one where neurons can have different types, and a simpler one where all of the neurons have the same type.\hypertarget{NeuralNet_AllNeuronsOfTheSameType}{}\paragraph{All neurons of the same type}\label{NeuralNet_AllNeuronsOfTheSameType}
Building the neuron layers simply consists of creating an S\-T\-L vector of integers with the number of neurons in each layer, including the output layer but excluding the input layer. The type of all of the neurons is set later when the network is built. So if a network takes 3 inputs, has two hidden layers with 6 neurons and 4 neurons respectively, and 2 outputs the layers would be set like this\-:


\begin{DoxyCode}
std::vector<int> neuronsInLayer;
neuronsInLayer.push\_back( 6 );
neuronsInLayer.push\_back( 4 );
neuronsInLayer.push\_back( 2 );
\textcolor{comment}{// The number of inputs is set later.}
\end{DoxyCode}
\hypertarget{NeuralNet_NeuronsWithDifferentTypes}{}\paragraph{Neurons with different types\}}\label{NeuralNet_NeuronsWithDifferentTypes}
These are set in a similar way, but instead of integers specifying the number of neurons in each layer, another S\-T\-L vector of strings specifying the name of each neuron type is used, with the number of neurons set by the size of the vector. Currently available types (descriptions are given in \hyperlink{NeuralNet_NeuronDescriptions}{Neuron Descriptions}) are\-:


\begin{DoxyCode}
LinearNeuron
SigmoidNeuron
TanSigmoidNeuron
\end{DoxyCode}


So if a network as in the previous example is to be built (with arbitrary neuron types)\-:


\begin{DoxyCode}
std::vector<std::string> layer1;
layer1.push\_back( \textcolor{stringliteral}{"LinearNeuron"} );
layer1.push\_back( \textcolor{stringliteral}{"SigmoidNeuron"} );
\textcolor{comment}{// ...and so on until there are six names in the vector -> six neurons in the layer}

std::vector<std::string> layer2;
layer2.push\_back( \textcolor{stringliteral}{"TanSigmoidNeuron"} );
\textcolor{comment}{// ... and so on another three times}

std::vector<std::string> outputlayer;
outputlayer.push\_back( \textcolor{stringliteral}{"LinearNeuron"} );
outputlayer.push\_back( \textcolor{stringliteral}{"LinearNeuron"} );

std::vector< std::vector<std::string> > neuronNames;
neuronNames.push\_back( layer1 );
neuronNames.push\_back( layer2 );
neuronNames.push\_back( outputlayer );
\textcolor{comment}{// The number of inputs is set later.}
\end{DoxyCode}
\hypertarget{NeuralNet_CreatingTheNetwork}{}\subsubsection{Creating the network}\label{NeuralNet_CreatingTheNetwork}
Once the layer structure has been set up, the network can be created as follows, depending on which layer specification method was used.\hypertarget{NeuralNet_AllNeuronsOfTheSameType}{}\paragraph{All neurons of the same type}\label{NeuralNet_AllNeuronsOfTheSameType}
The type of the neurons is set by creating a neuron builder and passing its address to the network constructor. The names of available builders are the same as for the neurons, but with {\ttfamily $<$tt$>$Builder$<$/tt$>$'' on the end, for example}{\ttfamily nnet\-::\-Linear\-Neuron\-Builder}'' will build ``{\ttfamily Linear\-Neuron}''s.


\begin{DoxyCode}
\textcolor{keywordtype}{int} numberOfInputs=3; \textcolor{comment}{// number of inputs set when the network is created}
nnet::SigmoidNeuronBuilder myNeuronBuilder; \textcolor{comment}{// the type of ALL the neurons is set here}

\textcolor{comment}{// now create the network using the neuronsInLayer vector from before}
nnet::NeuralNet sameNeuronsNet( numberOfInputs, neuronsInLayer, &myNeuronBuilder );
\end{DoxyCode}
\hypertarget{NeuralNet_NeuronsWithDifferentTypes}{}\paragraph{Neurons with different types\}}\label{NeuralNet_NeuronsWithDifferentTypes}
All that is needed here is the S\-T\-L vectors of neuron names previously initialised and the number of inputs.


\begin{DoxyCode}
\textcolor{keywordtype}{int} numberOfInputs=3; \textcolor{comment}{// number of inputs set when the network is created}

\textcolor{comment}{// now create the network using the neuronNames vector from before}
nnet::NeuralNet differentNeuronsNet( numberOfInputs, neuronNames );
\end{DoxyCode}
\hypertarget{NeuralNet_RandomNumberGeneration}{}\paragraph{Random number generation}\label{NeuralNet_RandomNumberGeneration}
In each case, you can optionally use a random seed for the random number generator that sets the initial neuron weights by adding a boolean parameter at the end of the constructor arguments. Default is to use a random seed.


\begin{DoxyCode}
\textcolor{comment}{//don't use a random seed}
nnet::NeuralNet sameNeuronsNet( numberOfInputs, neuronsInLayer, &myNeuronBuilder, \textcolor{keyword}{false} );

\textcolor{comment}{//use a random seed (default)}
nnet::NeuralNet differentNeuronsNet( numberOfInputs, neuronNames, \textcolor{keyword}{true} );
\end{DoxyCode}


Note that currently {\bfseries the default implementation uses {\ttfamily rand()} for random numbers}, and the random seed is taken from the current system time. If you require something more sophisticated modify the ``{\ttfamily \hyperlink{RandomNumberUtils_8h_source}{Random\-Number\-Utils.\-h}}'' file.\hypertarget{NeuralNet_BuildingTheTrainingSample}{}\subsubsection{Building the training sample}\label{NeuralNet_BuildingTheTrainingSample}
A network can be trained without setting out the data sample into a {\ttfamily nnet\-::\-Neural\-Net\-Data\-Set} using the {\ttfamily Back\-Propagation\-Algorithm}, but large scale training is easiest using the other algorithms so this will be covered here.\par
 Data is added to the {\ttfamily nnet\-::\-Neural\-Net\-Data\-Set} by calls to {\ttfamily add\-Data\-Item}, with a vector of inputs and a vector of the expected outputs as the arguments. All items in the data set must have the same number of inputs and outputs; the first item you add sets these sizes for the whole data set. If you try and add an item where the input or output vectors are not the correct size, then an error will be printed to standard error and the item will be ignored.

For example\-:


\begin{DoxyCode}
\textcolor{comment}{// for a network to calculate the probability a given animal is a donkey}
\textcolor{comment}{// with inputs, in order, of "number of legs", "height" and "length of tail"}
nnet::NeuralNetDataSet \(\backslash\)label\{donkeyNetDataSet\}animalSample;
std::vector<double> inputs;
std::vector<double> output;

\textcolor{comment}{// for donkey 1}
output.push\_back( 1 ); \textcolor{comment}{// I know for certain this animal is a donkey}
inputs.push\_back( 4 ); \textcolor{comment}{// it has four legs}
inputs.push\_back( 1.45 ); \textcolor{comment}{// it is 1.45m tall}
inputs.push\_back( 0.32 ); \textcolor{comment}{// it has a tail 32cm long}

\textcolor{comment}{// This call sets the sample to demand 3 inputs and 1 output}
animalSample.addDataItem( inputs, output ); \textcolor{comment}{// Add this animal to the training sample}

inputs.clear(); \textcolor{comment}{// Clear all the data so that the vectors can be reused}
output.clear();

\textcolor{comment}{//for Geoff}
output.push\_back( 0 ); \textcolor{comment}{// I know for certain Geoff isn't a donkey}
inputs.push\_back( 2 ); \textcolor{comment}{// he has 2 legs}
inputs.push\_back( 1.82 ); \textcolor{comment}{// he is 1.82m tall}

\textcolor{comment}{// This will not be added, because there are not enough inputs!}
animalSample.addDataItem( inputs, output ); \textcolor{comment}{// Add Geoff to the training sample}

\textcolor{comment}{// To add Geoff to the training sample we need to match the number of inputs}
inputs.push\_back( 0 ); \textcolor{comment}{// Geoff's tail is 0cm long}
animalSample.addDataItem( inputs, output ); \textcolor{comment}{// This will now work}
\end{DoxyCode}
\hypertarget{NeuralNet_TrainingTheNetwork}{}\subsubsection{Training the network}\label{NeuralNet_TrainingTheNetwork}
To train the network, a training algorithm is created with the network to be trained as the constructor argument, and a call to train is made with the number of training epochs and the training data. Currently available training algorithms are\-:\par



\begin{DoxyCode}
nnet::BackPropagationAlgorithm
nnet::BackPropagationCGAlgorithm
nnet::BatchBackPropagationAlgorithm
nnet::GeneticAlgorithm
\end{DoxyCode}
\hypertarget{NeuralNet_TrainingWithBackPropagationAlgorithm}{}\paragraph{Training with Back\-Propagation\-Algorithm}\label{NeuralNet_TrainingWithBackPropagationAlgorithm}
The Back Propagation Algorithm uses the back propagation method for determining the gradient of the error, and then gradient descent to modify the weights to minimise the error. It is very similar to the Batch\-Back\-Propagation\-Algorithm except that it only performs one training step at a time to give more control over the training parameters at each step.\par
 The algorithm class is constructed by giving it the network to be trained, and optionally values for $learningRate$ and $momentumConstant$ (defaults are 0.\-5 for both). The $learningRate$ parameter is just a multiplier applied to the calculated change required for each weight, larger values will mean the weights will change more rapidly with each step. The previous steps' calculated change is also added to the current steps', but multiplied by the $momentumConstant$ value. A value greater than or equal to one for this would stop the algorithm settling on a maximum because (at least) the full previous change is added as well.\par


The {\ttfamily train} method is used to perform one training run, and returns the error. It takes a vector of the inputs and a vector of the required outputs, so if the first data item in the previous example is used for the step\-:


\begin{DoxyCode}
std::vector<double> inputs;
std::vector<double> output;

\textcolor{comment}{// for donkey 1}
output.push\_back( 1 ); \textcolor{comment}{// I know for certain this animal is a donkey}
inputs.push\_back( 4 ); \textcolor{comment}{// it has four legs}
inputs.push\_back( 1.45 ); \textcolor{comment}{// it is 1.45m tall}
inputs.push\_back( 0.32 ); \textcolor{comment}{// it has a tail 32cm long}

\textcolor{comment}{//set learningRate to 0.6 and momentumConstant to 0.4}
nnet::BackPropagationAlgorithm myTrainer( myPreviouslyCreatedNetwork, 0.6, 0.4 );

\textcolor{keywordtype}{double} errorForThisStep=myTrainer.train( inputs, output );
\end{DoxyCode}
\hypertarget{NeuralNet_TrainingWithBatchPropagationAlgorithm}{}\paragraph{Training with Batch\-Back\-Propagation\-Algorithm}\label{NeuralNet_TrainingWithBatchPropagationAlgorithm}
This is essentially the same as {\ttfamily Back\-Propagation\-Algorithm}, except it is supplied with a training sample which it will loop over itself. It can also be set do so repeatedly by specifying the number of epochs to run when calling the {\ttfamily train} method. The error for the most recent epoch is returned by {\ttfamily train}, and the errors from previous epochs can be retrieved as a vector with the {\ttfamily get\-Training\-Error\-Values\-Per\-Epoch} method.


\begin{DoxyCode}
\textcolor{comment}{//created in the same way as for the single step version}
nnet::BatchBackPropagationAlgorithm myTrainer( myPreviouslyCreatedNetwork ) \textcolor{comment}{//use default learningRate and
       momentumConstant}

\textcolor{comment}{//train using the sample in the previous example and 50 epochs}
double finalError=myTrainer.train( 50, animalSample );

\textcolor{comment}{//get the errors from previous epochs to see how things are converging}
std::vector<\textcolor{keywordtype}{double}> errors=myTrainer.getTrainingErrorValuesPerEpoch();
\end{DoxyCode}
\hypertarget{NeuralNet_TrainingWithBackPropagationCGAlgorithm}{}\paragraph{Training with Back\-Propagation\-C\-G\-Algorithm}\label{NeuralNet_TrainingWithBackPropagationCGAlgorithm}
This algorithm is similar to {\ttfamily Batch\-Back\-Propagation\-Algorithm} except that it uses the conjugate gradient method to minimise the error instead of gradient descent. It offers three types of function to calculate the $\beta$ coefficient (see any detailed description of conjugate gradients) selected using the {\ttfamily set\-Beta\-Function} method. These are {\ttfamily Fletcher\-Reves'',}Polak\-Ribiere'', and ``\-Conjugate\-Gradient'', used as an enumeration as quoted. The default is Fletcher\-Reves.


\begin{DoxyCode}
\textcolor{comment}{//created in the same way as for the single step version}
nnet::BackPropagationCGAlgorithm myTrainer( myPreviouslyCreatedNetwork )

\textcolor{comment}{//set the beta function to Polak-Ribiere}
myTrainer.setBetaFunction( nnet::BackPropagationCGAlgorithm::PolakRibiere );

\textcolor{comment}{//train using the sample in the previous example and 500 epochs}
\textcolor{keywordtype}{double} finalError=myTrainer.train( 500, animalSample );
\end{DoxyCode}
\hypertarget{NeuralNet_ObtainingResults}{}\subsection{Obtaining results}\label{NeuralNet_ObtainingResults}
To get results from the neural network, the output method takes the inputs as an S\-T\-L vector of doubles, and provides the results as an S\-T\-L vector of doubles. So to determine if some animal is a donkey using a network trained from data of form of the data set in the previous example\-:


\begin{DoxyCode}
\textcolor{comment}{// Using the previously trained net "donkeyNet" to}
\textcolor{comment}{// find out if Fido is a donkey...}
std::vector<double> inputs;
inputs.push\_back( 4 ); \textcolor{comment}{// Fido has four legs}
inputs.push\_back( 0.54 ); \textcolor{comment}{// he is 54cm tall}
inputs.push\_back( 0.25 ); \textcolor{comment}{// his tail is 25cm long}

std::vector<double> output=donkeyNet( inputs );
\textcolor{comment}{// The network was set to have only one output, if there were any}
\textcolor{comment}{// more then they would be the higher elements in the vector.}
std::cout << \textcolor{stringliteral}{"Likelihood Fido is a donkey= "} << output[0] << std::endl;
\end{DoxyCode}
\hypertarget{NeuralNet_SavingANeuralNetToDisk}{}\subsection{Saving a neural net to disk}\label{NeuralNet_SavingANeuralNetToDisk}
Neural nets can either be saved as plain text or X\-M\-L files, with the default being X\-M\-L. To choose between the two make a call to {\ttfamily Neural\-Net\-::set\-Serialisation\-Mode} with either {\ttfamily nnet\-::\-Neural\-Net\-::\-Plain\-Text} or {\ttfamily nnet\-::\-Neural\-Net\-::\-X\-M\-L}.\par
 The network can then be saved to disk by passing a C++ stream to serialise. For example\-:


\begin{DoxyCode}
myNeuralNet.setSerialisationMode( nnet::NeuralNet::XML );
std::ofstream outputFile( \textcolor{stringliteral}{"/home/me/myNeuralNet.xml"} );
myNeuralNet.serialise( outputFile );
\end{DoxyCode}


The network can also of course be printed to standard output by calling {\ttfamily serialise( std\-::cout )}.\hypertarget{NeuralNet_LoadingANeuralNetFromDisk}{}\subsection{Loading a neural net from disk}\label{NeuralNet_LoadingANeuralNetFromDisk}
A network can be loaded from disk by simply passing the filename and the serialisation mode as the constructor arguments. If the serialisation mode is not specified then X\-M\-L is assumed. For example\-:


\begin{DoxyCode}
nnet::NeuralNet myXMLNet( \textcolor{stringliteral}{"/home/me/myNeuralNet.xml"}, nnet::NeuralNet::XML );
nnet::NeuralNet anotherXMLNet( \textcolor{stringliteral}{"/home/me/myOtherNeuralNet.xml"} );\textcolor{comment}{//XML is the default}
nnet::NeuralNet myTextNet( \textcolor{stringliteral}{"/home/me/myNeuralNet.txt"}, nnet::NeuralNet::PlainText );
\end{DoxyCode}


Note that there is currently no error checking when loading X\-M\-L nets, {\bfseries if you try and load a plain text net as X\-M\-L, or the file is not properly structured you will get a segmentation fault or runaway memory allocation}. This is still being looked into.\hypertarget{NeuralNet_NeuronDescriptions}{}\subsection{Neuron Descriptions}\label{NeuralNet_NeuronDescriptions}
The output from a neuron is given by its {\itshape threshold function} which is unique to each type of neuron. This is a function of the neurons {\itshape activation value}, which is calculated the same way for each type.

The activation value $ a $ for a neuron with $ N $ inputs, $ i_{n}$ , each with weights $ w_{n} $ is given by \[ a=\sum_{n=1}^{N} i_{n}\times w_{n}+b\times w_{b} \] Where $ b $ is a bias that can be assigned to a particular neuron (and $ w_{b}$ the bias' weight). The weights are initially random, and are then fine tuned by the training algorithms to try and give the desired output. The bias is set when the neuron is created but that process is done internally by the neuron builders. All current neuron builders set the bias to -\/1.

Some of the neurons have methods to change their behaviour. To get the neuron pointer to call these methods use {\ttfamily $<$tt$>$Neural\-Net\-::layer(layer\-Number)-\/$>$neuron(neuron\-Number)$<$/tt$>$'', where the numbers of available layers and neurons per layer can be found with}{\ttfamily Neural\-Net\-::number\-Of\-Layers()}'' and ``{\ttfamily Neural\-Net\-::layer(layer\-Number)-\/$>$number\-Of\-Neurons()}'' respectively.\hypertarget{NeuralNet_LinearNeuron}{}\subsubsection{Linear Neuron}\label{NeuralNet_LinearNeuron}
Linear neurons give, as the name suggests, a linear output between -\/1 and +1 with a gradient of $1/slopeEnd$. The value of {\itshape slope\-End} can be set using the {\ttfamily Linear\-Neuron\-::set\-Slope\-End(new\-Value)} method. If the output is greater than +slope\-End, then the output is limited to +1; any less than -\/slope\-End and the output is limited to -\/1. Anywhere in between gives the expected linear output of $activationvalue/slopeEnd$.\hypertarget{NeuralNet_SigmoidNeuron}{}\subsubsection{Sigmoid Neuron}\label{NeuralNet_SigmoidNeuron}
The sigmoid neuron gives sigmoid (sort of resembles a slanted ``\-S'') output, $ o $, of between 0 and 1 from the function

\[ o=\frac{1}{1+e^{-a/r}} \]

Where $ r $, the ``response'', can be set with the {\ttfamily Sigmoid\-Neuron\-::set\-Response(new\-Value)} method. The default is 1.\hypertarget{NeuralNet_TanSigmoidNeuron}{}\subsubsection{Tan Sigmoid Neuron}\label{NeuralNet_TanSigmoidNeuron}
This neuron gives a similar looking output to the sigmoid neuron, but between -\/1 and 1. The value is given by

\[ o=tanh(s\times a) \]

Where the value of $ s $ (the ``scale'') can be set with the {\ttfamily Tan\-Sigmoid\-Neuron\-::set\-Scale(new\-Value)} method. The default is 1.

\begin{DoxyAuthor}{Author}
Mark Grimes (\href{mailto:mark.grimes@bristol.ac.uk}{\tt mark.\-grimes@bristol.\-ac.\-uk}) 
\end{DoxyAuthor}
