FeedForward Neural Network

1. FeedForward Neural Network:
   The FeedForward Neural Network (FNN) is an artificial neural network wherein the connections between the nodes do not form a cycle, allowing the information to move only in one direction, forward, from the input layer to the subsequent layers.
   
   

2. Architecture:
   An FNN consists of one or more layers, each consisting of nodes (simulating biological neurons) that hold a certain wight value \(w_{ij}\). Those weights are usually multiplied by the input values (in the input layer) in each node and, then, summed; finally, one can apply some sort of activation function on the multiplied values to simulate a response (e.g. 1-0 classification).
   
   

3. Classes of FNNs:
   There are many variations of FNNs. As long as they utilize FeedForward control signals and have a layered structure (described above) they are a type of FNN:

   • Single-Layer Perceptron:
     A linear binary classifier, the single-layer perceptron is the simplest feedforward neural network. It consists of a single layer of output nodes; the inputs are multiplied by a series of weights, effectively, being fed directly to the outputs where they values are summed in each node, and if the value is above some threshold (typically 0) the neuron fires and takes the activated value (typically 1); otherwise it takes the deactivated value (typically 0).

     $$f(\mathbf{x})=\left\{\begin{array}{ll}{1} & {\text { if } \mathbf{w} \cdot \mathbf{x}+b>0} \\ {0} & {\text { otherwise }}\end{array}\right.$$

     In the context of neural networks, a perceptron is an artificial neuron using the Heaviside step function as the activation function.

   • Multi-Layer Perceptron:
     This class of networks consists of multiple layers of computational units, usually interconnected in a feed-forward way. Each neuron in one layer has directed connections to the neurons of the subsequent layer. In many applications the units of these networks apply a sigmoid function as an activation function.

Multilayer Perceptron

1. Multilayer Perceptron:
   The Multilayer Perceptron (MLP) is a class of FeedForward Neural Networks that is used for learning from data.
   
   

2. Architecture:
   The MLP consists of at least three layers of nodes: input layer, hidden layer, and an output layer.

   The layers in a neural network are connected by certain weights and the MLP is known as a fully-connected network where each neuron in one layer is connected with a weight \(w_{ij}\) to every node in the following layer.

   Each node (except for the input nodes) uses a non-linear activation function that were developed to model the frequency of action potential (firing) of biological neurons.
   
   

3. Learning:
   The MLP employs a supervised learning technique called backpropagation.
   Learning occurs by changing the weights, connecting the layers, based on the amount of error in the output compared to the expected result. Those weights are changed by using gradient-methods to optimize a, given, objective function (called the loss function).
   
   

4. Properties:

   • Due to their non-linearity, MLPs can distinguish and model non-linearly-separable data
   • According to Cybenko’s Theorem, MLPs are universal function approximators; thus, they can be used for regression analysis and, by extension, classification
   • Without the non-linear activation functions, MLPs will be identical to Perceptrons, since Linear Algebra shows that the linear transformations in many hidden layers can be collapsed into one linear-transformation
     
     

Deep FeedForward Neural Network

(Gradient-Based) Learning for FNNs