Why do we have to normalize the input for an artificial neural network

Question

Why do we have to normalize the input for a neural network  I understand that sometimes  when for example the input values are non-numerical a certain transformation must be performed  but when we have a numerical input  Why the numbers must be in a certain interval  What will happen if the data is not normalized

User · Answer

On a high level  if you observe as to where normalization standardization is mostly used  you will notice that  anytime there is a use of magnitude difference in model building process  it becomes necessary to standardize the inputs so as to ensure that important inputs with small magnitude don t loose their significance midway the model building process   example  v 3-1  2  1000-900  2    v 1000-900  2   Here   3-1  contributes hardly a thing to the result and hence the input corresponding to these values is considered futile by the model   Consider the following   Clustering uses euclidean or  other distance measures  NNs use optimization algorithm to minimise cost function ex  - MSE     Both distance measure Clustering  and cost function NNs  use magnitude difference in some way and hence standardization ensures that magnitude difference doesn t command over important input parameters and the algorithm works as expected

User · Answer

There are 2 Reasons why we have to Normalize Input Features before Feeding them to Neural Network   Reason 1  If a Feature in the Dataset is big in scale compared to others then this big scaled feature becomes dominating and as a result of that  Predictions of the Neural Network will not be Accurate   Example  In case of Employee Data  if we consider Age and Salary  Age will be a Two Digit Number while Salary can be 7 or 8 Digit  1 Million  etc     In that Case  Salary will Dominate the Prediction of the Neural Network  But if we Normalize those Features  Values of both the Features will lie in the Range from  0 to 1    Reason 2  Front Propagation of Neural Networks involves the Dot Product of Weights with Input Features  So  if the Values are very high  for Image and Non-Image Data   Calculation of Output takes a lot of Computation Time as well as Memory  Same is the case during Back Propagation  Consequently  Model Converges slowly  if the Inputs are not Normalized   Example  If we perform Image Classification  Size of Image will be very huge  as the Value of each Pixel ranges from 0 to 255  Normalization in this case is very important   Mentioned below are the instances where Normalization is very important    K-Means K-Nearest-Neighbours Principal Component Analysis  PCA  Gradient Descent

User · Answer

Some inputs to NN might not have a  naturally defined  range of values  For example  the average value might be slowly  but continuously increasing over time  for example a number of records in the database     In such case feeding this raw value into your network will not work very well  You will teach your network on values from lower part of range  while the actual inputs will be from the higher part of this range  and quite possibly above range  that the network has learned to work with     You should normalize this value  You could for example tell the network by how much the value has changed since the previous input  This increment usually can be defined with high probability in a specific range  which makes it a good input for network

User · Answer

I believe the answer is dependent on the scenario   Consider NN  neural network  as an operator F  so that F input    output  In the case where this relation is linear so that F A   input    A   output  then you might choose to either leave the input output unnormalised in their raw forms  or normalise both to eliminate A  Obviously this linearity assumption is violated in classification tasks  or nearly any task that outputs a probability  where F A   input    1   output  In practice  normalisation allows non-fittable networks to be fittable  which is crucial to experimenters programmers  Nevertheless  the precise impact of normalisation will depend not only on the network architecture algorithm  but also on the statistical prior for the input and output    What s more  NN is often implemented to solve very difficult problems in a black-box fashion  which means the underlying problem may have a very poor statistical formulation  making it hard to evaluate the impact of normalisation  causing the technical advantage  becoming fittable  to dominate over its impact on the statistics   In statistical sense  normalisation removes variation that is believed to be non-causal in predicting the output  so as to prevent NN from learning this variation as a predictor  NN does not see this variation  hence cannot use it

User · Answer

When you use unnormalized input features  the loss function is likely to have very elongated valleys  When optimizing with gradient descent  this becomes an issue because the gradient will be steep with respect some of the parameters  That leads to large oscillations in the search space  as you are bouncing between steep slopes  To compensate  you have to stabilize optimization with small learning rates   Consider features x1 and x2  where range from 0 to 1 and 0 to 1 million  respectively  It turns out the ratios for the corresponding parameters  say  w1 and w2  will also be large   Normalizing tends to make the loss function more symmetrical spherical  These are easier to optimize because the gradients tend to point towards the global minimum and you can take larger steps

User · Answer

In neural networks  it is good idea not just to normalize data but also to scale them  This is intended for faster approaching to global minima at error surface  See the following pictures      Pictures are taken from the coursera course about neural networks  Author of the course is Geoffrey Hinton

User · Answer

Looking at the neural network from the outside  it is just a function that takes some  arguments and produces a result  As with all functions  it has a domain  i e  a set of legal arguments   You have to normalize the values that you want to pass to the neural net in order to make sure it is in the domain  As with all functions  if the arguments are not in the domain  the result is not guaranteed to be appropriate    The exact behavior of the neural net on arguments outside of the domain depends on the implementation of the neural net  But overall  the result is useless if the arguments are not within the domain

User · Answer

The reason normalization is needed is because if you look at how an adaptive step proceeds in one place in the domain of the function  and you just simply transport the problem to the equivalent of the same step translated by some large value in some direction in the domain  then you get different results  It boils down to the question of adapting a linear piece to a data point   How much should the piece move without turning and how much should it turn in response to that one training point   It makes no sense to have a changed adaptation procedure in different parts of the domain  So normalization is required to reduce the difference in the training result   I haven t got this written up  but you can just look at the math for a simple linear function and how it is trained by one training point in two different places   This problem may have been corrected in some places  but I am not familiar with them   In ALNs  the problem has been corrected and I can send you a paper if you write to wwarmstrong AT shaw ca

User · Answer

Hidden layers are used in accordance with the complexity of our data  If we have input data which is linearly separable then we need not to use hidden layer e g  OR gate but if we have a non linearly seperable data then we need to use hidden layer for example ExOR logical gate  Number of nodes taken at any layer depends upon the degree of cross validation of our output

User · Answer

It s explained well here      If the input variables are combined linearly  as in an MLP  multilayer perceptron   then it is   rarely strictly necessary to standardize the inputs  at least in theory  The   reason is that any rescaling of an input vector can be effectively undone by   changing the corresponding weights and biases  leaving you with the exact   same outputs as you had before  However  there are a variety of practical   reasons why standardizing the inputs can make training faster and reduce the   chances of getting stuck in local optima  Also  weight decay and Bayesian   estimation can be done more conveniently with standardized inputs

[machine-learning] Why do we have to normalize the input for an artificial neural network?

example:

Examples related to machine-learning

Examples related to neural-network

Examples related to normalization