Where do I call the BatchNormalization function in Keras

Question

If I want to use the BatchNormalization function in Keras  then do I need to call it once only at the beginning   I read this documentation for it  http   keras io layers normalization   I don t see where I m supposed to call it  Below is my code attempting to use it   model   Sequential   keras layers normalization BatchNormalization epsilon 1e-06  mode 0  momentum 0 9  weights None  model add Dense 64  input dim 14  init  uniform    model add Activation  tanh    model add Dropout 0 5   model add Dense 64  init  uniform    model add Activation  tanh    model add Dropout 0 5   model add Dense 2  init  uniform    model add Activation  softmax     sgd   SGD lr 0 1  decay 1e-6  momentum 0 9  nesterov True  model compile loss  binary crossentropy   optimizer sgd  model fit X train  y train  nb epoch 20  batch size 16  show accuracy True  validation split 0 2  verbose   2    I ask because if I run the code with the second line including the batch normalization and if I run the code without the second line I get similar outputs  So either I m not calling the function in the right place  or I guess it doesn t make that much of a difference

User · Answer

Adding another entry for the debate about whether batch normalization should be called before or after the non-linear activation:

In addition to the original paper using batch normalization before the activation, Bengio's book Deep Learning, section 8.7.1 gives some reasoning for why applying batch normalization after the activation (or directly before the input to the next layer) may cause some issues:

It is natural to wonder whether we should apply batch normalization to the input X, or to the transformed value XW+b. Io?e and Szegedy (2015) recommend the latter. More speci?cally, XW+b should be replaced by a normalized version of XW. The bias term should be omitted because it becomes redundant with the ß parameter applied by the batch normalization reparameterization. The input to a layer is usually the output of a nonlinear activation function such as the recti?ed linear function in a previous layer. The statistics of the input are thus more non-Gaussian and less amenable to standardization by linear operations.

In other words, if we use a relu activation, all negative values are mapped to zero. This will likely result in a mean value that is already very close to zero, but the distribution of the remaining data will be heavily skewed to the right. Trying to normalize that data to a nice bell-shaped curve probably won't give the best results. For activations outside of the relu family this may not be as big of an issue.

Keep in mind that there are reports of models getting better results when using batch normalization after the activation, while others get best results when the batch normalization is placed before the activation. It is probably best to test your model using both configurations, and if batch normalization after activation gives a significant decrease in validation loss, use that configuration instead.

User · Answer

Keras now supports the use bias False option  so we can save some computation by writing like  model add Dense 64  use bias False   model add BatchNormalization axis bn axis   model add Activation  tanh      or   model add Convolution2D 64  3  3  use bias False   model add BatchNormalization axis bn axis   model add Activation  relu

User · Answer

It s almost become a trend now to have a Conv2D followed by a ReLu followed by a BatchNormalization layer  So I made up a small function to call all of them at once  Makes the model definition look a whole lot cleaner and easier to read    def Conv2DReluBatchNorm n filter  w filter  h filter  inputs       return BatchNormalization   Activation activation  relu   Convolution2D n filter  w filter  h filter  border mode  same   inputs

User · Answer

It is another type of layer  so you should add it as a layer in an appropriate place of your model  model add keras layers normalization BatchNormalization      See an example here  https   github com fchollet keras blob master examples kaggle otto nn py

User · Answer

Just to answer this question in a little more detail  and as Pavel said  Batch Normalization is just another layer  so you can use it as such to create your desired network architecture    The general use case is to use BN between the linear and non-linear layers in your network  because it normalizes the input to your activation function  so that you re centered in the linear section of the activation function  such as Sigmoid   There s a small discussion of it here  In your case above  this might look like       import BatchNormalization from keras layers normalization import BatchNormalization    instantiate model model   Sequential      we can think of this chunk as the input layer model add Dense 64  input dim 14  init  uniform    model add BatchNormalization    model add Activation  tanh    model add Dropout 0 5      we can think of this chunk as the hidden layer     model add Dense 64  init  uniform    model add BatchNormalization    model add Activation  tanh    model add Dropout 0 5      we can think of this chunk as the output layer model add Dense 2  init  uniform    model add BatchNormalization    model add Activation  softmax       setting up the optimization of our weights  sgd   SGD lr 0 1  decay 1e-6  momentum 0 9  nesterov True  model compile loss  binary crossentropy   optimizer sgd     running the fitting model fit X train  y train  nb epoch 20  batch size 16  show accuracy True  validation split 0 2  verbose   2      Hope this clarifies things a bit more

User · Answer

This thread has some considerable debate about whether BN should be applied before non-linearity of current layer or to the activations of the previous layer    Although there is no correct answer  the authors of Batch Normalization say that  It should be applied immediately before the non-linearity of the current layer  The reason   quoted from original paper  -       We add the BN transform immediately before the   nonlinearity  by normalizing x   Wu b  We could have   also normalized the layer inputs u  but since u is likely   the output of another nonlinearity  the shape of its distribution   is likely to change during training  and constraining   its first and second moments would not eliminate the covariate   shift  In contrast  Wu   b is more likely to have   a symmetric  non-sparse distribution  that is    more Gaussian       Hyv  arinen  amp  Oja  2000   normalizing it is likely to   produce activations with a stable distribution

User · Answer

This thread is misleading   Tried commenting on Lucas Ramadan s answer  but I don t have the right privileges yet  so I ll just put this here   Batch normalization works best after the activation function  and here or here is why  it was developed to prevent internal covariate shift  Internal covariate shift occurs when the distribution of the activations of a layer shifts significantly throughout training   Batch normalization is used so that the distribution of the inputs  and these inputs are literally the result of an activation function  to a specific layer doesn t change over time due to parameter updates from each batch  or at least  allows it to change in an advantageous way    It uses batch statistics to do the normalizing  and then uses the batch normalization parameters  gamma and beta in the original paper   to make sure that the transformation inserted in the network can represent the identity transform   quote from original paper    But the point is that we re trying to normalize the inputs to a layer  so it should always go immediately before the next layer in the network   Whether or not that s after an activation function is dependent on the architecture in question

User · Answer

Batch Normalization is used to normalize the input layer as well as hidden layers by adjusting mean and scaling of the activations  Because of this normalizing effect with additional layer in deep neural networks  the network can use higher learning rate without vanishing or exploding gradients  Furthermore  batch normalization regularizes the network such that it is easier to generalize  and it is thus unnecessary to use dropout to mitigate overfitting    Right after calculating the linear function using say  the Dense   or Conv2D   in Keras  we use BatchNormalization   which calculates the linear function in a layer and then we add the non-linearity to the layer using Activation     from keras layers normalization import BatchNormalization model   Sequential   model add Dense 64  input dim 14  init  uniform    model add BatchNormalization epsilon 1e-06  mode 0  momentum 0 9  weights None   model add Activation  tanh    model add Dropout 0 5   model add Dense 64  init  uniform    model add BatchNormalization epsilon 1e-06  mode 0  momentum 0 9  weights None   model add Activation  tanh    model add Dropout 0 5   model add Dense 2  init  uniform    model add BatchNormalization epsilon 1e-06  mode 0  momentum 0 9  weights None   model add Activation  softmax     sgd   SGD lr 0 1  decay 1e-6  momentum 0 9  nesterov True  model compile loss  binary crossentropy   optimizer sgd  model fit X train  y train  nb epoch 20  batch size 16  show accuracy True   validation split 0 2  verbose   2    How is Batch Normalization applied   Suppose we have input a l-1  to a layer l  Also we have weights W l  and bias unit b l  for the layer l  Let a l  be the activation vector calculated i e  after adding the non-linearity  for the layer l and z l  be the vector before adding non-linearity   Using a l-1  and W l  we can calculate z l  for the layer l Usually in feed-forward propagation we will add bias unit to the z l  at this stage like this z l  b l   but in Batch Normalization this step of addition of b l  is not required and no b l  parameter is used  Calculate z l  means and subtract it from each element Divide   z l  - mean  using standard deviation  Call it Z temp l  Now define new parameters   and    that will change the scale of the hidden layer as follows   z norm l      Z temp l         In this code excerpt  the Dense   takes the a l-1   uses W l  and calculates z l   Then the immediate BatchNormalization   will perform the above steps to give z norm l   And then the immediate Activation   will calculate tanh z norm l   to give a l  i e   a l    tanh z norm l

[python] Where do I call the BatchNormalization function in Keras?

Examples related to python

Examples related to keras

Examples related to neural-network

Examples related to data-science

Examples related to batch-normalization