Where do I call the BatchNormalization function in Keras

Question

If I want to use the BatchNormalization function in Keras  then do I need to call it once only at the beginning   I read this documentation for it  http   keras io layers normalization   I don t see where I m supposed to call it  Below is my code attempting to use it   model   Sequential   keras layers normalization BatchNormalization epsilon 1e-06  mode 0  momentum 0 9  weights None  model add Dense 64  input dim 14  init  uniform    model add Activation  tanh    model add Dropout 0 5   model add Dense 64  init  uniform    model add Activation  tanh    model add Dropout 0 5   model add Dense 2  init  uniform    model add Activation  softmax     sgd   SGD lr 0 1  decay 1e-6  momentum 0 9  nesterov True  model compile loss  binary crossentropy   optimizer sgd  model fit X train  y train  nb epoch 20  batch size 16  show accuracy True  validation split 0 2  verbose   2    I ask because if I run the code with the second line including the batch normalization and if I run the code without the second line I get similar outputs  So either I m not calling the function in the right place  or I guess it doesn t make that much of a difference

User · Accepted Answer

Just to answer this question in a little more detail, and as Pavel said, Batch Normalization is just another layer, so you can use it as such to create your desired network architecture.

The general use case is to use BN between the linear and non-linear layers in your network, because it normalizes the input to your activation function, so that you're centered in the linear section of the activation function (such as Sigmoid). There's a small discussion of it here

In your case above, this might look like:

# import BatchNormalization
from keras.layers.normalization import BatchNormalization

# instantiate model
model = Sequential()

# we can think of this chunk as the input layer
model.add(Dense(64, input_dim=14, init='uniform'))
model.add(BatchNormalization())
model.add(Activation('tanh'))
model.add(Dropout(0.5))

# we can think of this chunk as the hidden layer    
model.add(Dense(64, init='uniform'))
model.add(BatchNormalization())
model.add(Activation('tanh'))
model.add(Dropout(0.5))

# we can think of this chunk as the output layer
model.add(Dense(2, init='uniform'))
model.add(BatchNormalization())
model.add(Activation('softmax'))

# setting up the optimization of our weights 
sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='binary_crossentropy', optimizer=sgd)

# running the fitting
model.fit(X_train, y_train, nb_epoch=20, batch_size=16, show_accuracy=True, validation_split=0.2, verbose = 2)

Hope this clarifies things a bit more.

User · Answer

It s almost become a trend now to have a Conv2D followed by a ReLu followed by a BatchNormalization layer  So I made up a small function to call all of them at once  Makes the model definition look a whole lot cleaner and easier to read    def Conv2DReluBatchNorm n filter  w filter  h filter  inputs       return BatchNormalization   Activation activation  relu   Convolution2D n filter  w filter  h filter  border mode  same   inputs

User · Answer

Batch Normalization is used to normalize the input layer as well as hidden layers by adjusting mean and scaling of the activations  Because of this normalizing effect with additional layer in deep neural networks  the network can use higher learning rate without vanishing or exploding gradients  Furthermore  batch normalization regularizes the network such that it is easier to generalize  and it is thus unnecessary to use dropout to mitigate overfitting    Right after calculating the linear function using say  the Dense   or Conv2D   in Keras  we use BatchNormalization   which calculates the linear function in a layer and then we add the non-linearity to the layer using Activation     from keras layers normalization import BatchNormalization model   Sequential   model add Dense 64  input dim 14  init  uniform    model add BatchNormalization epsilon 1e-06  mode 0  momentum 0 9  weights None   model add Activation  tanh    model add Dropout 0 5   model add Dense 64  init  uniform    model add BatchNormalization epsilon 1e-06  mode 0  momentum 0 9  weights None   model add Activation  tanh    model add Dropout 0 5   model add Dense 2  init  uniform    model add BatchNormalization epsilon 1e-06  mode 0  momentum 0 9  weights None   model add Activation  softmax     sgd   SGD lr 0 1  decay 1e-6  momentum 0 9  nesterov True  model compile loss  binary crossentropy   optimizer sgd  model fit X train  y train  nb epoch 20  batch size 16  show accuracy True   validation split 0 2  verbose   2    How is Batch Normalization applied   Suppose we have input a l-1  to a layer l  Also we have weights W l  and bias unit b l  for the layer l  Let a l  be the activation vector calculated i e  after adding the non-linearity  for the layer l and z l  be the vector before adding non-linearity   Using a l-1  and W l  we can calculate z l  for the layer l Usually in feed-forward propagation we will add bias unit to the z l  at this stage like this z l  b l   but in Batch Normalization this step of addition of b l  is not required and no b l  parameter is used  Calculate z l  means and subtract it from each element Divide   z l  - mean  using standard deviation  Call it Z temp l  Now define new parameters   and    that will change the scale of the hidden layer as follows   z norm l      Z temp l         In this code excerpt  the Dense   takes the a l-1   uses W l  and calculates z l   Then the immediate BatchNormalization   will perform the above steps to give z norm l   And then the immediate Activation   will calculate tanh z norm l   to give a l  i e   a l    tanh z norm l

User · Answer

Adding another entry for the debate about whether batch normalization should be called before or after the non-linear activation  In addition to the original paper using batch normalization before the activation  Bengio s book Deep Learning  section 8 7 1 gives some reasoning for why applying batch normalization after the activation  or directly before the input to the next layer  may cause some issues   It is natural to wonder whether we should apply batch normalization to the input X  or to the transformed value XW b  Io e and Szegedy  2015  recommend the latter  More speci cally  XW b should be replaced by a normalized version of XW  The bias term should be omitted because it becomes redundant with the    parameter applied by the batch normalization reparameterization  The input to a layer is usually the output of a nonlinear activation function such as the recti ed linear function in a previous layer  The statistics of the input are thus more non-Gaussian and less amenable to standardization by linear operations   In other words  if we use a relu activation  all negative values are mapped to zero   This will likely result in a mean value that is already very close to zero  but the distribution of the remaining data will be heavily skewed to the right   Trying to normalize that data to a nice bell-shaped curve probably won t give the best results   For activations outside of the relu family this may not be as big of an issue  Keep in mind that there are reports of models getting better results when using batch normalization after the activation  while others get best results when the batch normalization is placed before the activation   It is probably best to test your model using both configurations  and if batch normalization after activation gives a significant decrease in validation loss  use that configuration instead

User · Answer

This thread has some considerable debate about whether BN should be applied before non-linearity of current layer or to the activations of the previous layer    Although there is no correct answer  the authors of Batch Normalization say that  It should be applied immediately before the non-linearity of the current layer  The reason   quoted from original paper  -       We add the BN transform immediately before the   nonlinearity  by normalizing x   Wu b  We could have   also normalized the layer inputs u  but since u is likely   the output of another nonlinearity  the shape of its distribution   is likely to change during training  and constraining   its first and second moments would not eliminate the covariate   shift  In contrast  Wu   b is more likely to have   a symmetric  non-sparse distribution  that is    more Gaussian       Hyv  arinen  amp  Oja  2000   normalizing it is likely to   produce activations with a stable distribution

User · Answer

This thread is misleading   Tried commenting on Lucas Ramadan s answer  but I don t have the right privileges yet  so I ll just put this here   Batch normalization works best after the activation function  and here or here is why  it was developed to prevent internal covariate shift  Internal covariate shift occurs when the distribution of the activations of a layer shifts significantly throughout training   Batch normalization is used so that the distribution of the inputs  and these inputs are literally the result of an activation function  to a specific layer doesn t change over time due to parameter updates from each batch  or at least  allows it to change in an advantageous way    It uses batch statistics to do the normalizing  and then uses the batch normalization parameters  gamma and beta in the original paper   to make sure that the transformation inserted in the network can represent the identity transform   quote from original paper    But the point is that we re trying to normalize the inputs to a layer  so it should always go immediately before the next layer in the network   Whether or not that s after an activation function is dependent on the architecture in question

User · Answer

Keras now supports the use bias False option  so we can save some computation by writing like  model add Dense 64  use bias False   model add BatchNormalization axis bn axis   model add Activation  tanh      or   model add Convolution2D 64  3  3  use bias False   model add BatchNormalization axis bn axis   model add Activation  relu

User · Answer

It is another type of layer  so you should add it as a layer in an appropriate place of your model  model add keras layers normalization BatchNormalization      See an example here  https   github com fchollet keras blob master examples kaggle otto nn py

[python] Where do I call the BatchNormalization function in Keras?

Examples related to python

Examples related to keras

Examples related to neural-network

Examples related to data-science

Examples related to batch-normalization