Why binary crossentropy and categorical crossentropy give different performances for the same problem

Question

I m trying to train a CNN to categorize text by topic  When I use binary cross-entropy I get  80  accuracy  with categorical cross-entropy I get  50  accuracy   I don t understand why this is  It s a multiclass problem  doesn t that mean that I have to use categorical cross-entropy and that the results with binary cross-entropy are meaningless     model add embedding layer  model add Dropout 0 25     convolution layers model add Conv1D nb filter 32                      filter length 4                      border mode  valid                       activation  relu    model add MaxPooling1D pool length 2     dense layers model add Flatten    model add Dense 256   model add Dropout 0 25   model add Activation  relu      output layer model add Dense len class id index    model add Activation  softmax      Then I compile it either it like this using categorical crossentropy as the loss function   model compile loss  categorical crossentropy   optimizer  adam   metrics   accuracy      or   model compile loss  binary crossentropy   optimizer  adam   metrics   accuracy      Intuitively it makes sense why I d want to use categorical cross-entropy  I don t understand why I get good results with binary  and poor results with categorical

User · Answer

As it is a multi-class problem  you have to use the categorical crossentropy  the binary cross entropy will produce bogus results  most likely will only evaluate the first two classes only   50  for a multi-class problem can be quite good  depending on the number of classes  If you have n classes  then 100 n is the minimum performance you can get by outputting a random class

User · Answer

Take a look at the equation you can find that binary cross entropy not only punish those label   1  predicted  0  but also label   0  predicted   1   However categorical cross entropy only punish those label   1 but predicted   1 That s why we make assumption that there is only ONE label positive

User · Answer

a simple example under a multi-class setting to illustrate  suppose you have 4 classes  onehot encoded  and below is just one prediction  true label    0 1 0 0  predicted label    0 0 1 0   when using categorical crossentropy  the accuracy is just 0   it only cares about if you get the concerned class right   however when using binary crossentropy  the accuracy is calculated for all classes  it would be 50  for this prediction  and the final result will be the mean of the individual accuracies for both cases   it is recommended to use categorical crossentropy for multi-class classes are mutually exclusive  problem but binary crossentropy for multi-label problem

User · Answer

After commenting  Marcin answer  I have more carefully checked one of my students code where I found the same weird behavior  even after only 2 epochs    So  Marcin s explanation was not very likely in my case    And I found that the answer is actually very simple  the accuracy computed with the Keras method evaluate is just plain wrong when using binary crossentropy with more than 2 labels  You can check that by recomputing the accuracy yourself  first call the Keras method  predict  and then compute the number of correct answers returned by predict   you get the true accuracy  which is much lower than the Keras  evaluate  one

User · Answer

The binary crossentropy y target  y predict  doesn t need to apply in binary classification problem     In the source code of binary crossentropy    the nn sigmoid cross entropy with logits labels target  logits output  TensorFlow function was actually used   And  in the documentation  it says that      Measures the probability error in discrete classification tasks in which each class is independent and not mutually exclusive  For instance  one could perform multilabel classification where a picture can contain both an elephant and a dog at the same time

User · Answer

I came across an  inverted  issue     I was getting good results with categorical crossentropy  with 2 classes  and poor with binary crossentropy  It seems that problem was with wrong activation function  The correct settings were    for binary crossentropy  sigmoid activation  scalar target  for categorical crossentropy  softmax activation  one-hot encoded target

User · Answer

It all depends on the type of classification problem you are dealing with  There are three main categories  binary classification  two target classes   multi-class classification  more than two exclusive targets   multi-label classification  more than two non exclusive targets   in which multiple target classes can be on at the same time   In the first case  binary cross-entropy should be used and targets should be encoded as one-hot vectors  In the second case  categorical cross-entropy should be used and targets should be encoded as one-hot vectors  In the last case  binary cross-entropy should be used and targets should be encoded as one-hot vectors  Each output neuron  or unit  is considered as a separate random binary variable  and the loss for the entire vector of outputs is the product of the loss of single binary variables  Therefore it is the product of binary cross-entropy for each single output unit  The binary cross-entropy is defined as  and categorical cross-entropy is defined as  where c is the index running over the number of classes C

User · Answer

when using the categorical crossentropy loss  your targets should be in categorical format  e g  if you have 10 classes  the target for each sample should be a 10-dimensional vector that is all-zeros except for a 1 at the index corresponding to the class of the sample

User · Answer

You are passing a target array of shape  x-dim  y-dim  while using as loss categorical crossentropy  categorical crossentropy expects targets to be binary matrices  1s and 0s  of shape  samples  classes   If your targets are integer classes  you can convert them to the expected format via   from keras utils import to categorical y binary   to categorical y int    Alternatively  you can use the loss function sparse categorical crossentropy instead  which does expect integer targets   model compile loss  sparse categorical crossentropy   optimizer  adam   metrics   accuracy

User · Answer

It s really interesting case  Actually in your setup the following statement is true   binary crossentropy   len class id index    categorical crossentropy   This means that up to a constant multiplication factor your losses are equivalent  The weird behaviour that you are observing during a training phase might be an example of a following phenomenon    At the beginning the most frequent class is dominating the loss - so network is learning to predict mostly this class for every example  After it learnt the most frequent pattern it starts discriminating among less frequent classes  But when you are using adam - the learning rate has a much smaller value than it had at the beginning of training  it s because of the nature of this optimizer   It makes training slower and prevents your network from e g  leaving a poor local minimum less possible    That s why this constant factor might help in case of binary crossentropy  After many epochs - the learning rate value is greater than in categorical crossentropy case  I usually restart training  and learning phase  a few times when I notice such behaviour or and adjusting a class weights using the following pattern   class weight   1   class frequency   This makes loss from a less frequent classes balancing the influence of a dominant class loss at the beginning of a training and in a further part of an optimization process   EDIT   Actually - I checked that even though in case of maths   binary crossentropy   len class id index    categorical crossentropy   should hold - in case of keras it s not true  because keras is automatically normalizing all outputs to sum up to 1  This is the actual reason behind this weird behaviour as in case of multiclassification such normalization harms a training

User · Answer

The reason for this apparent performance discrepancy between categorical  amp  binary cross entropy is what user xtof54 has already reported in his answer below  i e    the accuracy computed with the Keras method evaluate is just plain wrong when using binary crossentropy with more than 2 labels  I would like to elaborate more on this  demonstrate the actual underlying issue  explain it  and offer a remedy  This behavior is not a bug  the underlying reason is a rather subtle  amp  undocumented issue at how Keras actually guesses which accuracy to use  depending on the loss function you have selected  when you include simply metrics   accuracy   in your model compilation  In other words  while your first compilation option  model compile loss  categorical crossentropy   optimizer  adam   metrics   accuracy     is valid  your second one  model compile loss  binary crossentropy   optimizer  adam   metrics   accuracy     will not produce what you expect  but the reason is not the use of binary cross entropy  which  at least in principle  is an absolutely valid loss function   Why is that  If you check the metrics source code  Keras does not define a single accuracy metric  but several different ones  among them binary accuracy and categorical accuracy  What happens under the hood is that  since you have selected binary cross entropy as your loss function and have not specified a particular accuracy metric  Keras  wrongly     infers that you are interested in the binary accuracy  and this is what it returns - while in fact you are interested in the categorical accuracy  Let s verify that this is the case  using the MNIST CNN example in Keras  with the following modification  model compile loss  binary crossentropy   optimizer  adam   metrics   accuracy       WRONG way  model fit x train  y train            batch size batch size            epochs 2     only 2 epochs  for demonstration purposes           verbose 1            validation data  x test  y test      Keras reported accuracy  score   model evaluate x test  y test  verbose 0   score 1    0 9975801164627075    Actual accuracy calculated manually  import numpy as np y pred   model predict x test  acc   sum  np argmax y test i    np argmax y pred i   for i in range 10000    10000 acc   0 98780000000000001  score 1   acc   False      To remedy this  i e  to use indeed binary cross entropy as your loss function  as I said  nothing wrong with this  at least in principle  while still getting the categorical accuracy required by the problem at hand  you should ask explicitly for categorical accuracy in the model compilation as follows  from keras metrics import categorical accuracy model compile loss  binary crossentropy   optimizer  adam   metrics  categorical accuracy    In the MNIST example  after training  scoring  and predicting the test set as I show above  the two metrics now are the same  as they should be    Keras reported accuracy  score   model evaluate x test  y test  verbose 0   score 1    0 98580000000000001    Actual accuracy calculated manually  y pred   model predict x test  acc   sum  np argmax y test i    np argmax y pred i   for i in range 10000    10000 acc   0 98580000000000001  score 1   acc   True      System setup  Python version 3 5 3 Tensorflow version 1 2 1 Keras version 2 0 4  UPDATE  After my post  I discovered that this issue had already been identified in this answer

[machine-learning] Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?

Examples related to machine-learning

Examples related to keras

Examples related to neural-network

Examples related to deep-learning

Examples related to conv-neural-network