How to write a confusion matrix in Python

Question

I wrote a confusion matrix calculation code in Python   def conf mat prob arr  input arr             confusion matrix         conf arr     0  0    0  0            for i in range len prob arr                    if int input arr i      1                          if float prob arr i    lt  0 5                                  conf arr 0  1    conf arr 0  1    1                         else                                  conf arr 0  0    conf arr 0  0    1                 elif int input arr i      2                          if float prob arr i    gt   0 5                                  conf arr 1  0    conf arr 1  0   1                         else                                  conf arr 1  1    conf arr 1  1   1          accuracy   float conf arr 0  0    conf arr 1  1    len input arr     prob arr is an array that my classification code returned and a sample array is like this      1 0  1 0  1 0  0 41592955657342651  1 0  0 0053405015805891975  4 5321494433440449e-299  1 0  1 0  1 0  1 0  1 0  1 0  0 70943426182688163  1 0  1 0  1 0  1 0    input arr is the original class labels for a dataset and it is like this     2  1  1  1  1  1  2  1  1  2  1  1  2  1  2  1  1  1    What my code is trying to do is  i get prob arr and input arr and for each class  1 and 2  I check if they are misclassified or not   But my code only works for two classes  If I run this code for a multiple classed data  it doesn t work  How can I make this for multiple classes    For example  for a data set with three classes  it should return    21 7 3   3 38 6   5 4 19

User · Answer

This function creates confusion matrices for any number of classes.

def create_conf_matrix(expected, predicted, n_classes):
    m = [[0] * n_classes for i in range(n_classes)]
    for pred, exp in zip(predicted, expected):
        m[pred][exp] += 1
    return m

def calc_accuracy(conf_matrix):
    t = sum(sum(l) for l in conf_matrix)
    return sum(conf_matrix[i][i] for i in range(len(conf_matrix))) / t

In contrast to your function above, you have to extract the predicted classes before calling the function, based on your classification results, i.e. sth. like

[1 if p < .5 else 2 for p in classifications]

User · Answer

Here is a simple implementation that handles an unequal number of classes in the predicted and actual labels  see examples 3 and 4   I hope this helps   For folks just learning this  here s a quick review  The labels for the columns indicate the predicted class  and the labels for the rows indicate the correct class  In example 1  we have  3 1  on the top row  Again  rows indicate truth  so this means that the correct label is  0  and there are 4 examples with ground truth label of  0   Columns indicate predictions  so we have 3 4 of the samples correctly labeled as  0   but 1 4 was incorrectly labeled as a  1    def confusion matrix actual  predicted       classes         np unique np concatenate  actual predicted        confusion mtx   np empty  len classes  len classes   dtype np int      for i a in enumerate classes           for j p in enumerate classes               confusion mtx i j    np where  actual  a   predicted  p   0  shape 0      return confusion mtx     Example 1   actual      np array  1 1 1 1 0 0 0 0   predicted   np array  1 1 1 1 0 0 0 1   confusion matrix actual predicted      0  1 0  3  1 1  0  4     Example 2   actual      np array   a   a   a   a   b   b   b   b    predicted   np array   a   a   a   a   b   b   b   a    confusion matrix actual predicted      0  1 0  4  0 1  1  3     Example 3   actual      np array   a   a   a   a   b   b   b   b    predicted   np array   a   a   a   a   b   b   b   z       lt -- notice the 3rd class   z  confusion matrix actual predicted      0  1  2 0  4  0  0 1  0  3  1 2  0  0  0     Example 4   actual      np array   a   a   a   x   x   b   b   b       lt -- notice the 4th class   x  predicted   np array   a   a   a   a   b   b   b   z    confusion matrix actual predicted      0  1  2  3 0  3  0  0  0 1  0  2  0  1 2  1  1  0  0 3  0  0  0  0

User · Answer

A numpy-only solution for any number of classes that doesn t require looping   import numpy as np  classes   3 true   np random randint 0  classes  50  pred   np random randint 0  classes  50   np bincount true   classes   pred  reshape  classes  classes

User · Answer

You should map from classes to a row in your confusion matrix   Here the mapping is trivial   def row of class classe       return  1  0  2  1  classe    In your loop  compute expected row  correct row  and increment conf arr expected row  correct row   You ll even have less code than what you started with

User · Answer

In a general sense  you re going to need to change your probability array   Instead of having one number for each instance and classifying based on whether or not it is greater than 0 5  you re going to need a list of scores  one for each class   then take the largest of the scores as the class that was chosen  a k a  argmax    You could use a dictionary to hold the probabilities for each classification   prob arr     classification id  probability          Choosing a classification would be something like   for instance scores in prob arr       predicted classes    cls for  cls  score  in instance scores iteritems   if score   max instance scores values       This handles the case where two classes have the same scores  You can get one score  by choosing the first one in that list  but how you handle that depends on what you re classifying   Once you have your list of predicted classes and a list of expected classes you can use code like Torsten Marek s to create the confusion array and calculate the accuracy

User · Answer

Nearly a decade has passed  yet the solutions  without sklearn  to this post are convoluted and unnecessarily long  Computing a confusion matrix can be done cleanly in Python in a few lines  For example   import numpy as np  def compute confusion matrix true  pred        Computes a confusion matrix using numpy for two np arrays   true and pred     Results are identical  and similar in computation time  to        from sklearn metrics import confusion matrix     However  this function avoids the dependency on sklearn        K   len np unique true     Number of classes    result   np zeros  K  K      for i in range len true        result true i   pred i      1    return result

User · Answer

A small change of cgnorthcutt s solution  considering the string type variables def get confusion matrix l1  l2        assert len l1   len l2    quot Two lists have different size  quot       K   len np unique l1          create label-index value     label index   dict zip np unique l1   np arange K         result   np zeros  K  K       for i in range len l1            result label index l1 i    label index l2 i       1      return result

User · Answer

Scikit-Learn provides a confusion matrix function  from sklearn metrics import confusion matrix y actu    2  0  2  2  0  1  1  2  2  0  1  2  y pred    0  0  2  1  0  2  1  0  2  0  2  2  confusion matrix y actu  y pred    which output a Numpy array  array   3  0  0           0  1  2           2  1  3      But you can also create a confusion matrix using Pandas   import pandas as pd y actu   pd Series  2  0  2  2  0  1  1  2  2  0  1  2   name  Actual   y pred   pd Series  0  0  2  1  0  2  1  0  2  0  2  2   name  Predicted   df confusion   pd crosstab y actu  y pred    You will get a  nicely labeled  Pandas DataFrame   Predicted  0  1  2 Actual 0          3  0  0 1          0  1  2 2          2  1  3   If you add margins True like  df confusion   pd crosstab y actu  y pred  rownames   Actual    colnames   Predicted    margins True    you will get also sum for each row and column   Predicted  0  1  2  All Actual 0          3  0  0    3 1          0  1  2    3 2          2  1  3    6 All        5  2  5   12   You can also get a normalized confusion matrix using   df conf norm   df confusion   df confusion sum axis 1   Predicted         0         1         2 Actual 0          1 000000  0 000000  0 000000 1          0 000000  0 333333  0 333333 2          0 666667  0 333333  0 500000   You can plot this confusion matrix using  import matplotlib pyplot as plt def plot confusion matrix df confusion  title  Confusion matrix   cmap plt cm gray r       plt matshow df confusion  cmap cmap    imshow      plt title title      plt colorbar       tick marks   np arange len df confusion columns       plt xticks tick marks  df confusion columns  rotation 45      plt yticks tick marks  df confusion index       plt tight layout       plt ylabel df confusion index name      plt xlabel df confusion columns name   plot confusion matrix df confusion      Or plot normalized confusion matrix using   plot confusion matrix df conf norm        You might also be interested by this project https   github com pandas-ml pandas-ml and its Pip package https   pypi python org pypi pandas ml  With this package confusion matrix can be pretty-printed  plot  You can binarize a confusion matrix  get class statistics such as TP  TN  FP  FN  ACC  TPR  FPR  FNR  TNR  SPC   LR   LR-  DOR  PPV  FDR  FOR  NPV and some overall statistics  In  1   from pandas ml import ConfusionMatrix In  2   y actu    2  0  2  2  0  1  1  2  2  0  1  2  In  3   y pred    0  0  2  1  0  2  1  0  2  0  2  2  In  4   cm   ConfusionMatrix y actu  y pred  In  5   cm print stats   Confusion Matrix   Predicted  0  1  2    all   Actual 0          3  0  0        3 1          0  1  2        3 2          2  1  3        6   all      5  2  5       12   Overall Statistics   Accuracy  0 583333333333 95  CI   0 27666968568210581  0 84834777019156982  No Information Rate  ToDo P-Value  Acc  gt  NIR   0 189264302376 Kappa  0 354838709677 Mcnemar s Test P-Value  ToDo   Class Statistics   Classes                                        0          1          2 Population                                    12         12         12 P  Condition positive                          3          3          6 N  Condition negative                          9          9          6 Test outcome positive                          5          2          5 Test outcome negative                          7         10          7 TP  True Positive                              3          1          3 TN  True Negative                              7          8          4 FP  False Positive                             2          1          2 FN  False Negative                             0          2          3 TPR   Sensitivity  hit rate  recall            1  0 3333333        0 5 TNR SPC   Specificity                  0 7777778  0 8888889  0 6666667 PPV  Pos Pred Value  Precision               0 6        0 5        0 6 NPV  Neg Pred Value                            1        0 8  0 5714286 FPR  False-out                         0 2222222  0 1111111  0 3333333 FDR  False Discovery Rate                    0 4        0 5        0 4 FNR  Miss Rate                                 0  0 6666667        0 5 ACC  Accuracy                          0 8333333       0 75  0 5833333 F1 score                                    0 75        0 4  0 5454545 MCC  Matthews correlation coefficient  0 6831301  0 2581989  0 1690309 Informedness                           0 7777778  0 2222222  0 1666667 Markedness                                   0 6        0 3  0 1714286 Prevalence                                  0 25       0 25        0 5 LR   Positive likelihood ratio               4 5          3        1 5 LR-  Negative likelihood ratio                 0       0 75       0 75 DOR  Diagnostic odds ratio                   inf          4          2 FOR  False omission rate                       0        0 2  0 4285714   I noticed that a new Python library about Confusion Matrix named PyCM is out  maybe you can have a look

User · Answer

If you don t want scikit-learn to do the work for you         import numpy     actual   numpy array actual      predicted   numpy array predicted         calculate the confusion matrix  labels is numpy array of classification labels     cm   numpy zeros  len labels   len labels        for a  p in zip actual  predicted           cm a  p     1        also get the accuracy easily with numpy     accuracy    actual    predicted  sum     float len actual     Or take a look at a more complete implementation here in NLTK

User · Answer

I wrote a simple class to build a confusion matrix without the need to depend on a machine learning library   The class can be used such as   labels     cat    dog    velociraptor    kraken    pony   confusionMatrix   ConfusionMatrix labels   confusionMatrix update  cat    cat   confusionMatrix update  cat    dog       confusionMatrix update  kraken    velociraptor   confusionMatrix update  velociraptor    velociraptor    confusionMatrix plot     The class ConfusionMatrix   import pylab import collections import numpy as np   class ConfusionMatrix      def   init   self  labels           self labels   labels         self confusion dictionary   self build confusion dictionary labels       def update self  predicted label  expected label           self confusion dictionary expected label  predicted label     1      def build confusion dictionary self  label set           expected labels   collections OrderedDict            for expected label in label set              expected labels expected label    collections OrderedDict                for predicted label in label set                  expected labels expected label  predicted label    0 0          return expected labels      def convert to matrix self  dictionary           length   len dictionary          confusion dictionary   np zeros  length  length            i   0         for row in dictionary              j   0             for column in dictionary                  confusion dictionary i  j    dictionary row  column                  j    1             i    1          return confusion dictionary      def get confusion matrix self           matrix   self convert to matrix self confusion dictionary          return self normalize matrix       def normalize self  matrix           amin   np amin matrix          amax   np amax matrix           return      y - amin     1 - 0      amax - amin   for y in x  for x in matrix       def plot self           matrix   self get confusion matrix            pylab figure           pylab imshow matrix  interpolation  nearest   cmap pylab cm jet          pylab title  Confusion Matrix            for i  vi in enumerate matrix               for j  vj in enumerate vi                   pylab text j  i  1     1f    vj  fontsize 12           pylab colorbar            classes   np arange len self labels           pylab xticks classes  self labels          pylab yticks classes  self labels           pylab ylabel  Expected label           pylab xlabel  Predicted label           pylab show

User · Answer

Only with numpy  we can do as follow considering efficiency   def confusion matrix pred  label  nc None       assert pred size    label size     if nc is None          nc   len unique label           logging debug  Number of classes assumed to be     format nc        confusion   np zeros  nc  nc         avoid the confusion with  0      tran pred   pred   1     for i in xrange nc        current class         mask    label    i          masked pred   mask   tran pred         cls  counts   unique masked pred  return counts True            discard the first item         cls    cl - 1 for cl in cls  1           counts   counts 1           for cl  count in zip cls  counts               confusion i  cl    count     return confusion   For other features such as plot  mean-IoU  see my repositories

User · Answer

You can make your code more concise and  sometimes  to run faster using numpy  For example  in two-classes case your function can be rewritten as  see mply acc      def accuracy actual  predicted          accuracy    tp   tn    ts        where               ts - Total Samples         tp - True Positives         tn - True Negatives             return  actual    predicted  sum     float len actual       where   actual       numpy array input arr     2  predicted    numpy array prob arr   lt  0 5

User · Answer

A Dependency Free Multiclass Confusion Matrix   A Simple Confusion Matrix Implementation def confusionmatrix actual  predicted  normalize   False        quot  quot  quot      Generate a confusion matrix for multiple classification      params          actual      - a list of integers or strings for known classes         predicted   - a list of integers or strings for predicted classes         normalize   - optional boolean for matrix normalization      return          matrix      - a 2-dimensional list of pairwise counts      quot  quot  quot      unique   sorted set actual       matrix     0 for   in unique  for   in unique      imap      key  i for i  key in enumerate unique         Generate Confusion Matrix     for p  a in zip predicted  actual           matrix imap p   imap a      1       Matrix Normalization     if normalize          sigma   sum  sum matrix imap i    for i in unique           matrix    row for row in map lambda i  list map lambda j  j   sigma  i    matrix       return matrix  The approach here is to pair up the unique classes found in the actual vector into a 2-dimensional list  From there  we simply iterate through the zipped actual and predicted vectors and populate the counts using the indices to access the matrix positions  Usage cm   confusionmatrix       1  1  2  0  1  1  2  0  0  1     actual      0  1  1  0  2  1  2  2  0  2     predicted      And The Output print cm    2  1  0    0  2  1    1  2  1    Note  the actual classes are along the columns and the predicted classes are along the rows    Actual   0  1  2                2  1  0     0   0  2  1     1  Predicted   1  2  1     2  Class Names Can be Strings or Integers cm   confusionmatrix        quot B quot    quot B quot    quot C quot    quot A quot    quot B quot    quot B quot    quot C quot    quot A quot    quot A quot    quot B quot      actual       quot A quot    quot B quot    quot B quot    quot A quot    quot C quot    quot B quot    quot C quot    quot C quot    quot A quot    quot C quot      predicted      And The Output print cm    2  1  0    0  2  1    1  2  1    You Can Also Return The Matrix With Proportions  Normalization  cm   confusionmatrix        quot B quot    quot B quot    quot C quot    quot A quot    quot B quot    quot B quot    quot C quot    quot A quot    quot A quot    quot B quot      actual       quot A quot    quot B quot    quot B quot    quot A quot    quot C quot    quot B quot    quot C quot    quot C quot    quot A quot    quot C quot      predicted     normalize   True      And The Output print cm    0 2  0 1  0 0    0 0  0 2  0 1    0 1  0 2  0 1    A More Robust Solution Since writing this post  I ve updated my library implementation to be a class  that uses a confusion matrix representation internally to compute statistics  in addition to pretty printing the confusion matrix itself  See this Gist  Example Usage   Actual  amp  Predicted Classes actual          quot A quot    quot B quot    quot C quot    quot C quot    quot B quot    quot C quot    quot C quot    quot B quot    quot A quot    quot A quot    quot B quot    quot A quot    quot B quot    quot C quot    quot A quot    quot B quot    quot C quot   predicted       quot A quot    quot B quot    quot B quot    quot C quot    quot A quot    quot C quot    quot A quot    quot B quot    quot C quot    quot A quot    quot B quot    quot B quot    quot B quot    quot C quot    quot A quot    quot A quot    quot C quot      Initialize Performance Class performance   Performance actual  predicted     Print Confusion Matrix performance tabulate    With the output                                              A       B       C   A       3       2       1  B       1       4       1  C       1       0       4  Note  class    Predicted  class    Actual                                      And for the normalized matrix    Print Normalized Confusion Matrix performance tabulate normalized   True   With the normalized output                                              A       B       C   A       17 65   11 76   5 88   B       5 88    23 53   5 88   C       5 88    0 00    23 53   Note  class    Predicted  class    Actual

User · Answer

Scikit-learn  which I recommend using anyways  has it included in the metrics module    gt  gt  gt  from sklearn metrics import confusion matrix  gt  gt  gt  y true    0  1  2  0  1  2  0  1  2   gt  gt  gt  y pred    0  0  0  0  1  1  0  2  2   gt  gt  gt  confusion matrix y true  y pred  array   3  0  0           1  1  1           1  1  1

[python] How to write a confusion matrix in Python?

Examples related to python

Examples related to machine-learning