Plot correlation matrix using pandas

Question

I have a data set with huge number of features  so analysing the correlation matrix has become very difficult  I want to plot a correlation matrix which we get using dataframe corr   function from pandas library  Is there any built-in function provided by the pandas library to plot this matrix

User · Answer

Along with other methods it is also good to have pairplot which will give scatter plot for all the cases-  import pandas as pd import numpy as np import seaborn as sns rs   np random RandomState 0  df   pd DataFrame rs rand 10  10   sns pairplot df

User · Answer

You can use pyplot matshow    from matplotlib  import matplotlib pyplot as plt  plt matshow dataframe corr    plt show     Edit  In the comments was a request for how to change the axis tick labels  Here s a deluxe version that is drawn on a bigger figure size  has axis labels to match the dataframe  and a colorbar legend to interpret the color scale  I m including how to adjust the size and rotation of the labels  and I m using a figure ratio that makes the colorbar and the main figure come out the same height   EDIT 2  As the df corr   method ignores non-numerical columns   select dtypes   number    should be used when defining the x and y labels to avoid an unwanted shift of the labels  included in the code below   f   plt figure figsize  19  15   plt matshow df corr    fignum f number  plt xticks range df select dtypes   number    shape 1    df select dtypes   number    columns  fontsize 14  rotation 45  plt yticks range df select dtypes   number    shape 1    df select dtypes   number    columns  fontsize 14  cb   plt colorbar   cb ax tick params labelsize 14  plt title  Correlation Matrix   fontsize 16

User · Answer

For completeness  the simplest solution i know with seaborn as of late 2019  if one is using Jupyter   import seaborn as sns sns heatmap dataframe corr

User · Answer

You can use imshow   method from matplotlib  import pandas as pd import matplotlib pyplot as plt plt style use  ggplot    plt imshow X corr    cmap plt cm Reds  interpolation  nearest   plt colorbar   tick marks    i for i in range len X columns    plt xticks tick marks  X columns  rotation  vertical   plt yticks tick marks  X columns  plt show

User · Answer

Seaborn s heatmap version   import seaborn as sns corr   dataframe corr   sns heatmap corr               xticklabels corr columns values              yticklabels corr columns values

User · Answer

If you dataframe is df you can simply use   import matplotlib pyplot as plt import seaborn as sns  plt figure figsize  15  10   sns heatmap df corr    annot True

User · Answer

You can observe the relation between features either by drawing a heat map from seaborn or scatter matrix from pandas    Scatter Matrix   pd scatter matrix dataframe  alpha   0 3  figsize    14 8   diagonal    kde      If you want to visualize each feature s skewness as well - use seaborn pairplots    sns pairplot dataframe    Sns Heatmap   import seaborn as sns  f  ax   pl subplots figsize  10  8   corr   dataframe corr   sns heatmap corr  mask np zeros like corr  dtype np bool   cmap sns diverging palette 220  10  as cmap True               square True  ax ax    The output will be a correlation map of the features  i e  see the below example      The correlation between grocery and detergents is high  Similarly   Pdoducts With High Correlation    Grocery and Detergents     Products With Medium Correlation    Milk and Grocery Milk and Detergents Paper   Products With Low Correlation    Milk and Deli Frozen and Fresh  Frozen and Deli     From Pairplots  You can observe same set of relations from pairplots or scatter matrix  But from these we can say that whether the data is normally distributed or not      Note  The above is same graph taken from the data  which is used to draw heatmap

User · Answer

If your main goal is to visualize the correlation matrix  rather than creating a plot per se  the convenient pandas styling options is a viable built-in solution   import pandas as pd import numpy as np  rs   np random RandomState 0  df   pd DataFrame rs rand 10  10   corr   df corr   corr style background gradient cmap  coolwarm      RdBu r   amp   BrBG  are other good diverging colormaps     Note that this needs to be in a backend that supports rendering HTML  such as the JupyterLab Notebook   The automatic light text on dark backgrounds is from an existing PR and not the latest released version  pandas 0 23      Styling  You can easily limit the digit precision   corr style background gradient cmap  coolwarm   set precision 2      Or get rid of the digits altogether if you prefer the matrix without annotations   corr style background gradient cmap  coolwarm   set properties     font-size    0pt        The styling documentation also includes instructions of more advanced styles  such as how to change the display of the cell the mouse pointer is hovering over  To save the output you could return the HTML by appending the render   method and then write it to a file  or just take a screenshot for less formal purposes      Time comparison  In my testing  style background gradient   was 4x faster than plt matshow   and 120x faster than sns heatmap   with a 10x10 matrix  Unfortunately it doesn t scale as well as plt matshow    the two take about the same time for a 100x100 matrix  and plt matshow   is 10x faster for a 1000x1000 matrix      Saving  There are a few possible ways to save the stylized dataframe    Return the HTML by appending the render   method and then write the output to a file  Save as an  xslx file with conditional formatting by appending the to excel   method  Combine with imgkit to save a bitmap Take a screenshot  for less formal purposes       Update for pandas    0 24  By setting axis None  it is now possible to compute the colors based on the entire matrix rather than per column or per row   corr style background gradient cmap  coolwarm   axis None

User · Answer

Form correlation matrix  in my case zdf is the dataframe which i need perform correlation matrix   corrMatrix  zdf corr   corrMatrix to csv  sm zscaled correlation matrix csv    html   corrMatrix style background gradient cmap  RdBu   set precision 2  render      Writing the output to a html file  with open  test html    w   as f     print   lt  DOCTYPE html gt  lt html lang  en  gt  lt head gt  lt meta charset  UTF-8  gt  lt meta name  viewport  content  width device-widthinitial-scale 1 0  gt  lt title gt Document lt  title gt  lt  head gt  lt style gt table word-break  break-all   lt  style gt  lt body gt     html   lt  body gt  lt  html gt    file f    Then we can take screenshot  or convert html to an image file

User · Answer

statmodels graphics also gives a nice view of correlation matrix  import statsmodels api as sm import matplotlib pyplot as plt  corr   dataframe corr   sm graphics plot corr corr  xnames list corr columns   plt show

User · Answer

Surprised to see no one mentioned more capable  interactive and easier to use alternatives  A  You can use plotly   Just two lines and you get   interactivity   smooth scale   colors based on whole dataframe instead of individual columns   column names  amp  row indices on axes   zooming in   panning   built-in one-click ability to save it as a PNG format   auto-scaling   comparison on hovering   bubbles showing values so heatmap still looks good and you can see values wherever you want    import plotly express as px fig   px imshow df corr    fig show     B  You can also use Bokeh  All the same functionality with a tad much hassle  But still worth it if you do not want to opt-in for plotly and still want all these things  from bokeh plotting import figure  show  output notebook from bokeh models import ColumnDataSource  LinearColorMapper from bokeh transform import transform output notebook   colors      d7191c     fdae61     ffffbf     a6d96a     1a9641   TOOLS    quot hover save pan box zoom reset wheel zoom quot  data   df corr   stack   rename  quot value quot   reset index   p   figure x range list df columns   y range list df index   tools TOOLS  toolbar location  below              tooltips    Row  Column     level 0 x  level 1      value     value     height   500  width   500   p rect x  quot level 1 quot   y  quot level 0 quot   width 1  height 1         source data         fill color   field    value    transform   LinearColorMapper palette colors  low data value min    high data value max             line color None  color bar   ColorBar color mapper LinearColorMapper palette colors  low data value min    high data value max     major label text font size  quot 7px quot                        ticker BasicTicker desired num ticks len colors                         formatter PrintfTickFormatter format  quot  f quot                         label standoff 6  border line color None  location  0  0   p add layout color bar   right    show p

User · Answer

Try this function  which also displays variable names for the correlation matrix   def plot corr df size 10          Function plots a graphical correlation matrix for each pair of columns in the dataframe       Input          df  pandas DataFrame         size  vertical and horizontal size of the plot         corr   df corr       fig  ax   plt subplots figsize  size  size       ax matshow corr      plt xticks range len corr columns    corr columns       plt yticks range len corr columns    corr columns

[python] Plot correlation matrix using pandas

Examples related to python

Examples related to pandas

Examples related to matplotlib

Examples related to data-visualization

Examples related to information-visualization