Find empty or NaN entry in Pandas Dataframe

Question

I am trying to search through a Pandas Dataframe to find where it has a missing entry or a NaN entry   Here is a dataframe that I am working with   cl id       a           c         d         e        A1              A2             A3     0       1   -0 419279  0 843832 -0 530827    text76        1 537177      -0 271042     1       2    0 581566  2 257544  0 440485    dafN 6        0 144228       2 362259     2       3   -1 259333  1 074986  1 834653    system                       1 100353     3       4   -1 279785  0 272977  0 197011     Fifty       -0 031721       1 434273     4       5    0 578348  0 595515  0 553483   channel        0 640708       0 649132     5       6   -1 549588 -0 198588  0 373476     audio       -0 508501                    6       7    0 172863  1 874987  1 405923    Twenty             NaN            NaN     7       8   -0 149630 -0 502117  0 315323  file max             NaN            NaN   NOTE  The blank entries are empty strings - this is because there was no alphanumeric content in the file that the dataframe came from   If I have this dataframe  how can I find a list with the indexes where the NaN or blank entry occurs

User · Answer

Partial solution  for a single string column tmp   df  A1   fillna      isEmpty   tmp      gives boolean Series of True where there are empty strings or NaN values

User · Answer

Try this   df df  column name          index   and for NaNs you can try   pd isna df  column name

User · Answer

I ve resorted to   df   df column name  notnull     amp   df column name   u      index  lately   That gets both null and empty-string cells in one go

User · Answer

Another opltion covering cases where there might be severar spaces is by using the isspace   python function  df df col name apply lambda x x isspace      False     will only return cases without empty spaces  adding NaN values  df  df col name apply lambda x x isspace      False   amp    df col name isna

User · Answer

np where pd isnull df   returns the row and column indices where the value is NaN   In  152   import numpy as np In  153   import pandas as pd In  154   np where pd isnull df   Out 154    array  2  5  6  6  7  7    array  7  7  6  7  6  7     In  155   df iloc 2 7  Out 155   nan  In  160    df iloc i j  for i j in zip  np where pd isnull df     Out 160    nan  nan  nan  nan  nan  nan    Finding values which are empty strings could be done with applymap   In  182   np where df applymap lambda x  x         Out 182    array  5    array  7      Note that using applymap requires calling a Python function once for each cell of the DataFrame  That could be slow for a large DataFrame  so it would be better if you could arrange for all the blank cells to contain NaN instead so you could use pd isnull

User · Answer

you also do something good   text empty   df  column name   str len    gt  -1  df loc text empty  index  The results will be the rows which are empty  amp  it s index number

User · Answer

To obtain all the rows that contains an empty cell in in a particular column   DF new row DF raw loc DF raw  columnname          This will give the subset of DF raw  which satisfy the checking condition

User · Answer

Check if the columns contain Nan using  isnull   and check for empty strings using  eq      then join the two together using the bitwise OR operator     Sum along axis 0 to find columns with missing data  then sum along axis 1 to the index locations for rows with missing data   missing cols  missing rows          df2 isnull   sum x    df2 eq     sum x        loc lambda x  x gt 0   index     for x in  0  1      gt  gt  gt  df2 loc missing rows  missing cols           A2       A3 2            1 10035 5 -0 508501          6       NaN      NaN 7       NaN      NaN

[list] Find empty or NaN entry in Pandas Dataframe

Examples related to list

Examples related to python-2.7

Examples related to pandas

Examples related to indexing

Examples related to dataframe