Removing nan values from an array

Question

I want to figure out how to remove nan values from my array  My array looks something like this    x    1400  1500  1600  nan  nan  nan  1700   Not in this exact configuration   How can I remove the nan values from x

User · Answer

The accepted answer changes shape for 2d arrays. I present a solution here, using the Pandas dropna() functionality. It works for 1D and 2D arrays. In the 2D case you can choose weather to drop the row or column containing np.nan.

import pandas as pd
import numpy as np

def dropna(arr, *args, **kwarg):
    assert isinstance(arr, np.ndarray)
    dropped=pd.DataFrame(arr).dropna(*args, **kwarg).values
    if arr.ndim==1:
        dropped=dropped.flatten()
    return dropped

x = np.array([1400, 1500, 1600, np.nan, np.nan, np.nan ,1700])
y = np.array([[1400, 1500, 1600], [np.nan, 0, np.nan] ,[1700,1800,np.nan]] )


print('='*20+' 1D Case: ' +'='*20+'\nInput:\n',x,sep='')
print('\ndropna:\n',dropna(x),sep='')

print('\n\n'+'='*20+' 2D Case: ' +'='*20+'\nInput:\n',y,sep='')
print('\ndropna (rows):\n',dropna(y),sep='')
print('\ndropna (columns):\n',dropna(y,axis=1),sep='')

print('\n\n'+'='*20+' x[np.logical_not(np.isnan(x))] for 2D: ' +'='*20+'\nInput:\n',y,sep='')
print('\ndropna:\n',x[np.logical_not(np.isnan(x))],sep='')

Result:

==================== 1D Case: ====================
Input:
[1400. 1500. 1600.   nan   nan   nan 1700.]

dropna:
[1400. 1500. 1600. 1700.]


==================== 2D Case: ====================
Input:
[[1400. 1500. 1600.]
 [  nan    0.   nan]
 [1700. 1800.   nan]]

dropna (rows):
[[1400. 1500. 1600.]]

dropna (columns):
[[1500.]
 [   0.]
 [1800.]]


==================== x[np.logical_not(np.isnan(x))] for 2D: ====================
Input:
[[1400. 1500. 1600.]
 [  nan    0.   nan]
 [1700. 1800.   nan]]

dropna:
[1400. 1500. 1600. 1700.]

User · Answer

This is my approach to filter ndarray  X  for NaNs and infs      I create a map of rows without any NaN and any inf as follows   idx   np where  np isnan X   False   amp   np isinf X   False     idx is a tuple  It s second column  idx 1   contains the indices of the array  where no NaN nor inf where found across the row   Then   filtered X   X idx 1     filtered X contains X without NaN nor inf

User · Answer

If you re using numpy for your arrays  you can also use  x   x numpy logical not numpy isnan x      Equivalently   x   x  numpy isnan x      Thanks to chbrown for the added shorthand    Explanation  The inner function  numpy isnan returns a boolean logical array which has the value True everywhere that x is not-a-number  As we want the opposite  we use the logical-not operator    to get an array with Trues everywhere that x is a valid number   Lastly we use this logical array to index into the original array x  to retrieve just the non-NaN values

User · Answer

A simplest way is    numpy nan to num x    Documentation  https   docs scipy org doc numpy reference generated numpy nan to num html

User · Answer

As shown by others   x  numpy isnan x     works  But it will throw an error if the numpy dtype is not a native data type  for example if it is object  In that case you can use pandas   x  pandas isna x   or x  pandas isnull x

User · Answer

For me the answer by  jmetz didn t work  however using pandas isnull   did   x   x  pd isnull x

User · Answer

Try this   import math print  value for value in x if not math isnan value     For more  read on List Comprehensions

User · Answer

jmetz s answer is probably the one most people need  however it yields a one-dimensional array  e g  making it unusable to remove entire rows or columns in matrices   To do so  one should reduce the logical array to one dimension  then index the target array  For instance  the following will remove rows which have at least one NaN value   x   x  numpy isnan x  any axis 1     See more detail here

User · Answer

Doing the above    x   x  numpy isnan x     or  x   x numpy logical not numpy isnan x      I found that resetting to the same variable  x  did not remove the actual nan values and had to use a different variable  Setting it to a different variable removed the nans  e g    y   x  numpy isnan x

User · Answer

Simply fill with  x   numpy array     0 99929941  0 84724713  -0 1500044     -0 79709026  numpy NaN  -0 4406645     -0 3599013  -0 63565744  -0 70251352     x numpy isnan x      555  print x        0 99929941  0 84724713 -0 1500044       -0 79709026  0 555      -0 4406645       -0 3599013  -0 63565744 -0 70251352

User · Answer

If you re using numpy     first get the indices where the values are finite ii   np isfinite x     second get the values x   x ii

User · Answer

filter lambda v  v  v  x    works both for lists and numpy array since v  v only for NaN

[python] Removing nan values from an array

Examples related to python

Examples related to arrays

Examples related to numpy

Examples related to nan