NumPy array is not JSON serializable

Question

After creating a NumPy array  and saving it as a Django context variable  I receive the following error when loading the webpage   array     0   239   479   717   952  1192  1432  1667   dtype int64  is not JSON serializable   What does this mean

User · Answer

This is a different answer  but this might help to help people who are trying to save data and then read it again  There is hickle which is faster than pickle and easier  I tried to save and read it in pickle dump but while reading there were lot of problems and wasted an hour and still didn t find solution though I was working on my own data to create a chat bot    vec x and vec y are numpy arrays   data  vec x vec y  hkl dump  data   new data file hkl      Then you just read it and perform the operations   data2   hkl load   new data file hkl

User · Answer

I had a similar problem with a nested dictionary with some numpy ndarrays in it    def jsonify data       json data   dict       for key  value in data iteritems            if isinstance value  list     for lists             value     jsonify item  if isinstance item  dict  else item for item in value           if isinstance value  dict     for nested lists             value   jsonify value          if isinstance key  int     if key is integer   gt  to string             key   str key          if type value    module     numpy     if value is numpy     gt  to python list             value   value tolist           json data key    value     return json data

User · Answer

I found the best solution if you have nested numpy arrays in a dictionary   import json import numpy as np  class NumpyEncoder json JSONEncoder           Special json encoder for numpy types         def default self  obj           if isinstance obj  np integer               return int obj          elif isinstance obj  np floating               return float obj          elif isinstance obj  np ndarray               return obj tolist           return json JSONEncoder default self  obj   dumped   json dumps data  cls NumpyEncoder   with open path   w   as f      json dump dumped  f    Thanks to this guy

User · Answer

You could also use default argument for example   def myconverter o       if isinstance o  np float32           return float o   json dump data  default myconverter

User · Answer

use NumpyEncoder it will process json dump successfully without throwing - NumPy array is not JSON serializable  import numpy as np import json from numpyencoder import NumpyEncoder arr   array     0   239   479   717   952  1192  1432  1667   dtype int64   json dumps arr cls NumpyEncoder

User · Answer

TypeError  array   0 46872085  0 67374235  1 0218339   0 13210179  0 5440686   0 9140083   0 58720225  0 2199381     dtype float32  is not JSON serializable  The above-mentioned  error was thrown when i tried to pass of list of data to model predict   when i  was expecting the response in json format    gt  1        json file   open  model json   r    gt  2        loaded model json   json file read    gt  3        json file close    gt  4        loaded model   model from json loaded model json   gt  5         load weights into new model  gt  6        loaded model load weights  model h5    gt  7        loaded model compile optimizer  adam   loss  mean squared error    gt  8        X      874 12450 678 0 922500 0 113569    gt  9        d   pd DataFrame X   gt  10       prediction   loaded model predict d   gt  11       return jsonify prediction    But luckily found the hint to resolve the error that was throwing    The serializing of the objects is applicable only for the following conversion   Mapping should be in following way   object - dict   array - list   string - string   integer - integer  If you scroll up to see the line number 10    prediction   loaded model predict d   where this line of code was generating the output    of type array datatype   when you try to convert array to json format its not possible  Finally i found the solution just by converting obtained output to the type list by    following lines of code     prediction   loaded model predict d    listtype   prediction tolist     return jsonify listtype    Bhoom  finally got the expected output

User · Answer

This is not supported by default  but you can make it work quite easily  There are several things you ll want to encode if you want the exact same data back    The data itself  which you can get with obj tolist   as  travelingbones mentioned  Sometimes this may be good enough  The data type  I feel this is important in quite some cases  The dimension  not necessarily 2D   which could be derived from the above if you assume the input is indeed always a  rectangular  grid  The memory order  row- or column-major   This doesn t often matter  but sometimes it does  e g  performance   so why not save everything    Furthermore  your numpy array could part of your data structure  e g  you have a list with some matrices inside  For that you could use a custom encoder which basically does the above   This should be enough to implement a solution  Or you could use json-tricks which does just this  and supports various other types   disclaimer  I made it    pip install json-tricks   Then  data         arange 0  10  1  dtype int  reshape  2  5        datetime year 2017  month 1  day 19  hour 23  minute 00  second 00       1   2j      Decimal 42       Fraction 1  3       MyTestCls s  ub   dct   7   7       see later     set range 7        Encode with metadata to preserve types when decoding print dumps data

User · Answer

Here is an implementation that work for me and removed all nans  assuming these are simple object  list or dict     from numpy import isnan  def remove nans my obj  val None       if isinstance my obj  list           for i  item in enumerate my obj               if isinstance item  list  or isinstance item  dict                   my obj i    remove nans my obj i   val val               else                  try                      if isnan item                           my obj i    val                 except Exception                      pass      elif isinstance my obj  dict           for key  item in my obj iteritems                if isinstance item  list  or isinstance item  dict                   my obj key    remove nans my obj key   val val               else                  try                      if isnan item                           my obj key    val                 except Exception                      pass      return my obj

User · Answer

Store as JSON a numpy ndarray or any nested-list composition   class NumpyEncoder json JSONEncoder       def default self  obj           if isinstance obj  np ndarray               return obj tolist           return json JSONEncoder default self  obj   a   np array   1  2  3    4  5  6    print a shape  json dump   json dumps   a   a   aa    2   2  3  4   a    bb    2    cls NumpyEncoder  print json dump    Will output    2  3    a     1  2  3    4  5  6     aa    2   2  3  4     1  2  3    4  5  6      bb    2     To restore from JSON   json load   json loads json dump  a restored   np asarray json load  a    print a restored  print a restored shape    Will output     1 2 3    4 5 6    2  3

User · Answer

I regularly  jsonify  np arrays  Try using the   tolist    method on the arrays first  like this    import numpy as np import codecs  json   a   np arange 10  reshape 2 5    a 2 by 5 array b   a tolist     nested lists with same data  indices file path     path json     your path variable json dump b  codecs open file path   w   encoding  utf-8    separators             sort keys True  indent 4      this saves the array in  json format   In order to  unjsonify  the array use    obj text   codecs open file path   r   encoding  utf-8   read   b new   json loads obj text  a new   np array b new

User · Answer

You can use Pandas   import pandas as pd pd Series your array  to json orient  values

User · Answer

Use the json dumps default kwarg   default should be a function that gets called for objects that can   t otherwise be serialized       or raise a TypeError  In the default function check if the object is from the module numpy  if so either use ndarray tolist for a ndarray or use  item for any other numpy specific type  import numpy as np  def default obj       if type obj    module      np   name            if isinstance obj  np ndarray               return obj tolist           else              return obj item       raise TypeError  Unknown type    type obj    dumped   json dumps data  default default

User · Answer

May do simple for loop with checking types   with open  jsondontdoit json    w   as fp      for key in bests keys            if type bests key      np ndarray              bests key    bests key  tolist               continue         for idx in bests key               if type bests key  idx      np ndarray                  bests key  idx    bests key  idx  tolist       json dump bests  fp      fp close

User · Answer

Also  some very interesting information further on lists vs  arrays in Python    Python List vs  Array - when to use   It could be noted that once I convert my arrays into a list before saving it in a JSON file  in my deployment right now anyways  once I read that JSON file for use later  I can continue to use it in a list form  as opposed to converting it back to an array     AND actually looks nicer  in my opinion  on the screen as a list  comma seperated  vs  an array  not-comma seperated  this way    Using  travelingbones s  tolist   method above  I ve been using as such  catching a few errors I ve found too    SAVE DICTIONARY  def writeDict values  name       writeName   DIR name   json      with open writeName   w   as outfile          json dump values  outfile    READ DICTIONARY  def readDict name       readName   DIR name   json      try          with open readName   r   as infile              dictValues   json load infile              return dictValues      except IOError as e          print e          return  None       except ValueError as e          print e          return  None     Hope this helps

[python] NumPy array is not JSON serializable

Examples related to python

Examples related to json

Examples related to django

Examples related to numpy