How to extract multiple JSON objects from one file

Question

I am very new to Json files  If I have a json file with multiple json objects such as following     ID   12345   Timestamp   20140101    Usefulness   Yes     Code     event1   A   result   1           ID   1A35B   Timestamp   20140102    Usefulness   No     Code     event1   B   result   1           ID   AA356   Timestamp   20140103    Usefulness   No     Code     event1   B   result   0               I want to extract all  Timestamp  and  Usefulness  into a data frames       Timestamp    Usefulness  0   20140101      Yes  1   20140102      No  2   20140103      No        Does anyone know a general way to deal with such problems

User · Answer

Added streaming support based on the answer of  dunes   import re from json import JSONDecoder  JSONDecodeError  NOT WHITESPACE   re compile r    s      def stream json file obj  buf size 1024  decoder JSONDecoder         buf          ex   None     while True          block   file obj read buf size          if not block              break         buf    block         pos   0         while True              match   NOT WHITESPACE search buf  pos              if not match                  break             pos   match start               try                  obj  pos   decoder raw decode buf  pos              except JSONDecodeError as e                  ex   e                 break             else                  ex   None                 yield obj         buf   buf pos       if ex is not None          raise ex

User · Answer

So  as was mentioned in a couple comments containing the data in an array is simpler but the solution does not scale well in terms of efficiency as the data set size increases  You really should only use an iterator when you want to access a random object in the array  otherwise  generators are the way to go  Below I have prototyped a reader function which reads each json object individually and returns a generator  The basic idea is to signal the reader to split on the carriage character  quot  n quot   or  quot  r n quot  for Windows   Python can do this with the file readline   function  import json def json reader filename       with open filename  as f          for line in f              yield json loads line   However  this method only really works when the file is written as you have it -- with each object separated by a newline character  Below I wrote an example of a writer that separates an array of json objects and saves each one on a new line  def json writer file  json objects       with open file   quot w quot   as f          for jsonobj in json objects              jsonstr   json dumps jsonobj              f write jsonstr    quot  n quot    You could also do the same operation with file writelines   and a list comprehension          json strs    json dumps j     quot  n quot  for j in json objects      f writelines json strs       And if you wanted to append the data instead of writing a new file just change open file   quot w quot   to open file   quot a quot    In the end I find this helps a great deal not only with readability when I try and open json files in a text editor but also in terms of using memory more efficiently  On that note if you change your mind at some point and you want a list out of the reader  Python allows you to put a generator function inside of a list and populate the list automatically  In other words  just write lst   list json reader file

User · Answer

Update  I wrote a solution that doesn t require reading the entire file in one go  It s too big for a stackoverflow answer  but can be found here jsonstream  You can use json JSONDecoder raw decode to decode arbitarily big strings of  quot stacked quot  JSON  so long as they can fit in memory   raw decode stops once it has a valid object and returns the last position where wasn t part of the parsed object  It s not documented  but you can pass this position back to raw decode and it start parsing again from that position  Unfortunately  the Python json module doesn t accept strings that have prefixing whitespace  So we need to search to find the first none-whitespace part of your document  from json import JSONDecoder  JSONDecodeError import re  NOT WHITESPACE   re compile r    s     def decode stacked document  pos 0  decoder JSONDecoder         while True          match   NOT WHITESPACE search document  pos          if not match              return         pos   match start                    try              obj  pos   decoder raw decode document  pos          except JSONDecodeError                do something sensible if there s some error             raise         yield obj  s    quot  quot  quot     quot a quot   1           1      2      quot  quot  quot   for obj in decode stacked s       print obj   prints    a   1   1  2

User · Answer

Use a json array  in the format        ID   12345   Timestamp   20140101    Usefulness   Yes      Code     event1   A   result   1            ID   1A35B   Timestamp   20140102    Usefulness   No      Code     event1   B   result   1            ID   AA356   Timestamp   20140103    Usefulness   No      Code     event1   B   result   0                  Then import it into your python code  import json  with open  file json   as json file       data   json load json file    Now the content of data is an array with dictionaries representing each of the elements   You can access it easily  i e   data 0   ID

[python] How to extract multiple JSON objects from one file?

Examples related to python

Examples related to json

Examples related to pandas

Examples related to dataframe

Examples related to parsing