TypeError ObjectId is not JSON serializable

Question

My response back from MongoDB after querying an aggregated function on document using Python  It returns valid response and i can print it but can not return it   Error    TypeError  ObjectId  51948e86c25f4b1d1c0d303c   is not JSON serializable   Print     result       id   ObjectId  51948e86c25f4b1d1c0d303c     api calls with key   4   api calls per day   0 375   api calls total   6   api calls without key   2     ok   1 0    But When i try to return   TypeError  ObjectId  51948e86c25f4b1d1c0d303c   is not JSON serializable   It is RESTfull call    appv1 route   v1 analytics   def get api analytics          get handle to collections in MongoDB     statistics   sldb statistics      objectid   ObjectId  51948e86c25f4b1d1c0d303c        analytics   statistics aggregate          match     owner   objectid           project     owner     owner        api calls with key      cond       eq      apikey   None    0  1         api calls without key      cond       ne      apikey   None    0  1                  group      id     owner        api calls with key      sum     api calls with key         api calls without key      sum     api calls without key                  project     api calls with key     api calls with key        api calls without key     api calls without key        api calls total      add      api calls with key     api calls without key          api calls per day      divide       add      api calls with key     api calls without key        dayOfMonth   datetime now                           print analytics       return analytics   db is well connected and collection is there too and I got back valid expected result but when i try to return it gives me Json error  Any idea how to convert the response back into JSON  Thanks

User · Answer

SOLUTION for  mongoengine   marshmallow  If you use mongoengine and marshamallow then this solution might be applicable for you     Basically  I imported String field from marshmallow  and I overwritten default Schema id to be String encoded   from marshmallow import Schema from marshmallow fields import String  class FrontendUserSchema Schema        id   String        class Meta          fields     id    email

User · Answer

Most users who receive the  not JSON serializable  error simply need to specify default str when using json dumps  For example   json dumps my obj  default str    This will force a conversion to str  preventing the error  Of course then look at the generated output to confirm that it is what you need

User · Answer

Pymongo provides json util - you can use that one instead to handle BSON types def parse json data       return json loads json util dumps data

User · Answer

I would like to provide an additional solution that improves the accepted answer  I have previously provided the answers in another thread here   from flask import Flask from flask json import JSONEncoder  from bson import json util  from   import resources    define a custom encoder point to the json util provided by pymongo  or its dependency bson  class CustomJSONEncoder JSONEncoder       def default self  obj   return json util default obj   application   Flask   name    application json encoder   CustomJSONEncoder  if   name         main         application run

User · Answer

from bson objectid import ObjectId from core services db connection import DbConnectionService  class DbExecutionService       def   init   self           self db   DbConnectionService         def list self  collection  search           session   self db create connection collection          return list map lambda row   i  str row i   if isinstance row i   ObjectId  else row i  for i in row   session find search

User · Answer

Flask s jsonify provides security enhancement as described in JSON Security  If custom encoder is used with Flask  its better to consider the  points discussed in the JSON Security

User · Answer

If you will not be needing the  id of the records I will recommend unsetting it when querying the DB which will enable you to print the returned records directly e g   To unset the  id when querying and then print data in a loop you write something like this  records   mycollection find query     id   0    second argument    id  0  unsets the id from the query for record in records      print record

User · Answer

Posting here as I think it may be useful for people using Flask with pymongo  This is my current  best practice  setup for allowing flask to marshall pymongo bson data types   mongoflask py  from datetime import datetime  date  import isodate as iso from bson import ObjectId from flask json import JSONEncoder from werkzeug routing import BaseConverter   class MongoJSONEncoder JSONEncoder       def default self  o           if isinstance o   datetime  date                return iso datetime isoformat o          if isinstance o  ObjectId               return str o          else              return super   default o    class ObjectIdConverter BaseConverter       def to python self  value           return ObjectId value       def to url self  value           return str value    app py  from  mongoflask import MongoJSONEncoder  ObjectIdConverter  def create app        app   Flask   name        app json encoder   MongoJSONEncoder     app url map converters  objectid     ObjectIdConverter        Client sends their string  we interpret it as an ObjectId      app route   users  lt objectid user id gt        def show user user id             setup not shown  pretend this gets us a pymongo db object         db   get db              user id is a bson ObjectId ready to use with pymongo          result   db users find one    id   user id              And jsonify returns normal looking json               id    5b6b6959828619572d48a9da               name    Will               birthday    1990-03-17T00 00 00Z           return jsonify result        return app   Why do this instead of serving BSON or mongod extended JSON   I think serving mongo special JSON puts a burden on client applications  Most client apps will not care using mongo objects in any complex way  If I serve extended json  now I have to use it server side  and the client side  ObjectId and Timestamp are easier to work with as strings and this keeps all this mongo marshalling madness quarantined to the server         id    5b6b6959828619572d48a9da      created at    2018-08-08T22 06 17Z      I think this is less onerous to work with for most applications than         id      oid    5b6b6959828619572d48a9da       created at      date   1533837843000

User · Answer

As a quick replacement  you can change   owner   objectid  to   owner   str objectid     But defining your own JSONEncoder is a better solution  it depends on your requirements

User · Answer

For those who need to return the data thru Jsonify with Flask  cursor   db collection find   data      for doc in cursor      doc   id     str doc   id      This does the trick      data append doc  return jsonify data

User · Answer

You should define you own JSONEncoder and using it   import json from bson import ObjectId  class JSONEncoder json JSONEncoder       def default self  o           if isinstance o  ObjectId               return str o          return json JSONEncoder default self  o   JSONEncoder   encode analytics    It s also possible to use it in the following way    json encode analytics  cls JSONEncoder

User · Answer

from bson import json util import json   app route      def index        for   in  collection name  find            return json dumps i  indent 4  default json util default    This is the sample example for converting BSON into JSON object  You can try this

User · Answer

gt  gt  gt  from bson import Binary  Code  gt  gt  gt  from bson json util import dumps  gt  gt  gt  dumps    foo    1  2                 bar     hello    world                  code   Code  function x     return 1                     bin   Binary            foo    1  2      bar     hello    world       code      code    function x     return 1        scope           bin      binary    AQIDBA       type    00        Actual example from json util    Unlike Flask s jsonify   dumps  will return a string  so it cannot be used as a 1 1 replacement of Flask s jsonify   But this question shows that we can serialize using json util dumps    convert back to dict using json loads   and finally call Flask s jsonify on it    Example  derived from previous question s answer    from bson import json util  ObjectId import json   Lets create some dummy document to prove it will work page     foo   ObjectId     bar    ObjectId    ObjectId       Dump loaded BSON to valid JSON string and reload it as dict page sanitized   json loads json util dumps page   return page sanitized   This solution will convert ObjectId and others  ie Binary  Code  etc  to a string equivalent such as   oid     JSON output would look like this         id           oid    abc123

User · Answer

If you don t want  id in response  you can refactor your code something like this  jsonResponse   getResponse mock data  del jsonResponse   id     removes   id  from the final response return jsonResponse  This will remove the TypeError  ObjectId     is not JSON serializable error

User · Answer

This is how I ve recently fixed the error       app route          def home            docs              for doc in db person find                doc pop   id                docs append doc          return jsonify docs

User · Answer

in my case I needed something like this   class JsonEncoder        def encode self  o           if   id  in o              o   id     str o   id            return o

User · Answer

I know I m posting late but thought it would help at least a few folks   Both the examples mentioned by tim and defuz which are top voted  works perfectly fine  However  there is a minute difference which could be significant at times    The following method adds one extra field which is redundant and may not be ideal in all the cases   Pymongo provides json util - you can use that one instead to handle BSON types  Output        id           oid    abc123          Where as the JsonEncoder class gives the same output in the string format as we need and we need to use json loads output  in addition  But it leads to    Output        id    abc123       Even though  the first method looks simple  both the method need very minimal effort

[python] TypeError: ObjectId('') is not JSON serializable

Examples related to python

Examples related to json

Examples related to mongodb

Examples related to flask