TypeError Object of type bytes is not JSON serializable

Question

I just started programming Python  I want to use scrapy to create a bot and it showed  TypeError  Object of type  bytes  is not JSON serializable when I run the project    import json import codecs  class W3SchoolPipeline object      def   init   self         self file   codecs open  w3school data utf8 json    wb   encoding  utf-8      def process item self  item  spider         line   json dumps dict item       n          print line        self file write line decode  unicode escape          return item     from scrapy spiders import Spider from scrapy selector import Selector from w3school items import W3schoolItem  class W3schoolSpider Spider        name    w3school      allowed domains     w3school com cn        start urls              http   www w3school com cn xml xml syntax asp             def parse self  response           sel   Selector response          sites   sel xpath    div  id  navsecond   div  id  course   ul 1  li        items          for site in sites          item   W3schoolItem           title   site xpath  a text     extract           link   site xpath  a  href   extract           desc   site xpath  a  title   extract            item  title      t encode  utf-8   for t in title          item  link      l encode  utf-8   for l in link          item  desc      d encode  utf-8   for d in desc          items append item          return items   Traceback   TypeError  Object of type  bytes  is not JSON serializable 2017-06-23 01 41 15  scrapy core scraper  ERROR  Error processing         desc    b  x e4 xbd xbf xe7 x94 xa8 XSLT  xe6 x98 xbe xe7 xa4 xba XML      link    b  xml xml xsl asp      title    b XML XSLT     Traceback  most recent call last   File    c  users administrator appdata local programs python python36 lib site-p ackages twisted internet defer py   line 653  in  runCallbacks     current result   callback current result   args    kw  File  D  LZZZZB w3school w3school pipelines py   line 19  in process item     line   json dumps dict item       n  File   c  users administrator appdata local programs python python36 lib json    init   py   line 231  in dumps     return  default encoder encode obj  File   c  users administrator appdata local programs python python36 lib json e ncoder py   line 199  in encode     chunks   self iterencode o   one shot True  File    c  users administrator appdata local programs python python36 lib json e ncoder py   line 257  in iterencode     return  iterencode o  0  File        c  users administrator appdata local programs python python36 lib  json encoder py   line 180  in default     o   class     name      TypeError  Object of type  bytes  is not JSON serializable

User · Answer

I was dealing with this issue today  and I knew that I had something encoded as a bytes object that I was trying to serialize as json with json dump my json object  write to file json   my json object in this case was a very large json object that I had created  so I had several dicts  lists  and strings to look at to find what was still in bytes format   The way I ended up solving it  the write to file json will have everything up to the bytes object that is causing the issue   In my particular case this was a line obtained through  for line in text      json object  line     line strip     I solved by first finding this error with the help of the write to file json  then by correcting it to   for line in text      json object  line     line strip   decode

User · Answer

You are creating those bytes objects yourself   item  title      t encode  utf-8   for t in title  item  link      l encode  utf-8   for l in link  item  desc      d encode  utf-8   for d in desc  items append item    Each of those t encode    l encode   and d encode   calls creates a bytes string  Do not do this  leave it to the JSON format to serialise these   Next  you are making several other errors  you are encoding too much where there is no need to  Leave it to the json module and the standard file object returned by the open   call to handle encoding   You also don t need to convert your items list to a dictionary  it ll already be an object that can be JSON encoded directly   class W3SchoolPipeline object           def   init   self           self file   open  w3school data utf8 json    w   encoding  utf-8        def process item self  item  spider           line   json dumps item      n          self file write line          return item   I m guessing you followed a tutorial that assumed Python 2  you are using Python 3 instead  I strongly suggest you find a different tutorial  not only is it written for an outdated version of Python  if it is advocating line decode  unicode escape   it is teaching some extremely bad habits that ll lead to hard-to-track bugs  I can recommend you look at Think Python  2nd edition for a good  free  book on learning Python 3

User · Answer

I guess the answer you need is referenced here  Python sets are not json serializable  Not all datatypes can be json serialized    I guess pickle module will serve your purpose

[python] TypeError: Object of type 'bytes' is not JSON serializable

Examples related to python

Examples related to json

Examples related to serialization

Examples related to scrapy