TypeError can t pickle thread lock objects

Question

Trying to run two different functions at the same time with shared queue and get an error   how can I run two functions at the same time with a shared queue  This is Python version 3 6 on Windows 7    from multiprocessing import Process from queue import Queue import logging  def main        x   DataGenerator       try          x run       except Exception as e          logging exception  message     class DataGenerator       def   init   self           logging basicConfig filename  testing log   level logging INFO       def run self           logging info  Running Generator           queue   Queue           Process target self package  args  queue    start           logging info  Process started to generate data           Process target self send  args  queue    start           logging info  Process started to send data         def package self  queue            while True              for i in range 16                   datagram   bytearray                   datagram append i                  queue put datagram       def send self  queue           byte array   bytearray           while True              size of  queue   queue qsize               logging info   queue size  s   size of queue              if size of queue  gt  7                  for i in range 1  8                       packet   queue get                       byte array append packet                  logging info  Sending datagram                    print str datagram                   byte array 0   if   name         main         main     The logs indicate an error  I tried running console as administrator and I get the same message     INFO root Running Generator ERROR root message Traceback  most recent call last     File  test py   line 8  in main     x run     File  test py   line 20  in run     Process target self package  args  queue    start     File  C  ProgramData Miniconda3 lib multiprocessing process py   line 105  in start     self  popen   self  Popen self    File  C  ProgramData Miniconda3 lib multiprocessing context py   line 223  in  Popen     return  default context get context   Process  Popen process obj    File  C  ProgramData Miniconda3 lib multiprocessing context py   line 322  in  Popen     return Popen process obj    File  C  ProgramData Miniconda3 lib multiprocessing popen spawn win32 py   line 65  in   init       reduction dump process obj  to child    File  C  ProgramData Miniconda3 lib multiprocessing reduction py   line 60  in dump     ForkingPickler file  protocol  dump obj  TypeError  can t pickle  thread lock objects

User · Answer

You need to change from queue import Queue to from multiprocessing import Queue   The root reason is the former Queue is designed for threading module Queue while the latter is for multiprocessing Process module   For details  you can read some source code or contact me

User · Answer

multiprocessing Pool - PicklingError  Can  39 t pickle  lt type   39 thread lock  39  gt   attribute lookup thread lock failed  Move the queue to self instead of as an argument to your functions package and send

User · Answer

I had the same problem with Pool   in Python 3 6 3   Error received  TypeError  can t pickle  thread RLock objects  Let s say we want to add some number num to add to each element of some list num list in parallel  The code is schematically like this   class DataGenerator      def   init   self  num list  num to add          self num list   num list   e g   4 2 5 7          self num to add   num to add   e g  1           self run        def run self           new num list   Manager   list            pool   Pool processes 50          results    pool apply async run parallel   num  new num list                          for num in num list          roots    r get   for r in results          pool close           pool terminate           pool join        def run parallel self  num  shared new num list           new num   num   self num to add   uses class parameter         shared new num list append new num    The problem here is that self in function run parallel   can t be pickled as it is a class instance  Moving this parallelized function run parallel   out of the class helped  But it s not the best solution as this function probably needs to use class parameters like self num to add and then you have to pass it as an argument   Solution   def run parallel num  shared new num list  to add     to add is passed as an argument     new num   num   to add     shared new num list append new num   class DataGenerator      def   init   self  num list  num to add          self num list   num list   e g   4 2 5 7          self num to add   num to add   e g  1          self run        def run self           new num list   Manager   list            pool   Pool processes 50          results    pool apply async run parallel   num  new num list  self num to add     num to add is passed as an argument                       for num in num list          roots    r get   for r in results          pool close           pool terminate           pool join     Other suggestions above didn t help me

[python-3.x] TypeError: can't pickle _thread.lock objects

Examples related to python-3.x