How can I use threading in Python

Question

I am trying to understand threading in Python  I ve looked at the documentation and examples  but quite frankly  many examples are overly sophisticated and I m having trouble understanding them   How do you clearly show tasks being divided for multi-threading

User · Answer

Given a function  f  thread it like this   import threading threading Thread target f  start     To pass arguments to f  threading Thread target f  args  a b c   start

User · Answer

None of the previous solutions actually used multiple cores on my GNU Linux server  where I don t have administrator rights   They just ran on a single core   I used the lower level os fork interface to spawn multiple processes  This is the code that worked for me   from os import fork  values     different    values    for    threads    for i in range len values        p   fork       if p    0          my function values i           break

User · Answer

Since this question was asked in 2010  there has been real simplification in how to do simple multithreading with Python with map and pool  The code below comes from an article blog post that you should definitely check out  no affiliation  - Parallelism in one line  A Better Model for Day to Day Threading Tasks  I ll summarize below - it ends up being just a few lines of code  from multiprocessing dummy import Pool as ThreadPool pool   ThreadPool 4  results   pool map my function  my array   Which is the multithreaded version of  results      for item in my array      results append my function item     Description  Map is a cool little function  and the key to easily injecting parallelism into your Python code  For those unfamiliar  map is something lifted from functional languages like Lisp  It is a function which maps another function over a sequence  Map handles the iteration over the sequence for us  applies the function  and stores all of the results in a handy list at the end     Implementation  Parallel versions of the map function are provided by two libraries multiprocessing  and also its little known  but equally fantastic step child multiprocessing dummy   multiprocessing dummy is exactly the same as multiprocessing module  but uses threads instead  an important distinction - use multiple processes for CPU-intensive tasks  threads for  and during  I O    multiprocessing dummy replicates the API of multiprocessing  but is no more than a wrapper around the threading module   import urllib2 from multiprocessing dummy import Pool as ThreadPool  urls        http   www python org      http   www python org about       http   www onlamp com pub a python 2003 04 17 metaclasses html      http   www python org doc       http   www python org download       http   www python org getit       http   www python org community       https   wiki python org moin         Make the Pool of workers pool   ThreadPool 4     Open the URLs in their own threads   and return the results results   pool map urllib2 urlopen  urls     Close the pool and wait for the work to finish pool close   pool join    And the timing results  Single thread    14 4 seconds        4 Pool    3 1 seconds        8 Pool    1 4 seconds       13 Pool    1 3 seconds   Passing multiple arguments  works like this only in Python 3 3 and later   To pass multiple arrays  results   pool starmap function  zip list a  list b    Or to pass a constant and an array  results   pool starmap function  zip itertools repeat constant   list a    If you are using an earlier version of Python  you can pass multiple arguments via this workaround    Thanks to user136036 for the helpful comment

User · Answer

Just a note  A queue is not required for threading   This is the simplest example I could imagine that shows 10 processes running concurrently   import threading from random import randint from time import sleep   def print number number          Sleeps a random 1 to 10 seconds     rand int var   randint 1  10      sleep rand int var      print  Thread     str number      slept for     str rand int var      seconds   thread list       for i in range 1  10          Instantiates the thread        i  does not make a sequence  so  i       t   threading Thread target print number  args  i          Sticks the thread in a list so that it remains accessible     thread list append t     Starts threads for thread in thread list      thread start      This blocks the calling thread until the thread whose join   method is called is terminated    From http   docs python org 2 library threading html thread-objects for thread in thread list      thread join      Demonstrates that the main process waited for threads to complete print  Done

User · Answer

Here s a simple example  you need to try a few alternative URLs and return the contents of the first one to respond  import Queue import threading import urllib2    Called by each thread def get url q  url       q put urllib2 urlopen url  read     theurls     quot http   google com quot    quot http   yahoo com quot    q   Queue Queue    for u in theurls      t   threading Thread target get url  args    q u       t daemon   True     t start    s   q get   print s  This is a case where threading is used as a simple optimization  each subthread is waiting for a URL to resolve and respond  to put its contents on the queue  each thread is a daemon  won t keep the process up if the main thread ends -- that s more common than not   the main thread starts all subthreads  does a get on the queue to wait until one of them has done a put  then emits the results and terminates  which takes down any subthreads that might still be running  since they re daemon threads   Proper use of threads in Python is invariably connected to I O operations  since CPython doesn t use multiple cores to run CPU-bound tasks anyway  the only reason for threading is not blocking the process while there s a wait for some I O   Queues are almost invariably the best way to farm out work to threads and or collect the work s results  by the way  and they re intrinsically threadsafe  so they save you from worrying about locks  conditions  events  semaphores  and other inter-thread coordination communication concepts

User · Answer

Here is multi threading with a simple example which will be helpful  You can run it and understand easily how multi threading is working in Python  I used a lock for preventing access to other threads until the previous threads finished their work  By the use of this line of code      tLock   threading BoundedSemaphore value 4    you can allow a number of processes at a time and keep hold to the rest of the threads which will run later or after finished previous processes   import threading import time   tLock   threading Lock   tLock   threading BoundedSemaphore value 4  def timer name  delay  repeat       print    r nTimer     name    Started      tLock acquire       print   r n   name    has the acquired the lock      while repeat  gt  0          time sleep delay          print   r n   name        str time ctime time time             repeat -  1      print   r n   name    is releaseing the lock      tLock release       print   r nTimer     name    Completed   def Main        t1   threading Thread target timer  args   Timer1   2  5       t2   threading Thread target timer  args   Timer2   3  5       t3   threading Thread target timer  args   Timer3   4  5       t4   threading Thread target timer  args   Timer4   5  5       t5   threading Thread target timer  args   Timer5   0 1  5        t1 start       t2 start       t3 start       t4 start       t5 start        print   r nMain Complete   if   name         main         Main

User · Answer

For me  the perfect example for threading is monitoring asynchronous events  Look at this code     thread test py import threading import time  class Monitor threading Thread       def   init   self  mon           threading Thread   init   self          self mon   mon      def run self           while True              if self mon 0     2                  print  Mon   2                  self mon 0    3    You can play with this code by opening an IPython session and doing something like    gt  gt  gt  from thread test import Monitor  gt  gt  gt  a    0   gt  gt  gt  mon   Monitor a   gt  gt  gt  mon start    gt  gt  gt  a 0    2 Mon   2  gt  gt  gt a 0    2 Mon   2   Wait a few minutes   gt  gt  gt  a 0    2 Mon   2

User · Answer

Here is the very simple example of CSV import using threading   Library inclusion may differ for different purpose    Helper Functions   from threading import Thread from project import app import csv   def import handler csv file name       thr   Thread target dump async csv data  args  csv file name       thr start    def dump async csv data csv file name       with app app context            with open csv file name  as File              reader   csv DictReader File              for row in reader                    DB operation query   Driver Function   import handler csv file name

User · Answer

I would like to contribute with a simple example and the explanations I ve found useful when I had to tackle this problem myself   In this answer you will find some information about Python s GIL  global interpreter lock  and a simple day-to-day example written using multiprocessing dummy plus some simple benchmarks   Global Interpreter Lock  GIL   Python doesn t allow multi-threading in the truest sense of the word  It has a multi-threading package  but if you want to multi-thread to speed your code up  then it s usually not a good idea to use it   Python has a construct called the global interpreter lock  GIL   The GIL makes sure that only one of your  threads  can execute at any one time  A thread acquires the GIL  does a little work  then passes the GIL onto the next thread   This happens very quickly so to the human eye it may seem like your threads are executing in parallel  but they are really just taking turns using the same CPU core   All this GIL passing adds overhead to execution  This means that if you want to make your code run faster then using the threading package often isn t a good idea   There are reasons to use Python s threading package  If you want to run some things simultaneously  and efficiency is not a concern  then it s totally fine and convenient  Or if you are running code that needs to wait for something  like some I O  then it could make a lot of sense  But the threading library won t let you use extra CPU cores   Multi-threading can be outsourced to the operating system  by doing multi-processing   and some external application that calls your Python code  for example  Spark or Hadoop   or some code that your Python code calls  for example  you could have your Python code call a C function that does the expensive multi-threaded stuff    Why This Matters  Because lots of people spend a lot of time trying to find bottlenecks in their fancy Python multi-threaded code before they learn what the GIL is   Once this information is clear  here s my code      bin python from multiprocessing dummy import Pool from subprocess import PIPE Popen import time import os    In the variable pool size we define the  parallelness     For CPU-bound tasks  it doesn t make sense to create more Pool processes   than you have cores to run them on      On the other hand  if you are using I O-bound tasks  it may make sense   to create a quite a few more Pool processes than cores  since the processes   will probably spend most their time blocked  waiting for I O to complete   pool size   8  def do ping ip       if os name     nt           print   Using Windows Ping to     ip          proc   Popen   ping   ip   stdout PIPE          return proc communicate   0      else          print   Using Linux   Unix Ping to     ip          proc   Popen   ping   ip   -c    4    stdout PIPE          return proc communicate   0    os system  cls  if os name   nt  else  clear   print   Running using threads n   start time   time time   pool   Pool pool size  website names     www google com   www facebook com   www pinterest com   www microsoft com   result      for website name in website names      result website name    pool apply async do ping  args  website name    pool close   pool join   print    n--- Execution took    seconds ---  format  time time   - start time       Now we do the same without threading  just to compare time print    nRunning NOT using threads n   start time   time time   for website name in website names      do ping website name  print    n--- Execution took    seconds ---  format  time time   - start time       Here s one way to print the final output from the threads output      for key  value in result items        output key    value get   print    nOutput aggregated in a Dictionary    print  output  print    n    print    nPretty printed output     for key  value in output items        print  key     n       print  value

User · Answer

import threading import requests  def send       r   requests get  https   www stackoverlow com    thread      t   threading Thread target send    thread append t  t start

User · Answer

NOTE  For actual parallelization in Python  you should use the multiprocessing module to fork multiple processes that execute in parallel  due to the global interpreter lock  Python threads provide interleaving  but they are in fact executed serially  not in parallel  and are only useful when interleaving I O operations    However  if you are merely looking for interleaving  or are doing I O operations that can be parallelized despite the global interpreter lock   then the threading module is the place to start  As a really simple example  let s consider the problem of summing a large range by summing subranges in parallel   import threading  class SummingThread threading Thread        def   init   self low high            super SummingThread  self    init              self low low          self high high          self total 0       def run self            for i in range self low self high                self total  i   thread1   SummingThread 0 500000  thread2   SummingThread 500000 1000000  thread1 start     This actually causes the thread to run thread2 start   thread1 join      This waits until the thread has completed thread2 join     At this point  both threads have completed result   thread1 total   thread2 total print result   Note that the above is a very stupid example  as it does absolutely no I O and will be executed serially albeit interleaved  with the added overhead of context switching  in CPython due to the global interpreter lock

User · Answer

Using the blazing new concurrent futures module  def sqr val       import time     time sleep 0 1      return val   val  def process result result       print result   def process these asap tasks       import concurrent futures      with concurrent futures ProcessPoolExecutor   as executor          futures              for task in tasks              futures append executor submit sqr  task            for future in concurrent futures as completed futures               process result future result              Or instead of all this just do            results   executor map sqr  tasks            list map process result  results    def main        tasks   list range 10       print  Processing    tasks  format len tasks        process these asap tasks      print  Done       return 0  if   name         main         import sys     sys exit main      The executor approach might seem familiar to all those who have gotten their hands dirty with Java before   Also on a side note  To keep the universe sane  don t forget to close your pools executors if you don t use with context  which is so awesome that it does it for you

User · Answer

I saw a lot of examples here where no real work was being performed  and they were mostly CPU-bound  Here is an example of a CPU-bound task that computes all prime numbers between 10 million and 10 05 million  I have used all four methods here   import math import timeit import threading import multiprocessing from concurrent futures import ThreadPoolExecutor  ProcessPoolExecutor   def time stuff fn               Measure time of execution of a function             def wrapper  args    kwargs           t0   timeit default timer           fn  args    kwargs          t1   timeit default timer           print     seconds  format t1 - t0       return wrapper  def find primes in nmin  nmax               Compute a list of prime numbers between the given minimum and maximum arguments             primes             Loop from minimum to maximum     for current in range nmin  nmax   1              Take the square root of the current number         sqrt n   int math sqrt current           found   False            Check if the any number from 2 to the square root   1 divides the current numnber under consideration         for number in range 2  sqrt n   1                  If divisible we have found a factor  hence this is not a prime number  lets move to the next one             if current   number    0                  found   True                 break            If not divisible  add this number to the list of primes that we have found so far         if not found              primes append current         I am merely printing the length of the array containing all the primes  but feel free to do what you want     print len primes     time stuff def sequential prime finder nmin  nmax               Use the main process and main thread to compute everything in this case             find primes in nmin  nmax    time stuff def threading prime finder nmin  nmax               If the minimum is 1000 and the maximum is 2000 and we have four workers      1000 - 1250 to worker 1     1250 - 1500 to worker 2     1500 - 1750 to worker 3     1750 - 2000 to worker 4     so let   s split the minimum and maximum values according to the number of workers             nrange   nmax - nmin     threads          for i in range 8           start   int nmin   i   nrange 8          end   int nmin    i   1    nrange 8             Start the thread with the minimum and maximum split up to compute           Parallel computation will not work here due to the GIL since this is a CPU-bound task         t   threading Thread target   find primes in  args    start  end           threads append t          t start          Don   t forget to wait for the threads to finish     for t in threads          t join     time stuff def processing prime finder nmin  nmax               Split the minimum  maximum interval similar to the threading method above  but use processes this time             nrange   nmax - nmin     processes          for i in range 8           start   int nmin   i   nrange 8          end   int nmin    i   1    nrange 8          p   multiprocessing Process target   find primes in  args    start  end           processes append p          p start        for p in processes          p join     time stuff def thread executor prime finder nmin  nmax               Split the min max interval similar to the threading method  but use a thread pool executor this time      This method is slightly faster than using pure threading as the pools manage threads more efficiently      This method is still slow due to the GIL limitations since we are doing a CPU-bound task              nrange   nmax - nmin     with ThreadPoolExecutor max workers   8  as e          for i in range 8               start   int nmin   i   nrange 8              end   int nmin    i   1    nrange 8              e submit find primes in  start  end    time stuff def process executor prime finder nmin  nmax               Split the min max interval similar to the threading method  but use the process pool executor      This is the fastest method recorded so far as it manages process efficiently   overcomes GIL limitations      RECOMMENDED METHOD FOR CPU-BOUND TASKS             nrange   nmax - nmin     with ProcessPoolExecutor max workers   8  as e          for i in range 8               start   int nmin   i   nrange 8              end   int nmin    i   1    nrange 8              e submit find primes in  start  end   def main        nmin   int 1e7      nmax   int 1 05e7      print  Sequential Prime Finder Starting       sequential prime finder nmin  nmax      print  Threading Prime Finder Starting       threading prime finder nmin  nmax      print  Processing Prime Finder Starting       processing prime finder nmin  nmax      print  Thread Executor Prime Finder Starting       thread executor prime finder nmin  nmax      print  Process Executor Finder Starting       process executor prime finder nmin  nmax   main     Here are the results on my Mac nbsp OS nbsp X four-core machine  Sequential Prime Finder Starting 9 708213827005238 seconds Threading Prime Finder Starting 9 81836523200036 seconds Processing Prime Finder Starting 3 2467174359990167 seconds Thread Executor Prime Finder Starting 10 228896902000997 seconds Process Executor Finder Starting 2 656402041000547 seconds

User · Answer

The answer from Alex Martelli helped me  However  here is a modified version that I thought was more useful  at least to me    Updated  works in both Python nbsp 2 and Python nbsp 3  try        For Python 3     import queue     from urllib request import urlopen except        For Python 2      import Queue as queue     from urllib2 import urlopen  import threading  worker data     http   google com    http   yahoo com    http   bing com      Load up a queue with your data  This will handle locking q   queue Queue   for url in worker data      q put url     Define a worker function def worker url queue       queue full   True     while queue full          try                Get your data off the queue  and do some work             url   url queue get False              data   urlopen url  read               print len data            except queue Empty              queue full   False    Create as many threads as you want thread count   5 for i in range thread count       t   threading Thread target worker  args    q        t start

User · Answer

Most documentation and tutorials use Python s Threading and Queue module  and they could seem overwhelming for beginners   Perhaps consider the concurrent futures ThreadPoolExecutor module of Python nbsp 3   Combined with with clause and list comprehension it could be a real charm   from concurrent futures import ThreadPoolExecutor  as completed  def get url url         Your actual program here  Using threading Lock   if necessary     return       List of URLs to fetch urls     url1    url2    with ThreadPoolExecutor max workers   5  as executor         Create threads     futures    executor submit get url  url  for url in urls         as completed   gives you the threads once finished     for f in as completed futures             Get the results         rs   f result

User · Answer

I found this very useful  create as many threads as cores and let them execute a  large  number of tasks  in this case  calling a shell program    import Queue import threading import multiprocessing import subprocess  q   Queue Queue   for i in range 30     Put 30 tasks in the queue     q put i   def worker        while True          item   q get             Execute a task  call a shell program and wait until it completes         subprocess call  echo     str item   shell True          q task done    cpus   multiprocessing cpu count     Detect number of cores print  Creating  d threads    cpus  for i in range cpus        t   threading Thread target worker       t daemon   True      t start    q join     Block until all tasks are done

User · Answer

With borrowing from this post we know about choosing between the multithreading  multiprocessing  and async asyncio and their usage  Python 3 has a new built-in library in order to make concurrency and parallelism  concurrent futures So I ll demonstrate through an experiment to run four tasks  i e   sleep   method  by Threading-Pool  from concurrent futures import ThreadPoolExecutor  as completed from time import sleep  time  def concurrent max worker       futures          tic   time       with ThreadPoolExecutor max workers max worker  as executor          futures append executor submit sleep  2      Two seconds sleep         futures append executor submit sleep  1           futures append executor submit sleep  7           futures append executor submit sleep  3           for future in as completed futures               if future result   is not None                  print future result        print f Total elapsed time by  max worker  workers    time  -tic   concurrent 5  concurrent 4  concurrent 3  concurrent 2  concurrent 1   Output  Total elapsed time by 5 workers  7 007831811904907 Total elapsed time by 4 workers  7 007944107055664 Total elapsed time by 3 workers  7 003149509429932 Total elapsed time by 2 workers  8 004627466201782 Total elapsed time by 1 workers  13 013478994369507    NOTE    As you can see in the above results  the best case was 3 workers for those four tasks  If you have a process task instead of I O bound or blocking  multiprocessing instead of threading  you can change the ThreadPoolExecutor to ProcessPoolExecutor

User · Answer

Like others mentioned  CPython can use threads only for I O waits due to GIL   If you want to benefit from multiple cores for CPU-bound tasks  use multiprocessing   from multiprocessing import Process  def f name       print  hello   name  if   name         main         p   Process target f  args   bob         p start       p join

User · Answer

Python 3 has the facility of launching parallel tasks  This makes our work easier   It has thread pooling and process pooling   The following gives an insight   ThreadPoolExecutor Example  source   import concurrent futures import urllib request  URLS     http   www foxnews com             http   www cnn com             http   europe wsj com             http   www bbc co uk             http   some-made-up-domain com       Retrieve a single page and report the URL and contents def load url url  timeout       with urllib request urlopen url  timeout timeout  as conn          return conn read      We can use a with statement to ensure threads are cleaned up promptly with concurrent futures ThreadPoolExecutor max workers 5  as executor        Start the load operations and mark each future with its URL     future to url    executor submit load url  url  60   url for url in URLS      for future in concurrent futures as completed future to url           url   future to url future          try              data   future result           except Exception as exc              print   r generated an exception   s     url  exc           else              print   r page is  d bytes     url  len data      ProcessPoolExecutor  source   import concurrent futures import math  PRIMES         112272535095293      112582705942171      112272535095293      115280095190773      115797848077099      1099726899285419   def is prime n       if n   2    0          return False      sqrt n   int math floor math sqrt n        for i in range 3  sqrt n   1  2           if n   i    0              return False     return True  def main        with concurrent futures ProcessPoolExecutor   as executor          for number  prime in zip PRIMES  executor map is prime  PRIMES                print   d is prime   s     number  prime    if   name         main         main

[python] How can I use threading in Python?

Examples related to python

Examples related to multithreading

Examples related to concurrency

Examples related to python-multithreading