is python capable of running on multiple cores

Question

Question  Because of python s use of  GIL  is python capable running its separate threads simultaneously     Info   After reading this I came away rather uncertain on whether or not python is capable of taking advantage of a multi-core processor  As well done as python is  it feels really weird to think that it would lack such a powerful ability  So feeling uncertain  I decided to ask here  If I write a program that is multi threaded  will it be capable of executing simultaneously on multiple cores

User · Answer

example code taking all 4 cores on my ubuntu 14.04, python 2.7 64 bit.

import time
import threading


def t():
    with open('/dev/urandom') as f:
        for x in xrange(100):
            f.read(4 * 65535)

if __name__ == '__main__':
    start_time = time.time()
    t()
    t()
    t()
    t()
    print "Sequential run time: %.2f seconds" % (time.time() - start_time)

    start_time = time.time()
    t1 = threading.Thread(target=t)
    t2 = threading.Thread(target=t)
    t3 = threading.Thread(target=t)
    t4 = threading.Thread(target=t)
    t1.start()
    t2.start()
    t3.start()
    t4.start()
    t1.join()
    t2.join()
    t3.join()
    t4.join()
    print "Parallel run time: %.2f seconds" % (time.time() - start_time)

result:

$ python 1.py
Sequential run time: 3.69 seconds
Parallel run time: 4.82 seconds

User · Answer

The answer is  Yes  But      But cPython cannot when you are using regular threads for concurrency   You can either use something like multiprocessing  celery or mpi4py to split the parallel work into another process    Or you can use something like Jython or IronPython to use an alternative interpreter that doesn t have a GIL   A softer solution is to use libraries that don t run afoul of the GIL for heavy CPU tasks  for instance numpy can do the heavy lifting while not retaining the GIL  so other python threads can proceed   You can also use the ctypes library in this way     If you are not doing CPU bound work  you can ignore the GIL issue entirely  kind of  since python won t aquire the GIL while it s waiting for IO

User · Answer

As stated in prior answers - it depends on the answer to  quot cpu or i o bound  quot   but also to the answer to  quot threaded or multi-processing  quot   Examples run on Raspberry Pi 3B 1 2GHz 4-core with Python3 7 3 --  With other processes running including htop    For this test - multiprocessing and threading had similar results for i o bound  but multi-processing was more efficient than threading for cpu-bound   Using threads  Typical Result    Starting 4000 cycles of io-bound threading   Sequential run time  39 15 seconds   4 threads Parallel run time  18 19 seconds   2 threads Parallel - twice run time  20 61 seconds Typical Result    Starting 1000000 cycles of cpu-only threading   Sequential run time  9 39 seconds   4 threads Parallel run time  10 19 seconds   2 threads Parallel twice - run time  9 58 seconds Using multiprocessing  Typical Result    Starting 4000 cycles of io-bound processing   Sequential - run time  39 74 seconds   4 procs Parallel - run time  17 68 seconds   2 procs Parallel twice - run time  20 68 seconds Typical Result    Starting 1000000 cycles of cpu-only processing   Sequential run time  9 24 seconds   4 procs Parallel - run time  2 59 seconds   2 procs Parallel twice - run time  4 76 seconds compare io multiproc py     usr bin env python3    Compare single proc vs multiple procs execution for io bound operation   quot  quot  quot  Typical Result    Starting 4000 cycles of io-bound processing   Sequential - run time  39 74 seconds   4 procs Parallel - run time  17 68 seconds   2 procs Parallel twice - run time  20 68 seconds  quot  quot  quot  import time import multiprocessing as mp    one thousand cycles   1   1000  def t            with open   dev urandom    rb   as f                  for x in range cycles                           f read 4   65535   if   name         main         print  quot   Starting    cycles of io-bound processing quot  format cycles 4       start time   time time       t       t       t       t       print  quot   Sequential - run time    2f seconds quot     time time   - start time          four procs     start time   time time       p1   mp Process target t      p2   mp Process target t      p3   mp Process target t      p4   mp Process target t      p1 start       p2 start       p3 start       p4 start       p1 join       p2 join       p3 join       p4 join       print  quot   4 procs Parallel - run time    2f seconds quot     time time   - start time          two procs     start time   time time       p1   mp Process target t      p2   mp Process target t      p1 start       p2 start       p1 join       p2 join       p3   mp Process target t      p4   mp Process target t      p3 start       p4 start       p3 join       p4 join       print  quot   2 procs Parallel twice - run time    2f seconds quot     time time   - start time    compare cpu multiproc py    usr bin env python3    Compare single proc vs multiple procs execution for cpu bound operation   quot  quot  quot  Typical Result    Starting 1000000 cycles of cpu-only processing   Sequential run time  9 24 seconds   4 procs Parallel - run time  2 59 seconds   2 procs Parallel twice - run time  4 76 seconds  quot  quot  quot  import time import multiprocessing as mp    one million cycles   1000   1000  def t        for x in range cycles           fdivision   cycles   2 0         fcomparison    x  gt  fdivision          faddition   fdivision   1 0         fsubtract   fdivision - 2 0         fmultiply   fdivision   2 0  if   name         main         print  quot   Starting    cycles of cpu-only processing quot  format cycles       start time   time time       t       t       t       t       print  quot   Sequential run time    2f seconds quot     time time   - start time          four procs     start time   time time       p1   mp Process target t      p2   mp Process target t      p3   mp Process target t      p4   mp Process target t      p1 start       p2 start       p3 start       p4 start       p1 join       p2 join       p3 join       p4 join       print  quot   4 procs Parallel - run time    2f seconds quot     time time   - start time          two procs     start time   time time       p1   mp Process target t      p2   mp Process target t      p1 start       p2 start       p1 join       p2 join       p3   mp Process target t      p4   mp Process target t      p3 start       p4 start       p3 join       p4 join       print  quot   2 procs Parallel twice - run time    2f seconds quot     time time   - start time

User · Answer

Python threads cannot take advantage of many cores  This is due to an internal implementation detail called the GIL  global interpreter lock  in the C implementation of python  cPython  which is almost certainly what you use   The workaround is the multiprocessing module http   www python org dev peps pep-0371  which was developed for this purpose   Documentation  http   docs python org library multiprocessing html   Or use a parallel language

User · Answer

I converted the script to Python3 and ran it on my Raspberry Pi 3B    import time import threading  def t            with open   dev urandom    rb   as f                  for x in range 100                           f read 4   65535   if   name         main         start time   time time       t       t       t       t       print  Sequential run time    2f seconds     time time   - start time        start time   time time       t1   threading Thread target t      t2   threading Thread target t      t3   threading Thread target t      t4   threading Thread target t      t1 start       t2 start       t3 start       t4 start       t1 join       t2 join       t3 join       t4 join       print  Parallel run time    2f seconds     time time   - start time     python3 t py  Sequential run time  2 10 seconds Parallel run time  1 41 seconds   For me  running parallel was quicker

User · Answer

CPython  the classic and prevalent implementation of Python  can t have more than one thread executing Python bytecode at the same time   This means compute-bound programs will only use one core   I O operations and computing happening inside C extensions  such as numpy  can operate simultaneously   Other implementation of Python  such as Jython or PyPy  may behave differently  I m less clear on their details   The usual recommendation is to use many processes rather than many threads

User · Answer

Threads share a process and a process runs on a core  but you can use python s multiprocessing module to call your functions in separate processes and use other cores  or you can use the subprocess module  which can run your code and non-python code too

[python] is python capable of running on multiple cores?

Examples related to python

Examples related to multithreading