How to find out the number of CPUs using python

Question

I want to know the number of CPUs on the local machine using Python  The result should be user real as output by time 1  when called with an optimally scaling userspace-only program

User · Answer

Another option if you don t have Python 2 6   import commands n   commands getoutput  grep -c processor  proc cpuinfo

User · Answer

Can t figure out how to add to the code or reply to the message but here s support for jython that you can tack in before you give up     jython try      from java lang import Runtime     runtime   Runtime getRuntime       res   runtime availableProcessors       if res  gt  0          return res except ImportError      pass

User · Answer

You can also use  joblib  for this purpose    import joblib print joblib cpu count     This method will give you the number of cpus in the system  joblib needs to be installed though  More information on joblib can be found here       https   pythonhosted org joblib parallel html  Alternatively you can use numexpr package of python  It has lot of simple functions helpful for getting information about the system cpu   import numexpr as ne print ne detect number of cores

User · Answer

If you want to know the number of physical cores  not virtual hyperthreaded cores   here is a platform independent solution   psutil cpu count logical False    https   github com giampaolo psutil blob master INSTALL rst  Note that the default value for logical is True  so if you do want to include hyperthreaded cores you can use   psutil cpu count     This will give the same number as os cpu count   and multiprocessing cpu count    neither of which have the logical keyword argument

User · Answer

len os sched getaffinity 0   is what you usually want https   docs python org 3 library os html os sched getaffinity os sched getaffinity 0   added in Python 3  returns the set of CPUs available considering the sched setaffinity Linux system call  which limits which CPUs a process and its children can run on  0 means to get the value for the current process  The function returns a set   of allowed CPUs  thus the need for len    multiprocessing cpu count   and os cpu count   on the other hand just returns the total number of physical CPUs  The difference is especially important because certain cluster management systems such as Platform LSF limit job CPU usage with sched getaffinity  Therefore  if you use multiprocessing cpu count    your script might try to use way more cores than it has available  which may lead to overload and timeouts  We can see the difference concretely by restricting the affinity with the taskset utility  which allows us to control the affinity of a process  Minimal taskset example For example  if I restrict Python to just 1 core  core 0  in my 16 core system  taskset -c 0   main py  with the test script  main py    usr bin env python3  import multiprocessing import os  print multiprocessing cpu count    print os cpu count    print len os sched getaffinity 0     then the output is  16 16 1  Vs nproc nproc does respect the affinity by default and  taskset -c 0 nproc  outputs  1  and man nproc makes that quite explicit   print the number of processing units available  Therefore  len os sched getaffinity 0   behaves like nproc by default  nproc has the --all flag for the less common case that you want to get the physical CPU count without considering taskset  taskset -c 0 nproc --all  os cpu count documentation The documentation of os cpu count also briefly mentions this https   docs python org 3 8 library os html os cpu count  This number is not equivalent to the number of CPUs the current process can use  The number of usable CPUs can be obtained with len os sched getaffinity 0    The same comment is also copied on the documentation of multiprocessing cpu count  https   docs python org 3 library multiprocessing html multiprocessing cpu count From the 3 8 source under Lib multiprocessing context py we also see that multiprocessing cpu count just forwards to os cpu count  except that the multiprocessing one throws an exception instead of returning None if os cpu count fails      def cpu count self              Returns the number of CPUs in the system            num   os cpu count           if num is None              raise NotImplementedError  cannot determine number of cpus           else              return num  3 8 availability  systems with a native sched getaffinity function The only downside of this os sched getaffinity is that this appears to be UNIX only as of Python 3 8  cpython 3 8 seems to just try to compile a small C hello world with a sched setaffinity function call during configuration time  and if not present HAVE SCHED SETAFFINITY is not set and the function will likely be missing   https   github com python cpython blob v3 8 5 configure L11523 https   github com python cpython blob v3 8 5 Modules posixmodule c L6457  psutil Process   cpu affinity    third-party version with a Windows port The third-party psutil package  pip install psutil  had been mentioned at  https   stackoverflow com a 14840102 895245 but not the cpu affinity function  https   psutil readthedocs io en latest  psutil Process cpu affinity Usage  import psutil print len psutil Process   cpu affinity      This function does the same as the standard library os sched getaffinity on Linux  but they have also implemented it for Windows by making a call to the GetProcessAffinityMask Windows API function   https   github com giampaolo psutil blob ee60bad610822a7f630c52922b4918e684ba7695 psutil  psutil windows c L1112 https   docs microsoft com en-us windows win32 api winbase nf-winbase-getprocessaffinitymask  So in other words  those Windows users have to stop being lazy and send a patch to the upstream stdlib  -  Tested in Ubuntu 16 04  Python 3 5 2

User · Answer

multiprocessing cpu count   will return the number of logical CPUs  so if you have a quad-core CPU with hyperthreading  it will return 8   If you want the number of physical CPUs  use the python bindings to hwloc      usr bin env python import hwloc topology   hwloc Topology   topology load   print topology get nbobjs by type hwloc OBJ CORE    hwloc is designed to be portable across OSes and architectures

User · Answer

If you re interested into the number of processors available to your current process  you have to check cpuset first  Otherwise  or if cpuset is not in use   multiprocessing cpu count   is the way to go in Python 2 6 and newer  The following method falls back to a couple of alternative methods in older versions of Python   import os import re import subprocess   def available cpu count            Number of available virtual or physical CPUs on this system  i e      user real as output by time 1  when called with an optimally scaling     userspace-only program           cpuset       cpuset may restrict the number of  available  processors     try          m   re search r   m  Cpus allowed  s                               open   proc self status   read            if m              res   bin int m group 1  replace           16   count  1               if res  gt  0                  return res     except IOError          pass        Python 2 6      try          import multiprocessing         return multiprocessing cpu count       except  ImportError  NotImplementedError           pass        https   github com giampaolo psutil     try          import psutil         return psutil cpu count       psutil NUM CPUS on old versions     except  ImportError  AttributeError           pass        POSIX     try          res   int os sysconf  SC NPROCESSORS ONLN             if res  gt  0              return res     except  AttributeError  ValueError           pass        Windows     try          res   int os environ  NUMBER OF PROCESSORS             if res  gt  0              return res     except  KeyError  ValueError           pass        jython     try          from java lang import Runtime         runtime   Runtime getRuntime           res   runtime availableProcessors           if res  gt  0              return res     except ImportError          pass        BSD     try          sysctl   subprocess Popen   sysctl    -n    hw ncpu                                      stdout subprocess PIPE          scStdout   sysctl communicate   0          res   int scStdout           if res  gt  0              return res     except  OSError  ValueError           pass        Linux     try          res   open   proc cpuinfo   read   count  processor t             if res  gt  0              return res     except IOError          pass        Solaris     try          pseudoDevices   os listdir   devices pseudo            res   0         for pd in pseudoDevices              if re match r  cpuid  0-9      pd                   res    1          if res  gt  0              return res     except OSError          pass        Other UNIXes  heuristic      try          try              dmesg   open   var run dmesg boot   read           except IOError              dmesgProcess   subprocess Popen   dmesg    stdout subprocess PIPE              dmesg   dmesgProcess communicate   0           res   0         while   ncpu    str res        in dmesg              res    1          if res  gt  0              return res     except OSError          pass      raise Exception  Can not determine number of CPUs on this system

User · Answer

If you have python with a version    2 6 you can simply use  import multiprocessing  multiprocessing cpu count     http   docs python org library multiprocessing html multiprocessing cpu count

User · Answer

In Python 3 4   os cpu count     multiprocessing cpu count   is implemented in terms of this function but raises NotImplementedError if os cpu count   returns None   can t determine number of CPUs

User · Answer

This may work for those of us who use different os systems  but want to get the best of all worlds   import os workers   os cpu count   if  sched getaffinity  in dir os       workers   len os sched getaffinity 0

User · Answer

Another option is to use the psutil library  which always turn out useful in these situations    gt  gt  gt  import psutil  gt  gt  gt  psutil cpu count   2   This should work on any platform supported by psutil Unix and Windows    Note that in some occasions multiprocessing cpu count may raise a NotImplementedError while psutil will be able to obtain the number of CPUs  This is simply because psutil first tries to use the same techniques used by multiprocessing and  if those fail  it also uses other techniques

User · Answer

If you are using torch you can do  import torch multiprocessing as mp  mp cpu count

User · Answer

These give you the hyperthreaded CPU count   multiprocessing cpu count   os cpu count     These give you the virtual machine CPU count   psutil cpu count   numexpr detect number of cores     Only matters if you works on VMs

[python] How to find out the number of CPUs using python

Examples related to python

Examples related to system-information