How can you profile a Python script

Question

Project Euler and other coding contests often have a maximum time to run or people boast of how fast their particular solution runs  With Python  sometimes the approaches are somewhat kludgey - i e   adding timing code to   main     What is a good way to profile how long a Python program takes to run

User · Answer

It would depend on what you want to see out of profiling. Simple time metrics can be given by (bash).

time python python_prog.py

Even '/usr/bin/time' can output detailed metrics by using '--verbose' flag.

To check time metrics given by each function and to better understand how much time is spent on functions, you can use the inbuilt cProfile in python.

Going into more detailed metrics like performance, time is not the only metric. You can worry about memory, threads etc.
Profiling options:
1. line_profiler is another profiler used commonly to find out timing metrics line-by-line.
2. memory_profiler is a tool to profile memory usage.
3. heapy (from project Guppy) Profile how objects in the heap are used.

These are some of the common ones I tend to use. But if you want to find out more, try reading this book It is a pretty good book on starting out with performance in mind. You can move onto advanced topics on using Cython and JIT(Just-in-time) compiled python.

User · Answer

There s also a statistical profiler called statprof  It s a sampling profiler  so it adds minimal overhead to your code and gives line-based  not just function-based  timings  It s more suited to soft real-time applications like games  but may be have less precision than cProfile   The version in pypi is a bit old  so can install it with pip by specifying the git repository   pip install git git   github com bos statprof py 1a33eba91899afe17a8b752c6dfdec6f05dd0c01   You can run it like this   import statprof  with statprof profile        my questionable function     See also https   stackoverflow com a 10333592 320036

User · Answer

I recently created tuna for visualizing Python runtime and import profiles  this may be helpful here     Install with  pip install tuna   Create a runtime profile  python3 -m cProfile -o program prof yourfile py   or an import profile  Python 3 7  required   python3 -X importprofile yourfile py 2 gt  import log   Then just run tuna on the file  tuna program prof

User · Answer

Python includes a profiler called cProfile  It not only gives the total running time  but also times each function separately  and tells you how many times each function was called  making it easy to determine where you should make optimizations   You can call it from within your code  or from the interpreter  like this   import cProfile cProfile run  foo       Even more usefully  you can invoke the cProfile when running a script   python -m cProfile myscript py   To make it even easier  I made a little batch file called  profile bat    python -m cProfile  1   So all I have to do is run   profile euler048 py   And I get this   1007 function calls in 0 061 CPU seconds  Ordered by  standard name ncalls  tottime  percall  cumtime  percall filename lineno function      1    0 000    0 000    0 061    0 061  lt string gt  1  lt module gt    1000    0 051    0 000    0 051    0 000 euler048 py 2  lt lambda gt       1    0 005    0 005    0 061    0 061 euler048 py 2  lt module gt       1    0 000    0 000    0 061    0 061  execfile      1    0 002    0 002    0 053    0 053  map      1    0 000    0 000    0 000    0 000  method  disable  of   lsprof Profiler objects      1    0 000    0 000    0 000    0 000  range      1    0 003    0 003    0 003    0 003  sum    EDIT  Updated link to a good video resource from PyCon 2013 titled  Python Profiling Also via YouTube

User · Answer

gprof2dot magic  Magic function for gprof2dot to profile any Python statement as a DOT graph in JupyterLab or Jupyter Notebook     GitHub repo  https   github com mattijn gprof2dot magic  installation  Make sure you ve the Python package gprof2dot magic   pip install gprof2dot magic   Its dependencies gprof2dot and graphviz will be installed as well  usage  To enable the magic function  first load the gprof2dot magic module   load ext gprof2dot magic   and then profile any line statement as a DOT graph as such    gprof2dot print  hello world

User · Answer

pprofile  line profiler  already presented here  also inspired  pprofile  which is described as      Line-granularity  thread-aware deterministic and statistic pure-python   profiler   It provides line-granularity as line profiler  is pure Python  can be used as a standalone command or a module  and can even generate callgrind-format files that can be easily analyzed with  k q cachegrind   vprof  There is also vprof  a Python package described as            providing rich and interactive visualizations for various Python program characteristics such as running time and memory usage

User · Answer

When i m not root on the server  I use  lsprofcalltree py and run my program like this   python lsprofcalltree py -o callgrind 1 test py   Then I can open the report with any callgrind-compatible software  like qcachegrind

User · Answer

cProfile is great for profiling  while kcachegrind is great for visualizing the results  The pyprof2calltree in between handles the file conversion  python -m cProfile -o script profile script py pyprof2calltree -i script profile -o script calltree kcachegrind script calltree  Required system packages   kcachegrind  Linux   qcachegrind  MacOs   Setup on Ubuntu  apt-get install kcachegrind  pip install pyprof2calltree  The result

User · Answer

If you want to make a cumulative profiler  meaning to run the function several times in a row and watch the sum of the results   you can use this cumulative profiler decorator   it s python    3 6 specific  but you can remove nonlocal for it work on older versions   import cProfile  pstats  class  ProfileFunc      def   init   self  func  sort stats by           self func    func         self profile runs              self sort stats by   sort stats by      def   call   self   args    kwargs           pr   cProfile Profile           pr enable      this is the profiling section         retval   self func  args    kwargs          pr disable            self profile runs append pr          ps   pstats Stats  self profile runs  sort stats self sort stats by          return retval  ps  def cumulative profiler amount of times  sort stats by  time        def real decorator function           def wrapper  args    kwargs               nonlocal function  amount of times  sort stats by    for python 2 x remove this row              profiled func    ProfileFunc function  sort stats by              for i in range amount of times                   retval  ps   profiled func  args    kwargs              ps print stats               return retval    returns the results of the function         return wrapper      if callable amount of times      incase you don t want to specify the amount of times         func   amount of times    amount of times is the function in here         amount of times   5    the default amount         return real decorator func      return real decorator   Example  profiling the function baz   import time   cumulative profiler def baz        time sleep 1      time sleep 2      return 1  baz     baz ran 5 times and printed this            20 function calls in 15 003 seconds     Ordered by  internal time     ncalls  tottime  percall  cumtime  percall filename lineno function         10   15 003    1 500   15 003    1 500  built-in method time sleep          5    0 000    0 000   15 003    3 001  lt ipython-input-9-c89afe010372 gt  3 baz          5    0 000    0 000    0 000    0 000  method  disable  of   lsprof Profiler  objects    specifying the amount of times   cumulative profiler 3  def baz

User · Answer

There s a lot of great answers but they either use command line or some external program for profiling and or sorting the results  I really missed some way I could use in my IDE  eclipse-PyDev  without touching the command line or installing anything  So here it is  Profiling without command line def count        from math import sqrt     for x in range 10  5           sqrt x   if   name         main         import cProfile  pstats     cProfile run  quot count   quot    quot    profile quot  format   file         s   pstats Stats  quot    profile quot  format   file         s strip dirs       s sort stats  quot time quot   print stats 10   See docs or other answers for more info

User · Answer

A while ago I made pycallgraph which generates a visualisation from your Python code  Edit  I ve updated the example to work with 3 3  the latest release as of this writing   After a pip install pycallgraph and installing GraphViz you can run it from the command line   pycallgraph graphviz --   mypythonscript py   Or  you can profile particular parts of your code   from pycallgraph import PyCallGraph from pycallgraph output import GraphvizOutput  with PyCallGraph output GraphvizOutput         code to profile     Either of these will generate a pycallgraph png file similar to the image below

User · Answer

It s worth pointing out that using the profiler only works  by default  on the main thread  and you won t get any information from other threads if you use them   This can be a bit of a gotcha as it is completely unmentioned in the profiler documentation   If you also want to profile threads  you ll want to look at the threading setprofile   function in the docs   You could also create your own threading Thread subclass to do it   class ProfiledThread threading Thread         Overrides threading Thread run       def run self           profiler   cProfile Profile           try              return profiler runcall threading Thread run  self          finally              profiler dump stats  myprofile- d profile     self ident      and use that ProfiledThread class instead of the standard one   It might give you more flexibility  but I m not sure it s worth it  especially if you are using third-party code which wouldn t use your class

User · Answer

Simplest and quickest way to find where all the time is going   1  pip install snakeviz  2  python -m cProfile -o temp dat  lt PROGRAM gt  py  3  snakeviz temp dat   Draws a pie chart in a browser  Biggest piece is the problem function  Very simple

User · Answer

Ever want to know what the hell that python script is doing  Enter the   Inspect Shell  Inspect Shell lets you print alter globals and run   functions without interrupting the running script  Now with   auto-complete and command history  only on linux        Inspect Shell is not a pdb-style debugger    https   github com amoffat Inspect-Shell  You could use that  and your wristwatch

User · Answer

Maxy s comment on this answer helped me out enough that I think it deserves its own answer  I already had cProfile-generated  pstats files and I didn t want to re-run things with pycallgraph  so I used gprof2dot  and got pretty svgs     sudo apt-get install graphviz   git clone https   github com jrfonseca gprof2dot   ln -s   PWD  gprof2dot gprof2dot py   bin   cd  PROJECT DIR   gprof2dot py -f pstats profile pstats   dot -Tsvg -o callgraph svg   and BLAM   It uses dot  the same thing that pycallgraph uses  so output looks similar  I get the impression that gprof2dot loses less information though

User · Answer

cProfile is great for quick profiling but most of the time it was ending for me with the errors  Function runctx solves this problem by initializing correctly the environment and variables  hope it can be useful for someone   import cProfile cProfile runctx  foo     None  locals

User · Answer

A new tool to handle profiling in Python is PyVmMonitor  http   www pyvmmonitor com   It has some unique features such as   Attach profiler to a running  CPython  program On demand profiling with Yappi integration Profile on a different machine Multiple processes support  multiprocessing  django     Live sampling CPU view  with time range selection  Deterministic profiling through cProfile profile integration Analyze existing PStats results Open DOT files Programatic API access Group samples by method or line PyDev integration PyCharm integration   Note  it s commercial  but free for open source

User · Answer

I just developed my own profiler inspired from pypref time  https   github com modaresimr auto profiler By adding a decorator it will show a tree of time-consuming functions  Profiler depth 4  on disable show  Install by  pip install auto profiler  Example import time   line number 1 import random  from auto profiler import Profiler  Tree  def f1        mysleep  6 random random     def mysleep t       time sleep t   def fact i       f1       if i  1           return 1     return i fact i-1    def show p       print  Time    Hits   PerHit  Function name  Called from   Function Location  n               -----------------------------------------------------------------------       print Tree p root  threshold 0 5         Profiler depth 4  on disable show  def main        for i in range 5           f1        fact 3    if   name         main         main     Example Output  Time    Hits   PerHit  Function name  Called from   function location  ----------------------------------------------------------------------- 8 974s  1   8 974   main   auto-profiler profiler py 267     test t2 py 30   -- 5 954s  5   1 191   f1    test t2 py 34     test t2 py 14        -- 5 954s  5   1 191   mysleep    test t2 py 15     test t2 py 17            -- 5 954s  5   1 191    lt time sleep gt            The rest is for the example recursive function call fact  -- 3 020s  1   3 020   fact    test t2 py 36     test t2 py 20       -- 0 849s  1   0 849   f1    test t2 py 21     test t2 py 14            -- 0 849s  1   0 849   mysleep    test t2 py 15     test t2 py 17                -- 0 849s  1   0 849    lt time sleep gt       -- 2 171s  1   2 171   fact    test t2 py 24     test t2 py 20           -- 1 552s  1   1 552   f1    test t2 py 21     test t2 py 14                -- 1 552s  1   1 552   mysleep    test t2 py 15     test t2 py 17           -- 0 619s  1   0 619   fact    test t2 py 24     test t2 py 20               -- 0 619s  1   0 619   f1    test t2 py 21     test t2 py 14

User · Answer

A nice profiling module is the line profiler  called using the script kernprof py    It can be downloaded here   My understanding is that cProfile only gives information about total time spent in each function   So individual lines of code are not timed   This is an issue in scientific computing since often one single line can take a lot of time   Also  as I remember  cProfile didn t catch the time I was spending in say numpy dot

User · Answer

Also worth mentioning is the GUI cProfile dump viewer RunSnakeRun   It allows you to sort and select  thereby zooming in on the relevant parts of the program   The sizes of the rectangles in the picture is proportional to the time taken   If you mouse over a rectangle it highlights that call in the table and everywhere on the map   When you double-click on a rectangle it zooms in on that portion   It will show you who calls that portion and what that portion calls   The descriptive information is very helpful   It shows you the code for that bit which can be helpful when you are dealing with built-in library calls   It tells you what file and what line to find the code   Also want to point at that the OP said  profiling  but it appears he meant  timing    Keep in mind programs will run slower when profiled

User · Answer

Following Joe Shaw s answer about multi-threaded code not to work as expected  I figured that the runcall method in cProfile is merely doing self enable   and self disable   calls around the profiled function call  so you can simply do that yourself and have whatever code you want in-between with minimal interference with existing code

User · Answer

I ran into a handy tool called SnakeViz when researching this topic  SnakeViz is a web-based profiling visualization tool  It is very easy to install and use  The usual way I use it is to generate a stat file with  prun and then do analysis in SnakeViz   The main viz technique used is Sunburst chart as shown below  in which the hierarchy of function calls is arranged as layers of arcs and time info encoded in their angular widths   The best thing is you can interact with the chart  For example  to zoom in one can click on an arc  and the arc and its descendants will be enlarged as a new sunburst to display more details

User · Answer

The python wiki is a great page for profiling resources  http   wiki python org moin PythonSpeed PerformanceTips Profiling Code  as is the python docs  http   docs python org library profile html  as shown by Chris Lawlor cProfile is a great tool and can easily be used to print to the screen   python -m cProfile -s time mine py  lt args gt    or to file   python -m cProfile -o output file mine py  lt args gt    PS  If you are using Ubuntu  make sure to install python-profile  apt-get install python-profiler    If you output to file you can get nice visualizations using the following tools  PyCallGraph   a tool to create call graph images    install    pip install pycallgraph   run    pycallgraph mine py args   view    gimp pycallgraph png   You can use whatever you like to view the png file  I used gimp Unfortunately I often get   dot  graph is too large for cairo-renderer bitmaps  Scaling by 0 257079 to fit  which makes my images unusably small   So I generally create svg files   pycallgraph -f svg -o pycallgraph svg mine py  lt args gt    PS  make sure to install graphviz  which provides the dot program    pip install graphviz   Alternative Graphing using gprof2dot via  maxy    quodlibetor    pip install gprof2dot python -m cProfile -o profile pstats mine py gprof2dot -f pstats profile pstats   dot -Tsvg -o mine svg

User · Answer

My way is to use yappi  https   github com sumerc yappi   It s especially useful combined with an RPC server where  even just for debugging  you register method to start  stop and print profiling information  e g  in this way     staticmethod def startProfiler        yappi start     staticmethod def stopProfiler        yappi stop     staticmethod def printProfiler        stats   yappi get stats yappi SORTTYPE TTOT  yappi SORTORDER DESC  20      statPrint     n      namesArr    len str stat 0    for stat in stats func stats      log debug  namesArr  s   str namesArr       maxNameLen   max namesArr      log debug  maxNameLen   s   maxNameLen       for stat in stats func stats          nameAppendSpaces        for i in range maxNameLen - len stat 0             log debug  nameAppendSpaces   s   nameAppendSpaces          blankSpace              for space in nameAppendSpaces              blankSpace    space          log debug  adding spaces   s   len nameAppendSpaces           statPrint   statPrint   str stat 0     blankSpace         str stat 1   ljust 8      t    str              round stat 2   2   ljust 8 - len str stat 2         t    str round stat 3   2       n       log log 1000    nname       ljust maxNameLen - 4      ncall  tttot  ttsub       log log 1000  statPrint    Then when your program work you can start profiler at any time by calling the startProfiler RPC method and dump profiling information to a log file by calling printProfiler  or modify the rpc method to return it to the caller  and get such output   2014-02-19 16 32 24 128- SVR-MAIN   - Thread-3    -Level 1000   name                                                                                                                                      ncall     ttot    tsub 2014-02-19 16 32 24 128- SVR-MAIN   - Thread-3    -Level 1000   C  Python27 lib sched py run 80                                                                                                           22        0 11    0 05 M  02 documents  repos 09 aheadRepos apps ahdModbusSrv pyAheadRpcSrv xmlRpc py iterFnc 293                                                22        0 11    0 0 M  02 documents  repos 09 aheadRepos apps ahdModbusSrv serverMain py makeIteration 515                                                    22        0 11    0 0 M  02 documents  repos 09 aheadRepos apps ahdModbusSrv pyAheadRpcSrv PicklingXMLRPC py  dispatch 66                                       1         0 0     0 0 C  Python27 lib BaseHTTPServer py date time string 464                                                                                    1         0 0     0 0 c  users zasiec 1 appdata local temp easy install-hwcsr1 psutil-1 1 2-py2 7-win32 egg tmp psutil  psmswindows py  get raw meminfo 243     4         0 0     0 0 C  Python27 lib SimpleXMLRPCServer py decode request content 537                                                                          1         0 0     0 0 c  users zasiec 1 appdata local temp easy install-hwcsr1 psutil-1 1 2-py2 7-win32 egg tmp psutil  psmswindows py get system cpu times 148 4         0 0     0 0  lt string gt    new   8                                                                                                                        220       0 0     0 0 C  Python27 lib socket py close 276                                                                                                       4         0 0     0 0 C  Python27 lib threading py   init   558                                                                                                 1         0 0     0 0  lt string gt    new   8                                                                                                                        4         0 0     0 0 C  Python27 lib threading py notify 372                                                                                                   1         0 0     0 0 C  Python27 lib rfc822 py getheader 285                                                                                                   4         0 0     0 0 C  Python27 lib BaseHTTPServer py handle one request 301                                                                                  1         0 0     0 0 C  Python27 lib xmlrpclib py end 816                                                                                                      3         0 0     0 0 C  Python27 lib SimpleXMLRPCServer py do POST 467                                                                                         1         0 0     0 0 C  Python27 lib SimpleXMLRPCServer py is rpc path valid 460                                                                               1         0 0     0 0 C  Python27 lib SocketServer py close request 475                                                                                         1         0 0     0 0 c  users zasiec 1 appdata local temp easy install-hwcsr1 psutil-1 1 2-py2 7-win32 egg tmp psutil   init   py cpu times 1066               4         0 0     0 0    It may not be very useful for short scripts but helps to optimize server-type processes especially given the printProfiler method can be called multiple times over time to profile and compare e g  different program usage scenarios    In newer versions of yappi  the following code will work    staticmethod def printProfile        yappi get func stats   print all

User · Answer

To add on to https   stackoverflow com a 582337 1070617   I wrote this module that allows you to use cProfile and view its output easily  More here  https   github com ymichael cprofilev    python -m cprofilev  your python program   Go to http   localhost 4000 to view collected statistics    Also see  http   ymichael com 2014 03 08 profiling-python-with-cprofile html on how to make sense of the collected statistics

User · Answer

The terminal-only  and simplest  solution  in case all those fancy UI s fail to install or to run  ignore cProfile completely and replace it with pyinstrument  that will collect and display the tree of calls right after execution   Install       pip install pyinstrument   Profile and display result       python -m pyinstrument   prog py   Works with python2 and 3    EDIT  The documentation of the API  for profiling only a part of the code  can be found here

User · Answer

In Virtaal s source there s a very useful class and decorator that can make profiling  even for specific methods functions  very easy  The output can then be viewed very comfortably in KCacheGrind

User · Answer

With a statistical profiler like austin  no instrumentation is required  meaning that you can get profiling data out of a Python application simply with  austin python3 my script py   The raw output isn t very useful  but you can pipe that to flamegraph pl  to get a flame graph representation of that data that gives you a breakdown of where the time  measured in microseconds of real time  is being spent   austin python3 my script py   flamegraph pl  gt  my script profile svg

User · Answer

For getting quick profile stats on an IPython notebook  One can embed line profiler and memory profiler straight into their notebooks  Another useful package is Pympler  It is a powerful profiling package that s capable to track classes objects functions memory leaks etc  Examples below  Docs attached  Get it   pip install line profiler  pip install memory profiler  pip install pympler  Load it   load ext line profiler  load ext memory profiler  Use it    time  time print  Outputs CPU time Wall Clock time     CPU times  user 2   s  sys  0 ns  total  2   s Wall time  5 96   s  Gives   CPU times  CPU level execution time sys times  system level execution time total  CPU time   system time Wall time  Wall Clock Time    timeit  timeit -r 7 -n 1000 print  Outputs execution time of the snippet     1000 loops  best of 7  7 46 ns per loop   Gives best time out of given number of runs r  in looping  n  times  Outputs details on system caching   When code snippets are executed multiple times  system caches a few opearations and doesn t execute them again that may hamper the accuracy of the profile reports       prun  prun -s cumulative  Code to profile    Gives   number of function calls ncalls  has entries per function call distinct  time taken per call percall  time elapsed till that function call cumtime  name of the func module called etc        memit  memit  Code to profile   peak memory  199 45 MiB  increment  0 00 MiB  Gives   Memory usage    lprun  Example function def fun      for i in range 10       print i    Usage   lprun  lt name of the function gt  function  lprun -f fun fun    Gives   Line wise stats    sys getsizeof sys getsizeof  code to profile     64 bytes  Returns the size of an object in bytes   asizeof   from pympler from pympler import asizeof obj    1 2   quot hey quot   quot ha quot   3  print asizeof asizeof obj stats 4     pympler asizeof can be used to investigate how much memory certain Python objects consume  In contrast to sys getsizeof  asizeof sizes objects recursively   tracker from pympler from pympler import tracker tr   tracker SummaryTracker   def fun      li    1 2 3    di     quot ha quot   quot haha quot   quot duh quot   quot Umm quot   fun   tr print diff    Tracks the lifetime of a function   Pympler package consists of a huge number of high utility functions to profile code  All of which cannot be covered here  See the documentation attached for verbose profile implementations  Pympler doc

[python] How can you profile a Python script?

Examples related to python

Examples related to performance

Examples related to profiling

Examples related to time-complexity