Showing the stack trace from a running Python application

Question

I have this Python application that gets stuck from time to time and I can t find out where   Is there any way to signal Python interpreter to show you the exact code that s running    Some kind of on-the-fly stacktrace   Related questions    Print current call stack from a method in Python code Check what a running process is doing  print stack trace of an uninstrumented Python program

User · Answer

Take a look at the faulthandler module  new in Python 3 3   A faulthandler backport for use in Python 2 is available on PyPI

User · Answer

I am in the GDB camp with the python extensions  Follow https   wiki python org moin DebuggingWithGdb  which means   dnf install gdb python-debuginfo or sudo apt-get install gdb python2 7-dbg gdb python  lt pid of running process gt  py-bt   Also consider info threads and thread apply all py-bt

User · Answer

It can be done with excellent py-spy  It s a sampling profiler for Python programs  so its job is to attach to a Python processes and sample their call stacks  Hence  py-spy dump --pid  SOME PID is all you need to do to dump call stacks of all threads in the  SOME PID process  Typically it needs escalated privileges  to read the target process  memory     Here s an example of how it looks like for a threaded Python application     sudo py-spy dump --pid 31080 Process 31080  python3 7 -m chronologer -e production serve -u www-data -m Python v3 7 1   usr local bin python3 7   Thread 0x7FEF5E410400  active    MainThread       wait  cherrypy process wspbus py 370      wait  cherrypy process wspbus py 384      block  cherrypy process wspbus py 321      start  cherrypy daemon py 72      serve  chronologer cli py 27      main  chronologer cli py 84       lt module gt   chronologer   main   py 5       run code  runpy py 85       run module as main  runpy py 193  Thread 0x7FEF55636700  active     TimeoutMonitor      run  cherrypy process plugins py 518       bootstrap inner  threading py 917       bootstrap  threading py 885  Thread 0x7FEF54B35700  active    HTTPServer Thread-2      accept  socket py 212      tick  cherrypy wsgiserver   init   py 2075      start  cherrypy wsgiserver   init   py 2021       start http thread  cherrypy process servers py 217      run  threading py 865       bootstrap inner  threading py 917       bootstrap  threading py 885      Thread 0x7FEF2BFFF700  idle    CP Server Thread-10      wait  threading py 296      get  queue py 170      run  cherrypy wsgiserver   init   py 1586       bootstrap inner  threading py 917       bootstrap  threading py 885

User · Answer

How to debug any function in console   Create function where you use pdb set trace    then function you want debug    gt  gt  gt  import pdb  gt  gt  gt  import my function   gt  gt  gt  def f            pdb set trace           my function          Then call created function    gt  gt  gt  f    gt   lt stdin gt  3 f    Pdb  s --Call--  gt   lt stdin gt  1 my function    Pdb     Happy debugging

User · Answer

At the point where the code is run  you can insert this small snippet to see a nicely formatted printed stack trace  It assumes that you have a folder called logs at your project s root directory     DEBUG  START DEBUG -- gt  import traceback  with open  logs stack-trace log    w   as file      traceback print stack file file    DEBUG  END DEBUG --

User · Answer

gt  gt  gt  import traceback  gt  gt  gt  def x     gt  gt  gt     print traceback extract stack     gt  gt  gt  x       lt stdin gt    1    lt module gt    None      lt stdin gt    2   x   None     You can also nicely format the stack trace  see the docs   Edit  To simulate Java s behavior  as suggested by  Douglas Leeder  add this   import signal import traceback  signal signal signal SIGUSR1  lambda sig  stack  traceback print stack stack     to the startup code in your application  Then you can print the stack by sending SIGUSR1 to the running Python process

User · Answer

I would add this as a comment to haridsv s response  but I lack the reputation to do so   Some of us are still stuck on a version of Python older than 2 6  required for Thread ident   so I got the code working in Python 2 5  though without the thread name being displayed  as such   import traceback import sys def dumpstacks signal  frame       code          for threadId  stack in sys  current frames   items                code append   n  Thread   d     threadId           for filename  lineno  name  line in traceback extract stack stack               code append  File    s   line  d  in  s     filename  lineno  name               if line                  code append     s     line strip         print   n  join code   import signal signal signal signal SIGQUIT  dumpstacks

User · Answer

In Python 3  pdb will automatically install a signal handler the first time you use c ont inue   in the debugger  Pressing Control-C afterwards will drop you right back in there  In Python 2  here s a one-liner which should work even in relatively old versions  tested in 2 7 but I checked Python source back to 2 4 and it looked okay    import pdb  signal signal signal signal SIGINT  lambda sig  frame  pdb Pdb   set trace frame     pdb is worth learning if you spend any amount of time debugging Python  The interface is a bit obtuse but should be familiar to anyone who has used similar tools  such as gdb

User · Answer

pyringe is a debugger that can interact with running python processes  print stack traces  variables  etc  without any a priori setup   While I ve often used the signal handler solution in the past  it can still often be difficult to reproduce the issue in certain environments

User · Answer

The traceback module has some nice functions  among them  print stack   import traceback  traceback print stack

User · Answer

I was looking for a while for a solution to debug my threads and I found it here thanks to haridsv  I use slightly simplified version employing the traceback print stack     import sys  traceback  signal import threading import os  def dumpstacks signal  frame     id2name   dict  th ident  th name  for th in threading enumerate      for threadId  stack in sys  current frames   items        print id2name threadId       traceback print stack f stack   signal signal signal SIGQUIT  dumpstacks   os killpg os getpgid 0   signal SIGQUIT    For my needs I also filter threads by name

User · Answer

I hacked together some tool which attaches into a running Python process and injects some code to get a Python shell   See here  https   github com albertz pydbattach

User · Answer

There is no way to hook into a running python process and get reasonable results   What I do if processes lock up is hooking strace in and trying to figure out what exactly is happening   Unfortunately often strace is the observer that  fixes  race conditions so that the output is useless there too

User · Answer

use the inspect module                  import inspect       help inspect stack        Help on function stack in module inspect            stack context 1      Return a list of records for the stack above the caller s frame   I find it very helpful indeed

User · Answer

If you re on a Linux system  use the awesomeness of gdb with Python debug extensions  can be in python-dbg or python-debuginfo package   It also helps with multithreaded applications  GUI applications and C modules   Run your program with     gdb -ex r --args python  lt programname gt  py  arguments    This instructs gdb to prepare python  lt programname gt  py  lt arguments gt  and run it   Now when you program hangs  switch into gdb console  press Ctr C and execute    gdb  thread apply all py-list   See example session and more info here and here

User · Answer

Getting a stack trace of an unprepared python program  running in a stock python without debugging symbols can be done with pyrasite  Worked like a charm for me in on Ubuntu Trusty     sudo pip install pyrasite   echo 0   sudo tee  proc sys kernel yama ptrace scope   sudo pyrasite 16262 dump stacks py   dumps stacks to stdout stderr of the python program    Hat tip to  Albert  whose answer contained a pointer to this  among other tools

User · Answer

I have module I use for situations like this - where a process will be running for a long time but gets stuck sometimes for unknown and irreproducible reasons   Its a bit hacky  and only works on unix  requires signals    import code  traceback  signal  def debug sig  frame          Interrupt running process  and provide a python prompt for     interactive debugging         d    frame  frame            Allow access to frame object      d update frame f globals     Unless shadowed by global     d update frame f locals       i   code InteractiveConsole d      message     Signal received   entering python shell  nTraceback  n      message       join traceback format stack frame       i interact message   def listen        signal signal signal SIGUSR1  debug     Register handler   To use  just call the listen   function at some point when your program starts up  You could even stick it in site py to have all python programs use it   and let it run   At any point  send the process a SIGUSR1 signal  using kill  or in python       os kill pid  signal SIGUSR1    This will cause the program to break to a python console at the point it is currently at  showing you the stack trace  and letting you manipulate the variables   Use control-d  EOF  to continue running  though note that you will probably interrupt any I O etc at the point you signal  so it isn t fully non-intrusive   I ve another script that does the same thing  except it communicates with the running process through a pipe  to allow for debugging backgrounded processes etc    Its a bit large to post here  but I ve added it as a python cookbook recipe

User · Answer

The suggestion to install a signal handler is a good one  and I use it a lot   For example  bzr by default installs a SIGQUIT handler that invokes pdb set trace   to immediately drop you into a pdb prompt    See the bzrlib breakin module s source for the exact details    With pdb you can not only get the current stack trace  with the  w here command  but also inspect variables  etc  However  sometimes I need to debug a process that I didn t have the foresight to install the signal handler in   On linux  you can attach gdb to the process and get a python stack trace with some gdb macros   Put http   svn python org projects python trunk Misc gdbinit in    gdbinit  then   Attach gdb  gdb -p PID Get the python stack trace  pystack  It s not totally reliable unfortunately  but it works most of the time  Finally  attaching strace can often give you a good idea what a process is doing

User · Answer

You can try the faulthandler module  Install it using pip install faulthandler and add   import faulthandler  signal faulthandler register signal SIGUSR1    at the beginning of your program  Then send SIGUSR1 to your process  ex  kill -USR1 42  to display the Python traceback of all threads to the standard output  Read the documentation for more options  ex  log into a file  and other ways to display the traceback   The module is now part of Python 3 3  For Python 2  see http   faulthandler readthedocs org

User · Answer

I am almost always dealing with multiple threads and main thread is generally not doing much  so what is most interesting is to dump all the stacks  which is more like the Java s dump   Here is an implementation based on this blog   import threading  sys  traceback  def dumpstacks signal  frame       id2name   dict   th ident  th name  for th in threading enumerate         code          for threadId  stack in sys  current frames   items            code append   n  Thread   s  d      id2name get threadId      threadId           for filename  lineno  name  line in traceback extract stack stack               code append  File    s   line  d  in  s     filename  lineno  name               if line                  code append     s     line strip         print   n  join code   import signal signal signal signal SIGQUIT  dumpstacks

User · Answer

You can use the hypno package  like so  hypno  lt pid gt   quot import traceback  traceback print stack   quot   This would print a stack trace into the program s stdout  Alternatively  if you don t want to print anything to stdout  or you don t have access to it  a daemon for example   you could use the madbg package  which is a python debugger that allows you to attach to a running python program and debug it in your current terminal  It is similar to pyrasite and pyringe  but newer  doesn t require gdb  and uses IPython for the debugger  which means colors and autocomplete   To see the stack trace of a running program  you could run  madbg attach  lt pid gt   And in the debugger shell  enter  bt Disclaimer - I wrote both packages

User · Answer

It s worth looking at Pydb   an expanded version of the Python debugger loosely based on the gdb command set   It includes signal managers which can take care of starting the debugger when a specified signal is sent   A 2006 Summer of Code project looked at adding remote-debugging features to pydb in a module called mpdb

User · Answer

What really helped me here is spiv s tip  which I would vote up and comment on if I had the reputation points  for getting a stack trace out of an unprepared Python process  Except it didn t work until I modified the gdbinit script  So    download http   svn python org projects python trunk Misc gdbinit and put it in    gdbinit edit it  changing PyEval EvalFrame to PyEval EvalFrameEx  edit  no longer needed  the linked file already has this change as of 2010-01-14  Attach gdb  gdb -p PID Get the python stack trace  pystack

User · Answer

In case you need to do this with uWSGI  it has Python Tracebacker built-in and it s just matter of enabling it in the configuration  number is attached to the name for each worker    py-tracebacker  var run uwsgi pytrace   Once you have done this  you can print backtrace simply by connecting to the socket   uwsgi --connect-and-read  var run uwsgi pytrace1

User · Answer

python -dv yourscript py  That will make the interpreter to run in debug mode and to give you a trace of what the interpreter is doing   If you want to interactively debug the code you should run it like this   python -m pdb yourscript py  That tells the python interpreter to run your script with the module  pdb  which is the python debugger  if you run it like that the interpreter will be executed in interactive mode  much like GDB

User · Answer

It s worth looking at Pydb   an expanded version of the Python debugger loosely based on the gdb command set   It includes signal managers which can take care of starting the debugger when a specified signal is sent   A 2006 Summer of Code project looked at adding remote-debugging features to pydb in a module called mpdb

User · Answer

On Solaris  you can use pstack 1  No changes to the python code are necessary  eg     pstack 16000   grep     head 16000   usr bin python2 6  usr lib pkg depotd --cfg svc  application pkg serv    usr lib python2 6 vendor-packages cherrypy process wspbus py 282   wait       usr lib python2 6 vendor-packages cherrypy process wspbus py 295  wait       usr lib python2 6 vendor-packages cherrypy process wspbus py 242  block       usr lib python2 6 vendor-packages cherrypy  init  py 249  quickstart       usr lib pkg depotd 890   lt module gt        usr lib python2 6 threading py 256  wait       usr lib python2 6 Queue py 177  get       usr lib python2 6 vendor-packages pkg server depot py 2142  run       usr lib python2 6 threading py 477  run  etc

User · Answer

I don t know of anything similar to java s response to SIGQUIT  so you might have to build it in to your application  Maybe you could make a server in another thread that can get a stacktrace on response to a message of some kind

User · Answer

You can use PuDB  a Python debugger with a curses interface to do this  Just add   from pudb import set interrupt handler  set interrupt handler     to your code and use Ctrl-C when you want to break  You can continue with c and break again multiple times if you miss it and want to try again

[python] Showing the stack trace from a running Python application

Examples related to python

Examples related to debugging

Examples related to stack-trace

Examples related to traceback