Cron and virtualenv

Question

I am trying to run a Django management command from cron  I am using virtualenv to keep my project sandboxed   I have seen examples here and elsewhere that show running management commands from within virtualenv s like   0 3       source  home user project env bin activate  amp  amp   home user project manage py command arg   However  even though syslog shows an entry when the task should have started  this task never actually runs  the log file for the script is empty   If I run the line manually from the shell  it works as expected    The only way I can currently get the command to run via cron  is to break the commands up and put them in a dumb bash wrapper script      bin sh source  home user project env bin activate cd  home user project    manage py command arg   EDIT   ars came up with a working combination of commands   0 3       cd  home user project  amp  amp   home user project env bin python  home user project manage py command arg   At least in my case  invoking the activate script for the virtualenv did nothing  This works  so on with the show

User · Answer

Running source from a cronfile won t work as cron uses  bin sh as its default shell  which doesn t support source   You need to set the SHELL environment variable to be  bin bash   SHELL  bin bash   10         root source  path to virtualenv bin activate  amp  amp   path to build manage py some command  gt   dev null   It s tricky to spot why this fails as  var log syslog doesn t log the error details   Best to alias yourself to root so you get emailed with any cron errors   Simply add yourself to  etc aliases and run sendmail -bi   More info here  http   codeinthehole com archives 43-Running-django-cronjobs-within-a-virtualenv html  the link above is changed to  https   codeinthehole com tips running-django-cronjobs-within-a-virtualenv

User · Answer

python script   from datetime import datetime                                                                                                                                                                 import boto     check wheather its taking the virtualenv or not                                                                                                                                                                         import sys                                                                                                                                                                                    param1 sys argv 1       Param                                                                                                                                                                                                                                                                                                                                                                     myFile   open  appendtxt txt    a                                                                                                                                                         myFile write   nAccessed on     param1 str datetime now           Cron command      1          cd  Workspace testcron   amp  amp   Workspace testcron venvcron bin python3   Workspace testcron testcronwithparam py param        In above command        1         - Execute every one minte cd  Workspace testcron  - Path of the python script  Workspace testcron venvcron bin python3 - Virtualenv path Workspace testcron testcronwithparam py - File path param - parameter

User · Answer

The only correct way to run python cron jobs when using a virtualenv is to activate the environment and then execute the environment s python to run your code   One way to do this is use virtualenv s activate this in your python script  see  http   virtualenv readthedocs org en latest userguide html using-virtualenv-without-bin-python  Another solution is echoing the complete command including activating the environment and piping it into  bin bash  Consider this for your  etc crontab         root echo  source  env bin activate  python  your script     bin bash

User · Answer

Since a cron executes in its own minimal sh environment  here s what I do to run Python scripts in a virtual environment                 bash profile      path to venv bin activate  python   path to script py   Note  if      bash profile doesn t work for you  then try      bashrc or      profile depending on how your server is set up   This loads your bash shell environment  then activates your Python virtual environment  essentially leaving you with the same setup you tested your scripts in  No need to define environment variables in crontab and no need to modify your existing scripts

User · Answer

The best solution for me was to both    use the python binary in the venv bin  directory set the python path to include the venv modules directory    man python mentions modifying the path in shell at  PYTHONPATH or in python with sys path  Other answers mention ideas for doing this using the shell   From python  adding the following lines to my script allows me to successfully run it directly from cron   import sys sys path insert 0   path to venv lib python3 3 site-packages      Here s how it looks in an interactive session --  Python 3 3 2   default  Feb 28 2014  00 52 16    GCC 4 8 1  on linux Type  help    copyright    credits  or  license  for more information    gt  gt  gt  import sys   gt  gt  gt  sys path        usr lib python3 3     usr lib python3 3 plat-x86 64-linux-gnu     usr lib python3 3 lib-dynload     gt  gt  gt  import requests Traceback  most recent call last     File   lt stdin gt    line 1  in  lt module gt  ImportError  No module named  requests       gt  gt  gt  sys path insert 0   path to venv modules       gt  gt  gt  import requests  gt  gt  gt

User · Answer

Don t look any further   0 3        usr bin env bash -c  cd  home user project  amp  amp  source  home user project env bin activate  amp  amp    manage py command arg   gt   dev null 2 gt  amp 1   Generic approach              usr bin env bash -c  YOUR COMMAND HERE   gt   dev null 2 gt  amp 1   The beauty about this is you DO NOT need to change the SHELL variable for crontab from sh to bash

User · Answer

This is a solution that has worked well for me   source  root miniconda3 etc profile d conda sh  amp  amp    conda activate  lt your env gt   amp  amp    python  lt your application gt   amp    I am using miniconda with Conda version 4 7 12 on a Ubuntu 18 04 3 LTS   I am able to place the above inside a script and run it via crontab as well without any trouble

User · Answer

If you re on python and using a Conda Virtual Environment where your python script contains the shebang    usr bin env python the following works            cd  home user project  amp  amp   home user anaconda3 envs envname bin python script py 2 gt  amp 1  Additionally  if you want to capture any outputs in your script  e g  print  errors  etc  you can use the following            cd  home user project  amp  amp   home user anaconda3 envs envname bin python script py  gt  gt   home user folder script name log 2 gt  amp 1

User · Answer

Rather than mucking around with virtualenv-specific shebangs  just prepend PATH onto the crontab    From an activated virtualenv  run these three commands and python scripts should just work     echo  PATH  PATH   gt  myserver cron   crontab -l  gt  gt  myserver cron   crontab myserver cron   The crontab s first line should now look like this   PATH  home me virtualenv bin  usr bin  bin      etc

User · Answer

I d like to add this because I spent some time solving the issue and did not find an answer here for combination of variables usage in cron and virtualenv  So maybe it ll help someone   PATH  usr local sbin  usr local bin  sbin  bin  usr sbin  usr bin DIR SMTH  cd  smth  VENV    venv bin activate  CMD  some python bin do something    m h  dom mon dow   command 0          DIR SMTH  amp  amp   VENV  amp  amp   CMD -k2 some target  gt  gt   tmp crontest log 2 gt  amp 1   It did not work well when it was configured like     DIR SMTH  cd  smth  amp  amp    venv bin activate    Thanks  davidwinterbottom   reed-sandberg and  mkb for giving the right direction  The accepted answer actually works fine until your python need to run a script which have to run another python binary from venv bin directory

User · Answer

I ve added the following script as manage sh inside my Django project  it sources the virtualenv and then runs the manage py script with whatever arguments you pass to it  It makes it very easy in general to run commands inside the virtualenv  cron  systemd units  basically anywhere       bin bash    this is a convenience script that first sources the venv  assumed to be in      venv  and then executes manage py with whatever arguments you supply the   script with  this is useful if you need to execute the manage py from   somewhere where the venv isn t sourced  e g  system scripts     get the script s location DIR  quot    cd  quot    dirname  quot   BASH SOURCE 0   quot    quot   gt  dev null 2 gt  amp 1  amp  amp  pwd   quot     source venv  lt - UPDATE THE PATH HERE WITH YOUR VENV s PATH source  DIR    venv bin activate    run manage py script  DIR manage py  quot    quot   Then in your cron entry you can just run  0 3        home user project manage sh command arg  Just remember that you need to make the manage sh script executable

User · Answer

You should be able to do this by using the python in your virtual environment    home my virtual bin python  home my project manage py command arg   EDIT  If your django project isn t in the PYTHONPATH  then you ll need to switch to the right directory   cd  home my project  amp  amp   home my virtual bin python       You can also try to log the failure from cron   cd  home my project  amp  amp   home my virtual bin python  home my project manage py  gt   tmp cronlog txt 2 gt  amp 1   Another thing to try is to make the same change in your manage py script at the very top      home my virtual bin python

[python] Cron and virtualenv

Examples related to python

Examples related to django

Examples related to cron

Examples related to virtualenv