[bash] How do I write a bash script to restart a process if it dies?

I have a python script that'll be checking a queue and performing an action on each item:

# checkqueue.py
while True:
  check_queue()
  do_something()

How do I write a bash script that will check if it's running, and if not, start it. Roughly the following pseudo code (or maybe it should do something like ps | grep?):

# keepalivescript.sh
if processidfile exists:
  if processid is running:
     exit, all ok

run checkqueue.py
write processid to processidfile

I'll call that from a crontab:

# crontab
*/5 * * * * /path/to/keepalivescript.sh

I've used the following script with great success on numerous servers:

pid=`jps -v | grep $INSTALLATION | awk '{print $1}'`
echo $INSTALLATION found at PID $pid 
while [ -e /proc/$pid ]; do sleep 0.1; done

notes:

• It's looking for a java process, so I can use jps, this is much more consistent across distributions than ps
• $INSTALLATION contains enough of the process path that's it's totally unambiguous
• Use sleep while waiting for the process to die, avoid hogging resources :)

This script is actually used to shut down a running instance of tomcat, which I want to shut down (and wait for) at the command line, so launching it as a child process simply isn't an option for me.

In-line:

while true; do <your-bash-snippet> && break; done

e.g.

while true; do openconnect x.x.x.x:xxxx && break; done

if ! test -f $PIDFILE || ! psgrep `cat $PIDFILE`; then
    restart_process
    # Write PIDFILE
    echo $! >$PIDFILE
fi

You should use monit, a standard unix tool that can monitor different things on the system and react accordingly.

From the docs: http://mmonit.com/monit/documentation/monit.html#pid_testing

check process checkqueue.py with pidfile /var/run/checkqueue.pid
       if changed pid then exec "checkqueue_restart.sh"

You can also configure monit to email you when it does do a restart.

Have a look at monit (http://mmonit.com/monit/). It handles start, stop and restart of your script and can do health checks plus restarts if necessary.

Or do a simple script:

while true
do
/your/script
sleep 1
done

I use this for my npm Process

#!/bin/bash
for (( ; ; ))
do
date +"%T"
echo Start Process
cd /toFolder
sudo process
date +"%T"
echo Crash
sleep 1
done

I'm not sure how portable it is across operating systems, but you might check if your system contains the 'run-one' command, i.e. "man run-one". Specifically, this set of commands includes 'run-one-constantly', which seems to be exactly what is needed.

From man page:

run-one-constantly COMMAND [ARGS]

Note: obviously this could be called from within your script, but also it removes the need for having a script at all.

The easiest way to do it is using flock on file. In Python script you'd do

lf = open('/tmp/script.lock','w')
if(fcntl.flock(lf, fcntl.LOCK_EX|fcntl.LOCK_NB) != 0): 
   sys.exit('other instance already running')
lf.write('%d\n'%os.getpid())
lf.flush()

In shell you can actually test if it's running:

if [ `flock -xn /tmp/script.lock -c 'echo 1'` ]; then 
   echo 'it's not running'
   restart.
else
   echo -n 'it's already running with PID '
   cat /tmp/script.lock
fi

But of course you don't have to test, because if it's already running and you restart it, it'll exit with 'other instance already running'

When process dies, all it's file descriptors are closed and all locks are automatically removed.