What's the best way to run scheduled tasks in a Rails environment? Script/runner? Rake? I would like to run the task every few minutes.
This question is related to
ruby-on-rails
rake
daemon
cron
runner
The answer is
The problem with whenever (and cron) is that it reloads the rails environment every time it's executed, which is a real problem when your tasks are frequent or have a lot of initialization work to do. I have had issues in production because of this and must warn you.
Rufus scheduler does it for me ( https://github.com/jmettraux/rufus-scheduler )
When I have long jobs to run, I use it with delayed_job ( https://github.com/collectiveidea/delayed_job )
I hope this helps!
Use Craken (rake centric cron jobs)
That is interesting no one mentioned the Sidetiq. It is nice addition if you already using Sidekiq.
Sidetiq provides a simple API for defining recurring workers for Sidekiq.
Job will look like this:
class MyWorker
include Sidekiq::Worker
include Sidetiq::Schedulable
recurrence { hourly.minute_of_hour(15, 45) }
def perform
# do stuff ...
end
end
Use Craken (rake centric cron jobs)
In our project we first used whenever gem, but confronted some problems.
We then switched to RUFUS SCHEDULER gem, which turned out to be very easy and reliable for scheduling tasks in Rails.
We have used it for sending weekly & daily mails, and even for running some periodic rake tasks or any method.
The code used in this is like:
require 'rufus-scheduler'
scheduler = Rufus::Scheduler.new
scheduler.in '10d' do
# do something in 10 days
end
scheduler.at '2030/12/12 23:30:00' do
# do something at a given point in time
end
scheduler.every '3h' do
# do something every 3 hours
end
scheduler.cron '5 0 * * *' do
# do something every day, five minutes after midnight
# (see "man 5 crontab" in your terminal)
end
To learn more: https://github.com/jmettraux/rufus-scheduler
Probably the best way to do it is using rake to write the tasks you need and the just execute it via command line.
You can see a very helpful video at railscasts
Also take a look at this other resources:
Both will work fine. I usually use script/runner.
Here's an example:
0 6 * * * cd /var/www/apps/your_app/current; ./script/runner --environment production 'EmailSubscription.send_email_subscriptions' >> /var/www/apps/your_app/shared/log/send_email_subscriptions.log 2>&1
You can also write a pure-Ruby script to do this if you load the right config files to connect to your database.
One thing to keep in mind if memory is precious is that script/runner (or a Rake task that depends on 'environment') will load the entire Rails environment. If you only need to insert some records into the database, this will use memory you don't really have to. If you write your own script, you can avoid this. I haven't actually needed to do this yet, but I am considering it.
I have recently created some cron jobs for the projects I have been working on.
I found that the gem Clockwork very useful.
require 'clockwork'
module Clockwork
every(10.seconds, 'frequent.job')
end
You can even schedule your background job using this gem. For documentation and further help refer https://github.com/Rykian/clockwork
I use backgroundrb.
http://backgroundrb.rubyforge.org/
I use it to run scheduled tasks as well as tasks that take too long for the normal client/server relationship.
Assuming your tasks don't take too long to complete, just create a new controller with an action for each task. Implement the logic of the task as controller code, Then set up a cronjob at the OS level that uses wget to invoke the URL of this controller and action at the appropriate time intervals. The advantages of this method are you:
- Have full access to all your Rails objects just as in a normal controller.
- Can develop and test just as you do normal actions.
- Can also invoke your tasks adhoc from a simple web page.
- Don't consume any more memory by firing up additional ruby/rails processes.
I'm a big fan of resque/resque scheduler. You can not only run repeating cron-like tasks but also tasks at specific times. The downside is, it requires a Redis server.
Here's how I have setup my cron tasks. I have one to make daily backups of SQL database (using rake) and another to expire cache once a month. Any output is logged in a file log/cron_log. My crontab looks like this:
crontab -l # command to print all cron tasks
crontab -e # command to edit/add cron tasks
# Contents of crontab
0 1 * * * cd /home/lenart/izziv. whiskas.si/current; /bin/sh cron_tasks >> log/cron_log 2>&1
0 0 1 * * cd /home/lenart/izziv.whiskas.si/current; /usr/bin/env /usr/local/bin/ruby script/runner -e production lib/monthly_cron.rb >> log/cron_log 2>&1
The first cron task makes daily db backups. The contents of cron_tasks are the following:
/usr/local/bin/rake db:backup RAILS_ENV=production; date; echo "END OF OUTPUT ----";
The second task was setup later and uses script/runner to expire cache once a month (lib/monthly_cron.rb):
#!/usr/local/bin/ruby
# Expire challenge cache
Challenge.force_expire_cache
puts "Expired cache for Challenges (Challenge.force_expire_cache) #{Time.now}"
I guess I could backup database some other way but so far it works for me :)
The paths to rake and ruby can vary on different servers. You can see where they are by using:
whereis ruby # -> ruby: /usr/local/bin/ruby
whereis rake # -> rake: /usr/local/bin/rake
Both will work fine. I usually use script/runner.
Here's an example:
0 6 * * * cd /var/www/apps/your_app/current; ./script/runner --environment production 'EmailSubscription.send_email_subscriptions' >> /var/www/apps/your_app/shared/log/send_email_subscriptions.log 2>&1
You can also write a pure-Ruby script to do this if you load the right config files to connect to your database.
One thing to keep in mind if memory is precious is that script/runner (or a Rake task that depends on 'environment') will load the entire Rails environment. If you only need to insert some records into the database, this will use memory you don't really have to. If you write your own script, you can avoid this. I haven't actually needed to do this yet, but I am considering it.
I'm not really sure, I guess it depends on the task: how often to run, how much complicated and how much direct communication with the rails project is needed etc. I guess if there was just "One Best Way" to do something, there wouldn't be so many different ways to do it.
At my last job in a Rails project, we needed to make a batch invitation mailer (survey invitations, not spamming) which should send the planned mails whenever the server had time. I think we were going to use daemon tools to run the rake tasks I had created.
Unfortunately, our company had some money problems and was "bought" by the main rival so the project was never completed, so I don't know what we would eventually have used.
The problem with whenever (and cron) is that it reloads the rails environment every time it's executed, which is a real problem when your tasks are frequent or have a lot of initialization work to do. I have had issues in production because of this and must warn you.
Rufus scheduler does it for me ( https://github.com/jmettraux/rufus-scheduler )
When I have long jobs to run, I use it with delayed_job ( https://github.com/collectiveidea/delayed_job )
I hope this helps!
script/runner and rake tasks are perfectly fine to run as cron jobs.
Here's one very important thing you must remember when running cron jobs. They probably won't be called from the root directory of your app. This means all your requires for files (as opposed to libraries) should be done with the explicit path: e.g. File.dirname(__FILE__) + "/other_file". This also means you have to know how to explicitly call them from another directory :-)
Check if your code supports being run from another directory with
# from ~
/path/to/ruby /path/to/app/script/runner -e development "MyClass.class_method"
/path/to/ruby /path/to/rake -f /path/to/app/Rakefile rake:task RAILS_ENV=development
Also, cron jobs probably don't run as you, so don't depend on any shortcut you put in .bashrc. But that's just a standard cron tip ;-)
I have recently created some cron jobs for the projects I have been working on.
I found that the gem Clockwork very useful.
require 'clockwork'
module Clockwork
every(10.seconds, 'frequent.job')
end
You can even schedule your background job using this gem. For documentation and further help refer https://github.com/Rykian/clockwork
I used clockwork gem and it works pretty well for me. There is also clockworkd gem that allows a script to run as a daemon.
Both will work fine. I usually use script/runner.
Here's an example:
0 6 * * * cd /var/www/apps/your_app/current; ./script/runner --environment production 'EmailSubscription.send_email_subscriptions' >> /var/www/apps/your_app/shared/log/send_email_subscriptions.log 2>&1
You can also write a pure-Ruby script to do this if you load the right config files to connect to your database.
One thing to keep in mind if memory is precious is that script/runner (or a Rake task that depends on 'environment') will load the entire Rails environment. If you only need to insert some records into the database, this will use memory you don't really have to. If you write your own script, you can avoid this. I haven't actually needed to do this yet, but I am considering it.
Once I had to make the same decision and I'm really happy with that decision today. Use resque scheduler because not only a seperate redis will take out the load from your db, you will also have access to many plugins like resque-web which provides a great user interface. As your system develops you will have more and more tasks to schedule so you will be able to control them from a single place.
Here's how I have setup my cron tasks. I have one to make daily backups of SQL database (using rake) and another to expire cache once a month. Any output is logged in a file log/cron_log. My crontab looks like this:
crontab -l # command to print all cron tasks
crontab -e # command to edit/add cron tasks
# Contents of crontab
0 1 * * * cd /home/lenart/izziv. whiskas.si/current; /bin/sh cron_tasks >> log/cron_log 2>&1
0 0 1 * * cd /home/lenart/izziv.whiskas.si/current; /usr/bin/env /usr/local/bin/ruby script/runner -e production lib/monthly_cron.rb >> log/cron_log 2>&1
The first cron task makes daily db backups. The contents of cron_tasks are the following:
/usr/local/bin/rake db:backup RAILS_ENV=production; date; echo "END OF OUTPUT ----";
The second task was setup later and uses script/runner to expire cache once a month (lib/monthly_cron.rb):
#!/usr/local/bin/ruby
# Expire challenge cache
Challenge.force_expire_cache
puts "Expired cache for Challenges (Challenge.force_expire_cache) #{Time.now}"
I guess I could backup database some other way but so far it works for me :)
The paths to rake and ruby can vary on different servers. You can see where they are by using:
whereis ruby # -> ruby: /usr/local/bin/ruby
whereis rake # -> rake: /usr/local/bin/rake
I've used the extremely popular Whenever on projects that rely heavily on scheduled tasks, and it's great. It gives you a nice DSL to define your scheduled tasks instead of having to deal with crontab format. From the README:
Whenever is a Ruby gem that provides a clear syntax for writing and deploying cron jobs.
Example from the README:
every 3.hours do
runner "MyModel.some_process"
rake "my:rake:task"
command "/usr/bin/my_great_command"
end
every 1.day, :at => '4:30 am' do
runner "MyModel.task_to_run_at_four_thirty_in_the_morning"
end
Using something Sidekiq or Resque is a far more robust solution. They both support retrying jobs, exclusivity with a REDIS lock, monitoring, and scheduling.
Keep in mind that Resque is a dead project (not actively maintained), so Sidekiq is a way better alternative. It also is more performant: Sidekiq runs several workers on a single, multithread process while Resque runs each worker in a separate process.
That is interesting no one mentioned the Sidetiq. It is nice addition if you already using Sidekiq.
Sidetiq provides a simple API for defining recurring workers for Sidekiq.
Job will look like this:
class MyWorker
include Sidekiq::Worker
include Sidetiq::Schedulable
recurrence { hourly.minute_of_hour(15, 45) }
def perform
# do stuff ...
end
end
I'm a big fan of resque/resque scheduler. You can not only run repeating cron-like tasks but also tasks at specific times. The downside is, it requires a Redis server.
script/runner and rake tasks are perfectly fine to run as cron jobs.
Here's one very important thing you must remember when running cron jobs. They probably won't be called from the root directory of your app. This means all your requires for files (as opposed to libraries) should be done with the explicit path: e.g. File.dirname(__FILE__) + "/other_file". This also means you have to know how to explicitly call them from another directory :-)
Check if your code supports being run from another directory with
# from ~
/path/to/ruby /path/to/app/script/runner -e development "MyClass.class_method"
/path/to/ruby /path/to/rake -f /path/to/app/Rakefile rake:task RAILS_ENV=development
Also, cron jobs probably don't run as you, so don't depend on any shortcut you put in .bashrc. But that's just a standard cron tip ;-)
Use Craken (rake centric cron jobs)
Here's how I have setup my cron tasks. I have one to make daily backups of SQL database (using rake) and another to expire cache once a month. Any output is logged in a file log/cron_log. My crontab looks like this:
crontab -l # command to print all cron tasks
crontab -e # command to edit/add cron tasks
# Contents of crontab
0 1 * * * cd /home/lenart/izziv. whiskas.si/current; /bin/sh cron_tasks >> log/cron_log 2>&1
0 0 1 * * cd /home/lenart/izziv.whiskas.si/current; /usr/bin/env /usr/local/bin/ruby script/runner -e production lib/monthly_cron.rb >> log/cron_log 2>&1
The first cron task makes daily db backups. The contents of cron_tasks are the following:
/usr/local/bin/rake db:backup RAILS_ENV=production; date; echo "END OF OUTPUT ----";
The second task was setup later and uses script/runner to expire cache once a month (lib/monthly_cron.rb):
#!/usr/local/bin/ruby
# Expire challenge cache
Challenge.force_expire_cache
puts "Expired cache for Challenges (Challenge.force_expire_cache) #{Time.now}"
I guess I could backup database some other way but so far it works for me :)
The paths to rake and ruby can vary on different servers. You can see where they are by using:
whereis ruby # -> ruby: /usr/local/bin/ruby
whereis rake # -> rake: /usr/local/bin/rake
Once I had to make the same decision and I'm really happy with that decision today. Use resque scheduler because not only a seperate redis will take out the load from your db, you will also have access to many plugins like resque-web which provides a great user interface. As your system develops you will have more and more tasks to schedule so you will be able to control them from a single place.
I use backgroundrb.
http://backgroundrb.rubyforge.org/
I use it to run scheduled tasks as well as tasks that take too long for the normal client/server relationship.
Assuming your tasks don't take too long to complete, just create a new controller with an action for each task. Implement the logic of the task as controller code, Then set up a cronjob at the OS level that uses wget to invoke the URL of this controller and action at the appropriate time intervals. The advantages of this method are you:
- Have full access to all your Rails objects just as in a normal controller.
- Can develop and test just as you do normal actions.
- Can also invoke your tasks adhoc from a simple web page.
- Don't consume any more memory by firing up additional ruby/rails processes.
In our project we first used whenever gem, but confronted some problems.
We then switched to RUFUS SCHEDULER gem, which turned out to be very easy and reliable for scheduling tasks in Rails.
We have used it for sending weekly & daily mails, and even for running some periodic rake tasks or any method.
The code used in this is like:
require 'rufus-scheduler'
scheduler = Rufus::Scheduler.new
scheduler.in '10d' do
# do something in 10 days
end
scheduler.at '2030/12/12 23:30:00' do
# do something at a given point in time
end
scheduler.every '3h' do
# do something every 3 hours
end
scheduler.cron '5 0 * * *' do
# do something every day, five minutes after midnight
# (see "man 5 crontab" in your terminal)
end
To learn more: https://github.com/jmettraux/rufus-scheduler
Here's how I have setup my cron tasks. I have one to make daily backups of SQL database (using rake) and another to expire cache once a month. Any output is logged in a file log/cron_log. My crontab looks like this:
crontab -l # command to print all cron tasks
crontab -e # command to edit/add cron tasks
# Contents of crontab
0 1 * * * cd /home/lenart/izziv. whiskas.si/current; /bin/sh cron_tasks >> log/cron_log 2>&1
0 0 1 * * cd /home/lenart/izziv.whiskas.si/current; /usr/bin/env /usr/local/bin/ruby script/runner -e production lib/monthly_cron.rb >> log/cron_log 2>&1
The first cron task makes daily db backups. The contents of cron_tasks are the following:
/usr/local/bin/rake db:backup RAILS_ENV=production; date; echo "END OF OUTPUT ----";
The second task was setup later and uses script/runner to expire cache once a month (lib/monthly_cron.rb):
#!/usr/local/bin/ruby
# Expire challenge cache
Challenge.force_expire_cache
puts "Expired cache for Challenges (Challenge.force_expire_cache) #{Time.now}"
I guess I could backup database some other way but so far it works for me :)
The paths to rake and ruby can vary on different servers. You can see where they are by using:
whereis ruby # -> ruby: /usr/local/bin/ruby
whereis rake # -> rake: /usr/local/bin/rake
I'm not really sure, I guess it depends on the task: how often to run, how much complicated and how much direct communication with the rails project is needed etc. I guess if there was just "One Best Way" to do something, there wouldn't be so many different ways to do it.
At my last job in a Rails project, we needed to make a batch invitation mailer (survey invitations, not spamming) which should send the planned mails whenever the server had time. I think we were going to use daemon tools to run the rake tasks I had created.
Unfortunately, our company had some money problems and was "bought" by the main rival so the project was never completed, so I don't know what we would eventually have used.
you can use resque and resque-schedular gem for creating cron, this is very easy to do.
I Use script to run cron, that is the best way to run a cron. Here is some example for cron,
Open CronTab —> sudo crontab -e
And Paste Bellow lines:
00 00 * * * wget https://your_host/some_API_end_point
Here is some cron format, will help you
::CRON FORMAT::
Examples Of crontab Entries
15 6 2 1 * /home/melissa/backup.sh
Run the shell script /home/melissa/backup.sh on January 2 at 6:15 A.M.
15 06 02 Jan * /home/melissa/backup.sh
Same as the above entry. Zeroes can be added at the beginning of a number for legibility, without changing their value.
0 9-18 * * * /home/carl/hourly-archive.sh
Run /home/carl/hourly-archive.sh every hour, on the hour, from 9 A.M. through 6 P.M., every day.
0 9,18 * * Mon /home/wendy/script.sh
Run /home/wendy/script.sh every Monday, at 9 A.M. and 6 P.M.
30 22 * * Mon,Tue,Wed,Thu,Fri /usr/local/bin/backup
Run /usr/local/bin/backup at 10:30 P.M., every weekday.
Hope this will help you :)
Assuming your tasks don't take too long to complete, just create a new controller with an action for each task. Implement the logic of the task as controller code, Then set up a cronjob at the OS level that uses wget to invoke the URL of this controller and action at the appropriate time intervals. The advantages of this method are you:
- Have full access to all your Rails objects just as in a normal controller.
- Can develop and test just as you do normal actions.
- Can also invoke your tasks adhoc from a simple web page.
- Don't consume any more memory by firing up additional ruby/rails processes.
Probably the best way to do it is using rake to write the tasks you need and the just execute it via command line.
You can see a very helpful video at railscasts
Also take a look at this other resources:
script/runner and rake tasks are perfectly fine to run as cron jobs.
Here's one very important thing you must remember when running cron jobs. They probably won't be called from the root directory of your app. This means all your requires for files (as opposed to libraries) should be done with the explicit path: e.g. File.dirname(__FILE__) + "/other_file". This also means you have to know how to explicitly call them from another directory :-)
Check if your code supports being run from another directory with
# from ~
/path/to/ruby /path/to/app/script/runner -e development "MyClass.class_method"
/path/to/ruby /path/to/rake -f /path/to/app/Rakefile rake:task RAILS_ENV=development
Also, cron jobs probably don't run as you, so don't depend on any shortcut you put in .bashrc. But that's just a standard cron tip ;-)
Assuming your tasks don't take too long to complete, just create a new controller with an action for each task. Implement the logic of the task as controller code, Then set up a cronjob at the OS level that uses wget to invoke the URL of this controller and action at the appropriate time intervals. The advantages of this method are you:
- Have full access to all your Rails objects just as in a normal controller.
- Can develop and test just as you do normal actions.
- Can also invoke your tasks adhoc from a simple web page.
- Don't consume any more memory by firing up additional ruby/rails processes.
I use backgroundrb.
http://backgroundrb.rubyforge.org/
I use it to run scheduled tasks as well as tasks that take too long for the normal client/server relationship.
I Use script to run cron, that is the best way to run a cron. Here is some example for cron,
Open CronTab —> sudo crontab -e
And Paste Bellow lines:
00 00 * * * wget https://your_host/some_API_end_point
Here is some cron format, will help you
::CRON FORMAT::
Examples Of crontab Entries
15 6 2 1 * /home/melissa/backup.sh
Run the shell script /home/melissa/backup.sh on January 2 at 6:15 A.M.
15 06 02 Jan * /home/melissa/backup.sh
Same as the above entry. Zeroes can be added at the beginning of a number for legibility, without changing their value.
0 9-18 * * * /home/carl/hourly-archive.sh
Run /home/carl/hourly-archive.sh every hour, on the hour, from 9 A.M. through 6 P.M., every day.
0 9,18 * * Mon /home/wendy/script.sh
Run /home/wendy/script.sh every Monday, at 9 A.M. and 6 P.M.
30 22 * * Mon,Tue,Wed,Thu,Fri /usr/local/bin/backup
Run /usr/local/bin/backup at 10:30 P.M., every weekday.
Hope this will help you :)
Use Craken (rake centric cron jobs)
I'm not really sure, I guess it depends on the task: how often to run, how much complicated and how much direct communication with the rails project is needed etc. I guess if there was just "One Best Way" to do something, there wouldn't be so many different ways to do it.
At my last job in a Rails project, we needed to make a batch invitation mailer (survey invitations, not spamming) which should send the planned mails whenever the server had time. I think we were going to use daemon tools to run the rake tasks I had created.
Unfortunately, our company had some money problems and was "bought" by the main rival so the project was never completed, so I don't know what we would eventually have used.
I've used the extremely popular Whenever on projects that rely heavily on scheduled tasks, and it's great. It gives you a nice DSL to define your scheduled tasks instead of having to deal with crontab format. From the README:
Whenever is a Ruby gem that provides a clear syntax for writing and deploying cron jobs.
Example from the README:
every 3.hours do
runner "MyModel.some_process"
rake "my:rake:task"
command "/usr/bin/my_great_command"
end
every 1.day, :at => '4:30 am' do
runner "MyModel.task_to_run_at_four_thirty_in_the_morning"
end
you can use resque and resque-schedular gem for creating cron, this is very easy to do.
I'm not really sure, I guess it depends on the task: how often to run, how much complicated and how much direct communication with the rails project is needed etc. I guess if there was just "One Best Way" to do something, there wouldn't be so many different ways to do it.
At my last job in a Rails project, we needed to make a batch invitation mailer (survey invitations, not spamming) which should send the planned mails whenever the server had time. I think we were going to use daemon tools to run the rake tasks I had created.
Unfortunately, our company had some money problems and was "bought" by the main rival so the project was never completed, so I don't know what we would eventually have used.
I used clockwork gem and it works pretty well for me. There is also clockworkd gem that allows a script to run as a daemon.
Using something Sidekiq or Resque is a far more robust solution. They both support retrying jobs, exclusivity with a REDIS lock, monitoring, and scheduling.
Keep in mind that Resque is a dead project (not actively maintained), so Sidekiq is a way better alternative. It also is more performant: Sidekiq runs several workers on a single, multithread process while Resque runs each worker in a separate process.
Source: Stackoverflow.com