Ruby daemons using RobustThread
June 10, 2009
A daemon, or background process, typically follows a familiar pattern: fork a process that continuously loops over a block of code. In Ruby, this is pretty trivial:
pid = fork do
loop do
do_something_awesome
sleep 30
end
end
Process.detach pid
In this example, do_something_awesome is our application logic that is called
repeatedly, with 30 seconds of sleeping in between calls. While this works and
can be very effective, it is prone to failing due to broken code
(unless do_something_awesome adequately handles all exceptions).
However, a much larger problem is introduced: when this process is
killed—as it inevitably will be—we are not guaranteed to break the loop
cleanly—typically an Interrupt or SystemExit is raised. In other words, an
operation might be killed in the middle of doing something important, even if we
trap the signal.
Enter RobustThread, a
Ruby class that allows for the creation of Threads that will be automatically
re-joined when the process exits. Why is this a good thing? Here’s a quick
(and totally contrived) example:
require 'thread'
Thread.new do
sleep 10
puts "Hello from the spawned thread"
end
puts "Hello from the main thread"
If you were to run this code you would see that it never actually prints the
string “Hello from the spawned thread”. If we were to replace Thread with
RobustThread, the thread would be re-joined in Ruby’s exit handler, and the
string would print. Note: the thread will not be re-joined if
Kernel#exit! is called, or if the process receives and untrappable signal.
To use RobustThread in a less contrived example, let’s use the same code as in the first example, but with some modification:
require 'rubygems'
require 'robustthread'
RobustThread.logger = Logger.new('mydaemon.log')
RobustThread.exception_handler do |exception|
email_me_the_exception(exception)
end
pid = fork do
RobustThread.loop(:seconds => 30, :label => "Do something awesome!") do
do_something_awesome
end
sleep
end
Process.detach pid
Let’s go over the changes one by one. First, we implemented a logger for the
RobustThread output. This is helpful for debugging. Note: RobustThread
logs to STDOUT by default.
The next change is the exception handler, which allows us to gracefully deal with problems in the daemon. In this example, we’ve emailed ourselves the exception data so we know something bad happened. Note: If we don’t implement an exception handler, all exception backtraces will be piped into the log.
Inside the fork, we use RobustThread.loop, which loops over the
block in a RobustThread at the interval we specify. The label is what we’ll
see in the log. An additional benefit to RobustThread.loop is that it will cleanly exit
the loop on the next sleep call. Note: the loop sleep timer is actually a
timeout that sleeps in 0.1 second intervals, thus allowing us to break the loop
instead of waiting until the sleep finishes, which may be a very long time.
The last interesting change is the line that simply calls sleep with no
arguments, which causes the process to sleep until interrupted. This means that
when we send a TERM signal to the process, the infinite sleep is what’s
interrupted, not our code. When sleep is interrupted, the process continues and
will call the exit handler, at which point RobustThread tells all loops to
break when ready, and re-joins all remaining RobustThreads.
Examining the log we set up, mydaemon.log, we can see what’s happening after
we’ve killed the process:
I, [...] INFO -- : RobustThread: waiting on "Doing something awesome!"
I, [...] INFO -- : RobustThread: "Doing something awesome!" exited cleanly
I, [...] INFO -- : RobustThread: exited cleanly
The first line tells us about the thread we’re waiting on, the second line
is added when it completes, and finally RobustThread reports that all
threads were re-joined satisfactorily.
RobustThread allows us to create daemons that operate in a safe and sane
manner, with a fairly minimal amount of fuss. Since the project is so new,
expect it to change a bit. And please give me any feedback you might
have—especially in the form of cupcakes.
Add your comment
No HTML; Only URLs and line breaks are converted.
Comments
Awesome work! I'm getting this into daemon-kit [1] as soon as I possibly can.
[1] http://github.com/kennethkalmer/daemo...
Posted by Kenneth Kalmer
This got mentioned in the Rails Envy podcast, episode 83.
Posted by Carl
Great job! One question.. and this is kinda noob so sorry. So to run the daemon properly do I just run the ruby script? I know that works.. but just wanted to make sure that is the best way to run it as a daemon.
Posted by Nathan Leavitt