Ruby daemons using RobustThread

June 10, 2009

A daemon, or background process, typically follows a familiar pattern: fork a process that continuously loops over a block of code. In Ruby, this is pretty trivial:

pid = fork do
  loop do
    do_something_awesome
    sleep 30
  end
end
Process.detach pid

In this example, do_something_awesome is our application logic that is called repeatedly, with 30 seconds of sleeping in between calls. While this works and can be very effective, it is prone to failing due to broken code (unless do_something_awesome adequately handles all exceptions). However, a much larger problem is introduced: when this process is killed—as it inevitably will be—we are not guaranteed to break the loop cleanly—typically an Interrupt or SystemExit is raised. In other words, an operation might be killed in the middle of doing something important, even if we trap the signal.

Enter RobustThread, a Ruby class that allows for the creation of Threads that will be automatically re-joined when the process exits. Why is this a good thing? Here’s a quick (and totally contrived) example:

require 'thread'

Thread.new do
  sleep 10
  puts "Hello from the spawned thread"
end

puts "Hello from the main thread"

If you were to run this code you would see that it never actually prints the string “Hello from the spawned thread”. If we were to replace Thread with RobustThread, the thread would be re-joined in Ruby’s exit handler, and the string would print. Note: the thread will not be re-joined if Kernel#exit! is called, or if the process receives and untrappable signal.

To use RobustThread in a less contrived example, let’s use the same code as in the first example, but with some modification:

require 'rubygems'
require 'robustthread'

RobustThread.logger = Logger.new('mydaemon.log')
RobustThread.exception_handler do |exception|
  email_me_the_exception(exception)
end

pid = fork do
  RobustThread.loop(:seconds => 30, :label => "Do something awesome!") do
    do_something_awesome
  end
  sleep
end
Process.detach pid

Let’s go over the changes one by one. First, we implemented a logger for the RobustThread output. This is helpful for debugging. Note: RobustThread logs to STDOUT by default.

The next change is the exception handler, which allows us to gracefully deal with problems in the daemon. In this example, we’ve emailed ourselves the exception data so we know something bad happened. Note: If we don’t implement an exception handler, all exception backtraces will be piped into the log.

Inside the fork, we use RobustThread.loop, which loops over the block in a RobustThread at the interval we specify. The label is what we’ll see in the log. An additional benefit to RobustThread.loop is that it will cleanly exit the loop on the next sleep call. Note: the loop sleep timer is actually a timeout that sleeps in 0.1 second intervals, thus allowing us to break the loop instead of waiting until the sleep finishes, which may be a very long time.

The last interesting change is the line that simply calls sleep with no arguments, which causes the process to sleep until interrupted. This means that when we send a TERM signal to the process, the infinite sleep is what’s interrupted, not our code. When sleep is interrupted, the process continues and will call the exit handler, at which point RobustThread tells all loops to break when ready, and re-joins all remaining RobustThreads.

Examining the log we set up, mydaemon.log, we can see what’s happening after we’ve killed the process:

I, [...]  INFO -- : RobustThread: waiting on "Doing something awesome!"
I, [...]  INFO -- : RobustThread: "Doing something awesome!" exited cleanly
I, [...]  INFO -- : RobustThread: exited cleanly

The first line tells us about the thread we’re waiting on, the second line is added when it completes, and finally RobustThread reports that all threads were re-joined satisfactorily.

RobustThread allows us to create daemons that operate in a safe and sane manner, with a fairly minimal amount of fuss. Since the project is so new, expect it to change a bit. And please give me any feedback you might have—especially in the form of cupcakes.


Comments

Awesome work! I'm getting this into daemon-kit [1] as soon as I possibly can.

[1] http://github.com/kennethkalmer/daemo...

Posted by Kenneth Kalmer

This got mentioned in the Rails Envy podcast, episode 83.

Posted by Carl

Great job! One question.. and this is kinda noob so sorry. So to run the daemon properly do I just run the ruby script? I know that works.. but just wanted to make sure that is the best way to run it as a daemon.

Posted by Nathan Leavitt

Add your comment

No HTML; Only URLs and line breaks are converted.