This post originated from an RSS feed registered with Ruby Buzz
by Patrick Lenz.
Original Post: Use god to spawn FastCGI processes
Feed Title: poocs.net
Feed URL: http://feeds.feedburner.com/poocsnet
Feed Description: Personal weblog about free and open source software, personal development projects and random geek buzz.
For some reason we’re still on FastCGI with one of our apps and it’s not planned to change any time soon. But as is notorious about Ruby’s processes, they can get out of hand, both uptime wise and memory-usage wise. The former is greatly taken care of by the script/spinner utility that Rails ships with. The latter? Not so much.
System monitoring in general
We’ve traditionally employed Nagios as our cluster-wide monitoring tool. However, Nagios gets really clumsy dealing with anything that needs to happen locally on a machine like monitoring a specific process without outside connectivity (think 127.0.0.1) or something like restarting a service that’s unavailable for some reason (nagios-nrpe? Yech!).
A different approach
Then there are systems like Monit and God. Monit attempts to address the issue of not being able to do* anything when you *know something is wrong (like a service being down). And God, well, God is described by its author Tom Preston-Werner (of GitHub and Gravatar fame) as “like monit, only awesome”.
Enter god
God’s awesomeness (apart from its name) comes from the fact that it uses your favorite programming language (and all of its nice syntax tricks) for its config file. The god homepage has a plethora of configuration examples ranging from simple poll-based monitoring to using sophisticated event-based monitoring through its native support for kqueue/netlink, so I’m not going to repeat any of that here. In short: God rocks. (As if you didn’t know that already.)
Replacing the spinner/spawner duo
Here’s a short configuration recipe to replace an existing spinner/spawner configuration for FastCGI with God. Basically, I’m just writing this down for myself, since I’m probably the only one stuck with a FastCGI-based app. So you may just as well stop reading here and grab a latte.
# config.god
RAILS_ROOT = "/apps/yourapp/current"
7000.upto(7009) do |port|
God.watch do |w|
w.name = "#{`hostname`.strip}-fastcgi-#{port}"
w.pid_file = File.join(RAILS_ROOT, "tmp/pids/dispatch.#{port}.pid")
w.start = "spawn-fcgi -f #{RAILS_ROOT}/public/dispatch.fcgi -p #{port} -P #{w.pid_file}"
w.stop = "kill -9 `cat #{w.pid_file}`"
w.interval = 30.seconds
w.start_if do |start|
start.condition(:process_running) do |c|
c.interval = 5.seconds
c.running = false
end
end
w.restart_if do |restart|
restart.condition(:memory_usage) do |c|
c.above = 200.megabytes
c.times = [3, 5]
end
end
end
end
This config spawns and monitors 10 FastCGi processes on the machine we’re running god on. The name god shows in its logs, status reports and email notifications is derived from the machine name plus its port number. The process is monitored every 5 seconds to make sure it’s running. It’s also monitored every 30 seconds (the default) whether it’s above or beyond the set memory limit of 200MB, which was what originally got me playing with god.
Of course you can then start weaving in all sorts of additional niftyness such as email notification, monitoring for maximum CPU usage, etc. But that’s beyond the scope of this post and it’s also nicely outlined on the god website.
Live monitoring
==========
Getting a quick overview what’s up:
$ god status
http-3-fastcgi-7000: up
http-3-fastcgi-7001: up
http-3-fastcgi-7002: up
http-3-fastcgi-7003: up
http-3-fastcgi-7004: up
http-3-fastcgi-7005: up
http-3-fastcgi-7006: up
http-3-fastcgi-7007: up
http-3-fastcgi-7008: up
http-3-fastcgi-7009: up
Fetching log entries for a specific port (sub-string match):
$ god log 7004
I, [2008-02-04T12:31:30.361125 #20050] INFO -- : http-3-fastcgi-7004 [ok] process is running (ProcessRunning)
I, [2008-02-04T12:31:30.372208 #20050] INFO -- : http-3-fastcgi-7004 [ok] memory within bounds [92256kb, 92256kb, 92256kb, 92256kb, 92256kb] (MemoryUsage)
I, [2008-02-04T12:32:00.601712 #20050] INFO -- : http-3-fastcgi-7004 [ok] process is running (ProcessRunning)
I, [2008-02-04T12:32:00.617895 #20050] INFO -- : http-3-fastcgi-7004 [ok] memory within bounds [92256kb, 92256kb, 92256kb, 92256kb, 92256kb] (MemoryUsage)
I, [2008-02-04T12:32:30.816031 #20050] INFO -- : http-3-fastcgi-7004 [ok] process is running (ProcessRunning)
I, [2008-02-04T12:32:30.923948 #20050] INFO -- : http-3-fastcgi-7004 [ok] memory within bounds [92256kb, 92256kb, 92256kb, 92256kb, 92256kb] (MemoryUsage)
Previous Topic
