Last week, as I was about to deploy a small application, I got a case of "ughh". Well, if there is such a case, I had it. Normally, I deploy with mod_fcgid, Apache, and suexec. It's an awesome solution that's difficult to set up but easier to maintain.

mod_fcgid Deployment

Roughly, here's the steps to such a deployment:

• Download FastCGI. Compile and install it.
• Install fastcgi ruby gem.
• Download, compile, and install fcgid for Apache 2. Edit apache config to load the module.
• Deploy the source code / database. Make sure it's in the directory that suexec is set to "allow".
• Add a virtual host. Make sure you allow the .htaccess file to override. Set a user and group to run the app as.
• Check your .htaccess file. Comment out the "cgi" line and uncomment the "fcgi" line.
• Make dispatch.fcgi executable. Make the shebang point to a valid ruby path.
• Make dispatch.fcgi not publicly writable.
• Make public not publicly writable.
• Make your app not publicly writable.
• Add these steps to your capistrano deploy script to make sure this is always the case.
• apachectl graceful.
• Cross your fingers - didn't work.
• Check your apache error log.
• Oh crud... I forgot to do such and such.
• Fix it.
• apachectl graceful.
• Try again.
• See another error screen.
• Check your apache error log.
• Ugh, forgot that too.
• Fix that.
• apachectl graceful.
• Repeat several times until application starts.
• Enjoy!

You can imagine why I dreaded doing that for one tiny non-mission-critical application with one controller.

Mongrel Cluster with HTTP Proxy Deployment

The Mongrel Cluster / HTTP proxy is a great solution, perhaps the most popular now. I've never been a huge fan. Why? Well... I'll tell you (he's going to tell, he's going to tell... @ 7:30)

• It's one more thing to make sure is running.
• When you restart your cluster and don't redirect all the requests to a "we're down" page, Apache gets backlogged and all your users get a nice "Service not available" screen.
• Sometimes mongrels don't restart.
• Sometimes one doesn't come back up.
• Mongrels never cycle through in their life span, so if your app has a memory leak, you're memory is going to creep.

So, it's a great option, but more work than I'd like to do, and I'd still prefer to set up mod_fcgid because I'm much better at it than mongrel, and I just really like having Apache start my application servers.

Passenger (mod_rails) to the rescue

So I heard some buzz recently about Passenger (aka mod_rails): something about rails deployment as easy as uploading your application. That sounded really good. I didn't care about benchmarks for this app, but as it turns out, passenger is about on par with "thin", and is faster than mongrel! Sweet lovin'.

After 2 minutes of going through the setup instructions and deploying my code, I was incredulous. "Watch, I'm going to start the application up, and everything will just fall apart." I was pleasantly proven wrong!

Here's a summary of the install steps for mod_rails (Passenger)

• Gem install passenger.
• Passenger-install-apache-module.
• Copy and paste the 3 lines of apache config.
• Deploy your app source code / set-up db.
• Set up your virtual host (in 4 lines of code).
• apachectl graceful.
• That's it! Really!

And, I was pleasantly shocked to discover that it intelligently ran the rails processes as the user I deployed the application as. It checks config/environment.rb, and runs as the owner of that file. Hot stuff! And to reboot the server, just type "touch tmp/restart" - You could reboot your app with scp, rsync, ssh, ftp, samba, magnetic needles, anything! No need to have sudo access. No pid files. No "killall". Hot hot hot!

How's stability? I've converted our mongrel and fcgid deployments over to mod_rails now. Things are running smoothly. There seems to be a bit of a lag if the app isn't in use for a while (I don't think it keeps a minimum pool available, which would be a nice feature). I had a few issues with slowness on version 1.0.1, but 1.0.2 and onward has been rock solid. Oh - 1.0.4 doesn't work on the default install of Apache server for OS X Server - use 1.0.3 instead.

In Conclusion...

Here's a table of deployment scenarios with strengths and weaknesses, according to me and my experience.

	mod_rails	fcgid	Mongrel Cluster
Speed	Great	Great	Great
Reliability	Great	Great	Pretty good
Ease of setup	Easy	Difficult	Moderate
Graceful restarts? (restart app w/out dropping a single request)	Yes	Yes	No
App restarted on apache restart	Yes	Yes	No
Automatic process cycling	Yes	Yes	No
Process management	Allocates on demand	Allocates on demand	Fixed regardless of load
"Zombie" process recovery	Unnecessary	Excellent	Non-existant
Coolness factor	New kid on the block. Company logo looks totally hip-hop.	l77t hax0rs use it	Everyone's doing it.

I'm completely converted. I'm never going to go through that deployment nightmare again! Kudos to the Phusion folks for putting such a great solution together. If you've got a small app that you need to deploy, give it a spin. You won't regret it.

Posted by Tim Harper at 9:17 PM | Permalink | Comments

~ $ history | awk {'print $2'} | sort | uniq -c | sort -k1 -rn | head
 159 ll
 149 cd
  47 git
  27 rake
  23 mysql
  15 cat
   8 sudo
   7 ssh
   7 exit
   7 bg

OK, I tag Jeremy, David, and Chris

Posted by Tim Harper at 11:23 PM | Permalink | Comments

OK - a word to those as wise as I was previously to encountering this issue:

Do you have text blobs bigger than a few MBs that you'd like to log to the database? Guess who did? Here's a great thing to know about that:

max_allowed_packet - Maximum allowed packet size. In other words - if your query is bigger than this number, MySQL will clench it's little lips up and spit it's food back in your face (or is that my 1-year-old daughter?). This variable defaults to 1MB. You can set it up to 1GB.

Now for the fun part - since I didn't know about this value (nor that the data I was logging was going to get so big) mysql wouldn't take my 20MB+ insert query. Then, the ExceptionNotifier plugin (the helpful plugin that tells you when you have errors, and tells you what the errors were about) wanted to be kind enough to tell me about it. Not only did it try and email the failed query, but it multiplied it a few times, and spit out all kinds of other information. This ultimately sent our server into a tailspin and ate up all the cache. We had to call Rack-Space and ask them to get us back up and running again.

Moral of the story? Well, if you're gonna feed MySQL a 20MB chunk of data, make sure you set the max_allowed_packet variable higher. And, watch out for ExceptionNotifier. I've got a giant TODO on my list of things to do to go and patch ExceptionNotifier to prevent that from happening again.

Until then, may you be wiser than I was.

Posted by Tim Harper at 8:41 PM | Permalink | Comments

Yesterday I set up Munin, by recommendation of a good friend of mine. It's like an EKG, only for servers, and produces charts like the following:

It can produce charts for literally any resouce you can think of (provided you have some way to get to the data via scripting). It'll produce charts for the last 24 hours, week, month, and year.

Installation Hurdles:

No packages for RHEL5!

I installed Munin on a server running RHEL5. Which, for me, meant no packages! The instructions here worked well for me.

Perl library went missing: "Can't locate RRDs.pm in @INC"

On the host, I had to download rrdtool. Even after installation, the munin-cron process still failed to find it (even if I modified the PERL5LIB variable to include rrdtool's isolated include directory.

Cron jobs have a very bare environment. Modifying the PERL5LIB environment variable in the cron file to include the rrd-tool library did the job (hint: to edit the munin's user cron jobs, run crontab -e as the user munin).

PERL5LIB = /usr/local/rrdtool-1.2.27/lib/perl/5.8.8/i386-linux-thread-multi/
*/5 * * * * /opt/munin/bin/munin-cron # Update munin every 5 minutes

Warning spam: "Possible attempt to put comments in qw() list at /opt/munin/lib/munin-graph line 169"

Now munin was working. However, I started getting this email message every 5 minutes (since all cron system errors get directed to me):

Possible attempt to put comments in qw() list at /opt/munin/lib/munin-graph line 169

Turned out, the munin-graph perl script includes a -w flag in the shebang line. This flag tells perl to be extra-whiny (perl -h claims that the warnings are extra-useful). Removing this effectively suppressed the warning.

Conclusion

After I installed the nodes on both machines I wanted to monitor, and installed the master process, and hooked it into the cron job, all was working peachy king. I can watch every resource on all of our machines, and see potential problems before they happen.

But, the journey's not over! I'm going to get some plugins, either via discovery or via blazing my own trail, so we can monitor all of our rails processes - see what's running, cpu usage of each, memory usage, etc., to help us detect any problems with memory leaks or rampant processes.

Posted by Tim Harper at 10:19 AM | Permalink | Comments

Between mine and Geoff's efforts, we've finally reached a point where the Git OS X Installer is deemed "stable", and has now become an official source on Git's home page. There were a few major hurdles along the way:

HURDLES & SOLUTIONS:

---

HURDLE: Ruby script to update PATH and MANPATH failed on some machines

SOLUTION:

• Convert script to bash. Ruby was less than reliable, due to the inability to predict where it was installed.

---

HURDLE: Binaries still referencing libs from Darwin Ports, instead of the libs we've bundled with the installer.

SOLUTION:

• Add NO_DARWIN_PORTS=1 to the top of the Makefile
• Put LDFLAGS="-L/usr/local/git/lib,/usr/lib" in the arguments for each of the make commands

---

HURDLE: Package installer size was whopping 40MB download, and 89 MB to install:

SOLUTION:

• Converted the 80+ hardlinks (git-branch, git-remote, etc.) to symlinks. The OS X PackageMaker just doesn't respect hardlinks and was treating them all as individual duplicates of a file.
• Stripped binaries of debug symbols after compiling via "make strip"

---

CONCLUSION

Now, the package has worked very well on machines that don't have the developer tools installed. The download size is 3.6 MB (instead of the original 40 MB), and it takes 9 MB to install, instead of 87 MB. That's pretty great progress in my eyes! And considering the fact neither Geoff or I have built a package for OS X before, it's a reasonably great accomplishment.

There are a few minor glitches still happening:

• On some machines, it complains about not being able to create "/". This error does not mean git wasn't installed successfully, but it may mean that you need to download this script and run it manually, yourself. (or, set PATH and MANPATH in your environment to include /usr/local/git/bin and /usr/local/git/man, respectively)
• Unless you launch Git GUI from the shell, it fails on fetch / push / clone over the network. You can use the "Open in Git Gui" droplet to help overcome this.

Download Git for OS X

Posted by Tim Harper at 7:51 AM | Permalink | Comments

Yesterday's April Fools day was full of fun. My favorite jokes, in the following order:

Cobol on Cogs

HTTP://WWW.COBOLONCOGS.ORG/INDEX.HTM

This was my favorite one of all :) I got hours and hours of laughter with this one! My favorite was the excerpt under "Get Excited": Audio cassettes, screen-shots, and binders full of printouts! I remember the day when we used to get magazines full of printouts of computer programs so you could go and type them in on your own (note here Gentoo users: real programmers type in their own code for their operating system :P).

GitHub's SVNHub

What was especially funny about this one was that some people apparently thought it was serious!.

Google's Virgle

Virgle

Google had two pranks that I'm aware of this year. I love Google and appreciate their consistent playful nature around this time of year.

I didn't find out about virgle until after the fact, but was my favorite of their two. I got a great kick out of their quiz.

Google "Custom Time"

Not nearly as great as Last year's gag, but still got a laugh or two.

XCodeMate

I started this one, but it wouldn't have been fun without everyone who played along and had a good laugh with me. Essentially, XCodeMate installs a lot of EASILY REMOVABLE features into TextMate, all of which assign hot-keys in TextMate that open up Rick Astley's infamous video of Never gonna give you up. See, that's a feature? I'm never gonna give up on you code.

At first I had the bundle map enter, escape, tab, ctrl-option-command-b to Rick-Rollin commands, but I decided it was probably a bit too cruel and toned it down just a tad :)

---

Now back to seriousness!

Tim

Posted by Tim Harper at 10:20 PM | Permalink | Comments

No longer is it necessary to sacrifice elegance and simplicity for the power of a robust-IDE

I bring you:

The XCodeMate bundle

More details on the project page. Enjoy!

---

Update...

April Fools!

Thank you everyone who played along and had fun with this. If you're discovering this bundle after the fact: it was a all a joke.

However, if you're in the mood for some harmless belated-april-fools humour, the bundle install instructions are still available and should yield you with just that :)

Thanks,

Tim

Posted by Tim Harper at 2:46 PM | Permalink | Comments