~ $ history | awk {'print $2'} | sort | uniq -c | sort -k1 -rn | head
 159 ll
 149 cd
  47 git
  27 rake
  23 mysql
  15 cat
   8 sudo
   7 ssh
   7 exit
   7 bg

posted by Tim Harper  # 11:23 PM 2 Comments

OK - a word to those as wise as I was previously to encountering this issue:

Do you have text blobs bigger than a few MBs that you'd like to log to the database? Guess who did? Here's a great thing to know about that:

max_allowed_packet - Maximum allowed packet size. In other words - if your query is bigger than this number, MySQL will clench it's little lips up and spit it's food back in your face (or is that my 1-year-old daughter?). This variable defaults to 1MB. You can set it up to 1GB.

Now for the fun part - since I didn't know about this value (nor that the data I was logging was going to get so big) mysql wouldn't take my 20MB+ insert query. Then, the ExceptionNotifier plugin (the helpful plugin that tells you when you have errors, and tells you what the errors were about) wanted to be kind enough to tell me about it. Not only did it try and email the failed query, but it multiplied it a few times, and spit out all kinds of other information. This ultimately sent our server into a tailspin and ate up all the cache. We had to call Rack-Space and ask them to get us back up and running again.

Moral of the story? Well, if you're gonna feed MySQL a 20MB chunk of data, make sure you set the max_allowed_packet variable higher. And, watch out for ExceptionNotifier. I've got a giant TODO on my list of things to do to go and patch ExceptionNotifier to prevent that from happening again.

Until then, may you be wiser than I was.

posted by Tim Harper  # 8:41 PM 1 Comments

Yesterday I set up Munin, by recommendation of a good friend of mine. It's like an EKG, only for servers, and produces charts like the following:

It can produce charts for literally any resouce you can think of (provided you have some way to get to the data via scripting). It'll produce charts for the last 24 hours, week, month, and year.

Installation Hurdles:

No packages for RHEL5!

I installed Munin on a server running RHEL5. Which, for me, meant no packages! The instructions here worked well for me.

Perl library went missing: "Can't locate RRDs.pm in @INC"

On the host, I had to download rrdtool. Even after installation, the munin-cron process still failed to find it (even if I modified the PERL5LIB variable to include rrdtool's isolated include directory.

Cron jobs have a very bare environment. Modifying the PERL5LIB environment variable in the cron file to include the rrd-tool library did the job (hint: to edit the munin's user cron jobs, run crontab -e as the user munin).

PERL5LIB = /usr/local/rrdtool-1.2.27/lib/perl/5.8.8/i386-linux-thread-multi/
*/5 * * * * /opt/munin/bin/munin-cron # Update munin every 5 minutes

Warning spam: "Possible attempt to put comments in qw() list at /opt/munin/lib/munin-graph line 169"

Now munin was working. However, I started getting this email message every 5 minutes (since all cron system errors get directed to me):

Possible attempt to put comments in qw() list at /opt/munin/lib/munin-graph line 169

Turned out, the munin-graph perl script includes a -w flag in the shebang line. This flag tells perl to be extra-whiny (perl -h claims that the warnings are extra-useful). Removing this effectively suppressed the warning.

Conclusion

After I installed the nodes on both machines I wanted to monitor, and installed the master process, and hooked it into the cron job, all was working peachy king. I can watch every resource on all of our machines, and see potential problems before they happen.

But, the journey's not over! I'm going to get some plugins, either via discovery or via blazing my own trail, so we can monitor all of our rails processes - see what's running, cpu usage of each, memory usage, etc., to help us detect any problems with memory leaks or rampant processes.

posted by Tim Harper  # 10:19 AM 1 Comments

Between mine and Geoff's efforts, we've finally reached a point where the Git OS X Installer is deemed "stable", and has now become an official source on Git's home page. There were a few major hurdles along the way:

HURDLES & SOLUTIONS:

HURDLE: Ruby script to update PATH and MANPATH failed on some machines

SOLUTION:

• Convert script to bash. Ruby was less than reliable, due to the inability to predict where it was installed.

HURDLE: Binaries still referencing libs from Darwin Ports, instead of the libs we've bundled with the installer.

SOLUTION:

• Add NO_DARWIN_PORTS=1 to the top of the Makefile
• Put LDFLAGS="-L/usr/local/git/lib,/usr/lib" in the arguments for each of the make commands

HURDLE: Package installer size was whopping 40MB download, and 89 MB to install:

SOLUTION:

• Converted the 80+ hardlinks (git-branch, git-remote, etc.) to symlinks. The OS X PackageMaker just doesn't respect hardlinks and was treating them all as individual duplicates of a file.
• Stripped binaries of debug symbols after compiling via "make strip"

CONCLUSION

Now, the package has worked very well on machines that don't have the developer tools installed. The download size is 3.6 MB (instead of the original 40 MB), and it takes 9 MB to install, instead of 87 MB. That's pretty great progress in my eyes! And considering the fact neither Geoff or I have built a package for OS X before, it's a reasonably great accomplishment.

There are a few minor glitches still happening:

• On some machines, it complains about not being able to create "/". This error does not mean git wasn't installed successfully, but it may mean that you need to download this script and run it manually, yourself. (or, set PATH and MANPATH in your environment to include /usr/local/git/bin and /usr/local/git/man, respectively)
• Unless you launch Git GUI from the shell, it fails on fetch / push / clone over the network. You can use the "Open in Git Gui" droplet to help overcome this.

posted by Tim Harper  # 7:51 AM 7 Comments

Cobol on Cogs

This was my favorite one of all :) I got hours and hours of laughter with this one! My favorite was the excerpt under "Get Excited": Audio cassettes, screen-shots, and binders full of printouts! I remember the day when we used to get magazines full of printouts of computer programs so you could go and type them in on your own (note here Gentoo users: real programmers type in their own code for their operating system :P).

GitHub's SVNHub

What was especially funny about this one was that some people apparently thought it was serious!.

Google's Virgle

Virgle

Google had two pranks that I'm aware of this year. I love Google and appreciate their consistent playful nature around this time of year.

I didn't find out about virgle until after the fact, but was my favorite of their two. I got a great kick out of their quiz.

Google "Custom Time"

XCodeMate

I started this one, but it wouldn't have been fun without everyone who played along and had a good laugh with me. Essentially, XCodeMate installs a lot of EASILY REMOVABLE features into TextMate, all of which assign hot-keys in TextMate that open up Rick Astley's infamous video of Never gonna give you up. See, that's a feature? I'm never gonna give up on you code.

At first I had the bundle map enter, escape, tab, ctrl-option-command-b to Rick-Rollin commands, but I decided it was probably a bit too cruel and toned it down just a tad :)

Now back to seriousness!

posted by Tim Harper  # 10:20 PM 0 Comments

No longer is it necessary to sacrifice elegance and simplicity for the power of a robust-IDE

I bring you:

The XCodeMate bundle

More details on the project page. Enjoy!

Update...

April Fools!

Thank you everyone who played along and had fun with this. If you're discovering this bundle after the fact: it was a all a joke.

However, if you're in the mood for some harmless belated-april-fools humour, the bundle install instructions are still available and should yield you with just that :)

Thanks,

Tim

posted by Tim Harper  # 2:46 PM 1 Comments