SysAdmin

Interactive Cloud – Nicholas Weaver(@lynxbat) from Nicholas Weaver on Vimeo.

Gotta admit, I’m pretty impressed with this. Nice Hack!

Those that know me know I’m a big fan of 37Signals. Heck, I even applied for a job there once upon a time along with several hundred other people.

Over the last week or so they have been having some major issues with their campfire product. And in true 37signals style, they have an explanation and apology all wrapped up in one great blog post.

What’s so great about it?

Well, first off, its honest. There’s no marketing bullshit. No fake apology. Its sincere and straight forward. They had several issues, they laid them out, and took full responsibility for stuff breaking.

Some of your may have noticed that this site was down for about a day. Oh who am I trying to kid. 3 people noticed if that and most of you are probably reading this going, “oh, you had issues. I guess so, you’re writing about them.”

Since we’re all here on the same page, let’s continue shall with with the postmortem.

What Happened?

On Sunday morning, a little after 9:30 AM I noticed that I was no longer getting email from my server. Which was kind of a bummer as I had just gotten a cert setup for IMAP+SSL. I started poking around doing the usual pings and traceroutes. It was clear that I wasn’t having a connectivity issue to the internet, this was clearly an issue with my server. Web was down, smtp, ssh, everything. Ugh, not good. This server had issues about 6 months earlier where half of its ram died. I should have seen this as a sign to move everything off right then and there. But I didn’t.

Do you work in an environment where you bounce through a bunch of firewalls? Do you hang out on idle ssh connections that often times get dropped after a certain amount of idle time? I do and it has always annoyed me. To the point that once I connect to a box that I will be coming back to, I will run top and move on. Well, not anymore. You can set your SSH client to automatically send a bit of data over your connection every X seconds. Here is how it is done for Mac and Linux boxes.

Recently I had a chance to do a little monit foo with a co-worker for a rather interesting project that we will hopefully be sending off into the intertubes.

For one part of this project, I got the chance to get my hands dirty with my old friend monit. Monit, for those that don’t know, is a UNIX system administrators dream.

Here’s a brief run down of what monit can do from the web site: