I’ve been pretty swamped at work and side things that I have been neglecting the old blog here for a bit. But I should be posting some new content here over the next few weeks of some stuff that I have been working on and a certification that I’m picking up.

Good stuff to come…check back soon.

Anyone have a theme that they like?

So, as a good father, I promise to raise my son to learn all the important things. Things such as:

• How to count in binary
• How to compile a new kernel
• To root for the Packers
• How to shoot a wicked wrister
• How to get to second base with mom (sounds wrong but highly appropriate at this point in his life)

These and many other important life lessons.

And whether you wanted them or not, here are some photos. Enjoy, because I know I am.

You can see all the pictures of Nolan on my Flickr page.

So as every proud new father will tell you, I need some sleep. Later all.

Since we’re all here on the same page, let’s continue shall with with the postmortem.

What Happened?

On Sunday morning, a little after 9:30 AM I noticed that I was no longer getting email from my server. Which was kind of a bummer as I had just gotten a cert setup for IMAP+SSL. I started poking around doing the usual pings and traceroutes. It was clear that I wasn’t having a connectivity issue to the internet, this was clearly an issue with my server. Web was down, smtp, ssh, everything. Ugh, not good. This server had issues about 6 months earlier where half of its ram died. I should have seen this as a sign to move everything off right then and there. But I didn’t.

Luckily for me, this server is hosted by my old company Solinus who make the kick ass MailFoundry Anti-Spam appliances. So I had a good buddy take a look at this box and he noticed a few red flags right from the beginning. First and foremost, it was off. Yikes! After he booted it up, it took a LONG time just to get to a bios screen that would then half way load a lilo screen before freaking out again and going back down. And this happened only once. After that, it simply wouldn’t post. My friends, we have a system board failure.

Now, this is a sandbox server for me. Its used for backups and hosting a few low volume blogs for myself and a few friends. Its also pretty old when it comes to servers and their lifetime. I bought this off of ebay back in fall of 2006. It was several years old when I got it. When the system board finally gave up its ghost, it was at least 6 years old if not 7 or 8.

Recovery

So what do you do when you first lose a server and know its a lost cause. Well you go to your backups of coarse. Have you tested your backups lately? Most people forget this all too critical step. Sure you know that your backups are running, but do you know if you could really recover from them?

Luckily, I knew I had a good backup as I had to recover a file from it last week that I accidentally removed. Whew!

Recovering the files wasn’t too bad. The only down side is that the files are backed up a server I have sitting here at home. I have a cable modem connection with great download speeds. Guess what it doesn’t have…great UPLOAD speeds. So moving around the couple GB of data has been a bit of a pain but once on the server, I was able to parse out the data pretty quickly.

Lessons learned

• Always test your backups I really can’t stress this one enough. Just because your backup ran, doesn’t mean that you have a good backup. Try putting that data on another server and build up a clone of your current box.
• Documentation As with our previous example, can you restore the data to another server. Can you configure anything that you missed without looking at the live server? If its a one of a kind server, you may not have all those permissions documented properly. Its key to have good documentation how things were installed and configured not just the data that you were hosting.
• Recovery Plan Do you know where you are going to restore/rebuild that server. Do you have spare hardware? a VM maybe? It helps to have those plans in place so you know exactly where you can shift loads and get things going again. Sure this was a dev box for me, but I still like to have an idea of where things are going to go so my downtime is minimal.
• Did I mention backups? yup, its that important so it goes on the list twice

Final Thoughts

Overall, my server disaster wasn’t as bad as it could have been. I lost a few hours recovering data. But in the grand scheme of things, I lost very little of the imporatnt data. The one thing I didn’t have backed up was my home directory. There wasn’t too much that was really lost there. A little bit of mail and a few random scripts. But nothing that I would consider the end of the world.

More posts coming soon…promise!

We have some pretty big plans for the site and this is only the beginning. Revision one of the site allows you access to a lot of information that we have been collecting over the past several months. You can create an account, create a wish list to purchase apps later and rank and leave comments on apps that you have given a test run. Because let’s face it, they’re not all winners.

There is much more that we’re going to do with the site so we hope that you use it, tell us what you think, and spread the word to others!

-Matt

http://iapp100.com

http://twitter.com/iapp100

So sign up now, you can start making your picks on Monday!

http://nhl.usrlocal.com/

And spread the word if you could, let’s get as many people on this thing as we can!

-Matt

Work has been challenging but stressful. The company was bought, we battled floods, my team helped launch several major products, moved offices, and have continued to add on additional projects that have not been made public knowledge, but have taken up a bunch of time.

Yes, I’m very happy to be employed and busy. But there have been a number of challenges that I could do without in 2009. Let’s hope for a fairly dry spring so we can avoid the major flooding the drenched the midwest. Its a small step, but would save me a lot of sleep.

I have my goals for 2009 and most of them will eventually come out in this blog. But for the time being, set your goals high, spend time with the ones that you love and let’s hope for a prosperous 2009 for everyone.

So here it is, in all its glory. Unfortunately, there will be a few RSS feeds that will be broken most likely. I’ll do what I can in order to send those on to the correct places in the next couple of days.

But in the mean time, please update your links.

-Matt

I’ll have pictures posted to the flickr account here in the next few days and will begin my semi-regular posting again here shortly.