Wrestling with VMware High Availability (HA)

A few months back I had a little bit of trouble with an upgrade in our corporate VMware cluster that I thought I would share. The details of this upgrade was to add a new host to the mix and bring everything up to vSphere 4.1 update 1. It seemed pretty straight forward at the time but there were a few unexpected issues that sucked up more time than expected.

Now, we have several clusters here in our company and often times we move a host from one cluster to the next. Several of the clusters were originally setup to work within the various active directory domains. This has been rather annoying when moving a host from one domain to another and having to do all the DNS update foo. Its much easier to have one private domain to rule them all so most of our hosts have been updated and moved to this new private domain that can be resolved by all domains.

This is where the fun actually comes in. The cluster that I’m moving a new host into is a hold out in the old naming space. Adding a new node shouldn’t be a big deal as the new name resolves in the virtual center.

Now, here comes the rub.

When upgrading a particular host, for some reason I could not enable the HA configuration. It just wouldn’t work. No particular reason other than it just failed to find the primary node. Now, you would think that this was failing on the new host that was added to the cluster. Nope, that added just fine, no worries there what so ever. The node that was failing was actually the 3rd host in the group to be upgraded.

Apparently what was happening actually had nothing to do with the primary node. It happened to deal with the new node with the name in the private domain. The issue, somewhere deep in the bowels of vSphere, it attempts to look up by the “short name” meaning esx123456 instead of esx123456.domain.com. While I was able to resolve esx123456.private.domain.com, I was unable to resolve esx123456 as the rest of the cluster was still looking for esx123456.domain.com which didn’t exist.

So my advice to you is, when changing the domains of hosts in a cluster, make sure you have all entries in both the new and old domains so you can avoid this short name lookup failure.

Is there really that much money in it?

Disclaimer: I’m not a Comcast subscriber, but I play one on TV

Comcast has me scratching my head. A friend of mine pointed out the following post on the Comcast goofiness. For a long time now they have been messing with DNS and if you happen to screw up and look up a site that does not exist in DNS, you get the Comcast ad page. Many of the tech savvy folks out there simply got around that by putting up their own caching server or using opendns. I know I did when I found out that Mediacom started messing with DNS like this.

Apparently they have upped their game by routing all DNS traffic, no matter what, to their servers. You have no way around this, you WILL use their servers.

I disagree with the first practice of just having a * domain that has everything mis-spelled go to a certain site of yours. Now that they have taken this to a new level, I think it is quickly climbing up the all time ranks of dangerous and stupid.

Really, dangerous? Well, say I’m a person that has highly questionable morals and decides that the best way to attack a competative ISP is to go after their DNS servers. I could try and do some sort of DDoS. Or, the better, more devious approach is to attempt a DNS cache poisoning on their servers. Not always the easiest, but when done properly can have some pretty devastating effects. Now, I trust that Comcast has employed some top notch admins over there so I highly doubt that they are going to let their guard down here, but we’re all humans. We still make mistakes.

Why is this stupid? Honestly, do you click on the ads on one of these pages? Or do you swear to yourself, type it in correctly or load up google and search for what you really want? It just seems like a lot of hassle to implement and all they are really doing is pissing off their customers. They’re trying to milk every last cent out of them and the customers are not stupid. They know what comcast is trying to do. They’re pissed off and eventually they will leave.

The sad part is, other ISPs have already taken up the first goofy solution that Comcast put in place. Its only a matter of time until more people adopt this new tactic. For me personally, I’m going to smack the crap out of the first person that mentions this as a solution that we should deploy. I set up our DNS servers and I refuse to break the internet. I also have a higher respect for my customers. They’re all intelligent, reasonable, and good looking right?

Setting up a silent DNS master server

Recently, I have begun the process of moving my domains from the DNS servers at my previous employer. They have allowed me to continue hosting my domains there as I still send them a spam feed of unknown addresses from my various domains. Yes spammers, keep that mail coming. Its only doing you good, I promise 😉

The main reason for the change is that the former employer is locking down the admin access to a standard that as a non-employee, I can no longer get to the admin interface. That’s fine, very understandable. So its time to make a change and I decided to go the route of a silent master.

DNS Overview

If you don’t know what DNS is, then I highly recommend reading the wikipedia aricle. The short definition of what DNS is supposed to do is that for every name that is out there on the internet, there is a corresponding IP address. As an example, if you attempt to look up one of my domains usrlocal.com, you will find the IP address of 66.18.16.207.

What is a Silent Master server?

The way that our DNS system works is that for every domain out there, there are a set of servers that will answer for that domain authoritatively. If you have registered your domain at some of the big regisrars, they will often throw in DNS hosting for free or a very nominal fee. Depending on who your ISP is, they will often allow you to host your domains for pretty cheap as well. The idea of a silent master is that there is a server that you control that is _not_ listed in the authoritative DNS servers for your domain. What happens is that you set the servers that are going to answer authoritatively for your domain to load your domain as a secondary zone. The theory is that you can have a lower powered server that is protected from the outside world doing the updates on your domain while the rest of the world hits your authoritative DNS servers.

Why would you do this?

There are really two reasons why you would want something like this:

  • Control over your domain – It could be that your DNS provider doesn’t have a very nice interface for controlling your domain or doesn’t allow you to control it directly. It may be that you have to open a support ticket for each update.
  • Save on internet traffic – Basically, I don’t want all of that traffic hitting the lowly DNS server that I have. I would rather them hit the DNS servers at my ISP and pound on those bad boys. They can take it, they’re beefy for a reason.

Technical Details

So here is how you setup your DNS zone to work appropriately. I’m going to use BIND in my example as that is one of the more popular DNS servers out there.

I have setup my server with the following configuration options. First I setup an ACL list of the servers that are allowed to pull information from my server. My example looks something like this:

acl le_dns {
        11.22.33.123; // ns1.isphost.com
        22.11.33.124; // ns2.isphost.com
        33.11.22.125; // ns3.isphost.com
};

Now, I allow them to connect to the server and pull domains.

options {
        directory "/var/named";
        allow-query { le_dns; };
        allow_transfer { "le_dns"; };
        recursion no;
};

Then each zone file is treated as you normally would if you were serving it up authoritatively.

zone "usrlocal.com" IN
{
        type master;
        file "usrlocal.com";
        allow-update { none; };
};

Next, tell your authoritative DNS server to grab the zones as secondary zones. Depending on where you are hosting things, it may be a configuration that you need to put in place, something along the lines of this:

zone "usrlocal.com" IN
{
        type slave;
        file "usrlocal.com";
        masters {11.11.11.22; 11.22.33.11};
};

Finally, update your registrar to point to your DNS servers that you have setup that are serving up the secondary zone. It may take 24-72 hours for the new information to propagate, but after that you should have the full control, and bandwidth savings of a silent DNS master server.

Damn French!

For the most part, I like France. Its a nice place to visit, most of the people that I’ve met have been fairly nice, at least the ones that have come over here to study and visit. Heck, I even got engaged over there back in 1999.

However, today re-affirmed why most people hate the french. We have a few domains that are registered with a registrar that is based out of Paris France. We bought the company that used this registrar so this is some of the cleanup that just hasn’t happened over the years.

Today apparently they decided that one of our primary domains hadn’t had its whois information updated so it must be out of date which would put us in violation of the terms of service. Technically, the whois information was still 100% correct. The contacts were are still valid contacts handled by our NOC staff. The contact names, phone numbers and email addresses all still worked as you would expect them to. But since we haven’t made any updates, clearly it must be out of date so they removed the domain from the root servers. No notifications that this was being done, no notice that we needed to update our records, nothing! Another joy of dealing with this registrar, they only do email support. That’s right, no phone support is available. Awesome!

Needless to say, we’re filing complaints with ICANN and will hopefully have this issue resolved shortly. Once it is cleared up, you can bet your ass we’re dumping them. Or, as many of our staff have suggested, threatening to invade their country and expecting their unconditional surrender. 😉