I’ll be working on getting a screen cast of the demo up in the next week or so. Too many other things distracting me at this moment.

I’m going to be doing a presentation in the coming days for the local linux user group (CIALUG) on Puppet from Puppet Labs.

Free pizza and some swag will be thrown around.

The event is 100% free and should hopefully get you acquainted with all the cool stuff you can automate. Signup for your free ticket!

Puppetlabs has stepped up to offer sponsorship of the event so besides my insightful talk, there will be pizza and t-shirts that will be handed out while supplies last or whatever game we come up with to give them away. I have a whole box of them so hopefully many of you are sporting the latest in puppet wear before the night is through.

So put it in your calendar now to join me for Pizza and Puppet on January 18th at 7PM in the LightEdge corporate headquarters.

When I was brought in, it wasn’t powering up and throwing an error concerning a lock. So here is what I tried.

1. Remove the VM from inventory and re-add it. Based on past support calls with VMware, this trick seems to be a tried and true ‘do this first’ method of debugging a VM. I’m not going to go into great detail on why this might work, but needless to say, I’m dealing with a file system lock and re-adding it really does nothing for me here. In theory, it might remove the lock, but unfortunately method failed.
2. Snapshot shuffle. This is where you take a snapshot of the VM and then attempt to do a Delete All in the snapshot manager. The VM was acting as though it had a snapshot on the system and wasn’t letting it go. By doing a delete all, VMware will look for other snapshots and remove them if they are in the chain, even if they are not showing up in the snapshot manager. This method also failed.
3. vmkfstools for the win. I’m a unix admin so getting in the shell doesn’t bother me. For others, you might be a bit afraid by this. But I ssh’d into the host, found my VM directory and did a check on each of the VMDK files using vmkfstools. Each came back clean so then I moved onto the breaking of the lock file also with vmkfstools. The command technically stated that it didn’t complete successfully but I found that it did indeed work and I was able to power on the VM. Mission accomplished! Here is a run down of the commands used in this method:

   
To check a VMDK:
vmkfstools -x check virtualMachine.vmdk

   To remove a lock:
   vmkfstools -B virtualMachine.vmdk

   To list all vmkfstools options:
vmkfstools -h
-or-
man vmkfstools


Hopefully you don’t run into this issue anytime soon, but if you do, try a few of these methods and see where you end up. If all else fails, build a new VM and restore from backup. Or…call VMware support, they’re really good at this sort of thing.

Now, the reason for adding this new block is simply that we were out of WWNs and needed to turn up 4 new UCS B-200 blades. Well, since I have the new WWNs, might as well use them right? This is where the error came up that raised a bunch of alarms for our networking team which in turn, gave me a bit of ass-ache.

The fault code that was raised was Fault Code:F0283 and had a message in this format.

[transport] VIF [chassisId] / [slotId] [switchId]-[id] down, reason: [stateQual][transport] VIF [chassisId] / [id] [switchId]-[id] down, reason: [stateQual]

We ran into the stateQual being VIF Hold Down.

While looking for this error, I found the Cisco UCS Faults and Error Messages Reference which basically explained that “Endpoint(switch/fabric interconnect) reports the connectivity state on virtual interface as one of: a.down, b.errored, c.unavailable.”

Since this was a brand spanking new install and no zoning was present, all things are pointing a bit to the ports not being fully initialized since nothing has loaded a driver or attempted to make them active. Once we have a good first boot, these errors should clear themselves up as our Firmware has the appropriate bootcode for the HBAs, its just a bit of a chicken and egg issue. To test this theory, I simply loaded the ESXi boot CD with the Cisco boot drivers on it and lo and behold, the errors went away.

The confusing part is that this “looks” like an ethernet error when really it should be an HBA error. Someone on the support forum had a similiar issue as well. Also, UCS throws a major link down alarm when the link was never really up to begin with as this is a new install. A warning indication and a clearer error code on where the error actually is would make this error a lot easier to debug.

So, it appears to be harmless in our use case, but wanted to make a note of it here and maybe help out some other bastard with the same issue.

If you are a typical employee, there is a good chance that your job revolves not around day to day mundane tasks. There may be some of that, but there is a good chance that it revolves around completing projects. Yes, project oriented work is challenging and rewarding, but often done very poorly.

Not that your work is bad, its the management of the project that we suck at. And more to the point, its the time estimation that we’re the worst at doing.
This is one of the hardest thing a person has to do in their day to day jobs. If you are a contractor, you may have gotten pretty good at it.

Engineers, we suck at it.

I’ve been out of college now for 12 years. I’ve worked on a bunch of projects of all sizes. Even in college we did some time estimation practices and I can say with 100% certainty…we suck at time estimation.

If at first you don’t succeed, flying helicopters isn’t for you

So why do we keep going down the path of guessing what we think a project will take? Surely our ego is no longer getting in the way. We are very well aware that no matter what I tell you, it will probably take longer than that. Even when I’m bullshitting to myself that I think a project will take 1 week, I tell my manager it will take 2 and it ends up taking 3 or 4. Wow…I suck at this.

I don’t really have a good answer of why we keep doing this. The best that I can tell is, we need something on paper. Whether it has anything to do with reality, it helps the business move forward so they know what can get done in a reasonable amount of time, even if we miss it but a few weeks.

But what happens when those few weeks turn into a few months?

Solutions to the problem!

I don’t have all the answers, not by a long shot. But based on my experience, my work environment and knowing what my skills are, I can typically get a pretty good estimation of what my gut is tell me that a project will take. I think take that number and multiply by 3.

Yes, you heard that correctly, multiply by 3.

I have no idea why this works, but for some reason, it always seems to work out in my favor. Now I have a bit more luxury of if something derails what I thought would happen, I have some time to make up ground. If things go smoothly and I finish in the week that I thought it would take, now I look like a stellar employee. And we all want to look good at the office right?

Calling bullshit

I know that many of you are thinking, this is bullshit. It has no basis in reality and your manager knows you are making up numbers so why wouldn’t they just divide by 3 and bust your nuts. Well, they could. But I would say 75% of the time, my multiplication of 3 is actually what is the REAL time to take to get something done.

If someone wanted to bust me on it, I’d simply call out the bullshit of their business plan / sales forecast. After all, no one has a crystal ball.

So that’s my little tip to you on time estimation. Its far from perfect but has served me well over the years. Best of luck to you.

Last night due to a server maintenance that went south, I got 5 hours.

Today is not a good day.

Winter in the midwest gives you the opportunity to experience freezing cold temps that would make a fish stick shiver. Add on the occasional deluge of snow (in the 1-2 ft range) and you really get the pleasure of knowing Mother Nature. I have a few wonderful words to say to her right now and I haven’t even gone out to shovel yet.

One advantage to my job is the ability to work remotely when needed. Last week, the office had a LOT of people taking advantage of the working remotely. Because of this company perk, we ran into a rather interesting issue. Mainly, our SSL VPN licensure. Our typical usage doesn’t require us to have a license for everyone to be connected at the same time since we have offices around the midwest. The chances of that many people being out of the office at the same time is usually pretty minimal. But today, with so many people working from home, we hit our limit. Luckily we have an old PPTP VPN concentrator still hooked up that just need a few tweaks in the firewall to get back up and running.

If we didn’t have this technology, our company would have had a lot of people absent today hurting productivity. Or worse yet, a lot of employees putting themselves at risk to make it in for their jobs. No matter how much you love your job, its never worth giving your life for.

So midwesterners, what lessons did your company learn today? Do you have a decent disaster recovery plan in the case of a blizzard? How about a flood? Fire? Tornado?

There are many things to think about when disaster hits. Planning now will save you time and money down the road.

Programming note: I work for LightEdge Solutions which provides many IT services. One service is disaster recovery. This blog post is in no way an advertisement for this service. It is simply talking about how we handled it and hopefully sparking other to think about what they would do when disaster strikes.

Interactive Cloud – Nicholas Weaver(@lynxbat) from Nicholas Weaver on Vimeo.

Gotta admit, I’m pretty impressed with this. Nice Hack!

Over the last week or so they have been having some major issues with their campfire product. And in true 37signals style, they have an explanation and apology all wrapped up in one great blog post.

What’s so great about it?
Well, first off, its honest. There’s no marketing bullshit. No fake apology. Its sincere and straight forward. They had several issues, they laid them out, and took full responsibility for stuff breaking.

You could feel the pain. They rely on this product and were pissed it was down. More to the point, they were pissed that they were letting their customers down. Many of which, like me, are fans. Its easier to win back fans. But there are others that are on the fringe that they could have lost and they come right out and state that they need to earn their trust back.

The structure of the apology. Not many people realize that there is an art to writing this sort of message. The Pointy Haired Bosses (PHBs) will read a few sentences to maybe a couple of paragraphs. The first section is for them. The second paragraph is for those that have to answer to the PHBs. Middle managers that made the recommendation that are going to get the heat about it. They can state, “Yes, there was a problem and we’re getting a credit” The 3rd section is for the admins out there. The guys that want to hear what happened and learn from others issues. We’re guys that may run into this ourselves and knowledge is power here.

Conclusion
For those reading this thinking that it doesn’t apply to them because they’re not a tech company I say to you, wake up. Just take point number 1 as an example. No matter what profession you are in, you’re going to screw up at some point in your life. And when it comes time to apologize, be honest. Don’t sugar coat it or bury it in a non apology. Take responsibility, be sincere and admit you fucked up. Your customers will respect that sort of apology.