/usr/local.com

UCS – VIF Hold Down

November 22nd, 2011 | by | sysadmin

Nov
22

Last night I had a maintenance that went fine, but had a little hiccup that I wanted to write about. The maintenance was to simply add another block of WWN Initiators to our configuration and everything ran smooth.

Now, the reason for adding this new block is simply that we were out of WWNs and needed to turn up 4 new UCS B-200 blades. Well, since I have the new WWNs, might as well use them right? This is where the error came up that raised a bunch of alarms for our networking team which in turn, gave me a bit of ass-ache.

The fault code that was raised was Fault Code:F0283 and had a message in this format.

[transport] VIF [chassisId] / [slotId] [switchId]-[id] down, reason: [stateQual][transport] VIF [chassisId] / [id] [switchId]-[id] down, reason: [stateQual]

We ran into the stateQual being VIF Hold Down.

While looking for this error, I found the Cisco UCS Faults and Error Messages Reference which basically explained that “Endpoint(switch/fabric interconnect) reports the connectivity state on virtual interface as one of: a.down, b.errored, c.unavailable.”

Since this was a brand spanking new install and no zoning was present, all things are pointing a bit to the ports not being fully initialized since nothing has loaded a driver or attempted to make them active. Once we have a good first boot, these errors should clear themselves up as our Firmware has the appropriate bootcode for the HBAs, its just a bit of a chicken and egg issue. To test this theory, I simply loaded the ESXi boot CD with the Cisco boot drivers on it and lo and behold, the errors went away.

The confusing part is that this “looks” like an ethernet error when really it should be an HBA error. Someone on the support forum had a similiar issue as well. Also, UCS throws a major link down alarm when the link was never really up to begin with as this is a new install. A warning indication and a clearer error code on where the error actually is would make this error a lot easier to debug.

So, it appears to be harmless in our use case, but wanted to make a note of it here and maybe help out some other bastard with the same issue.

Comments Closed

Time Estimation

July 7th, 2011 | by | sysadmin, tips & tricks

Jul
07

How do you come up with a good time estimate?

If you are a typical employee, there is a good chance that your job revolves not around day to day mundane tasks. There may be some of that, but there is a good chance that it revolves around completing projects. Yes, project oriented work is challenging and rewarding, but often done very poorly.

Not that your work is bad, its the management of the project that we suck at. And more to the point, its the time estimation that we’re the worst at doing.
This is one of the hardest thing a person has to do in their day to day jobs. If you are a contractor, you may have gotten pretty good at it.

Engineers, we suck at it.

I’ve been out of college now for 12 years. I’ve worked on a bunch of projects of all sizes. Even in college we did some time estimation practices and I can say with 100% certainty…we suck at time estimation.

If at first you don’t succeed, flying helicopters isn’t for you

So why do we keep going down the path of guessing what we think a project will take? Surely our ego is no longer getting in the way. We are very well aware that no matter what I tell you, it will probably take longer than that. Even when I’m bullshitting to myself that I think a project will take 1 week, I tell my manager it will take 2 and it ends up taking 3 or 4. Wow…I suck at this.

I don’t really have a good answer of why we keep doing this. The best that I can tell is, we need something on paper. Whether it has anything to do with reality, it helps the business move forward so they know what can get done in a reasonable amount of time, even if we miss it but a few weeks.

But what happens when those few weeks turn into a few months?

Solutions to the problem!

I don’t have all the answers, not by a long shot. But based on my experience, my work environment and knowing what my skills are, I can typically get a pretty good estimation of what my gut is tell me that a project will take. I think take that number and multiply by 3.

Yes, you heard that correctly, multiply by 3.

I have no idea why this works, but for some reason, it always seems to work out in my favor. Now I have a bit more luxury of if something derails what I thought would happen, I have some time to make up ground. If things go smoothly and I finish in the week that I thought it would take, now I look like a stellar employee. And we all want to look good at the office right?

Calling bullshit

I know that many of you are thinking, this is bullshit. It has no basis in reality and your manager knows you are making up numbers so why wouldn’t they just divide by 3 and bust your nuts. Well, they could. But I would say 75% of the time, my multiplication of 3 is actually what is the REAL time to take to get something done.

If someone wanted to bust me on it, I’d simply call out the bullshit of their business plan / sales forecast. After all, no one has a crystal ball.

So that’s my little tip to you on time estimation. Its far from perfect but has served me well over the years. Best of luck to you.

1 Comment »

The dark hours of sleep

February 24th, 2011 | by | in the news, sysadmin

Feb
24

I’ve learned over the years that there is a timespan of hours that if my sleep falls into that range, I’m going to be a complete waste the next day. That range, 4-6 hours. If I get more or less sleep than 4-6 hours, I’m good to go. But for some reason that range is the death of me.

Last night due to a server maintenance that went south, I got 5 hours.

Today is not a good day.

Comments Closed

Snowpacolypse

February 9th, 2011 | by | in the news, sysadmin

Feb
09

Living in the midwest teaches you a lot of different things. One of those lessons, Mother Nature always wins. She can be awesome with warm springs and falls that make the trees turn all sorts of wonderful colors. But she can also be pretty mean. A tornado can devastate mile wide tracks across the midwest. Floods can wipe out farms, homes and everything it reaches.

Winter in the midwest gives you the opportunity to experience freezing cold temps that would make a fish stick shiver. Add on the occasional deluge of snow (in the 1-2 ft range) and you really get the pleasure of knowing Mother Nature. I have a few wonderful words to say to her right now and I haven’t even gone out to shovel yet.

One advantage to my job is the ability to work remotely when needed. Last week, the office had a LOT of people taking advantage of the working remotely. Because of this company perk, we ran into a rather interesting issue. Mainly, our SSL VPN licensure. Our typical usage doesn’t require us to have a license for everyone to be connected at the same time since we have offices around the midwest. The chances of that many people being out of the office at the same time is usually pretty minimal. But today, with so many people working from home, we hit our limit. Luckily we have an old PPTP VPN concentrator still hooked up that just need a few tweaks in the firewall to get back up and running.

If we didn’t have this technology, our company would have had a lot of people absent today hurting productivity. Or worse yet, a lot of employees putting themselves at risk to make it in for their jobs. No matter how much you love your job, its never worth giving your life for.

So midwesterners, what lessons did your company learn today? Do you have a decent disaster recovery plan in the case of a blizzard? How about a flood? Fire? Tornado?

There are many things to think about when disaster hits. Planning now will save you time and money down the road.

Programming note: I work for LightEdge Solutions which provides many IT services. One service is disaster recovery. This blog post is in no way an advertisement for this service. It is simply talking about how we handled it and hopefully sparking other to think about what they would do when disaster strikes.

Comments Closed

Older Posts
Newer Posts