Did someone hack our website? It looks like a Chinese news listing entry has been added with today’s date.

Ballsack!

Time to roll up the sleeves. How bad is the damage?

From the looks of it, other sites on the web server had not been damaged. Doing a search for modified files found nothing out of the ordinary had been changed. Log files sure but nothing out of the ordinary. A search through the web logs of the site showed that this appeared to be an attack on the Content Management System (CMS) for the site.

Whichever jackass wrote that code should be beaten! Get the torches and pitchforks, death to the programmer!! Oh wait, that was me. Death to the evil script kiddie that attacked my beautiful code!!!!!!!!!

Searching through the logs, I started seeing some interesting logs. A bunch of them that had variations such as this:

This one is actually after a lot of hits from the attacker of figuring out the tables and then getting down to the nitty gritty of pulling out a username and password.

The issue was that I had forgotten to sanitize my data. Like a jackass, I didn’t check that $id was an actual integer variable and when they ran thier script they were able to pull out a hex string that, with the use of such tools as this site, you can easily translate this into text.

The Quick Fix

I put in 2 fixes to ensure that we were dealing with an integer value here. First, I did some simple math to the variable that changes its type if it is not an integer.

If you have a string, you’ll get back a zero. If you have an integer value, you’re good to go.

Also in the code, I expanded the if statement that was around the code to grab the specific news item. Instead of just checking to see if the $id variable was set, I now check to see if it is set and greater than zero, another layer in ensuring that we have a number instead of a string of text.

We’re all human and humans make mistakes. This code that was exploited was written 8 years ago at a time when I should have known better, but missed it. It lived in the wild up until last year when it was finally exploited. I’m lucky that it took that long for it to expose itself, but kind of embarrassed that it was there in the first place. While I tend to come down hard on people for not doing these sort of things, its only because I’ve learned my lesson the hard way and have seen people continually mess this stuff up. While this happened to me a while ago, I’ve had friends get bit by this very recently so I figured it was time to finish off this post and get it out the door. An ounce of prevention sort of thing.

To lear more about SQL Inject attacks, here is a good article by Bhanu Mahesh on Quality Software Connection on how to prevent them.

I’ll be working on getting a screen cast of the demo up in the next week or so. Too many other things distracting me at this moment.

I’m going to be doing a presentation in the coming days for the local linux user group (CIALUG) on Puppet from Puppet Labs.

Free pizza and some swag will be thrown around.

The event is 100% free and should hopefully get you acquainted with all the cool stuff you can automate. Signup for your free ticket!

Puppetlabs has stepped up to offer sponsorship of the event so besides my insightful talk, there will be pizza and t-shirts that will be handed out while supplies last or whatever game we come up with to give them away. I have a whole box of them so hopefully many of you are sporting the latest in puppet wear before the night is through.

So put it in your calendar now to join me for Pizza and Puppet on January 18th at 7PM in the LightEdge corporate headquarters.

When I was brought in, it wasn’t powering up and throwing an error concerning a lock. So here is what I tried.

1. Remove the VM from inventory and re-add it. Based on past support calls with VMware, this trick seems to be a tried and true ‘do this first’ method of debugging a VM. I’m not going to go into great detail on why this might work, but needless to say, I’m dealing with a file system lock and re-adding it really does nothing for me here. In theory, it might remove the lock, but unfortunately method failed.
2. Snapshot shuffle. This is where you take a snapshot of the VM and then attempt to do a Delete All in the snapshot manager. The VM was acting as though it had a snapshot on the system and wasn’t letting it go. By doing a delete all, VMware will look for other snapshots and remove them if they are in the chain, even if they are not showing up in the snapshot manager. This method also failed.
3. vmkfstools for the win. I’m a unix admin so getting in the shell doesn’t bother me. For others, you might be a bit afraid by this. But I ssh’d into the host, found my VM directory and did a check on each of the VMDK files using vmkfstools. Each came back clean so then I moved onto the breaking of the lock file also with vmkfstools. The command technically stated that it didn’t complete successfully but I found that it did indeed work and I was able to power on the VM. Mission accomplished! Here is a run down of the commands used in this method:
   To check a VMDK:
   vmkfstools -x check virtualMachine.vmdk
   
   To remove a lock:
   vmkfstools -B virtualMachine.vmdk
   
   To list all vmkfstools options:
   vmkfstools -h
   -or-
   man vmkfstools

Hopefully you don’t run into this issue anytime soon, but if you do, try a few of these methods and see where you end up. If all else fails, build a new VM and restore from backup. Or…call VMware support, they’re really good at this sort of thing.

Now, the reason for adding this new block is simply that we were out of WWNs and needed to turn up 4 new UCS B-200 blades. Well, since I have the new WWNs, might as well use them right? This is where the error came up that raised a bunch of alarms for our networking team which in turn, gave me a bit of ass-ache.

The fault code that was raised was Fault Code:F0283 and had a message in this format.

[transport] VIF [chassisId] / [slotId] [switchId]-[id] down, reason: [stateQual][transport] VIF [chassisId] / [id] [switchId]-[id] down, reason: [stateQual]

We ran into the stateQual being VIF Hold Down.

While looking for this error, I found the Cisco UCS Faults and Error Messages Reference which basically explained that “Endpoint(switch/fabric interconnect) reports the connectivity state on virtual interface as one of: a.down, b.errored, c.unavailable.”

Since this was a brand spanking new install and no zoning was present, all things are pointing a bit to the ports not being fully initialized since nothing has loaded a driver or attempted to make them active. Once we have a good first boot, these errors should clear themselves up as our Firmware has the appropriate bootcode for the HBAs, its just a bit of a chicken and egg issue. To test this theory, I simply loaded the ESXi boot CD with the Cisco boot drivers on it and lo and behold, the errors went away.

The confusing part is that this “looks” like an ethernet error when really it should be an HBA error. Someone on the support forum had a similiar issue as well. Also, UCS throws a major link down alarm when the link was never really up to begin with as this is a new install. A warning indication and a clearer error code on where the error actually is would make this error a lot easier to debug.

So, it appears to be harmless in our use case, but wanted to make a note of it here and maybe help out some other bastard with the same issue.

If you are a typical employee, there is a good chance that your job revolves not around day to day mundane tasks. There may be some of that, but there is a good chance that it revolves around completing projects. Yes, project oriented work is challenging and rewarding, but often done very poorly.

Not that your work is bad, its the management of the project that we suck at. And more to the point, its the time estimation that we’re the worst at doing.
This is one of the hardest thing a person has to do in their day to day jobs. If you are a contractor, you may have gotten pretty good at it.

Engineers, we suck at it.

I’ve been out of college now for 12 years. I’ve worked on a bunch of projects of all sizes. Even in college we did some time estimation practices and I can say with 100% certainty…we suck at time estimation.

If at first you don’t succeed, flying helicopters isn’t for you

So why do we keep going down the path of guessing what we think a project will take? Surely our ego is no longer getting in the way. We are very well aware that no matter what I tell you, it will probably take longer than that. Even when I’m bullshitting to myself that I think a project will take 1 week, I tell my manager it will take 2 and it ends up taking 3 or 4. Wow…I suck at this.

I don’t really have a good answer of why we keep doing this. The best that I can tell is, we need something on paper. Whether it has anything to do with reality, it helps the business move forward so they know what can get done in a reasonable amount of time, even if we miss it but a few weeks.

But what happens when those few weeks turn into a few months?

Solutions to the problem!

I don’t have all the answers, not by a long shot. But based on my experience, my work environment and knowing what my skills are, I can typically get a pretty good estimation of what my gut is tell me that a project will take. I think take that number and multiply by 3.

Yes, you heard that correctly, multiply by 3.

I have no idea why this works, but for some reason, it always seems to work out in my favor. Now I have a bit more luxury of if something derails what I thought would happen, I have some time to make up ground. If things go smoothly and I finish in the week that I thought it would take, now I look like a stellar employee. And we all want to look good at the office right?

Calling bullshit

I know that many of you are thinking, this is bullshit. It has no basis in reality and your manager knows you are making up numbers so why wouldn’t they just divide by 3 and bust your nuts. Well, they could. But I would say 75% of the time, my multiplication of 3 is actually what is the REAL time to take to get something done.

If someone wanted to bust me on it, I’d simply call out the bullshit of their business plan / sales forecast. After all, no one has a crystal ball.

So that’s my little tip to you on time estimation. Its far from perfect but has served me well over the years. Best of luck to you.

Last night due to a server maintenance that went south, I got 5 hours.

Today is not a good day.

Winter in the midwest gives you the opportunity to experience freezing cold temps that would make a fish stick shiver. Add on the occasional deluge of snow (in the 1-2 ft range) and you really get the pleasure of knowing Mother Nature. I have a few wonderful words to say to her right now and I haven’t even gone out to shovel yet.

One advantage to my job is the ability to work remotely when needed. Last week, the office had a LOT of people taking advantage of the working remotely. Because of this company perk, we ran into a rather interesting issue. Mainly, our SSL VPN licensure. Our typical usage doesn’t require us to have a license for everyone to be connected at the same time since we have offices around the midwest. The chances of that many people being out of the office at the same time is usually pretty minimal. But today, with so many people working from home, we hit our limit. Luckily we have an old PPTP VPN concentrator still hooked up that just need a few tweaks in the firewall to get back up and running.

If we didn’t have this technology, our company would have had a lot of people absent today hurting productivity. Or worse yet, a lot of employees putting themselves at risk to make it in for their jobs. No matter how much you love your job, its never worth giving your life for.

So midwesterners, what lessons did your company learn today? Do you have a decent disaster recovery plan in the case of a blizzard? How about a flood? Fire? Tornado?

There are many things to think about when disaster hits. Planning now will save you time and money down the road.

Programming note: I work for LightEdge Solutions which provides many IT services. One service is disaster recovery. This blog post is in no way an advertisement for this service. It is simply talking about how we handled it and hopefully sparking other to think about what they would do when disaster strikes.

Interactive Cloud – Nicholas Weaver(@lynxbat) from Nicholas Weaver on Vimeo.

Gotta admit, I’m pretty impressed with this. Nice Hack!