vmkfstool for the win

Got a call tonight concerning a VM that was having some issues. We had p2v’d our Virtual Center and when the new vCenter came online, apparently DRS decided to balance the heck out of things and somewhere in the process, a VM got squished and had a variety of issues.

When I was brought in, it wasn’t powering up and throwing an error concerning a lock. So here is what I tried.

  1. Remove the VM from inventory and re-add it. Based on past support calls with VMware, this trick seems to be a tried and true ‘do this first’ method of debugging a VM. I’m not going to go into great detail on why this might work, but needless to say, I’m dealing with a file system lock and re-adding it really does nothing for me here. In theory, it might remove the lock, but unfortunately method failed.
  2. Snapshot shuffle. This is where you take a snapshot of the VM and then attempt to do a Delete All in the snapshot manager. The VM was acting as though it had a snapshot on the system and wasn’t letting it go. By doing a delete all, VMware will look for other snapshots and remove them if they are in the chain, even if they are not showing up in the snapshot manager. This method also failed.
  3. vmkfstools for the win. I’m a unix admin so getting in the shell doesn’t bother me. For others, you might be a bit afraid by this. But I ssh’d into the host, found my VM directory and did a check on each of the VMDK files using vmkfstools. Each came back clean so then I moved onto the breaking of the lock file also with vmkfstools. The command technically stated that it didn’t complete successfully but I found that it did indeed work and I was able to power on the VM. Mission accomplished! Here is a run down of the commands used in this method:
    To check a VMDK:
    vmkfstools -x check virtualMachine.vmdk

    To remove a lock:
    vmkfstools -B virtualMachine.vmdk

    To list all vmkfstools options:
    vmkfstools -h
    man vmkfstools

Hopefully you don’t run into this issue anytime soon, but if you do, try a few of these methods and see where you end up. If all else fails, build a new VM and restore from backup. Or…call VMware support, they’re really good at this sort of thing.

Matt Patterson avatar
About Matt Patterson
Husband, Father of 3, Programmer at heart, spends his days running ridiculously large data centers in the midwest.