vmkfstool for the win

Got a call tonight concerning a VM that was having some issues. We had p2v’d our Virtual Center and when the new vCenter came online, apparently DRS decided to balance the heck out of things and somewhere in the process, a VM got squished and had a variety of issues.

When I was brought in, it wasn’t powering up and throwing an error concerning a lock. So here is what I tried.

Remove the VM from inventory and re-add it. Based on past support calls with VMware, this trick seems to be a tried and true ‘do this first’ method of debugging a VM. I’m not going to go into great detail on why this might work, but needless to say, I’m dealing with a file system lock and re-adding it really does nothing for me here. In theory, it might remove the lock, but unfortunately method failed.
Snapshot shuffle. This is where you take a snapshot of the VM and then attempt to do a Delete All in the snapshot manager. The VM was acting as though it had a snapshot on the system and wasn’t letting it go. By doing a delete all, VMware will look for other snapshots and remove them if they are in the chain, even if they are not showing up in the snapshot manager. This method also failed.
vmkfstools for the win. I’m a unix admin so getting in the shell doesn’t bother me. For others, you might be a bit afraid by this. But I ssh’d into the host, found my VM directory and did a check on each of the VMDK files using vmkfstools. Each came back clean so then I moved onto the breaking of the lock file also with vmkfstools. The command technically stated that it didn’t complete successfully but I found that it did indeed work and I was able to power on the VM. Mission accomplished! Here is a run down of the commands used in this method:

To check a VMDK:
vmkfstools -x check virtualMachine.vmdk

To remove a lock:
vmkfstools -B virtualMachine.vmdk

To list all vmkfstools options:
vmkfstools -h
-or-
man vmkfstools

</div>

Hopefully you don’t run into this issue anytime soon, but if you do, try a few of these methods and see where you end up. If all else fails, build a new VM and restore from backup. Or…call VMware support, they’re really good at this sort of thing.

<div class="sd-content">
  <ul>
    <li class="share-facebook">
      <a rel="nofollow" data-shared="sharing-facebook-1633" class="share-facebook sd-button share-icon no-text" href="http://usrlocal.com/2011/12/vmkfstool-for-the-win/?share=facebook" target="_blank" title="Click to share on Facebook"><span></span><span class="sharing-screen-reader-text">Click to share on Facebook (Opens in new window)</span></a>
    </li>
    <li class="share-linkedin">
      <a rel="nofollow" data-shared="sharing-linkedin-1633" class="share-linkedin sd-button share-icon no-text" href="http://usrlocal.com/2011/12/vmkfstool-for-the-win/?share=linkedin" target="_blank" title="Click to share on LinkedIn"><span></span><span class="sharing-screen-reader-text">Click to share on LinkedIn (Opens in new window)</span></a>
    </li>
    <li class="share-google-plus-1">
      <a rel="nofollow" data-shared="sharing-google-1633" class="share-google-plus-1 sd-button share-icon no-text" href="http://usrlocal.com/2011/12/vmkfstool-for-the-win/?share=google-plus-1" target="_blank" title="Click to share on Google+"><span></span><span class="sharing-screen-reader-text">Click to share on Google+ (Opens in new window)</span></a>
    </li>
    <li class="share-pinterest">
      <a rel="nofollow" data-shared="sharing-pinterest-1633" class="share-pinterest sd-button share-icon no-text" href="http://usrlocal.com/2011/12/vmkfstool-for-the-win/?share=pinterest" target="_blank" title="Click to share on Pinterest"><span></span><span class="sharing-screen-reader-text">Click to share on Pinterest (Opens in new window)</span></a>
    </li>
    <li class="share-twitter">
      <a rel="nofollow" data-shared="sharing-twitter-1633" class="share-twitter sd-button share-icon no-text" href="http://usrlocal.com/2011/12/vmkfstool-for-the-win/?share=twitter" target="_blank" title="Click to share on Twitter"><span></span><span class="sharing-screen-reader-text">Click to share on Twitter (Opens in new window)</span></a>
    </li>
    <li class="share-email">
      <a rel="nofollow" data-shared="" class="share-email sd-button share-icon no-text" href="http://usrlocal.com/2011/12/vmkfstool-for-the-win/?share=email" target="_blank" title="Click to email this to a friend"><span></span><span class="sharing-screen-reader-text">Click to email this to a friend (Opens in new window)</span></a>
    </li>
    <li class="share-end">
    </li>
  </ul>
</div>

Share this: