The office move meant shutting down the VMware hosts and one of the side effects of this was that a couple of the virtual machines that had been happily running didn’t come back up.
One didn’t matter at all so I could safely ignore it, but the other was a demo server used by one of the company directors to show off our software to prospective clients and therefore needed to be working asap.
I was faced with quite a mystery as it seemed to have disappeared completely, it wasn’t listed in the inventory of the host in Vsphere and the datastores connected to the host didn’t contain the expected files either.
However I wasn’t completely at a loss as there was a message in the hosts inventory indicating that something had gone wrong and a virtual machine it had been hosting was missing and I knew that some of our VMs have been renamed and so the corresponding files in the datastores do not have matching names.
So I went looking for some orphan files. I found the files in question but it was just the VMDK files and they seemed to have been missing the VMDK metadata files also.
So in order to rebuild the virtual machine I needed to reconstruct the VMDK files so that I had usable virtual hard disk to the attach a new virtual machine to and given that we are using the free vSphere ESXi 4.0 version this meant using the unsupported Tech Support Mode and instructions from here http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002511
To use Tech Support Mode:
Log in to your ESXi host at the console.
Press Alt+F1 to switch to the console window.
Enter unsupported to start the Tech Support Mode login process. Note that no text will appear on the console window.
Enter the password for the root user. Tech Support Mode is now active.
Complete tasks in Tech Support Mode.
Enter the command clear to clear the screen of any residual data from step 5. This may be required by your local security policies.
Enter the command exit to exit Tech Support Mode.
Press Alt+F2 to return the server to DCUI mode.
So I logged into the terminal of the ESXi host.
Then to recreate the virtual machine disks I navigated to the directory that contained the virtual machine disks with the missing descriptor file using the following command (having previously found the relevant volume from browsing Storage in the vSphere client on my PC:
cd "/vmfs/volumes/4bfd0ee1-48e6535e-7d30-0026b97ee7d2/CRJ test/"
The instructions then asked me to identify the type of SCSI controller the virtual disk is using by examining the virtual machine configuration file (.vmx). But I didn’t have the .vmx file so I took a look at similar VMs and they used the SCSI controller type lsilogic.
I identified and recorded the exact size of the -flat file using the command:
# ls -l VS030-Srv08Tmpl-flat.vmdk
-rw——- 1 root root 32212254720 May 29 12:30 VS030-Srv08Tmpl-flat.vmdk
Then used the vmkfstools command to create a new virtual disk:
# vmkfstools -c 32212254720 -a lsilogic -d thin temp.vmdk
This command uses these flags:
-d thin (This creates the disk in a thin-provisioned format).
Note: To save disk space, the disk was created in a thin-provisioned format using the type thin. The resulting flat file then consumes minimal amounts of space (1MB) instead of immediately assuming the capacity specified with the -c switch. The only consequence, however, is the descriptor file contains an extra line that must be removed manually in a later step.
The files temp.vmdk and temp-flat.vmdk were created as a result.
I deleted the unneeded temp-flat.vmdk using the command:
# rm temp-flat.vmdk
And renamed temp.vmdk to the name that match the orphaned .flat file (or VS030-Srv08Tmpl-flat.vmdk, in my case):
# mv temp.vmdk VS030-Srv08Tmpl-flat.vmdk
Then the descriptor file needed to be edited to match the .flat file:
Under the Extent Description section, change the name of the .flat file to match the orphaned .flat file you have.
Find and remove the line ddb.thinProvisioned = “1″ if the original .vmdk was not a thin disk. If it was, retain this line.
This completed the reconstruction of the first virtual hard disk, I did the same for the second disk and then I was in a position to rebuild the machine.
Building the VM was straightforward and not worth writing in any great detail as it is simply a case of using the wizard in the vSphere client and then selecting “Use Existing” option instead of “Create New” when it gets to the section about virtual hard disks.