Case study: Cannot insert duplicate key in object 'dbo.VPX_IP_ADDRESS'
Johnny Zhang
This is a case I worked with one of the BCS customers on their vCenter issue.

Problem: vCenter starts, but soon after, it will crash on it's own. From the log we saw:

[2010-03-12 11:44:18.178 'App' 4500 error] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Check database connectivity before restarting. Error: Error[VdbODBCError] (-1) "ODBC error: (23000) - [Microsoft][SQL Native Client][SQL Server]Violation of PRIMARY KEY constraint 'PK_VPX_IP_ADDRESS'. Cannot insert duplicate key in object 'dbo.VPX_IP_ADDRESS'." is returned when executing SQL statement "INSERT INTO VPX_IP_ADDRESS (ENTITY_ID, DEVICE_ID, IP_ADDRESS) VALUES (?, ?, ?)"
[2010-03-12 11:44:18.178 'App' 4500 verbose] Backtrace:
[2010-03-12 11:44:18.178 'App' 4500 info] Forcing shutdown of VMware VirtualCenter now

So we know the cause of this problem was something try to insert into the table dbo.VPX_IP_ADDRESS. Note the thread ID is 4500, so we can trace back to the problem virtual machine or host. In our case, this was a VM

[2010-03-12 11:44:17.865 'App' 4500 verbose] [VpxdMoEventManager] Event[2567833]: SOME_VM on SOME_ESX_HOST in SOME_CLUSTER is powered on

Now we could:
  1. Shut down the vm
  2. Remove the host from vCenter
  3. Stop hostd/vpxa service(s) on that ESX server
  4. Start vCenter and confirm everything is working as designed
  5. Power down the VM, unregister it
  6. Add the host back in vCenter
  7. Check database to make sure the VM is not there
  8. Register the VM
Now, please don't ask me why one VM can bring down vCenter, it would be off topic :)
Labels: , 0 comments | edit post
Try this before look for the backup
Johnny Zhang
I was working with one of the BCS customer on a case. The customer was originally planing to extend a Windows 2003 VM's OS disk to a larger size. However, after extend the VMDK files with VI client, and extend the NTFS inside Windows (Since this was a Windows 2003 guest, the customer needs to use a helper VM to do the job) after failed to power on the VM, the customer found a shocking fact: there was a snapshot. Then the customer tried to shrink the VMDK file back to the original size... and number of other things was also tried. So when we start to work on this issue, the VMDK file was broken in many different ways alone with the snapshot. To make the thing worse the last and only backup was created right after the extension of the VMDK file.

The end story was, we did recover all the data on the VMDK, the VMDK was not good enough to power on as a boot disk, but that was good enough to save the day. Here was how we did it:

First, list the size of the flat file by using "ls -l" in the VM directory
Keep in mind the number we need is the size of the flat file. In my test set up it's 8589934592. We need this number later. Now we need open the VMDK file (The pointer). In my case snap_test.vmdk:
Note, we need the CID and the RW number.
The RW number is the flat disk size divide by 512.
in my case 8589934592 / 512 = 16777216. If this number is wrong, you will not able to power on the VM. (So first we fix this part).

Now let's open the snapshot VMDK file. snap-test-000001.vmdk

In this file the "parentCID" should be the same as the snap-test.vmdk's CID. In my case 4cc5f033. If this is wrong, then the VM will not power on. (We fixed this one as well). The RW size would be also wrong, since it will only record the original size. Change this to the same size as the parent RW (16777216).
Now we need a helper VM to attach this disk as a second VMDK file (Not boot disk). When the OS boots up it will automatically run check disk against the VMDK and auto fix the errors. Now you should able to mount the disk inside the helper VM and copy over the data.
Labels: , 0 comments | edit post
What "memory usage" in vCenter 4.0 really shows
Johnny Zhang
In vCenter 4.0 we now have some easy to used monitor tools for ESX server performance. One of the chart is the memory usage
This usages is NOT the active memory that is currently used by the ESX server or the VM. So when you see very high usage, don't be panic.
In fact, this chart shows the total "Consumed Host Memory" by all the VMs running on the host.
As we can see from the graph the "Consumed Host memory" is 2277.00MB and the "Active Guest Memory" is only 122.00MB. Why a big different?

This is by design, the "Consumed Host Memory" shows the highest memory used by that VM.
OS such like Windows will touch all the memory assigned to it during start up, so most likely you will see the consumed memory is close or little more then the memory assigned to the VM (Overhead to power on the VM is also added here). ESX server will NOT take back the memory if there is no resource contention on the server. The reason is, once the VM need more memory it can access them faster than send the request to vmkernel again.
If you want to know the real time memory usage you can look into the advanced chart to find it out.
Labels: 0 comments | edit post