Johnny Zhang
This is a case I worked with one of the BCS customers on their vCenter issue.

Problem: vCenter starts, but soon after, it will crash on it's own. From the log we saw:

[2010-03-12 11:44:18.178 'App' 4500 error] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Check database connectivity before restarting. Error: Error[VdbODBCError] (-1) "ODBC error: (23000) - [Microsoft][SQL Native Client][SQL Server]Violation of PRIMARY KEY constraint 'PK_VPX_IP_ADDRESS'. Cannot insert duplicate key in object 'dbo.VPX_IP_ADDRESS'." is returned when executing SQL statement "INSERT INTO VPX_IP_ADDRESS (ENTITY_ID, DEVICE_ID, IP_ADDRESS) VALUES (?, ?, ?)"
[2010-03-12 11:44:18.178 'App' 4500 verbose] Backtrace:
[2010-03-12 11:44:18.178 'App' 4500 info] Forcing shutdown of VMware VirtualCenter now

So we know the cause of this problem was something try to insert into the table dbo.VPX_IP_ADDRESS. Note the thread ID is 4500, so we can trace back to the problem virtual machine or host. In our case, this was a VM

[2010-03-12 11:44:17.865 'App' 4500 verbose] [VpxdMoEventManager] Event[2567833]: SOME_VM on SOME_ESX_HOST in SOME_CLUSTER is powered on

Now we could:
  1. Shut down the vm
  2. Remove the host from vCenter
  3. Stop hostd/vpxa service(s) on that ESX server
  4. Start vCenter and confirm everything is working as designed
  5. Power down the VM, unregister it
  6. Add the host back in vCenter
  7. Check database to make sure the VM is not there
  8. Register the VM
Now, please don't ask me why one VM can bring down vCenter, it would be off topic :)
0 Responses

Post a Comment