This is a case I worked with one of the BCS customers on their vCenter issue.
Problem: vCenter starts, but soon after, it will crash on it's own. From the log we saw:
[2010-03-12 11:44:18.178 'App' 4500 error] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Check database connectivity before restarting. Error: Error[VdbODBCError] (-1) "ODBC error: (23000) - [Microsoft][SQL Native Client][SQL Server]Violation of PRIMARY KEY constraint 'PK_VPX_IP_ADDRESS'. Cannot insert duplicate key in object 'dbo.VPX_IP_ADDRESS'." is returned when executing SQL statement "INSERT INTO VPX_IP_ADDRESS (ENTITY_ID, DEVICE_ID, IP_ADDRESS) VALUES (?, ?, ?)"
[2010-03-12 11:44:18.178 'App' 4500 verbose] Backtrace:
[2010-03-12 11:44:18.178 'App' 4500 info] Forcing shutdown of VMware VirtualCenter now
So we know the cause of this problem was something try to insert into the table dbo.VPX_IP_ADDRESS. Note the thread ID is 4500, so we can trace back to the problem virtual machine or host. In our case, this was a VM
[2010-03-12 11:44:17.865 'App' 4500 verbose] [VpxdMoEventManager] Event[2567833]: SOME_VM on SOME_ESX_HOST in SOME_CLUSTER is powered on
Now we could:
Problem: vCenter starts, but soon after, it will crash on it's own. From the log we saw:
[2010-03-12 11:44:18.178 'App' 4500 error] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Check database connectivity before restarting. Error: Error[VdbODBCError] (-1) "ODBC error: (23000) - [Microsoft][SQL Native Client][SQL Server]Violation of PRIMARY KEY constraint 'PK_VPX_IP_ADDRESS'. Cannot insert duplicate key in object 'dbo.VPX_IP_ADDRESS'." is returned when executing SQL statement "INSERT INTO VPX_IP_ADDRESS (ENTITY_ID, DEVICE_ID, IP_ADDRESS) VALUES (?, ?, ?)"
[2010-03-12 11:44:18.178 'App' 4500 verbose] Backtrace:
[2010-03-12 11:44:18.178 'App' 4500 info] Forcing shutdown of VMware VirtualCenter now
So we know the cause of this problem was something try to insert into the table dbo.VPX_IP_ADDRESS. Note the thread ID is 4500, so we can trace back to the problem virtual machine or host. In our case, this was a VM
[2010-03-12 11:44:17.865 'App' 4500 verbose] [VpxdMoEventManager] Event[2567833]: SOME_VM on SOME_ESX_HOST in SOME_CLUSTER is powered on
Now we could:
- Shut down the vm
- Remove the host from vCenter
- Stop hostd/vpxa service(s) on that ESX server
- Start vCenter and confirm everything is working as designed
- Power down the VM, unregister it
- Add the host back in vCenter
- Check database to make sure the VM is not there
- Register the VM
Post a Comment