How to find the VM's ip address from ESX host?
Johnny Zhang
Now, you are not suppose to find the VM's IP address from the host. But if you really want to. Here is the steps.
We talked about the use of "ft-stats" and "vmware-vimdump". so we can use those two command together to find the guest IP address that we are looking for

1. Let's find the VM's name by using "ft-stats -lv" (let's use BCS-VC40-1)
2. Now we can use "vmware-vimdump | less" and search for hostname (/hostname)
Labels: 3 comments | edit post
What ft-stats gives you even you don't have a ft VM?
Johnny Zhang
Not many people are running FT VMs. and even less people know about the ft-stats command. Now if you don't use a FT vm what this command can give you?

Answer: a fast report on the vmid and path to your VMs

"ft-stats -lv"
Now Why the state is set as "Not Configured"? Because none of the VM was configured as a FT VM
Labels: 1 comments | edit post
vmware-vimdump. You now know too much
Johnny Zhang
This tip is just for Scott. We had a conference meeting, Scott requested an update on my blog. I save the best for you!

We talked about collecting information from vCenter. What about ESX? the "vmware-vimdump" command will print out what vmkernel knows about the system it's running on, which is EVERYTHING!

Let's show some examples:

How do I know which path I'm using to connect to the storage?

How about I want to see the policy used?

What about the storage team ask me for the WWP?

Ok enough on storage, what about virtual switch?

You got the idea. Now you know too much!
Labels: , 0 comments | edit post
Fast way to collect vCenter information
Johnny Zhang
From time to time, you may want to know how many ESX servers in your vCenter, what are their build levels, how many VMs are registered, how many powered on, etc. There is a fast way to give you all those information. All you need to look is the vpxd-profiler.log on your vCenter server. This log is in the same directory as your vpxd.log file

for example on a Windows 2008 server the log is at
\Users\All Users\VMware\VMware VirtualCenter\Logs\

What you need look for is the "InventoryStats". You are going to see something like this
those logs update every 5 minutes, you can show your boss, you are on top of things.

Note: this log can be large enough to hung notepad or wordpad. I do suggest you to move to a Linux to read it.
Labels: 2 comments | edit post
How to setup sudo with active directory accounts in ESX 4.1
Johnny Zhang
Customer using Active directory integration function in 4.1. the user logged in is DOMAIN/user_name, and since the user is not in the sudoer file, they can not view logs

You can add AD groups into sudoer file
for example, you have a domain called "MYDOMAIN" and the group call "ESX Admins" (or MYDOMAIN\ESX Admins)
you add this under group %wheel:
%MYDOMAIN//ESX/ Admins ALL=(ALL) ALL
Note: if it did not prompt for password, you may need to restart the server, and everything will work from here)
Labels: 0 comments | edit post
Change virtaul hardware version from 7 back to 4
Johnny Zhang
Some times customers may want to change the virtual hardware version back from 7 to 4, there is a way to do this

the first is the VM's vmx file
the part you need to change is:

virtualHW.version = "7" you change this to
virtualHW.version = "4"

No need to change config.version = "8" since ESX 3.x already using this version

Next what you need to change is:
scsi0.virtualDev = "lsisas1068"

this is default for version 7 of the virtual hardware. Version 4 will not understand this virtual controller

You can change to
scsi0.virtualDev = "lsilogic"

The next step is change the .vmdk file of the VM (pointer file)
ddb.virtualHWVersion = "7"

you need to change this to
ddb.virtualHWVersion = "4"

and after that, you should able to power on the VM as hardware version 4

(please note, this is not supported by VMware)
Labels: 2 comments | edit post
vCenter and SQL 2008
Johnny Zhang
I was setting up the BCS team lab, and one of the steps is to install a new vCenter 4.0 U1 server with a SQL 2008 back end. During the installation, the installer complained about port 80 was in use. From the past experience, I quickly checked if IIS is running on the server. There was not.

A quick “netstat –a” shows port 80 is listening to any incoming request. To find what service was using the port you can use “netstat –bna” the result shows the service that using the port was “SQLserver.exe” Wow, since when SQL server starts to use port 80? I took a look at SQL server configuration manager. There a list of services. One that came into the spot light was “SQL server reporting service” I recalls on SQL 2005 when Microsoft introduces the reporting service, IIS was a required service. Since IIS is not running, how did reporting service start itself? Right after I turn off SQL reporting service I can move the installation forward. (So the reporting service was the one holding port 80). Did a quick research on SQL 2008’s reporting service, here is the reason:

“Reporting Services no longer depends on IIS to provide access to the SOAP endpoint. URLs no longer include Web sites in IIS. Reporting Services uses HTTP.SYS directly to listen for requests on a specific port that you define for report server URLs.”

In case you need to use SQL 2008 reporting service on the same server where vCenter installed, keep in mind this could prevent vCenter from starting even it already installed. Make sure you either change the port for the reporting service or your vCenter’s port to avoid any issues
Labels: , 1 comments | edit post
“A general system error occurred” when try to add a ESX 3.5 server into vCenter 4.0
Johnny Zhang
Since most of you are upgrading form VI 3 to vSphere, I'd like to share with you on this issue I worked with one of the BCS customers.

I received this call when customer failed to add their ESX 3.5 U4 host to their new vCenter 4.0 server. The error message they hit was “A general system error occurred” (Always nice to see a message does not give any direction). So for the first step, we tested viclient connect directly to ESX server and that worked for us. Normally, that means hostd is working (This is actually the key issue). So we focused on vpxa.
from the vpxa logs:

[2010-05-20 11:29:01.793 server_name error 'App'] [VpxaHalCnxHostagent] caught InvalidProperty::Exception - filter creati
on failed, temptorary versioning workaround
[2010-05-20 11:29:01.820 server_name verbose 'App'] [VpxVmomi] Invoke error: vpxapi.VpxaService.login session: 5238dc5c-bd8b-fcdd-f450-ab23a826a413 Throw: vmodl.fault.SystemError

So we tried many things and nothing worked... I then want to see what hostd report during that time

Here is the hostd log:
[2010-05-20 13:04:36.281 'App' 52411312 error] AdapterServer caught unexpected exception: vector::reserve
[2010-05-20 13:04:36.281 'App' 52411312 trivia] Completing task haTask vim.event.EventManager.createColle
ctor-389 as 3
[2010-05-20 13:04:36.281 'PropertyProvider' 52411312 trivia] RecordOp 2: info, haTask--vim.event.EventMana
ger.createCollector-389
[2010-05-20 13:04:36.281 'Vmomi' 52411312 info] Activation [N5Vmomi10ActivationE:0x9fd9780] : Invoke done
[createCollector] on [vim.event.EventManager:ha-eventmgr]
[2010-05-20 13:04:36.281 'Vmomi' 52411312 info] Throw vmodl.fault.SystemError
[2010-05-20 13:04:36.281 'Vmomi' 52411312 info] Result:
(vmodl.fault.SystemError) {
dynamicType = ,
reason = "",
msg = ""
}
[2010-05-20 13:04:36.325 'Proxysvc Req00153' 18353072 trivia] Client HTTP stream read error
[2010-05-20 13:04:36.325 'Proxysvc Req00153' 18353072 trivia] Closed

The error message "AdapterServer caught unexpected exception: vector::reserve" lead me to a problem report. However, the report had no vCenter involved. I asked customer to bring up the ESX server via viclient, click on any virtual machine, then click on "event" tab, sure enough, it hits “A general system error occurred” message, just like the problem report. The good news is patch 19 fixed this issue. We installed patch 19 ( ESX350-200912401-BG - PATCH ) and able to add those ESX server to the new vCenter 4.0.

If you already have patch 19 installed, you should not have this issue. If you don't, then put this on the side, it would be handy in case you need it.
Labels: 0 comments | edit post
Case study: Cannot insert duplicate key in object 'dbo.VPX_IP_ADDRESS'
Johnny Zhang
This is a case I worked with one of the BCS customers on their vCenter issue.

Problem: vCenter starts, but soon after, it will crash on it's own. From the log we saw:

[2010-03-12 11:44:18.178 'App' 4500 error] An unrecoverable problem has occurred, stopping the VMware VirtualCenter service. Check database connectivity before restarting. Error: Error[VdbODBCError] (-1) "ODBC error: (23000) - [Microsoft][SQL Native Client][SQL Server]Violation of PRIMARY KEY constraint 'PK_VPX_IP_ADDRESS'. Cannot insert duplicate key in object 'dbo.VPX_IP_ADDRESS'." is returned when executing SQL statement "INSERT INTO VPX_IP_ADDRESS (ENTITY_ID, DEVICE_ID, IP_ADDRESS) VALUES (?, ?, ?)"
[2010-03-12 11:44:18.178 'App' 4500 verbose] Backtrace:
[2010-03-12 11:44:18.178 'App' 4500 info] Forcing shutdown of VMware VirtualCenter now

So we know the cause of this problem was something try to insert into the table dbo.VPX_IP_ADDRESS. Note the thread ID is 4500, so we can trace back to the problem virtual machine or host. In our case, this was a VM

[2010-03-12 11:44:17.865 'App' 4500 verbose] [VpxdMoEventManager] Event[2567833]: SOME_VM on SOME_ESX_HOST in SOME_CLUSTER is powered on

Now we could:
  1. Shut down the vm
  2. Remove the host from vCenter
  3. Stop hostd/vpxa service(s) on that ESX server
  4. Start vCenter and confirm everything is working as designed
  5. Power down the VM, unregister it
  6. Add the host back in vCenter
  7. Check database to make sure the VM is not there
  8. Register the VM
Now, please don't ask me why one VM can bring down vCenter, it would be off topic :)
Labels: , 0 comments | edit post