CPU Ready time & VM performance with multiple vCPU’s

Wow, first blog article for quite some time! Time has been stretched over the last year or so with family and new work commitments, so something had to slip! So, hopefully this is the start of me finding more time to blog! I’ve been working on plenty of scripts and other bits and pieces that’ll make some good articles, so fingers crossed they’ll be blogged soon!

I’ve been delving more and more into the world of performance monitoring with relation to VMware vSphere, and CPU Ready times has always been a topic of heated conversation at work… people over commit CPU resource as if it’s free, but don’t realise the consequences.

To prove a point I’ve made an example of an Exchange server. It runs for a business of about 20 users, running Exchange 2010. They also use Lync, and SharePoint, so there’s some integration going on too. It’s a fairly busy machine, and was configured with 4 virtual CPU’s, and a load of RAM (12GB). I’d argued the configuration of machines like this for some time, trying to explain that more CPU’s may mean fewer CPU time for the VM, but it was falling on deaf ears, so, I decided it was time to make a change, and prove a point ūüôā

Now, for a very simple overview…

In case you don’t know how CPU scheduling works, regardless of the number of vCPU’s granted, or their workloads, ALL vCPU’s must be scheduled to run on pCPU’s at the same time, even if the vCPU would be idle. So, if you have 4 pCPU’s, and 3 VM’s with a single pCPU, all is OK, each virtual machine can always get CPU resource, as there will always be 3 CPU’s available. Add in a virtual machine with 2 vCPU’s, and immediately you’d need 5 pCPU’s for all machines to always get pCPU time. Luckily, the VMware scheduler will deal with this and queue pCPU requests. As our new machine will always need time on 2 pCPU’s, it’s “easier” for VMware to schedule pCOU time to the VM’s with 1 vCPU, so they’ll end up getting more CPU time than the 2 vCPU VM. This waiting time, is what’s known as CPU Ready time, and when this get’s too high, you’ll find your VM’s with more vCPU’s will get slower…

Here’s an example:

This is the previously mentioned Exchange server, with 4 vCPU’s. It’s a one hour capture of both CPU Usage, and CPU Ready time:

EX02 4 vCPU

As you can see, CPU ready time was anywhere between 180ms and 1455ms, averaging 565ms. This lead to slow CPU response for the machine.

So, looking at the average CPU usage for a couple of months, it was at ~30%. So that’s 30% of 4 CPU’s.. just over a single CPU. So, 2 vCPU’s needed to be removed… and this is the result:

EX02 with 2 vCPU

So, the result? CPU ready time was between 28ms and 578ms, a vast improvement, and averaged just 86ms, far better than 565ms! CPU usage was higher, but then it’s now using more of the CPU’s it’s granted, so this was to be expected.

Now, CPU Ready time on this machine still isn’t great, but I’ve a lot more VM’s to sort through, reducing vCPU allocation, and hopefully it’ll just get better!

Copy Files Between Datastores – PowerCLI

I had the need to automate moving about 50 ISO files from one datastore to another during a storage array migration a short while ago, so I wanted to share this script with you all in case you ever find the need for this or similar.

It’s rather simple, and you just need to edit this with the names of your datastores and folder structure (top folder only):

#Set's Old Datastore
$oldds = get-datastore "Old Datastore Name"

#Set's New Datastore
$newds = get-datastore "New Datastore Name"

#Set's ISO Folder Location
$ISOloc = "Subfolder_Name\"

#Map Drives
new-psdrive -Location $oldds -Name olddrive -PSProvider VimDatastore -Root "\"
new-psdrive -Location $newds -Name newdrive -PSProvider VimDatastore -Root "\"
#Copies Files from Old to New
copy-datastoreitem -recurse -item olddrive:\$ISOloc* newdrive:\$ISOloc

Line 1: Change the script to have the name of the datastore you are moving the files FROM.
Line 5: Change the script to have the name of the datastore you are moving the files TO.
Line 8: Change the script to have the name of your ISO subdirectory. Do not remove the “\” unless you have no subfolder.
Lines 11 & 12: Maps PowerShell drives to those datastores.
Line 14: Copies the files.

Preparing for VMware vCenter Database Disaster ‚Äď Part Two: Export Host information via PowerCLI

Hopefully you’ve all read Part One of this series, where I provide examples of gathering information from vCenter mainly for VM’s in order to recreate your environment from scratch, just in case you have a major vCenter database corruption or the like. If you have, sorry part two has taken so long!
Part Two will show how to export information regarding your ESX(i) hosts, including networking information, so that this part of your setup is also easy to recreate. I should note here, that I’ll be trying to export VSS information, as well as Service Console and VM Kernel port configuration, and get this all exported into CSV files.
So… Here goes…!
Exporting physcial NIC info for the vDS switch
This is a pretty simple script that uses the get-vmhostpnic function from the Distributed Switch module in I mentioned in part one (Thanks again Luc Dekens :¬)).
import-module distributedswitch

write-host "Getting vDS pNIC Info"

$vdshostfilename = "C:\vdshostinfo.csv"
$pnics = get-cluster "<em>ClusterName</em>" | get-vmhost | get-vmhostpnic
foreach ($pnic in $pnics) {
if ($pnic.Switch -eq "<em>dVS-Name</em>") {
$strpnic = $strpnic + $pnic.pnic + "," + $pnic.VMhost + "," + $pnic.Switch + "`n"
#Writes to CSV file
out-file -filepath $vdshostfilename -inputobject $strpnic -encoding ASCII

Simply change “ClusterName” to match that of your cluster, and change “dVS-Name” to match that of your dVS (vDS – whichever). Then the info exported will contain the physical nic info for your distributed switch.

Next it’s time for simply getting a list of hosts in the cluster, I know, it’s nothing major, but at least it’s in a CSV I can import later, and it makes life much easier!!!

$hostfilename = "c:\filename.csv"
write-host "Getting Host List"
$hosts = get-cluster $cluster | get-vmhost
foreach ($vmhost in $hosts) {
$outhost = $outhost + $vmhost.Name + "`n"

out-file -filepath $hostfilename -inputobject $outhost -encoding ASCII

Simply put, gather a list of hosts in the cluster called “ClusterName”¬†and output their names to “c:\filename.csv”

OK, so now that we have that info, all I need to gather is a list of Standard Switches and their port groups, including IP information to make life easy… So, here goes:

$vssoutfile = "vssoutfile.csv"
$cluster = "Cluster Name"
$vmhosts = get-cluster $cluster | get-vmhost

$vssout = "Host Name, VSS Name, VSS Pnic, VSS PG" + "`n"
foreach ($vmhost in $vmhosts) {
$vmhostname = $vmhost.name
$switches = get-virtualswitch $vmhost
foreach ($switch in $switches) {
$vssname = $switch.name
$Nic = $switch.nic
$pgs = get-virtualportgroup -virtualswitch $switch
foreach ($pg in $pgs) {
$pgname = $pg.name
$vssout = $vssout + "$vmhostname" + "," + `
        "$vssname" + "," + "$Nic" + "," + `
        "$pgName" + "`n"

out-file -filepath $vssoutfile -inputobject $vssout -encoding ASCII
Now we just need the host IP’s. At the moment, I can find this info for VM Kernel ports on ESX hosts, but I can get service console information, and the vmkernel IP in ESXi hosts (it’s pulled from the same PowerCLI script, so that’s this one here:

$hostipoutfile = "hostip.csv"
$cluster = "Cluster Name"
$output = "Host Name" + "," + "IP Addresses" + "`n"

$vmhosts = get-cluster $cluster | get-vmhost
foreach ($vmhost in $vmhosts) {
$vmhostname = $vmhost.name
$ips = Get-VMHost $vmhostname | `
     Select @{N="ConsoleIP";E={(Get-VMHostNetwork $_).VirtualNic | `
$ipaddrs = $ips.ConsoleIP
$output = $output + "$vmhostname" + "," + "$ipaddrs" + "`n"

out-file -filepath $hostipoutfile -inputobject $output -encoding ASCII

Now, I’m slowly working on this project in my spare time at work (it’s actually for work but not as important as everything else I’m doing!), so part 3 is probably going to be some time away, and that’ll show you how to import all this info back into vCenter to reconfigure your hosts… bear with me, I’ll get this written ūüôā

vApps in vSphere 4, and why they’re very, very useful

The week before last I attended the vSphere 4 Design Workshop at QA in Reading and came across something I’ve rarely actually¬†seen in use… vApps. It’s not something that many people pay attention to I don’t think, but in all honesty, they’re pretty awesome when you think about it even for internal use, in fact, the only place I’ve seen them is when downloading pre-built appliances from the marketplace… They’ve certainly made me re-think a few things…

Imagine this:

You have several ESX hosts running¬†a bunch of¬†virtual machines, and for some reason the power fails in the middle of the night and the UPS systems don’t have enough power to last until you get to the office in the morning (I’m talking worst case here basically, and you should have far more protection than that ideally)…

When you come in the next morning (if you haven’t had a call in the middle of the night), and your systems are¬†finally¬†powered on, you’re going to have to boot each virtual machine to restore the network’s functionality, taking the usual route of Domain Controllers first, then mail servers, file servers, print servers so on and so forth until the network is operational again, each one being booted manually, or via some sort of PowerCLI script perhaps? Well, what if you could make that process 30 times easier? Well then, go take a look at vApps…

A vApp for all intents are purposes is a container of one, or more, virtual machines. BUT, what you can do with a vApp is specify boot order of the machines within that vApp… So, for instance, we all know that to boot an Exchange server we need Active Directory and DNS servers to be operational right?

Well… create a vApp, add the Domain Controllers, DNS servers and Exchange Mailbox Server, as well as the Exchange CAS server (just drag and drop them¬†in the vCenter console). Edit¬†the vApp’s¬†settings and you’ll find a tab called “Start Order”. Now, here you’ll find some “Groups” and all of the VM’s you added are probably listed in their own group. Make sure that your VM’s are listed in the correct order (use the up and down arrows), so that¬†Domain Controllers at the top, and the mailbox server at the bottom in this case. Now, if you put two machines in the same group, they’ll boot at the same time, otherwise it’s a top to bottom list (and reverse for shut down). My preference here is to change the settings for each VM so that the next machine will boot once VMware tools has loaded in the VM, so, tick the “VMware Tools are ready” check box. Whilst you’re doing this, set the “Shutdown Action” to “Guest Shutdown”.

That’s it… now that the machines are in a vApp and the start order is set, all you have to do is power on the vApp and¬†it’ll then automatically boot each VM in turn, waiting for either 2 minutes to pass (that’s the default which can be changed) or for VMware Tools to be started by the OS. Simple huh?

Now… I hear you say “But I have power on options for when my hosts boot”… yeah, but… what happens when DRS or manual vMotion is implemented and the VM is moved to another host, oh¬† yeah, it loses that setting for eternity (or at least until you manually add the rule on the host again)…

Oh… and you can¬†nest vApps too…

Taking the previous example, you may want to segregate Exchange from the Domain Controllers to allow you to easily power on or shut down each type of system separately (for maintenance for example), so just create 3 vApps: one as a “Master”, one for the Domain Controllers, and the third for the Exchange Servers. Populate the latter two with the¬†correct virtual machines, and set the start order and shut down options as before, giving you two vApps that are independent of each other. Now, drag those vApps¬†into the “Master” vApp and set the start order here too, with your DC’s vApp in a group higher than the Exchange Servers vApp. You don’t get the same options here, as the settings from the nested vApps will still apply. You now have an easy method to boot just the domain controllers, just the Exchange Servers, or the whole lot in one click, or shut them down in reverse order too. Nice!

That’s not the only benefit, there’s¬†a couple¬†more…

vApps also give you another security boundary. You can create roles that have access to specific tasks with vApps, so you can give “Power On” rights to a member of the IT Department who may not have any other access, but in an emergency, can still boot specific vApps and therefore boot the VM’s in the correct sequence.

They also have built-in resource pools, so all the usual benefits still apply here too, and yes, you can nest resource pools inside vApps too if you really want or need to!

Now, this does alter the way VM’s appear in the vCenter console, much to my own disappointment in fact. The “Hosts and Clusters” view doesn’t change much, other than the fact that each vApp becomes another level to expand in the console, but, the VM’s and Templates view is changed. Now, in the left hand pane where the VM’s used to reside, you can only see the vApps, and to see which VM’s are in which vApps you have to click on the vApp and then on the “Virtual Machines” tab. Why a vApp in this view doesn’t act as a folder I don’t know, especially when it does in the “Hosts and Clusters”¬†view, which doesn’t usually show folders!!

From a disaster recovery scenario, and from a systems maintenance point of view, I think vApps are fantastic… Being able to boot all of my machines in one click, and also having the option to shut them all down the same way is fantastic, moving servers, or having to shut them down for electrical systems maintenance makes life easier, and that’s the whole idea of virtualization isn’t it?

Hiding VMware Tools from the user on a virtualised Terminal Services/Citrix XenApp Server

I¬†came across this one earlier today, and I must say, I was surprised that this option is available to users without administrative rights to vCenter/ESX or the Virtual Machine… but it would appear that the VMTools application that appears by default in the notification area for any user logged onto the virtual machine allows ANY user to perform any actions within that app… including disconnecting devices such as IDE controllers, but more importantly for TS/XenApp servers… the network card.

There are simple ways to block this though, but it takes some effort, especially if you have lots of TS/XenApp servers!

So, there are 3 things you can do to help:

1. Hide the VMware Tools icon in the system tray.
2. Restrict access to the Control Panel applet.
3. Restrict access to the VMWareTray.exe application

I’ll talk you through each one:

Hiding the VMware Tools icon:

This unfortunately isn’t as simple as opening the tools application, and unchecking the “Show VMware Tools in the taskbar” box… this action only applies to the user performing it… not for the whole system, so, we have to manually edit the registry to get this to take effect for all users… Now, don’t forget, editing the registry without knowing what you’re doing can be very dangerous, always backup your system first…

1. Open regedit.exe
2. Browse to the following key:

HKEY_LOCAL_MACHINESoftwareVMware, Inc.VMware Tools

3. Edit the “ShowTray” subkey and change the value to a zero, click OK.

When you log back into the server, the VMware Tools icon shouldn’t display in the notification area.

Restrict Access to the Control Panel Applet:

You have several options here, this can be done as a local policy (meaning no one including the administrator can access the applet) or via a Group Policy which can be filtered to specific users, these instructions are for Windows 2008 R2, but will be very similar for Server 2003 and Server 2008 R1.

1. Open an MMC and either add the Local Policy or Group Policy Management consoles.
2. If using a Group Policy create a new policy and link it to the OU as required.
3. Browse to the following area in the policy:

User ConfigurationAdministrative TemplatesControl Panel

4. Open the “Hide Specified Control Panel items” setting.
5. Click “Enabled”, then click “Show”.
6. In the “Value” field type “VMware Tools” (no quotes). Click OK.
7. Click OK again and close the policy.
8. Reboot the server to test that the Applet is no longer accessible.

Restrict access to the executable:

Even with all of this, the user could (if you don’t restrict access to local disks) find the executable and run it, which will open the GUI for VMware Tools… shame really! So, the other options are to set the file permissions to block the user’s group from accessing these files, or at least allow administrators, domain admins, etc. and the user account that runs the VMware Tools service, and block all other users. Personally, I always hide the local disk from the users, so this part isn’t an issue for me, but there are admins out there that perhaps aren’t as “strict” as me!

And that’s it, one blocked application and no users disconnecting NIC’s and CD ROM’s etc. whilst the server is in use!