Can’t configure VMware HA – unknown error

Whilst running though the upgrade to vCenter 4.1 the installer will update the VPXA agent on each host when you reconnect it within the vSphere client. Great! Unless that is, you start getting HA errors….

If you had HA running before, you can pretty much guarantee that the settings are correct, DNS/Host files are all set correctly and there shouldn’t be any IP connectivity issues… well, sometimes the agent upgrade doesn’t quite work correctly, and removing the host from the cluster means moving all your VM’s to other hosts, placing it in maintenance mode, and then finally removing the host from vCenter. If you’ve got Virtual Distributed Switches, this becomes much more of a pain too! Well, you could just temporarily stop host monitoring from HA in the HA settings on the cluster and then perform the following to remove the vCenter and HA agents from the affected host:

From a CLI (SSH session):

  1. Login to your host and sudo up to root level access.
  2. Type the following and press enter:rpm -qa | grep -i aam
  3. This should give two lines for an answer, both need removing using the following (altering <version number> with your version listed from the above command):rpm -e VMware-aam-vcint-<version number>
    rpm -e VMware-aam-haa-<version number>
  4. You then need to run this command:rpm -qa | grep -i vpxa
  5. And then remove that package with this command (altering <version number> with your version listed from the above command):rpm -e VMware-vpxa-<version number>
  6. Wait a few minutes and vCenter should show that host as “Disconnected”.
  7. Right click on the host, and click Connect. Follow the wizard and the agents should be re-installed.
  8. Re-enable host monitoring from within the cluster settings, and HA should be configured on the affected host.

Hope that helps others out there…!

Call “PropertyCollector.RetrieveContents” for object “propertyCollector” on vCenter Server failed.

OK… annoying one today… but simple to resolve!

This error popped up after our upgrade to vCenter 4.1… when converting templates into VM’s to perform updates and then trying to edit the settings we started receiving this error:

Call “PropertyCollector.RetrieveContents” for object “propertyCollector” on vCenter Server “vcenter_server_name” failed.

How annoying! Anyhow, I found that it seemed to be down to the fact that the VM had an ISO attached to its CD drive… But, I can’t access the settings to alter it! So, I had two options:

  1. Use PowerCLI to change the settings.
  2. Remove the Template from the Inventory, and re-add it to see what happens.

I went with option 2 to test this theory out. So, removed the VM, browsed the templates data store and re-added it again. Voila! I could access its settings, and there it was, an ISO file connected to the CD drive…. Now to do the rest of them!!!

Looks like it’s time to send a note around the tech team again reminding them to remove ISO’s from the VM’s once they’re finished with them!!!

Upgrading VMware vCenter 4.0 to 4.1

With the recent(ish) release of vSphere 4.1 comes the task of upgrading ESX/ESXi hosts to this new build, fortunately, with some of the tools available from VMware (VMware vCenter Update Manager) the host upgrade process is fairly simple, and takes approximately half an hour per host (plus some time afterwards for testing).

The latest version of vCenter however is slightly more difficult for some people… The latest edition requires a 64-bit OS, and more RAM than the previous version. Now RAM is easy, but you can’t “upgrade” a 32-bit OS to a 64-bit OS which can make migrating it slightly difficult!

I’ve got this arduous task to come but here’s my proposed plan…

  1. Check your physical host (if using one) complies with the HCL for vcenter: http://www.vmware.com/resources/compatibility/
  2. Use your favourite VM conversion tool to get a copy of your current vCenter server as a virtual machine (if it’s physical that is).
  3. Take a backup of your vCenter (and Update Manager if applicable) databases. Also backup your vCenter server’s SSL certificate information as you’ll need this later if you have your own certificates.
  4. Shut down the vCenter physical host and then boot the VM version.
  5. Check that the VM version is working correctly, and then power it down.
  6. Rebuild your vCenter server with a 64-bit OS such as MS Windows Server 2008 R2. If you need to give it the same name in Active Directory then reset the computer account first in AD so that this machine can join in its place.
  7. Create an ODBC connection to the vCenter DB on your DB server (or install it locally and restore the DB). If you were previously using Update Manager, then create the ODBC connector or restore the DB for this too.
  8. Install vCenter 4.1, as well as vCenter Converter, and vCenter Update Manager.
  9. Copy in the SSL Certificates you backed up earlier.
  10. Open the vCenter client and install the plug-ins for Update Manager and the Converter.
  11. Check that all the hosts can be accessed correctly, if not, right-click the host, choose “Connect” and then follow the wizard.
  12. Once that’s done, check through all your client settings, and the settings for Update Manager to check all is normal.

That should be about it, as I said, I’ll be performing this task shortly (about mid September) so I’ll update this entry if anything does need changing! Next it’s time to upgrade the hosts…..

UPDATE:

Ok, so this wasn’t as easy as I first thought it would be… VUM was a pain, in fact, I ended up creating a new database and scrapping the old one (not too much hassle for me though when my VUM server is sat behind a 1Gbps internet connection…). So, here’s what I did and how I did it, starting with my previous config:

I had a vCenter 4.0U1 server running on Server 2003 Standard x86 with 4GB of RAM. There was also a server running server 2008 R1 x64 running SQL 2005 for the vCenter and VUM databases. All sat on a 1Gbps network.

The perceived final result would be a vCenter server running version 4.1 with Windows Server 2008 R2 Standard x64, connected to the same, unaltered, SQL server as before. This sounded really simple!

So, here are the steps I took to rebuild the vCenter server:

  1. I started by using VMware Converter to convert the vCenter host to a VM so that I knew, should all else fail, I’d have a backup copy.
  2. I then backed up the SQL databases so that I knew they were also protected.
  3. I shut down the vCenter server and booted the VM copy to check that this still worked ok.
  4. Once I was happy that this was alright, I shut down the VM and booted the physical server with the 2008 DVD.
  5. Once Windows was installed I did all the usual bits: Windows Updates, teaming the NICs etc. I then reset the computer account in AD and joined the server to the domain.
  6. I then set about recreating the ODBC connections. This became a little bit of a task… vCenter still requires a 32-bit DSN and having not needed this before, setting up was a challenge… Anyway, you need to run odbcad32.exe rather than the ODBC tool under Administrative Tools. Simply running this from the Run prompt wasn’t working though, so I ended up running it from c:windowssyswow64odbcad32.exe.
  7. I then installed vCenter and the vSphere client, which worked a charm after fixing the ODBC connector. At this point, I wasn’t installing Converter, or Update Manager, I wanted to deal with vCenter first.
  8. If you have your own certificates for vCenter this is when you copy them back in… If you don’t, just reconnect the hosts by right clicking, choosing “Connect” and running through the wizard. Easy. Well, unless (like me) you have a host that won’t join the HA cluster… more on that in another article!
  9. OK, so by this point you can start testing that vCenter is working OK, so I started performing some vMotion operations, and checking through the console that configurations could be changed etc. Once happy that all was OK, I moved on…
  10. So, now I installed Update Manager and VMware Converter.
  11. After this, I logged back into the vSphere client and isntalled the required plug-ins, then restarted the vSphere client.
  12. This is when the problems struck…. VUM wouldn’t work correctly. I couldn’t scan machines, or hosts for updates. I then noticed that the service wasn’t running in an account that had access to the SQL database… so I changed this, still no luck! In the end, I gave up, created a blank database on SQL, altered the ODBC connection to point to this DB and job done, Update Manager worked again. So, then I kicked off a patch definition download (again, not a problem when you’re sat behind a massive internet connection).

In essence, the migration had gone as planned, the server had been rebuilt, and was using the old database like I was hoping it would. Yes, I had some issues with a host not joining HA correctly, and yes I had to rebuild the VUM database (which let’s face it, isn’t the end of the world unless you have a lot of custom baselines). So, where from here? Oh yes… upgrading the hosts… and of course VMware Tools on each and every VM… that’s a time consuming task when you have over 200 VM’s in a single cluster, especially when there’s only 4 ESX hosts!

So… you can probably guess what will be coming up soon on this blog… yep… Upgrading ESX or ESX(i) 4.0 to 4.1…

VMware Data Recovery is awesome!

I came across this application some time ago when we first upgraded to vSphere 4. At the time I thought, “I’ll have to take a look at that at some point”, and since then I’ve been testing each version as it’s been released, and playing around with it. We’re now at version 2.1, and I have to say, it’s fantastic! It also now has features that make it much more useable…

VMware add-on small apps to vSphere all the time, and this one has to be one of the most useful apps I’ve seen. Basically, it’s a pre-built VMware ready appliance that can perform backups and restorations of your VM’s. You can run backups to vDisks, NFS/CIFS shares, or Raw Device Mappings (RDM’s). On top of that, it’s really simple to set up, and provides deduplication of all your backup data.

Sounds good huh?

Well, what if I said, that by default, it also provides File Level Restore capabilities too? Well… it does… via a really simple tool. In fact, this tool is literally an executable file. You don’t even have to install it! Just run it, connect to your VDR appliance, and hey presto, you can mount any backup to a folder on your local disk, browse it, copy the files out that you need, and then unmount the backup. Simple!

Setting up VDR couldn’t be simpler either. Download the image from the VMware website (your licensed for it with Advanced and upwards versions of ESX/ESX(i) and any Essentials Plus and upwards kit). It comes with a copy of the FLR tools for Linux and Windows, the release notes for the current version, the appliance itself, and the vCenter plug-in you need to manage it.

Install the plugin, and deploy the OVF template using the vSphere client. Then, add some hard disks (up to 2 x 1TB disks in vDisk or RDM format, or 2 x 0.5TB CIFS shares are currently supported). Boot the appliance and open the console to it. Once it’s booted there’s a simple option to give the VDR appliance an IP and host name, follow this through. The rest will be done in the vSphere client.

Within the client (with the plug-in installed), navigate to Solutions and Applications -> VMware Data Recovery. Then, either choose your VDR appliance in the left pane, and double-click, or type its name or IP into the box in the right hand pane, and click connect. A short wizard will appear to configure some extra bits.

Supply a username and password to connect with, then click on each hard disk you added earlier and click format at the top of the screen. Depending on the size of your disks, this may take a while! You can also add a network share from this pane if needs be. You can click next in the mean time, and come back to configuring a backup job later on if it’s going to take a while!

Once the appliance is on the network, and has its disks formatted, all you need to do is set up a backup job and time window. So click on the backup tab, and click the “New” button in the top right. Give the backup job a name, choose the VM’s you want to back up, when to back them up, and then how many versions to keep in the store. Remember at this point that VDR uses incremental backups, so in theory you can keep a lot of backups, also remember that it’s going to deduplicate the backups, so there should be even less data if all of your servers run the same OS etc.

That’s about it…! Just run the backup and test it, to ensure its working ok, and job done! Restoring a VM couldn’t be simpler wither, just run through the wizard on the restore tab! Oh, and file level restore… simple.

As you can see, this appliance is very simple to use, configure and restore from. The next step is for VMware to add some sort of alerting from the appliance so that you know when VM’s haven’t been backed up etc. without logging into the console all the time… Once that’s done, this appliance will become truly fantastic!

Hiding VMware Tools from the user on a virtualised Terminal Services/Citrix XenApp Server

I came across this one earlier today, and I must say, I was surprised that this option is available to users without administrative rights to vCenter/ESX or the Virtual Machine… but it would appear that the VMTools application that appears by default in the notification area for any user logged onto the virtual machine allows ANY user to perform any actions within that app… including disconnecting devices such as IDE controllers, but more importantly for TS/XenApp servers… the network card.

There are simple ways to block this though, but it takes some effort, especially if you have lots of TS/XenApp servers!

So, there are 3 things you can do to help:

1. Hide the VMware Tools icon in the system tray.
2. Restrict access to the Control Panel applet.
3. Restrict access to the VMWareTray.exe application

I’ll talk you through each one:

Hiding the VMware Tools icon:

This unfortunately isn’t as simple as opening the tools application, and unchecking the “Show VMware Tools in the taskbar” box… this action only applies to the user performing it… not for the whole system, so, we have to manually edit the registry to get this to take effect for all users… Now, don’t forget, editing the registry without knowing what you’re doing can be very dangerous, always backup your system first…

1. Open regedit.exe
2. Browse to the following key:

HKEY_LOCAL_MACHINESoftwareVMware, Inc.VMware Tools

3. Edit the “ShowTray” subkey and change the value to a zero, click OK.

When you log back into the server, the VMware Tools icon shouldn’t display in the notification area.

Restrict Access to the Control Panel Applet:

You have several options here, this can be done as a local policy (meaning no one including the administrator can access the applet) or via a Group Policy which can be filtered to specific users, these instructions are for Windows 2008 R2, but will be very similar for Server 2003 and Server 2008 R1.

1. Open an MMC and either add the Local Policy or Group Policy Management consoles.
2. If using a Group Policy create a new policy and link it to the OU as required.
3. Browse to the following area in the policy:

User ConfigurationAdministrative TemplatesControl Panel

4. Open the “Hide Specified Control Panel items” setting.
5. Click “Enabled”, then click “Show”.
6. In the “Value” field type “VMware Tools” (no quotes). Click OK.
7. Click OK again and close the policy.
8. Reboot the server to test that the Applet is no longer accessible.

Restrict access to the executable:

Even with all of this, the user could (if you don’t restrict access to local disks) find the executable and run it, which will open the GUI for VMware Tools… shame really! So, the other options are to set the file permissions to block the user’s group from accessing these files, or at least allow administrators, domain admins, etc. and the user account that runs the VMware Tools service, and block all other users. Personally, I always hide the local disk from the users, so this part isn’t an issue for me, but there are admins out there that perhaps aren’t as “strict” as me!

And that’s it, one blocked application and no users disconnecting NIC’s and CD ROM’s etc. whilst the server is in use!

F-Secure Mobile Security and Anti-Theft

On Monday I flew over to Sweden for a 2 day conference being held at the Hotel Rival in Stockholm by the guys and gals at F-Secure. We had a fantastic time! They talked about a lot of subjects over those few days, new features in their Mail Security Gateway product, including e-mail encryption, as well as some fairly new products. They now have a fantastic product called F-Secure Anti-Theft.

This tool is FREE to download, and runs on any Symbian, Android or Windows Phone/Mobile devices. It allows you to remotely wipe, lock or even locate your lost/stolen mobile, by simply sending it a text message…! Oh, and the locate feature sends a reply, with a link to a map to show you exactly where the phone is!!! On top of that, if the thief changes the SIM card… it will send a text to a pre-defined mobile number, giving you the new number that the thief has put into your phone, allowing you to lock, wipe or locate your phone still!!! Good huh?

Well, they’ve also released another product, F-Secure Mobile Security. This incorporates the Anti-Theft product, as well as a small, lightweight Anti-Virus scanner and some other features such as F-Secure’s Browsing Protection service which helps you to know if a site you are visiting is dangerous, and will block it for you, for your own security.

There’s also a business version available, with a central management console, the only feature missing here is the location option (this is still available via text message though).

With the increasing numbers of mobile phone viruses out there, it’s time to start protecting ourselves now, before it’s too late.

You can find out more about these products here:

F-Secure Mobile Security
F-Secure Anti-Theft
F-Secure Mobile Security for Business

To use the Anti-Theft features, you simply text once of the following statements to your phone:

#wipe#<password.
#lock#<password>
#locate#<password>

Just replace the <password> section with your preset security code that you configure once the software is installed.