Changing VMware ESX Service Console IP’s without downtime

Very recently I came across a situation where I needed to change the IP range that a group of ESX hosts were using for Service Console connections, without causing a disruption in service to the VM’s.

This was something I thought would be possible, but I hit a few issues, so thought I’d share the process that worked without failure, in case anyone else out there has the same issues.

Let’s start with some background…

The customer had a single IP range for their entire network, and now wanted to change the addressing so that management services are segregated from “normal” network traffic. So, we had to migrate the Service Console, vCenter IP and the VM Kernel port group IP’s to a whole new range. The first port of call was to ensure that the new range could still communicate with the current IP range, so there wouldn’t be a problem getting traffic across the boundary. For now, this was open to all traffic to ensure that there were no port blocking rules interfering with the transition, and this would be locked down once all migrations were complete.

The site uses both HA and DRS and so these had to be worked around when it came to moving the IP range. On top of all this, they had turned off admission control as they were running low on resources and the new host hadn’t arrived yet, so they didn’t have a spare host that they could temporarily put into maintenance mode…

So, we made a plan, and we tried to stick to it…

The plan was:

  1. Disable HA as a precautionary step. We actually removed HA from the cluster altogether.
  2. Set DRS to partially automated.
  3. Clear the host files in each ESX host (HA fills these in but doesn’t empty it when removing HA). Also clear the host files in vCenter if needed.
  4. Change the IP of the vCenter host, and alter the DNS records for it. Ensure that vCenter can ping each ESX host.
  5. Check that each ESX host can resolve the new vCenter IP.
  6. Add an additional Service Console port group onto each ESX host with its new IP. Each time, add the new IP to the vCenter server’s host file after an error is generated.
  7. Right click the host and choose “Connect”. It should now reconnect using its new IP.
  8. Add the new VM Kernel port groups, with their new IP’s and enable them for vMotion.
  9. Alter the DNS entries for each host in your DNS system, and flush the DNS cache.
  10. Change the Gateway in each Service Console to the new router IP. Change the “Gateway Device” setting to the new vSwif (most likely vSwif1).
  11. Remove the old Service Console port groups.
  12. Re-enable DRS.
  13. Re-enable HA.

The reason for modifying the host file in vCenter if you’re wondering was because I found that whilst you add a service console and the vCenter server is on its new IP address, even if DNS resolves the ESX hostname to the old IP for the host, the new service console would respond, causing errors with routing and the firewalling that was in place on this site. Changing the host file was far easier than editing DNS and flushing all the relevant DNS caches each time, as it provided an immediate change. Once DNS was then altered, caches cleared, the host file could be emptied, and the system would continue to function.

Preparing for vCenter Database Disaster – Part One: Export information via PowerCLI

In this new blog post I’m going to start showing you how to export data from vCenter using PowerCLI to provide you with all sorts of useful information that you may need in case of a vCenter database loss. I’m going to include the loss of a Virtual Distributed Switch (vDS) as well in case you’re using this fantastic option from VMware.

I’m starting this because recently I had just this problem. The vCenter DB at a customer’s site had corrupted, and this meant they lost the vDS configuration plane, and although they could work on systems, managing them just got a whole lot more difficult… The SQL database had actually been corrupted for a long time… their 3 months of archived backups wouldn’t restore and get the vCenter services operational again, all with the same error, leading to a VMware KB article telling them to reinitialize the database. Ouch.

So, using some PowerCLI commands I started gathering information about the VM’s, their networking information as well as the vDS information I needed (they had several VLAN’s in place which all needed recreating).

It was a tiresome process as I didn’t have time to get proper scripts running to export nicely to CSV, so I decided I’d change this later, once I had time to write-up this blog post!!

The main items the customer needed at the time were VM networking information (which VM was on which port group), the configuration of their vDS and the folder structure within vCenter. Below, I’ve listed scripts which will help with these three operations, scripts I’m going to implement on every install I ever do from now on – I certainly don’t want to go through all the pain I faced first time around…!!!

So, if you want to get networking information from each VM, into a CSV file, here’s a script I’ve done it with, and this takes multiple NIC’s into account too:

#Script to backup all VM Network info

#Connect to vCenter Server
Connect-VIServer -Server "servername" -User "username" -Password "password"

#Set Variables
$cluster = "Cluster Name"
$strout = "VMName,NIC Name,Network`n"
$vmsfilename = "filename.csv"

#Backup VM Networking information
$nics = get-cluster $cluster | get-vm | sort-object Name | get-networkadapter
foreach ($nic in $nics)
{
$strout = $strout + $nic.Parent + "," + $nic.Name + "," + $nic.networkname + "`n"
}

#Outputs $strtest into the file named in $vmsfilename
out-file -filepath $vmsfilename -inputobject $strout -encoding ASCII

#Disconnects from vCenter server
disconnect-viserver * -force -confirm:$false

The next set of information I’d like to have easily available would be the vDS switch configuration information. Now, we always setup vDS port groups with the “Route based on Physical NIC load” load balancing policy, so this wasn’t information I needed and so isn’t included below. Now, you’ll need Luc Dekens’ Distributed Switch module for PowerCLI to do this (his blog is here and I highly recommend it):

#Script to Export vDS config

#Import vDS module from Luc Dekens
import-module vmwaredistributedswitch.psm1

#Connect to vCenter Server
Connect-VIServer -Server "server" -User "username" -Password "password"

#Set Variables
$strdvpg = "PGName,VLAN ID,NumPorts,vDSName`n"
$vdsfilename = "filename.csv"

#Gets Distributed Virtual Switch information
$dvpg = get-distributedswitchportgroup
#Sets CSV output for each DVPG in $dvpg
foreach ($pg in $dvpg)
{
$strdvpg = $strdvpg + $pg.Name + "," + $pg.VLANID + "," + $pg.NumOfPorts + "," `
+ $pg.DistributedSwitch + "`n"
}

#Outputs $strdvpg to the file named in $vdsfilename
out-file -filepath $vdsfilename -inputobject $strdvpg -encoding ASCII

disconnect-viserver * -force -confirm:$false

So, that’s the first part over and done with. Networking info for the VM’s and vDS has been exported to CSV files in case we ever need to re-import them again, next we’ll take a look at how to export your folder configuration from under the “VM’s and Templates” view in the vSphere client:

#Script to Export vSphere folder structure to CSV

# Connect to vCenter
connect-viserver "servername" -user "username" -password "password"
#Set Variables
</em>$filename = "filename.csv"
$strout = "Name,Parent`n"

# Get Folder Structure
$list = get-folder
foreach ($folder in $list)
{
$strout = $strout + $folder.name + "," + $folder.parent + "`n"
}
out-file -filepath $filename -inputobject $strout -encoding ASCII

#Disconnect from vCenter
disconnect-viserver * -force -confirm:$false

So, that’s a lot of the VM side complete, folders are exported, VM network information and vDS configuration is all sat in CSV files. Now, you can e-mail that out using standard Powershell e-mail cmdlets if you’d like, and set this up as a scheduled task too should you so wish… whatever you do, just ensure it’s not all held on the same server as your SQL database 🙂

Part Two will come along shortly, where I’ll be looking into pulling out the host configuration, and getting ready to import all of this back into a reinitialized database later on in part 3…

UPDATE: Part two is available here: Part Two