I am not a VMware basher, just the opposite actually.  I serve as a primary VMware advocate in my company.   But, my company has not embraced the vCloud vision of VMware.   I am a VCP3 and VCP4 and hope to be a VCP5 in the near future.  I know their products well and use them on a daily basis, both at work (vSphere) and at home (Fusion).  But it is harder for me to make a technical or business case for their product.  The first issue is cost.  The second is the “good enough” factor, since we are not using some of the additional value they have added to their product in vSphere 4 and 5.

There are already good cases in the datacenter where all the advanced VMware features don’t matter, and in those cases my company has already adopted XenServer as a secondary hypervisor.  And XenServer works well, which becomes a problem.  We have proven its ability to run our workloads and consolidate servers.    In some cases, the applications we run on it were built with high availability and fail-over and the tried and true VMware features like clustering, HA and DRS do not matter, specifically our XenApp servers .

In other some ways, VMware is erroding the existing value of their vSphere product suite by pulling features its customers are using.  The primary reason I have heard to do this is because there is overlap with new products they have purchased or developed.  Guest patch management is an example of this.  Since their Configuration Management product handles patch management, a feature that has existed in vSphere for two generations, Update Manager is now being downgraded to only patch vSphere hosts.  But the kicker in this case is that Configuration Manager does much more than patch management and is priced as such.  We aren’t seeking the additional features and VMware has priced themselves out of the game for us.

VMware’s decision on patch management leaves companies with a big void to fill.  But no solution, including the VMware Configuration Manager, fills the void as seamlessly as the Update Manager product that once patched my systems.  Because we have firewalls in between our vCenter and hosts, Update Manager worked well because it used the same vCenter ports for patching.  Configuration Manager and other solutions do not, which is actually kind of a pain.

VMware has cast a vision of the vCloud and added API sets for storage, security and networking that help to pave the path to the cloud for companies.  In our case, we have not embraced the cloud vision and while we may in the future, today, the enhancements added to vSphere have not added real value to us.  Unless a company embraces VMware’s vision and adopts these technologies, the vSphere suite continues to erode value.

The cloud is a vision I have written about before and I stated then that it’s one that systems administration groups have little to do with influencing in organizations (see my original post here).  This is a particular challenge for VMware and its advocates.  It is, frankly, a problem that worries me as a VMware believer and administrator.  But, as the tides change – such as the transition from Netware to Active Directory, we as administrators move where we need to and adapt, like the chameleons that we are.  But I am also wondering what other VMware administrators are feeling?  There was a time my company used all the features in vSphere.  Has VMware left you to in a corner while they focus on the cloud?  Talk back to me…

Why would you want to do a long distance VMotion?  In my case, we have two data-centers – geographically close to one another.  We currently stretch our cluster between the two locations and it allows us to float VM’s using VMotion between the two.  The problem is that if we lose our primary datacenter, all storage is presented from here.  Long Distance VMotion is the notion of having two separate clusters, one in each datacenter, and being able to VMotion between them.

What was really news to me from this session (I’ll get to what was presented) was that we can present the same data stores to two different clusters and have them recognized on both clusters.  I am pretty sure I tried this way back in the 3.0 days and it failed to work.  This must have been added in 3.5 or 4.0 – I have not tried in recent years.

So, what was presented?  The three companies worked together to identify and trial a solution to allow for long distance VMotion.  At this point, there is a very narrow set of criteria must be satisified to be support and for this to occur.  Much of the restriction comes on the storage side, but network also presents some problems.  Apparently, everything you need in vSphere is there, if you separate each datacenter into its own set of hosts.

Requirements

• Distance between datacenters must be less than 200 km
• A single instance of vCenter to control both clusters
• Each site must be configured in it’s own cluster – the cluster cannot be stretched
• Dedicated gigabit ethernet network
• A single VMware Distributed Switch stretched across both clusters (ok, didn’t know we could that either)
• Same IP subnet configured on both clusters for the VM to run
• Cisco DCI (Datacenter Interconnect) type technology – if you have something similiar by another vendor, you’ll be supported – this means that you should have a core network that can handle routing traffic to either location for the IP – the VM networks must be stretched between datacenters
• Datacenter storage should be R/W on both sides
• VMFS Storage is presented to both clusters – VM’s in each datacenter are run from the local storage LUNs.
• Must have less than 5 ms latency and at least 622 mbps bandwidth (OC12)
• No FT across sites!

Surprisingly, we have most of this configured in our environment and its been status-quo for us for several years.  The biggest difference between our environment and this spec configuration is that we run a stretched cluster to achieve this.  Our datacenters are very close to one another and we only present storage from our primary datacenter so that we don’t have a split-brain scenario.  But, it does give me new things to think about and talk about with co-workers.  We currently don’t run two clusters or SRM because we like the flexibility to VMotions between datacenters – with that now a possiblity, we may have something new to investigate…

Multiple VLANs on a single NIC
When using Virtual Connect, the default configuration sets VLAN tagging support to “Tunnel VLAN Tags.”  The mode is self-explanitory and just means that the “Ethernet Network” chosen and assigned to the server profile in Virtual Connect is the only network visible to the blade.  For most blade users, this setting works fine and many ESX deployments might be ok with this configuration.  But for many ESX deployments, people require multiple VLANs to be brought up on the same physical NIC.  The “Tunnel VLAN Tags” mode does not allow for this functionality.  

To allow for multiple VLANs on a single NIC, you must login to Virtual Connect, expand Ethernet Settings, select Advanced Settings and change the VLAN Tagging Support from Tunnel to “Map VLAN Tags.”  Map VLAN Tags exposes a Multiple VLANs option in the network assignment drop-down under your server profile.  Once you select Mutliple VLANs, a new window appears and you may select as many VLANs as you need exposed to the server.  The ESX host is then required to tag its traffic on these NICs.

Number of nodes per enclosure
Thou shalt not run more than 4 nodes of an ESX cluster in the same blade enclosure.   No, its not the 11th commandement, but it is an important rule to know.  Thanks to Duncan Epping’s article on the topic, we discovered a major implementation hazard for ESX on blade architecture that we didn’t have to experience first hand.  We did have our own enclosure failure, which made us aware that we could have been affected, however.  The pitfall is that HA has primary and secondary nodes in a cluster.  An ESX HA cluster can have up to five primary nodes, but never more.  The first five nodes in the HA cluster become primaries and these roles never get reassigned if a primary node fails.  The primary nodes are responsible for directing the HA activites.  So, you don’t want all your HA primary nodes running in the same enclosure.  If all five HA nodes are running in the same enclosure and it fails, you will not get the desired HA restarts on the other ESX nodes in your cluster.  Duncan’s article gives a great overview of the HA clustering architecture and sheds light on a little known consideration.  

Service Console & VMotion
Perhaps the favorite thing about Virtual Connect and ESX is the ability to creatively configure Service Console and VMotion using just two NICs, and providing redundancy and isolation as needed for these functions.  Lets look at the best practices:  

1. Service Console should have two NICs teamed for redundancy of this network link.
2. VMotion should have its own dedicated NIC for the best performance of VMotion traffic.

So, what makes Virtual Connect suit this well?  
First, Virtual Connect is redundant on the VC-Ethernet side.  We can create a single “Shared Uplink Set” with both Service Console and VMotion tagged traffic.  The stacking link between the two VC-Ethernet module will provide for traffic on NIC0 to be rerouted on Bay 2 if the uplink on Bay 1 is down.  As long as both VC-Ethernet modules are functioning, the stacking links would be utilized.  

Second, we can use ESX NIC Teaming failover settings to keep the traffic separate, except when a failure occurs.   If you lost a VC-Ethernet module, your Service Console and VMotion traffic would be failed over using ESX onto the same NIC on the other VC-Ethernet module.  

There are really a lot of options in this space and this is my high-level implementation.  I think its a great solution and can’t find many trade-offs for this.  In an blade environment where NICs are a premium, this is a wonderful solution. 

(By the way, I didn’t devise this SC/VMotion configuration – its something someone else posted, but I can’t find the original blog post to give you credit… If it was you, please let me know.)

Of course, my story has to begin on Friday the 13th…  Sometime around 9:00am, we started getting calls for both our SQL 2005 database cluster and our Exchange cluster.  After investigation, we found that the active nodes were both in the same enclosure and a third ESX host in the same was experiencing problems, too.  The problems were affecting both network and disk IO on the blades.  All of our blades are boot from SAN, so the IO had to be a fiber-channel issue.  

Several hours later, we were finally able to get enough response out of the nodes to be able to force a failover of services for Exchange, shortly followed by SQL 2005.  As I worked with HP support, nothing improved on the affected servers.  We were finally diagnosed with a problem mid-plane on the enclosure.  

 While waiting for the mid-plane to be dispatched to the field service folks, I requested that we go ahead and do a complete power-down on the enclosure and bring it up clean.  This required physically removing power from the enclosure after powering down everything that I could from the onboard administrator.  

After the reboot, everything looked much healthier.  The blades came back to life and everything began operating as expected.  After intense discussions on the HP side, we reseated our OA’s and the sleeve that they plug into on the back side of the enclosure.  Net outcome was the same – everything still operating well.  The OA’s nor the sleeve were loose, so we doubted that was the cause.  

One nugget I learned from HP support (please vett this information on your own), is that the Virtual Connect interconnect modules require communication with the onboard administrators (OA’s).  I’m still not sure I fully understand, but HP support did tell us that if VC lost communication to the OA, its possible that it caused our problems.  If this is so, this smells like very, very bad engineering and design…

Continued investigation on HP’s part has pointed us back to the original diagnosis – a faulty mid-plane.  Only by default did we return to that conculsion, however.  This is the only piece of hardware common to the problems.  Our only other conclusion was that this was a very bad, “hiccup” — which obviously buys us no real peace of mind…  

So, sometime soon, we will be replacing the mid-plane of our enclosure.   I have, of course, lost some faith in the HP blade ecosystem.  We have plans to migrate our corporate VMware cluster onto blades, as well as some Citrix and other servers.  Losing an enclosure like this has un-nerved those plans.  We were fortunate to have drug our feet  to only have 3 blades populated and serving anything at the time this happened.  I will post updates as we move forward…

That’s how my 2009 started… about 13 hours later, I finally left work and resumed my long-interrupted nap.   We had what seems to have been a storage meltdown behind our VMware farm yesterday.  Our file sharing cluster was also affected and so our few employees who were working on New Years Day, well, weren’t working at all.  The short version of the story goes like this.  Our scheduled backup process, using EMC Networker, kicked off VCB backups on the ESX 3.5 hosts around 1:30 am.  By 2:00 am, the process was trying to create snapshots on VMs and this caused some sort of meltdown due to SCSI reservations (found the SCSI reservation problem after VMware analysis).  Turns out the HP Insight Agents loaded on our VMware hosts were causing these SCSI reservation issues.  The agents were checking the disks at a consistent interval and we had not upgraded the agents to the latest revision, which was supported with ESX 3.5 – so not VMware’s fault – they have a great KB article about this issue (see KB 1005009).   As an immediate resolution, one of my co-workers removed the HP agents from our hosts and worked our way through rebooting the entire farm, one host at a time to remove the SCSI reservations.  I cross my fingers on VCB backups working when they kick off in an hour.  Had this been the only issue, we would have been fine.

Unfortunately, at around 4:30 a.m., while I was unaware, our cluster began experiencing troubles, too.  And this is where our detective skills have come up short.  We have been sleuthing to find the cause of some weirdness in both our file sharing and Exchange clusters for several weeks, now.  The file share cluster is, dumbly enough, critical in our environment.  Without it, our users home directories are inaccessible and, since these home directories are defined in Active Directory, it seemingly hoses up our employee’s workstations.  Things that should otherwise be speedy, say opening a program — any program — or browsing to your local hard drive, become unbearably slow.  Even running applications sometimes lock up as it attempts to access some unknown part of Windows during normal operation.  It brings our entire business operation to crawl and that’s unacceptable.  (BTW, if all this sounds familiar, please leave comments or send me an email with suggestions.)

So, what actually happened to our Windows file sharing cluster?  We have an issue where we see the network utilization on the file share cluster drop to nothing, but the cluster nodes still respond to ping and other non-storage related network services – but what we found out later in the process – not to anything which needed IO to respond.  After repeated network sniffs, we were seeing that traffic would come to the cluster, it would be acknowledge, but the node would not start sending data.  The break between the request and data could be as long as 20 or 30 seconds.  And that was consistent with our ‘outage’ periods.  So, I decided to fail over the cluster shares from one node to the node that had been ‘solving’ the problems in the past.  When attempting to fail the share, they locked and never became accessible again.  After waiting for almost an hour, I rebooted a node trying to clear up the locks and let the other node take control.  That was never possible either.  A reboot of the second node only served to cause it to stall during boot, and never provide me a login screen.  Rebooting the other node, same result.  And then, a lengthy phone call with HP support after driving into the office.

The short version of this is that we are running Windows Server 2003 with SP2.  Apparently, therein lies our problems with a) clustering and b) storport.sys.  The StorPort driver issues are pretty well documented and it in combination with several other hotfixes are HP’s recommendation to us.  The hotfixes were released outside of Microsoft’s normal patch schedule to the large number of customers having issues similiar to this.  HP’s recommendation to us was to install the list of suggested hotfixes for post-SP2 (Microsoft KB 935640).  My co-workers successfully completed that on the file share cluster this evening without incident.  (Hallelujia!)  I got the call shortly around 10:15 pm with the all clear.

The file sharing problem has been plauging us for several weeks and we have not been able to deduct the full cause.  We have had theories and as soon as we seem to figure it out or know how it will act, we’re proven wrong.  At least until yesterday, we hope.  The next week will tell for sure.

I also mentioned issues on our Exchange cluster.  We’re not actually sure its having issues.  It may only be showing issues at the same time the file share cluster was having issues.  We believe the above issues on file shares are causing lock ups on client machines and their programs, so we are currently thinking Exchange’s perceived network issues are just the fact that our employee’s Outlook is locked and can’t connect, so we see what looks like Exchange issues.  But, then again, we’re not 100% sure.  We still have some detective work to do here.  Where’s Sherlock Holmes when you need him?

As for VMware, we still have a punch list for a few additional things to do – putting newer HP agents on the ESX hosts for one.  There are some things we may want to customize here based on feedback from VMware.  We may want to disable the storage monitoring agent on these hosts, but more research is required.  Removing these agents all together for now is a our preferred fix.  So, for the next hour, I just need to keep myself awake and hope that VCB backups will go well tonight.