In the latest realignment, HP is consolidating these release sets with its Proliant Support Packs (PSP’s) and the Firmware releases that were previously three separate distributions.  In addition, HP is adopting a date style of numbering, matching the release set scheme, rather than the version numbering previously used with PSP and Firmware releases.  This will allow administrators to quickly recognize how behind a system is from the current release.

Consolidation also allows a single interface, qualification schedule and unified release for customers on all systems.  Although a solid step, the release sets did not completely match the release of Firmware DVD’s from HP and sometimes there were issues with support where case workers asked that you update firmware out of compliance with a release set.  During their first year of existence, release sets came out often and in many ways, shored up stability in our blade environment, but in the past year, they seemed to have lagged behind in releases.

Back when I first covered release sets, I also noted that it the release sets were a very good thing for customers, since the sets gave a supportable configuration and schedule of compliance.  Although I didn’t say it, compliance meant that the customer could finally force the support side of HP to work a case instead of hiding behind the ‘please patch to latest firmware’ excuse.  My hope is that the SPP will do the same in a unified firmware and driver software release.

On many of our critical systems, we have observed an increased interdependency between the firmware versions and the driver versions which need to be loaded together to gain stability.  Consolidating all these separate releases into a single engine and distribution makes logical sense as these dependencies increase.

In an effort to target exactly what customers need, there will be a master distribution of the SPP along with 6 additional subset versions targeted towards specific needs, per the HP website.    These will be smaller, which translates to faster downloads and a more customized installation, including only what is required for these solutions.

HP touts four major areas of improvement by introducing the SPP — it reduces customer qualification cycles, reduces resource usage, maintenance windows and downtime.  Since firmware patching is typically not the most fun part of a system administrators job, reducing the frequency of these upgrades is a welcome change.

HP plans to offer 3 to 5 of these SPP’s per year.  The first version released is version 2011.09.0 and 2012.01.0 is expected soon.

Unfortunately, when multiple vendors’ products are coupled together as a solution, support can become nasty as vendors point back and forth at each other while trying to get to a resolution.  The more complex the solution, for instance a SAN, the more difficult to troubleshoot through the multiple layers of software, firmware and hardware, even multiple vendors of the solution.  And, I believe, the hassle has made customers seek a better way.

Finding a better way

In my employer’s case, they chose to standardize with a single vendor long before I joined the staff.   We have stuck with servers and storage hardware from the single vendor, including their certified part upgrades (no third party upgrade components).  We chose to do this to simplify our support and avoid finger-pointing.

The vendor we standardized with was HP, and the reason was that they offered an entire line of products under their umbrella to meet our needs.  By the time I joined the staff in 2006, we were already HP heavy, except where a specific Unix was required by another vendor.   What we wanted as a customer was the quickest and easiest route  to a resolution, with the least resistance and finger-pointing, when a problem came up.  Even beyond the hardware solutions, HP has handled our software support for Microsoft, RedHat and VMware for many years.  We wanted this because the software companies could not finger point at the hardware or vice versa – HP was doing it all.  Sure, it might happen between teams in HP occasionally, but we could easily escalate our case and have a manager bring this to a resolution.  It has worked well for our needs.

Having all this expertise in-house is an advantage that HP is now branding under the name “Converged Systems” or the “Instant-On Enterprise”.  Earlier this week, I attended a webinar for the Blogger Reality Contest where HP unpacked more of its converged solutions strategies.  HP is bringing together all of the pieces spread throughout its portfolio into specialized solutions.  Its not a new concept, in my opinion, but one that some customers have been already using for years on their own.  HP has improved on this by tweaking configurations  to squeeze performance out of configurations and adding software to ease installation and management of the solutions.

Building Upwards – HP VirtualSystem

HP introduced VirtualSystem in June as a modular, easy and quick way to implement virtualization in customer datacenters.  The VirtualSystem solution is a full package of storage and compute resources plus the software tools to quickly and easily deploy a virtual stack in an environment.

For HP VirtualSystem, the key benefits are:

• Quick built out timeframe
• Automation through Insight Control suite components
• Monitoring through the Insight Dynamics suite components
• Improved virtual machine performance, cost and scale due to purpose built hardware
• Ability to upgrade to CloudSystem for fully automated IT
• Single point of contact for support – HP for compute, storage and software, including hypervisor

HP VirtualSystem comes in 3 levels (shown below).  The VS1 is built out using rack-mount, Proliant hardware for both the server hosts and for the storage and features a P4000 series iSCSI storage array.  It is rated to handle up to 750 virtual machines and can scale up to 8 physical hosts.  The VS2 is built out using HP BladeSystem with a P4800 iSCSI storage array (covered in depth last week).  It is rated for up to 2500 virtual machines and can scale up to 24 physical hosts.  The third offering is the VS3 which is built on HP BladeSystem and the 3PAR Utility Storage platform to provide ultimate scale and performance.  VS3 introduces fiber channel storage capability and scales up to 6000 virtual machines with up to 64 hosts.

In terms of choice, VirtualSystem supports all three major hypervisors from VMware, Microsoft and Citrix.  Using my company as an example again, the multi-hypervisor datacenter already exists.  We are utilizing VMware vSphere heavily and then some Citrix XenServer.  When it came to planning upgrades for our aging MetaFrame/XenApp farm, we looked at virtualization.  As we evaluated XenServer, we found it to be “good enough” for running Citrix XenApp on top of it.  XenApp has its own failover and redundancy built into the application layer, so many of the VMware advanced features did not matter.

For VirtualSystem, HP is also handling all support for both the hardware and software for these solutions.  Having experience with HP’s software support teams, I can report that they do a good job at it.  I would not say they are always perfect, but in general, they have solved our issues and advised us well, so in reality this is a big benefit.  For those who want not on break/fix support, HP offers Proactive 24 Services for an additional level of preventative support.

Building to the cloud – HP CloudSystem

As I learned at HP Discover, just because you have a large virtualization pool in your datacenter does not mean you have a private “cloud.”  The critical difference between a virtual infrastructure and a cloud is orchestration and automation.  Built on top of HP VirtualSystem, HP CloudSystem is a solution that offers all of the necessary orchestration, service catalog and workflows to turn virtual infrastructure into a true cloud.  There is a clear and clean upgrade path from VirtualSystem into CloudSystem.  And for those starting fresh or who want to evaluate the HP solution, there is even an HP CloudStart service which will deliver a rack with CloudSystem into their datacenter and have it fully operational in 30 days or less.

CloudSystem is offered in three levels: CloudSystem Matrix, CloudSystem Enterprise and CloudSystem Service Provider.  CloudSystem Matrix is targeted towards those looking to automate the private cloud, customers who are looking to add automation and orchestration to their existing virtual systems.  It provides infrastructure as a service (IaaS) and basic application provisioning in minutes.  CloudSystem Enterprise extends upon Matrix and allows for private and hybrid cloud, enabling the bursting of workloads to public cloud.  It is a platform for hosting not only IaaS, but Platform as a Service (Paas) and Software as a Service (SaaS).  CloudSystem Enterprise provides application and infrastructure lifecycle management and allows for management of traditional IT resources in addition to virtualized resources.   The CloudSystem Service Provider edition extends upon the Enterprise edition and allows for multiple tenants on a single infrastructure, securely without exposing customer data between customers.  It is intended to host public and hosted private clouds for customers.  The editions in CloudSystem are more about capabilities and less about limits, compared to VirtualSystem.

Since automation and orchestration is the key of CloudSystem, that is where I wanted to focus.  The base of CloudSystem is the Matrix Operating System, which is the same combination of HP software found in the HP VirtualSystem solution.  On top of the Matrix Operating System, the CloudSystem Matrix solution includes Cloud Service Automation for Matrix.  This software includes Server Automation for lifecycle management for physical and virtual assets via a single portal and set of processes and HP SiteScope, an agent-less monitoring solution for performance and availability.

The enterprise and service provider editions include a beefed up version called, simply, Cloud Service Automation.  It includes the entire orchestration, database and middleware automation pieces of the pie and a cloud controller software.  These additional pieces allow not only the automatic and streamlined provisioning of physical and virtual servers but also the provisioning of the required glue that sits in between the apps and the servers.  The diagram below from HP shows all the moving parts of Cloud Service Automation better than I can explain in words.  And because, Cloud Service Automation is total lifecycle management, there are the pieces for monitoring and performance management which would be needed.  In addition, the centralized portals serve as point for both end users and IT professionals to manage the cloud.

Cloud Maps are another feature of CloudSystem and these are predefined automation workflows for deploying software and platforms easily.  These are the piece of the puzzle that allows for improved deployment times and also allow for drag and drop creation of new workflows and processes in the cloud.  HP has worked with its software partners to create these maps of requirements and automate the process of deploying their solutions.

Beyond all of the capabilities, HP is working hard to make this an open solution by making it compatible to burst workloads into third party clouds, whether its Amazon’s EC3 or a vCloud service provider.  This was a point stressed during the announcements at HP Discover and during the call on Tuesday.

This is post number two for Thomas Jones’ Blogger Reality Show sponsored by HP and Ivy Worldwide. I ask that readers be as engaged and responsive as possible during this contest.  I would like to see comments and conversations that these entries spark, tweets and retweets if it interests you and I also request that you vote for this entry using the thumbs up/thumbs at the top of this page.  As I said earlier, our readers play a large part in scoring, so participate in my blog and all the others!

This isn’t the first time I’ve written about CloudSystem.  In June,  I posted about my take on CloudSystem Service Provider from a potential service provider’s perspective.  I encourage you to take a look at that post, too, after you take a minute to comment and/or vote on this post.

I worked with HP to deploy two HP BladeSystem C7000 chassis in my work datacenters about three years ago.  Our group has two enclosures in to separate datacenters which are mirrors of each other in configuration.  At first, we only ran Windows clusters on the BladeSystems with one node in each enclosure.  We considered this a safe way to get familiar with the technology, management and understand the reliability of the BladeSystem in our environment.  We, as others have, did find the enclosures to be very reliable, eventually reconfiguring them to add Virtual Connect Flex-10 and moving some of our ESX onto BladeSystem.  Today, HP BladeSystem runs a very large percentage of our overall infrastructure.

Some of the BladeSystem advantages we found are:

• Reduced energy costs and better energy management by collapsing multiple servers into a single enclosure
• Reduced footprint in the datacenter and increased density
• Reduced cabling and no need to re-cable enclosures if using the VirtualConnect technology (wire once)
• Condensed space required in the datacenter – in a C7000 chassis, up to 16 servers in 10U of space.  In the past, 16 servers would require a minimum of 16U (1U servers) and worst case of up to 64U (4U servers) in the rack
• Allows users to mix and match workloads and types of servers while keeping management tools the same

Just as with the Converged Storage solution I covered last week, it is a small incremental cost to scale up, after the initial chassis investment.  The incremental cost to add a server is much smaller than buying an entire rack mount server.  And we reap the benefits of buying the newest, fastest servers with the most memory possible as those limits increase and we only buy the blades as we need them.

It appears that management was a primary focus when HP created the BladeSystems.  The OA or Onboard Administrator is the primary resource for configuring, managing and monitoring the chassis.  From the OA, you can manage the system IP addresses, configuration of the blades and interconnect bays and launch into other management interfaces.  The standard iLO or Integrated Lights Out management processor, which is a separate processor found on all Proliant servers (rackmount or blade) is the individual server blade administration point and it is very well integrated into the OA.  For those customers who choose HP’s Virtual Connect technology, you can launch into the Virtual Connect Administration panel from within the OA, too.  The VC administrator is where you configure virtualized MAC address and Fiber Channel WWID assignments which allows you to create a network and SAN profile for a particular server which contains virtual addresses assigned to the blade.  If the blade has a hardware problem, the profile can be reassigned and the blade can be booted to another blade, if one is available.  In addition, Insight Control is available which allows for enhanced automation of deployments and management.

The only downside I can report is that we have found firmware upgrades to be fairly invasive and disruptive due to the amount of infrastructure that has been condensed into a single enclosure.  Because of this, we have designed the workloads on the BladeSystem so that we can fail-over workloads from on enclosure to the other with minimal disruption.  We use Microsoft Failover Clusters, vSphere (which allows vMotion of workloads), and XenApp which allows us to put hosts in maintenance mode and move users off of them.  The firmware process has improved since HP introduced the concept of release sets for BladeSystem firmware last year, but it still takes some coordination on our part.

HP BladeSystem is the market leader in blades according to IDC.  It has consistently held the number one spot in blade market share quarter after quarter, with 50% of the blade servers shipped during Q1 2011.  It is not just market fluff or creative interpretation when HP says they are the leader in blades.

The HP BladeSystem has been a rock solid solution for our needs.  I commented last week to my boss that by far our most capable servers in the datacenter are in our blade centers.  Where blade computing began and is today is a huge leap.  There is almost nothing that cannot fit into a blade server.

Converge.  That theme is a tougher sell in my employer’s environment.  There is resistance to converging, whether it’s IP telephony on our data network versus converging our fiber channel with fiber channel over Ethernet and putting it on our same core Ethernet network.  Same would go for iSCSI, if we had it.  We tend to separate for simplicity reasons, but there are certainly cost savings in convergence.

 Why converge?

Convergence is a major trend in IT, today, although it goes by many names.  But like most trends and buzzwords (think Cloud), your mileage will vary among vendors and interpretations of the buzz.  HP’s approach to convergence is largely centered around standardized x86 hardware for both server and storage platforms.  In addition, the converged storage platforms within HP are about scale out, with multiple controllers to handle unpredictable and unruly disk I/O with ease.

Before moving into a discussion of converged storage, though, it is worth taking a moment to talk about how things have been done in the past.  For the past 20 years, storage has been largely created around a monolithic model.  This model consisted of dual controllers and shelves of JBOD’s for capacity.  The entire workload and orchestration of the array was trusted to the controllers.  With the traditional workloads, the controllers performed well.  In the old world, capacity was the limitation on data arrays.

Today, the demands of virtualization and cloud architectures on storage have considerably changed the workloads.  The I/O is unpredictable and burst large amounts of traffic to the arrays.  This is not what our traditional arrays were designed for and the controllers were paying the price.  In a large number cases, including my company, the controllers become oversubscribed before the capacity of the disks are exhausted so you don’t realize your full investment. Monolithic arrays come with a high up-front price tag.  When one is “full”, it is a big hassle and cost to bring in a new array and migrate. But these have been the work horses of our IT operations.  They are trusted.

Hitting the wall

Within the past couple years, I have found the limitation of the controllers to be a significant problem within our environment.  And even after significant upgrades to high-end HP EVA within our company, we can still see times when the disk I/O overwhelms our controllers to a point that disk latency increases and response slows.

The controller pain points are one of the driving forces behind converged storage.  Converged storage is the “ability to provide a single infrastructure, that provides with server storage and networking and rack that in a common management framework,” says Lee Johns, HP’s Director of Converged Storage.  “It enables a much simpler delivery of IT.”

What is different with converged storage?

Across the board, convergence leverages standardized, commodity hardware to lower costs and improve the ability to scale out.  Converged storage is about taking those same x86 servers and creating a SAN that can adapt to the demands of today’s cloud and virtualization.  Instead of the limitations of a single set of controllers, intelligently clustered server nodes enables each server in the array to serve as a controller.

Through distribution of control, the cluster is able to handle the bursts of I/O more easily across all of its members than a monolithic array controller is able.  The software becomes the major player in the array operations and it really is a paradigm shift for storage administrators.  No longer is storage a basic building block, it is just another application running on x86 hardware.

Diving deep into the HP P4800 G2 SAN solution

Perhaps the best way to understand converged storage is to look at a highly evolved converged data array.  On Tuesday, Dale Degen, the worldwide product manager for the HP P4000 arrays, introduced our blogger crew to the P4800 G2, built on HP’s C7000 Bladesystem chassis.

The core of the LeftHand Networks and now P4000 series arrays is the SAN/iQ software.  The SAN/iQ takes individual storage blade servers and clusters them into an array of controllers. This clustering allows for scale out as you need additional processing ability to handle the workload.  Each of the storage blades is connected to its own MDS-600 disk enclosure via a SATA switch on the interconnect bays of the blade center.  The individual nodes of the array mirror and spread the data over the entire environment.  One of the best things about the SAN/iQ software is its ability to replicate to a different datacenter and handle seamless failover if one site is lost.  (Today, in my fiber channel world, if I lose an array, it involves presentation changes to bring up my replica from another EVA, so this is a huge plus.)

By leveraging the HP Bladesystem for the P4800 G2, you can also leverage its native abilities, such as the shared 10Gb Ethernet interconnects and Flex-10.  For blades in the same chassis with the P4800, the iSCSI traffic never has to leave the enclosure and it is allowed to flow at speeds up to 10Gb (unless you have split your connection into multiple NICs).

From an administrative standpoint, the P4800 is managed just like any other blade server in the C7000 enclosure.  These blades are standard servers, except that they include the SATA interface cards.  They include standard features like iLO (Integrated Lights Out) management, VirtualConnect for Ethernet network configuration, and the Onboard Administrator for overall blade health and management.

Within a single chassis, the P4800 can scale up to 8 storage blades (half of the chassis).  The iSCSI SAN is not limited to presentation within the same C7000 or within the BladeSystems at all.  It is a standard iSCSI SAN which can be presented to any iSCSI capable server in the datacenter.

The P4800 G2 is available in two ways.  For customers new to the C7000, they may purchase a factory integrated P4800 G2 and C7000 chassis together.  For existing customers with a C7000 and available blade slots, the P4800 G2 can be integrated with the purchase of blades, SATA interconnects and one or more MDS-600 disk enclosures.  For existing customers, you must also purchase the installation services for the P4800 G2.

The P4800 is a scale up technology also.   Customers do not need to migrate everything at one time.  It allows for a single infrastructure and allows you to move onto it over time by adding additional storage blades and MDS-600 disk enclosures.

As a quick side note, this is the first entry for Thomas Jones’ Blogger Reality Show sponsored by HP and Ivy Worldwide. I ask that readers be as engaged and responsive as possible during this contest.  I would like to see comments and conversations that these entries spark, tweets and retweets if it interests you and I also request that you vote for this entry using the thumbs up/thumbs at the top of this page.  As I said earlier, our readers play a large part in scoring, so participate in my blog and all the others!

HP added the idea of release sets for Bladesystem firmware starting in January.  On a conference call yesterday, we were alerted to the new release set certification process.  Previously, HP had been releasing firmware for the Bladesystem as components and firmware were updated, which they still do, but the idea of a release set adds an additional cross-testing process to ensure that firmware from each component works together correctly.  There was no publicly disclosed certification process prior to January.

To quote the HP engineer on the call — there were just too many combinations and possibilities to be able to certify all available firmwares — and so the idea of a release set began in January.  The release sets are available in a compatibility matrix on the HP Bladesystem page (http://h18004.www1.hp.com/products/blades/components/c-class.html) — look at the Compatibility tab.

The good news is that in our environment, we are close to compliance with the January release set.  The only thing out of compliance for the January release is our blade ROM, PMC and iLO firmware, but, as we were also informed, this is not a good situation to be in.

Something I already knew is that we have a bottom-up firmware topology for the HP Bladesystem — meaning that we have to update the bottom level components first and then move up.  I knew this applied to the interconnects and onboard administrator modules, but it also extends to blade servers also.   In addition, the blade servers are at the lowest level and should be updated first — particularly the iLO.

The reason for this is that new OA and interconnect firmware may introduce features and if the PMC or iLO is not aware of these features due to out-dated firmware, erroneous might be passed and all sorts of things could happen — at worst, random server reboots.

iLO firmware 1.81 is particularly susceptible to the random reboot for Windows blades.  If you are running 1.81 and have Windows OS loaded, you should upgrade to 1.82 as soon as possible.  See http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?locale=en_US&objectID=c01802766 for more information.  Other OSes – Linux, VMware, XenServer – are not affected.

http://h18004.www1.hp.com/products/servers/service_packs/en/index.html

HP Virtual Connect Flex-10 allows an administrator to take a physical 10G connection and partition it into up to 4 more appropriately sized logical NICs presented to a server blade.  Virtual Connect Flex-10 incorporates all of the benefits of Virtual Connect (previously explained here) while allowing for additional logical NICs.  A Flex-10 Ethernet interconnect module with two onboard Flex-10 NICs on a blade will allow for 8 total NICs – something formerly only available if all 8 interconnect bays and the two mezzanine slots were populated with Ethernet modules.  This greatly reduces the amount of cabling required at the blade enclosure and condenses the amount of connections required.

The Flex-10 interconnect modules include SFP connectors which are capable of copper or fiber connections depending on the GBIC installed.

Upgrade Process
We expected going into the installation that we could need to recreate the Virtual Connect domain (a domain is the configuration, server profiles and settings for a Virtual Connect deployment on a specific enclosure or set of enclosures).  Since we only had 5 server blades installed and operating, we weren’t overly concerned.  Our primary concerns were to ensure that the same Virtual Connect MAC and WWID settings were reinstalled and that our recreated server profiles matched their MAC address and WWID settings from the prior installation.  So, to ensure we had all the information, we made printouts of the server profiles, Ethernet networks and fiber channel configuration and then proceeded to installing the modules.

We shutdown all of the existing blades from service.  We removed the existing VC-Ethernet modules from Bays 1 and 2 and we installed the new VC-Ethernet Flex-10 modules in these bays.  The VC-Eth modules in Bays 1 and 2 contain the domain configuration.  Once removed, the domain had to be completely recreated.  The new modules were at a higher firmware (2.12) versus the original modules (2.01).

What surprised us during installation was that we were able to successfully restore our original Virtual Connect domain configuration onto the Flex-10 modules.  Once restored, we were able to see all configuration and the only thing showing an error were the Ethernet networks and shared uplink sets.  These didn’t have any active ports assigned to the uplinks.  The new VC-Eth Flex 10 modules have a different port numbering to differentiate from the original VC-Eth modules.  After reassigning ports to the uplink sets, all of the Ethernet networks appeared to be back online.  The firmware differences didn’t appear to be a problem after initial restore.

After booting servers onto the enclosure (all are boot-from-SAN and booted fine), we determined that we were having some network connectivity issues.  After troubleshooting, we upgraded VC firmware on the modules.  This proceeded without problems, but upon reboot, the VC-Ethernet Flex-10 modules would not come back online.  After troubleshooting, we installed upgraded firmware on the enclosure.  The newer firmware on all seemed to resolve all issues and the enclosure was back online.  Final time to upgrade was around 5 hours, but much of that was waiting on Virtual Connect modules to reboot and wondering why they wouldn’t come back online.

Our second enclosure was expected to be a simple slam dunk after our largely successful and simple upgrade procedure — especially since we knew about firmware before on this round, but unexpected problems were there.  This enclosure had a mid-plane replacement early in its life and apparently there was confusion with the Virtual Connect domain and the enclosure serial number.  We received errors during restore and even after we forced the restore and upgraded firmware, the Ethernet networks never came back and responded.

This forced us into a recreate scenario.  After recreating this VC domain by hand, everything came online and worked as wanted.  I still believe that this is the cleaner way to handle the upgrade, although the restore worked on the first enclosure and presented very few problems.

Next…  Configuring Flex-10 FlexNICs

After a month of continued support and working the case, we escalated the case to a level where an engineer reviewed all the steps, troubleshooting, and case information to ensure nothing had been missed and to help diagnose the issue.  He came back to the original conclusion – that after all else was eliminated, the mid-plane must be the culprit.  So, we scheduled the replacement. 

The actual hardware replacement went smoothly and took less than an hour to complete.  The midplane is a lot more bulky the I expected when I first saw it.  It is a single piece of hardware with interconnects on both sides that connect blades to interconnect bays, power sources to power consumers and LCD display to the logic.  But, I guess I was surprised that it was a good 2 to 3 inches thick.  In my mind, I expected a single piece of copper sitting in the middle – yes I realize now that’s stupid.  

Something interesting occurred after replacing the mid-plane.  Apparently, Virtual Connect did not see the system serial number that it expected and so it reverted to its default configuration.  So, word of advice to anyone replacing a mid-plane.  Leave your VC modules ejected so that you don’t lose your domain configuration.  From talking with support, Virtual Connect needs constant communication with the OA to function (another dependancy we were not aware of).  The serial number stored reported by the OA from the enclosure is also very important to VC.  Its part of the configuration file for VC.  It all makes logical sense, but it was not spelled out in the support document detailing the mid-plane replacement.  

After unsuccessfully attempting to restore my backup for the Virtual Connect domain, I opted to build it from scratch by hand.  It took about an hour and half to do for my 5 blades, but I feel better about it.  I am still worried about not being able to restore my VC domain configuration, but I attribute that problem to the hardware replacement.

Multiple VLANs on a single NIC
When using Virtual Connect, the default configuration sets VLAN tagging support to “Tunnel VLAN Tags.”  The mode is self-explanitory and just means that the “Ethernet Network” chosen and assigned to the server profile in Virtual Connect is the only network visible to the blade.  For most blade users, this setting works fine and many ESX deployments might be ok with this configuration.  But for many ESX deployments, people require multiple VLANs to be brought up on the same physical NIC.  The “Tunnel VLAN Tags” mode does not allow for this functionality.  

To allow for multiple VLANs on a single NIC, you must login to Virtual Connect, expand Ethernet Settings, select Advanced Settings and change the VLAN Tagging Support from Tunnel to “Map VLAN Tags.”  Map VLAN Tags exposes a Multiple VLANs option in the network assignment drop-down under your server profile.  Once you select Mutliple VLANs, a new window appears and you may select as many VLANs as you need exposed to the server.  The ESX host is then required to tag its traffic on these NICs.

Number of nodes per enclosure
Thou shalt not run more than 4 nodes of an ESX cluster in the same blade enclosure.   No, its not the 11th commandement, but it is an important rule to know.  Thanks to Duncan Epping’s article on the topic, we discovered a major implementation hazard for ESX on blade architecture that we didn’t have to experience first hand.  We did have our own enclosure failure, which made us aware that we could have been affected, however.  The pitfall is that HA has primary and secondary nodes in a cluster.  An ESX HA cluster can have up to five primary nodes, but never more.  The first five nodes in the HA cluster become primaries and these roles never get reassigned if a primary node fails.  The primary nodes are responsible for directing the HA activites.  So, you don’t want all your HA primary nodes running in the same enclosure.  If all five HA nodes are running in the same enclosure and it fails, you will not get the desired HA restarts on the other ESX nodes in your cluster.  Duncan’s article gives a great overview of the HA clustering architecture and sheds light on a little known consideration.  

Service Console & VMotion
Perhaps the favorite thing about Virtual Connect and ESX is the ability to creatively configure Service Console and VMotion using just two NICs, and providing redundancy and isolation as needed for these functions.  Lets look at the best practices:  

1. Service Console should have two NICs teamed for redundancy of this network link.
2. VMotion should have its own dedicated NIC for the best performance of VMotion traffic.

So, what makes Virtual Connect suit this well?  
First, Virtual Connect is redundant on the VC-Ethernet side.  We can create a single “Shared Uplink Set” with both Service Console and VMotion tagged traffic.  The stacking link between the two VC-Ethernet module will provide for traffic on NIC0 to be rerouted on Bay 2 if the uplink on Bay 1 is down.  As long as both VC-Ethernet modules are functioning, the stacking links would be utilized.  

Second, we can use ESX NIC Teaming failover settings to keep the traffic separate, except when a failure occurs.   If you lost a VC-Ethernet module, your Service Console and VMotion traffic would be failed over using ESX onto the same NIC on the other VC-Ethernet module.  

There are really a lot of options in this space and this is my high-level implementation.  I think its a great solution and can’t find many trade-offs for this.  In an blade environment where NICs are a premium, this is a wonderful solution. 

(By the way, I didn’t devise this SC/VMotion configuration – its something someone else posted, but I can’t find the original blog post to give you credit… If it was you, please let me know.)

Back in Janauary, I talked a bit about our VMware View conference call.  We evaluated and chose to implement VMware View  to support the purchase of 50 Pano Logic thin client devices that we purchased end-of-year.  The VMware View licensing model made more economic sense than purchasing ESX licenses outright to deploy virtual desktops.  Our other option was to deploy these desktops onto our existing ESX clusters, but I wasn’t a big fan of this.  I like keeping things silo-ed, at times.  

Making the decision to virtualize desktops
When making the decision to move towards virtual desktops, we decided to make a baby step (not a head-first leap).  We identified a strong business use for virtual desktops in our business operation.  That case is our central offices.  Being a telephone company, we have central offices (CO’s) all throughout our geographic area.  These CO’s house our telephone equipment, jumpers, and now fiber and network equipment as our plant becomes more sophisticated.   Each one of these locations has traditionally had a desktop PC at them.  Maintenance for these PC’s requires sending one of our 3 PC technicians to remote parts of our coverage area to work on these PC’s.  

We felt that the thinnest, dumbest possible terminal at that location – with no moving parts or software – was our best choice for device.  Last May, several of us attended the Charlotte, NC, VMware Users Group meeting.  Pano Logic was there and demo’d their device to us at that time.  I don’t know of another time when we’ve all come to a unanimous opinion of a technology so quickly.  All three of our group was impressed.  Fortunately for us, our CIO also came to the opinion that we should be investigating virtualized desktops, so so our search began.  

Ultimately, after looking at many thin clients, we made the decision to go with the Pano Logic units that initially sparked our interest.  While many of the other thin clients we evaluated were very thin, none were really zero software like the Pano solution.  And that was a selling point to us.  Zero to administer, patch or upgrade at the end-point.  And for the remote locations we’re talking about using these devices, the fewer trips down the remote dirt roads, the better.  

We need VMware, but View is not required
Now that we’d chosen our device, we knew that we’d be sticking with the VMware ESX that we know well and are comfortable with.  From my initial research from a year ago, I knew that the VDI licensing model would probably be a good fit, but VMware had released View in the interim.  

So, we got VMware and our partner on the phone to talk more about the additions, changes and pricing.  The View starter packs come with vCenter Foundations license (which can manage up to 3 ESX hosts, I believe), an ESX 2-socket license and the View desktop licenses.  The VDI model brings you a big price break on the ESX servers bundled, but comes with a license restriction that you can only run desktop OS virtual machines on those servers.  You should create it as a sepearate cluster.  In the end, this was a better choice for us since we’d be running no more than 20 or 30 virtual desktops in our deployment.  We saved initially.  This isn’t always the case.  Had we jumped head-first and planned a deployment of several hundred devices, ESX licenses might have been a better choice with Pano devices.  It depends on how much density you can get onto a single ESX host. 

VMware View comes in two flavors.  There is an enterprise and premium.  We chose the premium package that afforded us the linked-clone technology, View Composer, and ThinApp – among other additions.  The enterprise edition is largely the solution offered as the VDI 2 product and the premium brings all the new advanced features.  

We planned to make use of the pooled options for our CO desktops.  All of our installers and CO personnel have been deployed laptops in the past year as they’ve assumed new duties installing cable modems and DSL for customers.  We believe that the CO desktops are not being used often, now.   So, we hope to be able to deploy about 50 PanoLogic devices and only need 15 to 20 virtual desktops to serve those.  

Our Implementation
We decided to silo the desktops into their own ESX cluster – just two nodes – but also majorly overkill for just 20 virtual desktops.  But this allows us to plan for the future also.  We can incrementally step up the number of View licenses as needed.

From our proof-of-concept deployment, we knew that the Pano deployment was largely just a matter of importing the Pano Manager virtual appliance and configuring the right DHCP options.  We implemented these on one of our production VLAN’s and we were in business.  The Pano devices came to life.

The View deployment went smoothly, also.  There are relatively few pieces to install – just the View Composer service to install on the vCenter server and importing the VMware View Manager virtual appliance.  We are running both of these on our corporate ESX farm.   

The configuration piece between Pano Manager and VMware View had a few gotcha’s.  For the most part, Pano Logic’s documentation gave us just what we needed, but there were a few small changes with the new VMware View version (thanks to Jason for providing these).  After configuring the connection to View Manager (VDM), when you click configure, you don’t get a success message like you do when configuring the directory settings.  The status also doesn’t show connected, although the settings are working and integration is setup.   Pano Logic support assured us they were working on that for the next release.   The Pano Manager works a little differently, too, after successful configuration.  With VDI, the Pano Manager automatically imported all the virtual desktop machines.  With Pano Manager connected to View Manager, the desktops are only shown in Pano Manager after being connected for the first time.

To make use of the linked-clone technology,  you cannot control the provisioning of VM’s from Pano Manager.  You have to do this using View Manager.  Its not a big deal, since simliar functionality already exists in View Manager, but seemeless integration between the two would be nice.  

Next Steps
As next steps, we’re continuing to evaluate other technologies, devices and solutions that might better fit in other areas of our business.  For training rooms, we may need a different solution than the Pano Logic one for CO’s.  We maintain a call center, information workers, customer service reps, etc.   So, wer’e not looking at a one size fits all.  We are continuing to look at what fits best for us.  And, as always,  I’ll keep you posted.   

Of course, my story has to begin on Friday the 13th…  Sometime around 9:00am, we started getting calls for both our SQL 2005 database cluster and our Exchange cluster.  After investigation, we found that the active nodes were both in the same enclosure and a third ESX host in the same was experiencing problems, too.  The problems were affecting both network and disk IO on the blades.  All of our blades are boot from SAN, so the IO had to be a fiber-channel issue.  

Several hours later, we were finally able to get enough response out of the nodes to be able to force a failover of services for Exchange, shortly followed by SQL 2005.  As I worked with HP support, nothing improved on the affected servers.  We were finally diagnosed with a problem mid-plane on the enclosure.  

 While waiting for the mid-plane to be dispatched to the field service folks, I requested that we go ahead and do a complete power-down on the enclosure and bring it up clean.  This required physically removing power from the enclosure after powering down everything that I could from the onboard administrator.  

After the reboot, everything looked much healthier.  The blades came back to life and everything began operating as expected.  After intense discussions on the HP side, we reseated our OA’s and the sleeve that they plug into on the back side of the enclosure.  Net outcome was the same – everything still operating well.  The OA’s nor the sleeve were loose, so we doubted that was the cause.  

One nugget I learned from HP support (please vett this information on your own), is that the Virtual Connect interconnect modules require communication with the onboard administrators (OA’s).  I’m still not sure I fully understand, but HP support did tell us that if VC lost communication to the OA, its possible that it caused our problems.  If this is so, this smells like very, very bad engineering and design…

Continued investigation on HP’s part has pointed us back to the original diagnosis – a faulty mid-plane.  Only by default did we return to that conculsion, however.  This is the only piece of hardware common to the problems.  Our only other conclusion was that this was a very bad, “hiccup” — which obviously buys us no real peace of mind…  

So, sometime soon, we will be replacing the mid-plane of our enclosure.   I have, of course, lost some faith in the HP blade ecosystem.  We have plans to migrate our corporate VMware cluster onto blades, as well as some Citrix and other servers.  Losing an enclosure like this has un-nerved those plans.  We were fortunate to have drug our feet  to only have 3 blades populated and serving anything at the time this happened.  I will post updates as we move forward…