To quote the HP engineer on the call — there were just too many combinations and possibilities to be able to certify all available firmwares — and so the idea of a release set began in January.  The release sets are available in a compatibility matrix on the HP Bladesystem page (http://h18004.www1.hp.com/products/blades/components/c-class.html) — look at the Compatibility tab.

The good news is that in our environment, we are close to compliance with the January release set.  The only thing out of compliance for the January release is our blade ROM, PMC and iLO firmware, but, as we were also informed, this is not a good situation to be in.

Something I already knew is that we have a bottom-up firmware topology for the HP Bladesystem — meaning that we have to update the bottom level components first and then move up.  I knew this applied to the interconnects and onboard administrator modules, but it also extends to blade servers also.   In addition, the blade servers are at the lowest level and should be updated first — particularly the iLO.

The reason for this is that new OA and interconnect firmware may introduce features and if the PMC or iLO is not aware of these features due to out-dated firmware, erroneous might be passed and all sorts of things could happen — at worst, random server reboots.

iLO firmware 1.81 is particularly susceptible to the random reboot for Windows blades.  If you are running 1.81 and have Windows OS loaded, you should upgrade to 1.82 as soon as possible.  See http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?locale=en_US&objectID=c01802766 for more information.  Other OSes – Linux, VMware, XenServer – are not affected.

Our errors look like this:

After several cases with VMware and HP technical support, we are no closer to resolving the issues.  VMware support, for its part, has done a good job of telling us what ESX is reacting to and seeing.  HP support, on the other hand, has been circling around the problem but has made little progress in diagnosing the issue.  We have had an ongoing case for several months and our primary EVA resource at HP has continually examined the EVAperf information and SSSU output that we have sent to HP for analysis.  Those have turned up nothing, and yet the messages continue from VMware.

The errors in the log make sense to me – we are losing a path to a data disk (sometimes even a boot-from-SAN ESX OS disk!) – but why HP cannot see anything in our Brocade switches or within the EVA is beyond me.   Our ESX hosts, whether blade or rack-mounted hardware, are seeing the problems across the board.  The one cluster we waited to upgrade never saw the issues in ESX3.5, but sees them now in ESX4.  And perhaps it is a VMware issue that is just too sensitive in monitoring its storage, but I suspect its something else.   The messages don’t seem to affect operation on the hosts, but it certainly makes investigating problems difficult when trying to determine what is a real problem versus just another failed path message.  Anyone else seeing this?

HP Virtual Connect Flex-10 allows an administrator to take a physical 10G connection and partition it into up to 4 more appropriately sized logical NICs presented to a server blade.  Virtual Connect Flex-10 incorporates all of the benefits of Virtual Connect (previously explained here) while allowing for additional logical NICs.  A Flex-10 Ethernet interconnect module with two onboard Flex-10 NICs on a blade will allow for 8 total NICs – something formerly only available if all 8 interconnect bays and the two mezzanine slots were populated with Ethernet modules.  This greatly reduces the amount of cabling required at the blade enclosure and condenses the amount of connections required.

The Flex-10 interconnect modules include SFP connectors which are capable of copper or fiber connections depending on the GBIC installed.

Upgrade Process
We expected going into the installation that we could need to recreate the Virtual Connect domain (a domain is the configuration, server profiles and settings for a Virtual Connect deployment on a specific enclosure or set of enclosures).  Since we only had 5 server blades installed and operating, we weren’t overly concerned.  Our primary concerns were to ensure that the same Virtual Connect MAC and WWID settings were reinstalled and that our recreated server profiles matched their MAC address and WWID settings from the prior installation.  So, to ensure we had all the information, we made printouts of the server profiles, Ethernet networks and fiber channel configuration and then proceeded to installing the modules.

We shutdown all of the existing blades from service.  We removed the existing VC-Ethernet modules from Bays 1 and 2 and we installed the new VC-Ethernet Flex-10 modules in these bays.  The VC-Eth modules in Bays 1 and 2 contain the domain configuration.  Once removed, the domain had to be completely recreated.  The new modules were at a higher firmware (2.12) versus the original modules (2.01).

What surprised us during installation was that we were able to successfully restore our original Virtual Connect domain configuration onto the Flex-10 modules.  Once restored, we were able to see all configuration and the only thing showing an error were the Ethernet networks and shared uplink sets.  These didn’t have any active ports assigned to the uplinks.  The new VC-Eth Flex 10 modules have a different port numbering to differentiate from the original VC-Eth modules.  After reassigning ports to the uplink sets, all of the Ethernet networks appeared to be back online.  The firmware differences didn’t appear to be a problem after initial restore.

After booting servers onto the enclosure (all are boot-from-SAN and booted fine), we determined that we were having some network connectivity issues.  After troubleshooting, we upgraded VC firmware on the modules.  This proceeded without problems, but upon reboot, the VC-Ethernet Flex-10 modules would not come back online.  After troubleshooting, we installed upgraded firmware on the enclosure.  The newer firmware on all seemed to resolve all issues and the enclosure was back online.  Final time to upgrade was around 5 hours, but much of that was waiting on Virtual Connect modules to reboot and wondering why they wouldn’t come back online.

Our second enclosure was expected to be a simple slam dunk after our largely successful and simple upgrade procedure — especially since we knew about firmware before on this round, but unexpected problems were there.  This enclosure had a mid-plane replacement early in its life and apparently there was confusion with the Virtual Connect domain and the enclosure serial number.  We received errors during restore and even after we forced the restore and upgraded firmware, the Ethernet networks never came back and responded.

This forced us into a recreate scenario.  After recreating this VC domain by hand, everything came online and worked as wanted.  I still believe that this is the cleaner way to handle the upgrade, although the restore worked on the first enclosure and presented very few problems.

Next…  Configuring Flex-10 FlexNICs

Virtual Connect is about consolidation by reducing the number of physical connections required.  From the fiber channel modules, two 4-port VC-Fiber modules connect an entire chassis to the fabric, and then using NPIV, the fiber channel traffic is sent to individual blade servers within the chassis.  The 16 blade slots all share the 4 ports of each VC module.

The new development on the ethernet Virtual Connect side is a technology known as Flex-10.  Flex-10  is a new technology which helps to overcome some of the traditional challenges of blades (such as lack of NICs) when looking at blades for virtualization projects.  Virtual Connect Flex-10 allows up to 4 times as many NIC connections per blade server.  In a simliar way to the fiber channel implementation, a single Flex-10 physical NIC (pNIC) on a blade server can be divided into 4 logical NICs or FlexNICs.  These FlexNICs can be configured to different port speeds and capacities.

A blade server with two integrated Flex-10 NICs will have 8 logicial FlexNICs on the motherboard, and each one of those has it’s own MAC address.  These MAC addresses, like earlier blade NICs, can be overridden in the Virtual Connect server profiles and can be assigned virtualized MAC addresses, so the Flex-10 servers get the same portability that earlier server profiles in Virtual Connect offered.

The Flex-10 technology is a two parter – requiring both Flex-10 Ethernet modules on the interconnect bays of the blade enclosure and Flex-10 NICs in the blade servers.  Fortunately for companies with existing investments, the Flex-10 Ethernet interconnect modules are backwards compatible with earlier integrated NICs on your existing blades.

Each FlexNIC can be sized from 100Mb up to a full 10Gb speeds in 100MB increments.  The incredible thing, is that with the two available mezzanine cards and the integrated pNIC’s, Flex-10 technology allows for up to 24 FlexNICs per blade – an incredible amount in a such a compact space.

We have plans of implementing this technology, so hopefully I can post a follow up from personal experiences once we begin that project.

HP is busy working on new technology to integrate vCenter and the HP hardware giving administrators greater visiblity into the hardware state and ease of management from a central pane of glass. This software is part of the Insight Control suite of products which HP already produces.  The new plug-in will sit on a Virtual Center client and will talk directly to the ILO, OA, and other agents on HP hardware to report back things like temperature, processor state, memory errors, etc. into the vCenter console.  HP reps also told me that the software goes bidirectional and can send information from vCenter over into HP SIM.

Power control also gets integrated as we continue to see HP work on power management in its rack mount and blade enclosures.  Power and heat were major issues addressed with the C7000 chassis and the functionality is being extended into the view from vCenter.  Today, if you enabled VMware Distributed Power Management (DPM) and it shuts down a host because the virtual machines are idle, HP SIM sees this as an error state and begins to alert administrators due to a failure of a host.  In the future, the two-way communication will allow for HP SIM to be smart enough to recognize that the host is offline due to DPM and not to a critical error.  We’ll gain insight (sorry, had to)  into vCenter from the HP tools.

HP also touted its new VDI hardware package which includes Proliant server along with a Lefthand SAN solution to provide for a very flexible, single-rack solution for deploying VDI.  The press release touts it as the first sub-$1000 per seat solution for deploying VDI.

For those who haven’t heard of HP’s Lefthand technology – Lefthand Networks was a start-up SAN provider who used commodity hardware to create iSCSI SANs with advanced snapshotting and replication features.  HP acquired this company last year and is steadily working to bring their iSCSI solution into their Storageworks lineup.  This solution seemed to use a purpose built Lefthand solution with what appeared to be a 3 to 5 U enclosure with two or three sliding shelves housing multiple mini SAS drives to accommodate terrabytes of data.  The disk enclosure shown during the presentation looked more like HP’s SFS enclosures than the Lefthand SAN offerings – the P4000, etc. – but we were told it was Lefthand powered.

For more information, see the original press release: http://www.hp.com/hpinfo/newsroom/press/2009/090901a.html

For information about the Insight Control for VMware, see: http://h18013.www1.hp.com/products/servers/management/integration.html