I had been worried about losing my digital home movies.  That data is really too large to really push out to a backup service like Mozy and when I load new movies, its usually to the tune of 20 or 30GB at a time, which would take weeks to push up.  In addition to that data, I also have Movies and TV shows that we have purchased through iTunes.

So, I knew that I wanted some sort of external storage with at least mirroring capabilities to protect the movies as best I could.  Back in January, I was close to buying a Western Digital mirrored external hard drive from the Apple store.  I chose instead to get a 1TB, single external drive with Firewire as an upgrade to my 500GB Time Machine which was almost full, and wait for a better solution.

Since it was first released, I have always been in love with the Drobo.  I describe it as an mini-EVA to my co-workers, because it mirrors a lot of the HP Storageworks EVA functionality – like drives auto-leveling and automatically striping data across disks in the disk group.  Maybe, I just like the idea of having something that advanced attached to my home computers…  and I’ll be the first to admit that they don’t compare — they are apples and oranges.

But I do love what the Drobo offers, and so I have been watching them for some time.  Since its introduction, the Drobo had gone from a single device with USB only, to an enhanced version which has USB2 and Firewire (my choice of devices), a version which adds eSATA with USB and Firewire, and several larger devices, the Drobo Elite and the Drobo Pro, which feature iSCSI and NAS functionality targeted towards SMB’s.

As a side note, the Drobo Pro is even VMware certified, and I feel like it is a great solution for small to medium businesses looking for shared storage for an vSphere deployment.  This past week, we attended the Charlotte Regional VMware Users Group meeting, and actually got to see one of the Drobo Elite units on display.

The price has held steady on the Drobo that I have been watching.  At a $399 price point, I couldn’t justify buying it and then having to purchase drives to go inside.  But, as I said before, B&H was offering a great deal – the Drobo device at $100 off its normal MSRP.  It was a good $50 less than I could find it anywhere else, and so I bit — err bought.  I was able to equip it with two Western Digital Caviar Green (my wife would be proud) 1TB drives from NewEgg for a cost of $138.  And so, last night I finished up my transition of data and have everything I wanted protected on the Drobo – and I’m happy…

Our errors look like this:

After several cases with VMware and HP technical support, we are no closer to resolving the issues.  VMware support, for its part, has done a good job of telling us what ESX is reacting to and seeing.  HP support, on the other hand, has been circling around the problem but has made little progress in diagnosing the issue.  We have had an ongoing case for several months and our primary EVA resource at HP has continually examined the EVAperf information and SSSU output that we have sent to HP for analysis.  Those have turned up nothing, and yet the messages continue from VMware.

The errors in the log make sense to me – we are losing a path to a data disk (sometimes even a boot-from-SAN ESX OS disk!) – but why HP cannot see anything in our Brocade switches or within the EVA is beyond me.   Our ESX hosts, whether blade or rack-mounted hardware, are seeing the problems across the board.  The one cluster we waited to upgrade never saw the issues in ESX3.5, but sees them now in ESX4.  And perhaps it is a VMware issue that is just too sensitive in monitoring its storage, but I suspect its something else.   The messages don’t seem to affect operation on the hosts, but it certainly makes investigating problems difficult when trying to determine what is a real problem versus just another failed path message.  Anyone else seeing this?

In previous version of ESX, the desired storage setting was fixed path for the EVA.  In our case, we simultaneously presented the ESX3.5 and ESX4 hosts to the same LUNs, meaning some were fixed and some were set to the ESX4 default, which was MRU.  This caused problems.  After initial issues, we backed away and presented one LUN at a time, performed our VMotions and then unpresented the LUN from the old cluster.  This prevented any flapping issues between controllers.

The best practices document describes two settings for best practice with an EVA that is ALUA aware (EVA 4000/6000/8000 and EVA 4400/6400/8400):

• Set the default path selection policy to round robin (VMW_PSP_RR)
• Set the IOPS Limit to 1

The path selection policy is the bigger deal.   The default path selection policy in ESX4 is most recently used (MRU).  This is still a valid policy and works with the EVA, but it is not their best recommendation because Round Robin allows the ESX host to use two paths or four paths (depending on EVA model) to send traffic to the ESX host which can provide greater performance.

I was not sure what the IOPS setting actually does based on the documentation, but Duncan Epping has since posted about the IOPS=1 setting and about how in a real-world environment this setting may not make a difference.  I agree with his conclusions.

Now, I should not and do not want to take credit for the instructions below… I did not come up with this solution — simply a user.  The esxcli commands came directly from the HP Best Practices Guide, but the scripted IOPS setting changes for the LUNs came from another blog - Virtual IEF.  This solution has worked brilliantly in our environment.  Virutal IEF has a great, in-depth posting about how the EVA LUN ownership works and what makes an optimal path.

Setting Changes for Round Robin PSP
To change the default path selection policy (PSP) to Round Robin. If LUNs are already presented, you will need to reboot to allow the LUNs to rescan and set to Round Robin instead of the default most recently used (MRU) setting.

 esxcli nmp satp setdefaultpsp --satp VMW_SATP_ALUA --psp VMW_PSP_RR

Change the IOPS limit value for the interactive session.

 for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device $i ;done

Verify the IOPS value is correct, run this script and check values.

 for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp roundrobin getconfig --device $i ;done

Your output should look like:

 Device: naa.600508b4001087e400009000012b0000
 I/O Operation Limit: 1
 Limit Type: Iops
 Use Active Unoptimized Paths: false
 Errors:
 Unable to find device with the name naa.600508b4001087e400009000012b0000:1
 Byte Limit: 10485760

Unfortunately, the IOPS value will reset on reboot many times. To ensure that this value is correctly set, you are able to set a value in the post boot script. Add the same for statement to/etc/rc.local to set the IOPS value on each reboot.

 for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device $i ;done

Background
At work, we use raw disk mappings (RDM) some within our VMware virtual environment.   The last VMworld, I was inundated with folks saying that using an RDM for any reason other than a Microsoft cluster is a bad idea anymore.  None of them really gave concrete reasons why, other than saying that VMDK performance had improved so much that there was really no reason to use them.  Their recommendation was to use them only where required, such as clustering.  

In our datacenter, we have chosen to implement RDM’s for large filesystems, typically database applications.  All of our SQL server VM’s and all of our Oracle VM’s run raw disk mappings for their data storage.   Before we felt comfortable about running these workloads in VM’s and knowing how they’d perform, RDM’s gave us a way to quickly move back to physical server, if needed.  That reasons still stands, though our confidence in our virtual infrastructure has increased considerably.  We have no qualms about moving SQL into a VM, as a matter of fact, we prefer it in most cases for redundancy and disaster recovery reasons.   The only physical Microsoft SQL Server is our multi-instance cluster, something that has redundancy and disaster recovery covered.

Growing, growing, grown
We initially liked RDM’s for Windows VM’s so that we could grow our disks using the SAN array and then extend the partition using diskpart.  This always worked well with our physical database servers, so we continued the practice into the virtual world.  We initally implemented physical mode RDM’s for our SQL servers, but we quickly found out that physical RDM’s prohibit snapshots and other features, so we changed to virtual mode RDM’s. 

With the virtual mode RDM, we found that growing the filesystem and then getting the VM to recognize the new space was more difficult.  Most information I saw indicated that you’d need to reboot the VM to recongize the space, but then I happened upon a great post in the VMware forums.  This post said to rescan the host, VMotion the VM and then rescan the VM and the disk space would be recognized.  This worked great.

To use or not to use
I realize that today, growing a VMDK is an easy task.   ESX 3.5 and 4.0 allow you to grow a VMDK from the Edit Settings… menu, but when we began in the 3.0 days, it wasn’t an easy thing to grow a VMDK.   So, we went this direction and we continue today to keep things consistent.  I think its safe to recommend that others use VMDK’s, since I do agree that the performance hit is small for using a VMDK versus a raw LUN.   That advice is sound, but I wanted to post this for others who might be in our situation, with VM’s that were already setup with virtual RDM’s and trying to figure the way to grow their filesystems.

An EVA normally has two (maybe more, I’m not primarily a storage guy) controllers and those handle all the requests received through the SAN.  For every LUN, one controller is its master.  Both controllers can handle requests for the LUN, but only one actually handles the access.  If the controller on fabric A is the primary but the controller on fabric B is getting more requests, eventually the EVA swaps control for the LUN to fabric B — wherever the majority of requests are coming.

This behavior would only become a problem if you had hosts configured to access the LUN on different fabrics.  ESX4 is ALUA (asymmetric logical unit access) aware, meaning it should automatically determine the optimal path and in the case of an EVA.  The EVA, I’m told by HP support, is supposed to respond an ALUA request for the optimal path by responding with the controller that is the master over the LUN.

If you, like us, have an ESX 3.5 cluster with preferred paths setup, you should proceed with caution.  The ALUA information isn’t apparently shared between clusters.  And if your clusters get different optimal paths, you could end up with controller ping-pong as requests are sent down both fabrics and the volume changes between the two, resulting in more on Fabric A followed by more on Fabric B — forcing the controller to switch masters.

So, while in a migratory state, I think my safest route is to configure the ESX4 hosts to use a preferred path like the ESX3.5 cluster nodes.  I hate to move from the default ESX configuration and this isn’t an official recommendation from HP support, but it certainly makes the most sense to define the paths being used (except in a failure).

I post this because I feel like there have to be other HP Storageworks customers who have the same situation or have experienced something similar.  I would love to hear from you…

Lesson # 1: vCenter Server doesn’t cluster well with Microsoft Cluster Services
Well, its not so much vCenter Server as Update Manager that gave me problems.  I can get Update Manager to work on one node but it doesn’t want to cooperate on the other node of my Microsoft cluster. This is not a major issue for us since we plan to implement vCenter Server Heartbeat on our vCenter install.  We view this as the official clustering solution for vCenter and so I believe it is the better option.

Lesson #2: Use the upgrade path in vCenter Update Manager to upgrade your ESX hosts – makes life really easy!
I was pretty surprised at how easily my four nodes upgraded to ESX4 using Update Manager.  It is well worth the bit of time to get things setup and push the upgrade out.  Upgrades is a nice new feature of Update Manager in vSphere that I had not heard any real information on.  You are able to upgrade virtual appliances and ESX hosts with the latest installs. On the ESX side, you load the ISO distribution from VMware onto the Update Manager server and then ESX captures configuration information, handles a scripted install using the hosted ISO on the Update Manager server and then loads back your configuration overtop the upgraded server.  Given the architecture changes between ESX 3.5 and vSphere ESX, it is a clean install — which was one of my concerns.  My boxes were 3.0.0 boxes, upgraded through 3.0.2 and then to 3.5 and 4 updates.

Lesson #3: Storage behavior in ESX4 and vCenter4 has changed…
One nice change, and I learned part of this as trial by fire (a separate post on that later), is that storage behavior changes in vCenter 4. Storage is a more unified system in vCenter 4.  If a host makes or detects a change in the storage LUNs, it automatically kicks off storage rescans on other nodes to recognize the change you just made on a single host.  It also seems to do periodic rescans on its own in ESX4 to recognize LUN changes.  These are welcome administration changes to the operation of vCenter and ESX.

Lesson #4: Be very careful about presenting LUNs to two different ESX clusters when using HP Storageworks EVA’s.
Talking about trial by fire, we experienced a pretty major failure with our storage a couple weekends ago.  ESX4 is ALUA (asymmetric logical unit access) aware – meaning it should determine its optimal path automatically and use those optimal paths.  The EVA storage should respond back with the controller who ‘owns’ that LUN as the optimal path.  So, this should present little trouble.

While this sounds great, mixing the ESX4 behavior with ESX3.5 caused us to experiencing controller ping-pong where the controllers were flipping back and forth for particular LUNs.  LUNs not used by the ESX4 nodes, yet it happened.  EVA storage will automatically move control of a LUN from one controller to another depending on where it is receiving the majority of requests.  In a shared storage environment like ESX, that can be somewhat problematic if your two clusters are using different paths and thus a majority of requests changes every few minutes.  While, its my belief that we have a sick EVA and that’s more the problem, it did bring back the notion of planned storage pathing for our fiber channel, but that’s for a longer post.

Lesson #5: Rolling out slow, especially around the holidays, is SMART…
I mentioned early that we were rolling out our ESX4 slowly.  We have installed ESX4 on our new blade cluster and we only moved test and development virtual servers over to this new environment and we are glad.  As part of our migration, we moved the backend storage for the test and development to our disaster recovery storage array and that appears to be the sick EVA – it caused a couple complete outages for these virtual servers, but because we’d done a slow migration, production (running safely on our tried and true ESX 3.5 cluster) didn’t see the problems.

We don’t plan on touching much this week with Christmas coming Thursday.  It gives me a time to update the ole blog with some of the work that has been occupying my time.   This migration has a lot more to go, so there will be more meat as I move through the process.