I had been worried about losing my digital home movies.  That data is really too large to really push out to a backup service like Mozy and when I load new movies, its usually to the tune of 20 or 30GB at a time, which would take weeks to push up.  In addition to that data, I also have Movies and TV shows that we have purchased through iTunes.

So, I knew that I wanted some sort of external storage with at least mirroring capabilities to protect the movies as best I could.  Back in January, I was close to buying a Western Digital mirrored external hard drive from the Apple store.  I chose instead to get a 1TB, single external drive with Firewire as an upgrade to my 500GB Time Machine which was almost full, and wait for a better solution.

Since it was first released, I have always been in love with the Drobo.  I describe it as an mini-EVA to my co-workers, because it mirrors a lot of the HP Storageworks EVA functionality – like drives auto-leveling and automatically striping data across disks in the disk group.  Maybe, I just like the idea of having something that advanced attached to my home computers…  and I’ll be the first to admit that they don’t compare — they are apples and oranges.

But I do love what the Drobo offers, and so I have been watching them for some time.  Since its introduction, the Drobo had gone from a single device with USB only, to an enhanced version which has USB2 and Firewire (my choice of devices), a version which adds eSATA with USB and Firewire, and several larger devices, the Drobo Elite and the Drobo Pro, which feature iSCSI and NAS functionality targeted towards SMB’s.

As a side note, the Drobo Pro is even VMware certified, and I feel like it is a great solution for small to medium businesses looking for shared storage for an vSphere deployment.  This past week, we attended the Charlotte Regional VMware Users Group meeting, and actually got to see one of the Drobo Elite units on display.

The price has held steady on the Drobo that I have been watching.  At a $399 price point, I couldn’t justify buying it and then having to purchase drives to go inside.  But, as I said before, B&H was offering a great deal – the Drobo device at $100 off its normal MSRP.  It was a good $50 less than I could find it anywhere else, and so I bit — err bought.  I was able to equip it with two Western Digital Caviar Green (my wife would be proud) 1TB drives from NewEgg for a cost of $138.  And so, last night I finished up my transition of data and have everything I wanted protected on the Drobo – and I’m happy…

While we were first deploying VI3 and performing our consolidation, the primary focus was on the non-critical systems and moving them into the virtual infrastructure to get the best utilization of hardware.  Since completion of this phase, the next focus became moving some of our mission critical systems to VMware in order to establish disaster recovery for our non-clustered systems.  Disaster recovery through VMware is accomplished by 1) relocating the boot and data onto SAN storage which is replicated to our secondary data center and 2) by the ability to utilized VMware HA in the event of hardware failure to establish resiliency we do not have on a single-server, hardware deployment.

As we have expanded VMware’s role in our data center, new challenges have emerged.  First, when a network issue is occurring, we don’t have our traditional monitoring tools (like PRTG) in a position where they are able to alert for large changes in traffic.  In our physical environment, HP agents are run and PRTG is able to query against these systems with SNMP to retrieve information about traffic.  In the virtual environment, we don’t run these agents (because they are largely non-applicable since these are virtual boxes).  Our preferred way to monitor is through something that can look directly into the virtualization layer and retrieve information.

When we began our virtualization initiative, we saw some monitoring solutions at regional VMware user group meetings, but at that point – we didn’t see a lot of value in the products.  As we progressed through the beginning of phase two, we knew that monitoring and visibility into the virtualization beyond what vCenter allowed was a big thing for us.  We demoed several products from companies I had seen or talked with at VMworld and we finally decided upon two candidates to demo within our environment.

The first product we tested was Hyper9 — and we were really impressed with this product.  This product began as a search tool – basically a Google for virtual infrastructure.  It was really cool and the search interface was really good for answering questions as asked or posed by management.  It also was quick to drill down and get to information that we were searching for.  One of its most powerful features is it’s vmDNA – where it tracks changes in the VM and can compare two point-in-time views to see what has changed on the server.  This, we felt, was really powerful.  The place where we saw a deficiency in the product was related to alerts and alarming.  Hyper9 had separated this into a separate product which is good from a flexibility stand-point, but bad from a central management and alarming standpoint.  This piece of the puzzle just didn’t seem to be enterprise ready.  The second deficiency was in relation to network monitoring – something that was on our must-have list of requirements.   Hyper9 came pre-populated with lots of common pain point searches that allowed us to quickly look for problem areas — large snapshot files, datastores that are low on disk space, and tools that are out of date or not running.  The problem we saw with this was that the searches were passive (had to be executed at a specific time — scheduled — in the secondary application) and did not actively alert when a condition was met.

The second product we had decided on demoing was Vizioncore’s vFoglight application.  vFoglight is based on Quest Software’s Foglight engine (Quest owns Vizioncore), which is an enterprise class monitoring and visualization engine.  Vizioncore wrote the VMware cartridge which handles the monitoring for VMware infrastructure by pulling information directly from vCenter — with no impact to the ESX hosts.   It is strong in reporting and alarming.  The configuration, terminology and organization of the application is different from anything else we looked at, so learning the layout and how it operates was a large learning curve.  Fortunately, Vizioncore offers good training for free on their website.

vFoglight was exponentially more complex than the Hyper9 product, but it offered the additional features we were looking for in the networking areas.  It also had the integrated alarming and notifications within a centralized console – not a in a separate application.   Most of vFoglight’s intelligence was built around the alert and alarm conditions which triggered alarms in an alarms panel which could be tracked and researched (how often has this happened before), where notes could be stored and where individual members of our team could acknowledge an alarm or add notes on how the issue was resolved.   In addition, one of the internal developer had created a “vBundle” of common, useful reports which is available free to add to the system.

Because this software was built on the Foglight engine, it was not only possible but may be very advantageous for us to add additional cartridges to monitor specific databases or application or physical hosts.  The fewer monitoring systems we have to maintain or watch, the better, in my opinion.  Time will tell.

Ultimately, we made the decision to go with vFoglight.  The downside to the decision was its price.  We paid well for all this functionality.  But, so far, it has worked extremely well.  We finished our production installation last week and have about a week’s worth of data.  The Foglight system works much better over a long time as statistics and predictions become more accurate because of historical data warehoused.  At this point, we are still learning the application and how to do things in it and letting it collect data.

The service engineer for our account was extremely helpful in setting up the application.  They have a sizer application which determines the number of objects in your infrastructure and then provides recommendation on how to best hardware configuration, database options and estimated data growth over 1 year and 3 years for planning.  I hope to have more information on vFoglight as I continue to learn about it.

Our errors look like this:

After several cases with VMware and HP technical support, we are no closer to resolving the issues.  VMware support, for its part, has done a good job of telling us what ESX is reacting to and seeing.  HP support, on the other hand, has been circling around the problem but has made little progress in diagnosing the issue.  We have had an ongoing case for several months and our primary EVA resource at HP has continually examined the EVAperf information and SSSU output that we have sent to HP for analysis.  Those have turned up nothing, and yet the messages continue from VMware.

The errors in the log make sense to me – we are losing a path to a data disk (sometimes even a boot-from-SAN ESX OS disk!) – but why HP cannot see anything in our Brocade switches or within the EVA is beyond me.   Our ESX hosts, whether blade or rack-mounted hardware, are seeing the problems across the board.  The one cluster we waited to upgrade never saw the issues in ESX3.5, but sees them now in ESX4.  And perhaps it is a VMware issue that is just too sensitive in monitoring its storage, but I suspect its something else.   The messages don’t seem to affect operation on the hosts, but it certainly makes investigating problems difficult when trying to determine what is a real problem versus just another failed path message.  Anyone else seeing this?

Transition
The transition to the dvSwitches was an unexpected complication.  In vCenter, the port group names, although identical, are presented differently in vCenter.  Because of this, you could not simply VMotion from an ESX3.5 node to ESX4.  As a solution, I decided to make an temporary standard vSwitch on one node of the new ESX4 cluster as the destination for all my vMotions.  Each of my dvSwitches on the ESX4 cluster had two uplinks, so I stole one uplink for each of my temp standard vSwitches.  This allowed me to seemlessly change the network configuration of each VM after it vMotioned from the standard port group to the distributed port group.   Although it was more steps and trouble, it allowed me to make the transition during the day while production workloads stayed online with no impact to them.

Design Considerations
vNetwork Distributed Switch has certainly saved me time, and its only been in my VMware environment a really short time.  I like several things that the distributed switch allows – such as persistent ports, port statistics and the ability to maintain the distributed switch in one place across all nodes in my cluster.  The downside to this centralized administration is an increased dependence on vCenter Server.

Rich Bramley of VM/ETC posted on the challenges of distributed switches with virtualized vCenter instances — largely the problem of a cyclical relationship that could leave you between “the vDS rock and a hard place,” as he puts it.   He and I drew the same conclusions and ultimately, I have decided to keep a mix of distributed and standard vSwitches within my environment.

Perhaps I am overly cautious, however, I run my service console, VMotion, FT Logging all across VMware Standard Switches.  All VM traffic across the vNetwork Distributed Switch.  The exception to the “all VM traffic” rule will be when  if we introduce a virtualized vCenter instance on our cluster.  The dependance on vCenter for dvSwitches and having a virtualized vCenter assigned to a distributed port group makes for a potential disaster, since the VM cannot access or bind to his port on the distributed vSwitch without vCenter running.  Also of note,  Rich reports that vCenter on a distributed switch is NOT supported by VMware because of the cyclical relationship between the two.

In previous version of ESX, the desired storage setting was fixed path for the EVA.  In our case, we simultaneously presented the ESX3.5 and ESX4 hosts to the same LUNs, meaning some were fixed and some were set to the ESX4 default, which was MRU.  This caused problems.  After initial issues, we backed away and presented one LUN at a time, performed our VMotions and then unpresented the LUN from the old cluster.  This prevented any flapping issues between controllers.

The best practices document describes two settings for best practice with an EVA that is ALUA aware (EVA 4000/6000/8000 and EVA 4400/6400/8400):

• Set the default path selection policy to round robin (VMW_PSP_RR)
• Set the IOPS Limit to 1

The path selection policy is the bigger deal.   The default path selection policy in ESX4 is most recently used (MRU).  This is still a valid policy and works with the EVA, but it is not their best recommendation because Round Robin allows the ESX host to use two paths or four paths (depending on EVA model) to send traffic to the ESX host which can provide greater performance.

I was not sure what the IOPS setting actually does based on the documentation, but Duncan Epping has since posted about the IOPS=1 setting and about how in a real-world environment this setting may not make a difference.  I agree with his conclusions.

Now, I should not and do not want to take credit for the instructions below… I did not come up with this solution — simply a user.  The esxcli commands came directly from the HP Best Practices Guide, but the scripted IOPS setting changes for the LUNs came from another blog - Virtual IEF.  This solution has worked brilliantly in our environment.  Virutal IEF has a great, in-depth posting about how the EVA LUN ownership works and what makes an optimal path.

Setting Changes for Round Robin PSP
To change the default path selection policy (PSP) to Round Robin. If LUNs are already presented, you will need to reboot to allow the LUNs to rescan and set to Round Robin instead of the default most recently used (MRU) setting.

 esxcli nmp satp setdefaultpsp --satp VMW_SATP_ALUA --psp VMW_PSP_RR

Change the IOPS limit value for the interactive session.

 for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device $i ;done

Verify the IOPS value is correct, run this script and check values.

 for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp roundrobin getconfig --device $i ;done

Your output should look like:

 Device: naa.600508b4001087e400009000012b0000
 I/O Operation Limit: 1
 Limit Type: Iops
 Use Active Unoptimized Paths: false
 Errors:
 Unable to find device with the name naa.600508b4001087e400009000012b0000:1
 Byte Limit: 10485760

Unfortunately, the IOPS value will reset on reboot many times. To ensure that this value is correctly set, you are able to set a value in the post boot script. Add the same for statement to/etc/rc.local to set the IOPS value on each reboot.

 for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device $i ;done

On a personal note, my wife and I are in the final stages of planning our new home, which we hope to begin building in the coming months.  Any free time that I would have had to blog about my experiences has been consumed with house plans, builder meetings and other items to prepare for this major undertaking.  My wife is attempting to chronicle our build on her blog, My Green Glasses.  The house will be a certified green home, Energy Star certified and *possibly* LEED certified.  We know that we will be close to meeting requirements for a LEED certification and we are looking at what additional things need to be done to make it happen and whether it is worth it or not.

Thanks to vm/etc for posting this originally…

Background
At work, we use raw disk mappings (RDM) some within our VMware virtual environment.   The last VMworld, I was inundated with folks saying that using an RDM for any reason other than a Microsoft cluster is a bad idea anymore.  None of them really gave concrete reasons why, other than saying that VMDK performance had improved so much that there was really no reason to use them.  Their recommendation was to use them only where required, such as clustering.  

In our datacenter, we have chosen to implement RDM’s for large filesystems, typically database applications.  All of our SQL server VM’s and all of our Oracle VM’s run raw disk mappings for their data storage.   Before we felt comfortable about running these workloads in VM’s and knowing how they’d perform, RDM’s gave us a way to quickly move back to physical server, if needed.  That reasons still stands, though our confidence in our virtual infrastructure has increased considerably.  We have no qualms about moving SQL into a VM, as a matter of fact, we prefer it in most cases for redundancy and disaster recovery reasons.   The only physical Microsoft SQL Server is our multi-instance cluster, something that has redundancy and disaster recovery covered.

Growing, growing, grown
We initially liked RDM’s for Windows VM’s so that we could grow our disks using the SAN array and then extend the partition using diskpart.  This always worked well with our physical database servers, so we continued the practice into the virtual world.  We initally implemented physical mode RDM’s for our SQL servers, but we quickly found out that physical RDM’s prohibit snapshots and other features, so we changed to virtual mode RDM’s. 

With the virtual mode RDM, we found that growing the filesystem and then getting the VM to recognize the new space was more difficult.  Most information I saw indicated that you’d need to reboot the VM to recongize the space, but then I happened upon a great post in the VMware forums.  This post said to rescan the host, VMotion the VM and then rescan the VM and the disk space would be recognized.  This worked great.

To use or not to use
I realize that today, growing a VMDK is an easy task.   ESX 3.5 and 4.0 allow you to grow a VMDK from the Edit Settings… menu, but when we began in the 3.0 days, it wasn’t an easy thing to grow a VMDK.   So, we went this direction and we continue today to keep things consistent.  I think its safe to recommend that others use VMDK’s, since I do agree that the performance hit is small for using a VMDK versus a raw LUN.   That advice is sound, but I wanted to post this for others who might be in our situation, with VM’s that were already setup with virtual RDM’s and trying to figure the way to grow their filesystems.

An EVA normally has two (maybe more, I’m not primarily a storage guy) controllers and those handle all the requests received through the SAN.  For every LUN, one controller is its master.  Both controllers can handle requests for the LUN, but only one actually handles the access.  If the controller on fabric A is the primary but the controller on fabric B is getting more requests, eventually the EVA swaps control for the LUN to fabric B — wherever the majority of requests are coming.

This behavior would only become a problem if you had hosts configured to access the LUN on different fabrics.  ESX4 is ALUA (asymmetric logical unit access) aware, meaning it should automatically determine the optimal path and in the case of an EVA.  The EVA, I’m told by HP support, is supposed to respond an ALUA request for the optimal path by responding with the controller that is the master over the LUN.

If you, like us, have an ESX 3.5 cluster with preferred paths setup, you should proceed with caution.  The ALUA information isn’t apparently shared between clusters.  And if your clusters get different optimal paths, you could end up with controller ping-pong as requests are sent down both fabrics and the volume changes between the two, resulting in more on Fabric A followed by more on Fabric B — forcing the controller to switch masters.

So, while in a migratory state, I think my safest route is to configure the ESX4 hosts to use a preferred path like the ESX3.5 cluster nodes.  I hate to move from the default ESX configuration and this isn’t an official recommendation from HP support, but it certainly makes the most sense to define the paths being used (except in a failure).

I post this because I feel like there have to be other HP Storageworks customers who have the same situation or have experienced something similar.  I would love to hear from you…

Lesson # 1: vCenter Server doesn’t cluster well with Microsoft Cluster Services
Well, its not so much vCenter Server as Update Manager that gave me problems.  I can get Update Manager to work on one node but it doesn’t want to cooperate on the other node of my Microsoft cluster. This is not a major issue for us since we plan to implement vCenter Server Heartbeat on our vCenter install.  We view this as the official clustering solution for vCenter and so I believe it is the better option.

Lesson #2: Use the upgrade path in vCenter Update Manager to upgrade your ESX hosts – makes life really easy!
I was pretty surprised at how easily my four nodes upgraded to ESX4 using Update Manager.  It is well worth the bit of time to get things setup and push the upgrade out.  Upgrades is a nice new feature of Update Manager in vSphere that I had not heard any real information on.  You are able to upgrade virtual appliances and ESX hosts with the latest installs. On the ESX side, you load the ISO distribution from VMware onto the Update Manager server and then ESX captures configuration information, handles a scripted install using the hosted ISO on the Update Manager server and then loads back your configuration overtop the upgraded server.  Given the architecture changes between ESX 3.5 and vSphere ESX, it is a clean install — which was one of my concerns.  My boxes were 3.0.0 boxes, upgraded through 3.0.2 and then to 3.5 and 4 updates.

Lesson #3: Storage behavior in ESX4 and vCenter4 has changed…
One nice change, and I learned part of this as trial by fire (a separate post on that later), is that storage behavior changes in vCenter 4. Storage is a more unified system in vCenter 4.  If a host makes or detects a change in the storage LUNs, it automatically kicks off storage rescans on other nodes to recognize the change you just made on a single host.  It also seems to do periodic rescans on its own in ESX4 to recognize LUN changes.  These are welcome administration changes to the operation of vCenter and ESX.

Lesson #4: Be very careful about presenting LUNs to two different ESX clusters when using HP Storageworks EVA’s.
Talking about trial by fire, we experienced a pretty major failure with our storage a couple weekends ago.  ESX4 is ALUA (asymmetric logical unit access) aware – meaning it should determine its optimal path automatically and use those optimal paths.  The EVA storage should respond back with the controller who ‘owns’ that LUN as the optimal path.  So, this should present little trouble.

While this sounds great, mixing the ESX4 behavior with ESX3.5 caused us to experiencing controller ping-pong where the controllers were flipping back and forth for particular LUNs.  LUNs not used by the ESX4 nodes, yet it happened.  EVA storage will automatically move control of a LUN from one controller to another depending on where it is receiving the majority of requests.  In a shared storage environment like ESX, that can be somewhat problematic if your two clusters are using different paths and thus a majority of requests changes every few minutes.  While, its my belief that we have a sick EVA and that’s more the problem, it did bring back the notion of planned storage pathing for our fiber channel, but that’s for a longer post.

Lesson #5: Rolling out slow, especially around the holidays, is SMART…
I mentioned early that we were rolling out our ESX4 slowly.  We have installed ESX4 on our new blade cluster and we only moved test and development virtual servers over to this new environment and we are glad.  As part of our migration, we moved the backend storage for the test and development to our disaster recovery storage array and that appears to be the sick EVA – it caused a couple complete outages for these virtual servers, but because we’d done a slow migration, production (running safely on our tried and true ESX 3.5 cluster) didn’t see the problems.

We don’t plan on touching much this week with Christmas coming Thursday.  It gives me a time to update the ole blog with some of the work that has been occupying my time.   This migration has a lot more to go, so there will be more meat as I move through the process.