A couple months ago now, I sent out a cryptic tweet: “Its like Christmas in September. I’m spending some time this evening with a new product. More details to come soon on the blog…”  Soon being a relative term, I didn’t expect almost 3 months to pass before I revealed the device and my experiences with it.  But I am excited to finally talk about my experiences with the NextIO vNet I/O Maestro, officially released in mid-October.

The NextIO vNet is a converged PCI interconnect solution which takes standard PCIe network and fiber channel adapters and provisions them in a shared way back to individual rack mount servers.  The vNet extends PCI outside of the physical server through a special (albeit simple) riser PCI card then via cables to the vNet.   The provisioned resources can be reassigned to different physical hardware, if required, offering a level of portability for systems and disaster recovery.  In addition, the vNet does not add any specialized drivers or software on the hosts and the host simply see a PCI device presented to it from the vNet.   The nControl management software handles creating virtualized WWID and MAC addresses that are assigned to the individual rack mount servers through an intuitive interface.

Configuration & Testing

Installation of the vNet was very straight forward.  The unit is 4U in size and installs into any rack.  To bring the unit online, we connected power and a single network cable for management.  Once connected, the unit retrieved an address by DHCP.  Once it got an address, we were free to configure it to a static address (as most enterprise would like do).

The unit required very little configuration on our part.  The unit had two Ethernet and two Fiber Channel cards loaded on the interconnect slots.  These dual port cards were presented in the nControl management console along with all 30 possible servers which could be connected via PCI on the back-end.  The administrator is free to create profiles on any of the available server connections and these virtualized WWID and MAC addresses are portable between profiles, meaning that in the event of a failure for a critical system, the addresses could be reassigned to new hardware and the system brought back online (assuming similar enough hardware that the OS will not complain and the ability to move the OS disks to a compatible model server).

On the rackmount server side, we only needed to install a small PCI pass-through card and connect the cable from it to the vNet and it was ready to be powered on.  After this, all configuration is done in nControl.  Drivers for the Fiber Channel and Network adapters is done as normal in the operating system.  The cards I received in my demo were newer than ESX 4.1 and required us to add a OEM vendor supplied driver to ESX and the same would be true for Windows.  But in many cases, the OS vendor will bundle appropriate drivers and the solution will just work.

The management interface is easy to use with drag and drop and the vNet also supports command line configuration through SSH.  The command line interface reminded me of a configuration similar to our HP or Cisco managed switches.  Commands were easy to understand and I found no limitations of what could be configured and viewed from command line, which I know will make some Unix administrators happy.

I have always believed that a picture is worth a thousand words, so below is a logical view of how a vNet connects to a set of rack mount servers.

For my internal testing, I had two ESX hosts setup on the vNet and I was able to test network throughput between virtual machines on both boxes at amazing speeds. I was never able to get the device SAN connected to truly test some advanced features such as vMotion and disk IO due to limitations of my test hardware.  The vNet requires PCIe card slots, so users should know that going in they will not be repurposing a lot of older equipment onto a vNet, although anything produced in the last 3 year should be compatible with the solution.  I ran into issues with available hardware for my abbreviated testing since my systems were older Proliant DL380 G4′s.

I did not see boot from SAN as an option since the fiber channel card did not show during the POST of the test rack systems.  I believe this may be an option depending on the fiber channel card, but I cannot recall the exact discussion I had with NextIO about this.   Boot from SAN would greatly improve the ability to make OS profiles portable between rack servers since you no longer have to manually swap OS disk drives.  Boot from SAN could mean a remote administrator could perform a hardware failover while offsite.

Advantages

The advantages that I see with the technology are very similar to the benefits I initially show with HP Virtual Connect, although a little different since it is with more industry standard hardware.  Some of the advantages include:

• Reduced need of ports for Fiber Channel and Network, which is really useful when few applications can consume the capacity of an 8Gb fiber channel port or a 10Gb Ethernet port.   As with virtualization in its first phase which sought to increase utilization of processor and memory in hosts, this technology can be used to increase utilization of under-utilized connectivity.
• Less costly upgrades to the newest technology allowed by adding a single PCI card of a specific technology and then sharing the new features to the backend rack mount systems.
• Server to server traffic not interacting with systems outside of the vNet’s domain does not need to leave the vNet device and can travel between systems at PCIe native speeds.
• Very intuitive and easy to learn interface to provision interfaces to backend nodes.
• The special PCI riser cards are pass-through modules with no firmware or intelligence to patch and maintain.
• Interconnect cards are industry standard PCI cards – nothing special or proprietary.
• PCI traffic passes through the vNet unit even without the management module in service, meaning that a firmware upgrade or other outage in the module would not cause downtime.

Disadvantages

To be fair, I can see some downsides to this approach and these are downsides I have found with the HP Virtual Connect solutions.

• Converged solutions sometimes introduce new complexities for maintenance and patching firmware on the PCI interconnect cards and due to share nature, it could be difficult from a coordination stand-point.  This is a consideration when deciding what to run on a vNet, and in my mind makes cluster nodes and virtual hosts good candidates so that you can fail workloads onto nodes on a different vNet to enable maintenance periods.
• The solution could introduce a single point of failure, but this is easily overcome with the use of two vNet devices, although that does double the cost for both interconnect cards and vNet units.
• In some ways, it introduces a new black box to the environment whose traffic cannot be inspected which is a downside on the security end of things.

For a number of reasons, I think this solution could have a good use case with virtualization.  Particularly with a customer who is just adopting virtualization, this solution could give them the ability to repurpose fairly new servers and connect them to SAN and 10Gb Ethernet with ease.

In addition, I think that a customer who needs to limit the number of SAN and 10Gb port investment would find the technology beneficial.

Unfortunately, when multiple vendors’ products are coupled together as a solution, support can become nasty as vendors point back and forth at each other while trying to get to a resolution.  The more complex the solution, for instance a SAN, the more difficult to troubleshoot through the multiple layers of software, firmware and hardware, even multiple vendors of the solution.  And, I believe, the hassle has made customers seek a better way.

Finding a better way

In my employer’s case, they chose to standardize with a single vendor long before I joined the staff.   We have stuck with servers and storage hardware from the single vendor, including their certified part upgrades (no third party upgrade components).  We chose to do this to simplify our support and avoid finger-pointing.

The vendor we standardized with was HP, and the reason was that they offered an entire line of products under their umbrella to meet our needs.  By the time I joined the staff in 2006, we were already HP heavy, except where a specific Unix was required by another vendor.   What we wanted as a customer was the quickest and easiest route  to a resolution, with the least resistance and finger-pointing, when a problem came up.  Even beyond the hardware solutions, HP has handled our software support for Microsoft, RedHat and VMware for many years.  We wanted this because the software companies could not finger point at the hardware or vice versa – HP was doing it all.  Sure, it might happen between teams in HP occasionally, but we could easily escalate our case and have a manager bring this to a resolution.  It has worked well for our needs.

Having all this expertise in-house is an advantage that HP is now branding under the name “Converged Systems” or the “Instant-On Enterprise”.  Earlier this week, I attended a webinar for the Blogger Reality Contest where HP unpacked more of its converged solutions strategies.  HP is bringing together all of the pieces spread throughout its portfolio into specialized solutions.  Its not a new concept, in my opinion, but one that some customers have been already using for years on their own.  HP has improved on this by tweaking configurations  to squeeze performance out of configurations and adding software to ease installation and management of the solutions.

Building Upwards – HP VirtualSystem

HP introduced VirtualSystem in June as a modular, easy and quick way to implement virtualization in customer datacenters.  The VirtualSystem solution is a full package of storage and compute resources plus the software tools to quickly and easily deploy a virtual stack in an environment.

For HP VirtualSystem, the key benefits are:

• Quick built out timeframe
• Automation through Insight Control suite components
• Monitoring through the Insight Dynamics suite components
• Improved virtual machine performance, cost and scale due to purpose built hardware
• Ability to upgrade to CloudSystem for fully automated IT
• Single point of contact for support – HP for compute, storage and software, including hypervisor

HP VirtualSystem comes in 3 levels (shown below).  The VS1 is built out using rack-mount, Proliant hardware for both the server hosts and for the storage and features a P4000 series iSCSI storage array.  It is rated to handle up to 750 virtual machines and can scale up to 8 physical hosts.  The VS2 is built out using HP BladeSystem with a P4800 iSCSI storage array (covered in depth last week).  It is rated for up to 2500 virtual machines and can scale up to 24 physical hosts.  The third offering is the VS3 which is built on HP BladeSystem and the 3PAR Utility Storage platform to provide ultimate scale and performance.  VS3 introduces fiber channel storage capability and scales up to 6000 virtual machines with up to 64 hosts.

In terms of choice, VirtualSystem supports all three major hypervisors from VMware, Microsoft and Citrix.  Using my company as an example again, the multi-hypervisor datacenter already exists.  We are utilizing VMware vSphere heavily and then some Citrix XenServer.  When it came to planning upgrades for our aging MetaFrame/XenApp farm, we looked at virtualization.  As we evaluated XenServer, we found it to be “good enough” for running Citrix XenApp on top of it.  XenApp has its own failover and redundancy built into the application layer, so many of the VMware advanced features did not matter.

For VirtualSystem, HP is also handling all support for both the hardware and software for these solutions.  Having experience with HP’s software support teams, I can report that they do a good job at it.  I would not say they are always perfect, but in general, they have solved our issues and advised us well, so in reality this is a big benefit.  For those who want not on break/fix support, HP offers Proactive 24 Services for an additional level of preventative support.

Building to the cloud – HP CloudSystem

As I learned at HP Discover, just because you have a large virtualization pool in your datacenter does not mean you have a private “cloud.”  The critical difference between a virtual infrastructure and a cloud is orchestration and automation.  Built on top of HP VirtualSystem, HP CloudSystem is a solution that offers all of the necessary orchestration, service catalog and workflows to turn virtual infrastructure into a true cloud.  There is a clear and clean upgrade path from VirtualSystem into CloudSystem.  And for those starting fresh or who want to evaluate the HP solution, there is even an HP CloudStart service which will deliver a rack with CloudSystem into their datacenter and have it fully operational in 30 days or less.

CloudSystem is offered in three levels: CloudSystem Matrix, CloudSystem Enterprise and CloudSystem Service Provider.  CloudSystem Matrix is targeted towards those looking to automate the private cloud, customers who are looking to add automation and orchestration to their existing virtual systems.  It provides infrastructure as a service (IaaS) and basic application provisioning in minutes.  CloudSystem Enterprise extends upon Matrix and allows for private and hybrid cloud, enabling the bursting of workloads to public cloud.  It is a platform for hosting not only IaaS, but Platform as a Service (Paas) and Software as a Service (SaaS).  CloudSystem Enterprise provides application and infrastructure lifecycle management and allows for management of traditional IT resources in addition to virtualized resources.   The CloudSystem Service Provider edition extends upon the Enterprise edition and allows for multiple tenants on a single infrastructure, securely without exposing customer data between customers.  It is intended to host public and hosted private clouds for customers.  The editions in CloudSystem are more about capabilities and less about limits, compared to VirtualSystem.

Since automation and orchestration is the key of CloudSystem, that is where I wanted to focus.  The base of CloudSystem is the Matrix Operating System, which is the same combination of HP software found in the HP VirtualSystem solution.  On top of the Matrix Operating System, the CloudSystem Matrix solution includes Cloud Service Automation for Matrix.  This software includes Server Automation for lifecycle management for physical and virtual assets via a single portal and set of processes and HP SiteScope, an agent-less monitoring solution for performance and availability.

The enterprise and service provider editions include a beefed up version called, simply, Cloud Service Automation.  It includes the entire orchestration, database and middleware automation pieces of the pie and a cloud controller software.  These additional pieces allow not only the automatic and streamlined provisioning of physical and virtual servers but also the provisioning of the required glue that sits in between the apps and the servers.  The diagram below from HP shows all the moving parts of Cloud Service Automation better than I can explain in words.  And because, Cloud Service Automation is total lifecycle management, there are the pieces for monitoring and performance management which would be needed.  In addition, the centralized portals serve as point for both end users and IT professionals to manage the cloud.

Cloud Maps are another feature of CloudSystem and these are predefined automation workflows for deploying software and platforms easily.  These are the piece of the puzzle that allows for improved deployment times and also allow for drag and drop creation of new workflows and processes in the cloud.  HP has worked with its software partners to create these maps of requirements and automate the process of deploying their solutions.

Beyond all of the capabilities, HP is working hard to make this an open solution by making it compatible to burst workloads into third party clouds, whether its Amazon’s EC3 or a vCloud service provider.  This was a point stressed during the announcements at HP Discover and during the call on Tuesday.

This is post number two for Thomas Jones’ Blogger Reality Show sponsored by HP and Ivy Worldwide. I ask that readers be as engaged and responsive as possible during this contest.  I would like to see comments and conversations that these entries spark, tweets and retweets if it interests you and I also request that you vote for this entry using the thumbs up/thumbs at the top of this page.  As I said earlier, our readers play a large part in scoring, so participate in my blog and all the others!

This isn’t the first time I’ve written about CloudSystem.  In June,  I posted about my take on CloudSystem Service Provider from a potential service provider’s perspective.  I encourage you to take a look at that post, too, after you take a minute to comment and/or vote on this post.

Converge.  That theme is a tougher sell in my employer’s environment.  There is resistance to converging, whether it’s IP telephony on our data network versus converging our fiber channel with fiber channel over Ethernet and putting it on our same core Ethernet network.  Same would go for iSCSI, if we had it.  We tend to separate for simplicity reasons, but there are certainly cost savings in convergence.

 Why converge?

Convergence is a major trend in IT, today, although it goes by many names.  But like most trends and buzzwords (think Cloud), your mileage will vary among vendors and interpretations of the buzz.  HP’s approach to convergence is largely centered around standardized x86 hardware for both server and storage platforms.  In addition, the converged storage platforms within HP are about scale out, with multiple controllers to handle unpredictable and unruly disk I/O with ease.

Before moving into a discussion of converged storage, though, it is worth taking a moment to talk about how things have been done in the past.  For the past 20 years, storage has been largely created around a monolithic model.  This model consisted of dual controllers and shelves of JBOD’s for capacity.  The entire workload and orchestration of the array was trusted to the controllers.  With the traditional workloads, the controllers performed well.  In the old world, capacity was the limitation on data arrays.

Today, the demands of virtualization and cloud architectures on storage have considerably changed the workloads.  The I/O is unpredictable and burst large amounts of traffic to the arrays.  This is not what our traditional arrays were designed for and the controllers were paying the price.  In a large number cases, including my company, the controllers become oversubscribed before the capacity of the disks are exhausted so you don’t realize your full investment. Monolithic arrays come with a high up-front price tag.  When one is “full”, it is a big hassle and cost to bring in a new array and migrate. But these have been the work horses of our IT operations.  They are trusted.

Hitting the wall

Within the past couple years, I have found the limitation of the controllers to be a significant problem within our environment.  And even after significant upgrades to high-end HP EVA within our company, we can still see times when the disk I/O overwhelms our controllers to a point that disk latency increases and response slows.

The controller pain points are one of the driving forces behind converged storage.  Converged storage is the “ability to provide a single infrastructure, that provides with server storage and networking and rack that in a common management framework,” says Lee Johns, HP’s Director of Converged Storage.  “It enables a much simpler delivery of IT.”

What is different with converged storage?

Across the board, convergence leverages standardized, commodity hardware to lower costs and improve the ability to scale out.  Converged storage is about taking those same x86 servers and creating a SAN that can adapt to the demands of today’s cloud and virtualization.  Instead of the limitations of a single set of controllers, intelligently clustered server nodes enables each server in the array to serve as a controller.

Through distribution of control, the cluster is able to handle the bursts of I/O more easily across all of its members than a monolithic array controller is able.  The software becomes the major player in the array operations and it really is a paradigm shift for storage administrators.  No longer is storage a basic building block, it is just another application running on x86 hardware.

Diving deep into the HP P4800 G2 SAN solution

Perhaps the best way to understand converged storage is to look at a highly evolved converged data array.  On Tuesday, Dale Degen, the worldwide product manager for the HP P4000 arrays, introduced our blogger crew to the P4800 G2, built on HP’s C7000 Bladesystem chassis.

The core of the LeftHand Networks and now P4000 series arrays is the SAN/iQ software.  The SAN/iQ takes individual storage blade servers and clusters them into an array of controllers. This clustering allows for scale out as you need additional processing ability to handle the workload.  Each of the storage blades is connected to its own MDS-600 disk enclosure via a SATA switch on the interconnect bays of the blade center.  The individual nodes of the array mirror and spread the data over the entire environment.  One of the best things about the SAN/iQ software is its ability to replicate to a different datacenter and handle seamless failover if one site is lost.  (Today, in my fiber channel world, if I lose an array, it involves presentation changes to bring up my replica from another EVA, so this is a huge plus.)

By leveraging the HP Bladesystem for the P4800 G2, you can also leverage its native abilities, such as the shared 10Gb Ethernet interconnects and Flex-10.  For blades in the same chassis with the P4800, the iSCSI traffic never has to leave the enclosure and it is allowed to flow at speeds up to 10Gb (unless you have split your connection into multiple NICs).

From an administrative standpoint, the P4800 is managed just like any other blade server in the C7000 enclosure.  These blades are standard servers, except that they include the SATA interface cards.  They include standard features like iLO (Integrated Lights Out) management, VirtualConnect for Ethernet network configuration, and the Onboard Administrator for overall blade health and management.

Within a single chassis, the P4800 can scale up to 8 storage blades (half of the chassis).  The iSCSI SAN is not limited to presentation within the same C7000 or within the BladeSystems at all.  It is a standard iSCSI SAN which can be presented to any iSCSI capable server in the datacenter.

The P4800 G2 is available in two ways.  For customers new to the C7000, they may purchase a factory integrated P4800 G2 and C7000 chassis together.  For existing customers with a C7000 and available blade slots, the P4800 G2 can be integrated with the purchase of blades, SATA interconnects and one or more MDS-600 disk enclosures.  For existing customers, you must also purchase the installation services for the P4800 G2.

The P4800 is a scale up technology also.   Customers do not need to migrate everything at one time.  It allows for a single infrastructure and allows you to move onto it over time by adding additional storage blades and MDS-600 disk enclosures.

As a quick side note, this is the first entry for Thomas Jones’ Blogger Reality Show sponsored by HP and Ivy Worldwide. I ask that readers be as engaged and responsive as possible during this contest.  I would like to see comments and conversations that these entries spark, tweets and retweets if it interests you and I also request that you vote for this entry using the thumbs up/thumbs at the top of this page.  As I said earlier, our readers play a large part in scoring, so participate in my blog and all the others!

Up until January, we had scripted everything for VMware in VI-Perl.  I don’t begin to call myself a Perl programmer, but I can crawl in and modify a canned Perl script and I am able to make it do what I want.  That said, it was always a pain to do it.

For our VMware environment, we have monitoring scripts that report low disk space for our guests, a daily snapshot report and what VM’s have been provisioned or deleted from the system.  All of these were things I had to find scripts that had already been written in VI-Perl, and so we relegated scripting to simply monitoring in our environment. After we added the VMware Management Appliance to our network, we used it occasionally for Storage VMotion in the ESX 3.5 days, before it was added to the GUI.

With PowerCLI, however, we have moved into doing more actual configuration and changes — real work — using it.  Because PowerShell support piping the objects between commands, we have quickly realized gains in efficiency by retrieving a list of targets and issuing a command against the group.  Some things we have done manually in the past make much more sense in PowerShell.  Performing a VMotion or Storage VMotion is a good example of this.  Storage VMotion involves a number of steps (click, click, click, click, click) to perform in the GUI.  In Powershell, its a simple, one-line and easy to understand command.

PowerCLI is much more user friendly to me than writing Perl.  Soon, I plan on rewriting my management scripts using PowerCLI. There are some security related things I must figure out (haven’t even looked into them yet) – like signing PowerShell scripts with a certificate, which will be required for executing scripts on a schedule.  (In our environment, we plan on making all of our PowerShell instances run in remote-signed execution policy to maintain security.

Another great reason to move to PowerCLI for VMware management in our environment is that current and future version of Microsoft products, which my group also administers, will be largely manageable from PowerShell.

Some examples that I have found useful so far:

Storage VMotion a VM

Connect to vCenter or an ESX server using connect-viserver.

Get-VM "VMNAME" | Move-VM -datastore (Get-datastore "DATASTORENAME") -runasync

Show VMs that have Raw Disk Mappings

Connect to vCenter (its most useful in an entire datacenter, but works against an ESX hosts also).

Get-VM | Get-HardDisk | Where-Object { $_.DiskType -like "Raw*"} |
   `Select-Object -property @{N="VM";E={$_.Parent}},DiskType,
   Name,ScsiCanonicalName,DeviceName,CapacityKB

Another big reason I’m glad we were exposed to PowerCLI at this time is we all know the future roadmap of vSphere.  The next major release will remove the ESX version of the hypervisor and only include the ESXi (console-less) hypervisor.  Once we make that migration, things we have done in the past will no longer be possible.  By moving to remote command-line administration today, we are curbing some of our pain of moving there when we also move to the new vSphere release.

PowerCLI for HP EVA best practices of VMware settings

But, as I wrote about when we found the EVA best practices, we still need to modify some other settings on the hypervisor — specifically chaning the IOPS value for LUNs is recommended in HP’s best practices.  So, in the past week, I have been looking at how to do this from PowerCLI.

I have not written about this, but our EVA’s have been having a large amount of issues in the recent months.  One reason I have not written anything is that we still have not gotten a root cause for our problems.    But, we have been working extensively with HP’s senior level support team for a few months now.  One of the things we looked at was our ESX configuration and once again, we were sent to their best practices document for VMware.  Fortunately, we already had their recommendations in place, so there was little to worry about.

But, going back and looking at these settings again, we were able to find ways to set these settings (without reboot to boot) using PowerCLI instead of using command line from the ESX console.  So, I certainly wanted to share these, along with the sources where I found them,  for anyone using HP EVA in their environments.

Set the Multipath Policies of a Host to Round Robin from PowerCLI (for HP EVA best practices)

Connect to vCenter

Get-VMHost <hostname> | Get-ScsiLun -CanonicalName "naa.600*" |
   Set-ScsiLun -MultipathPolicy "roundrobin"

Remember, that for the hostname, you can use wildcards to get all the hosts in a cluster or simply a * for all hosts!

>> Source

Set IOPS values of disks to 1 from PowerCLI (for HP EVA best practices)

Connect to an individual ESX host for this.  Its cumbersome, but provides the functionality.

$esxcli = Get-EsxCli
$esxcli.nmp.device.list() | where {$_.device -like "naa.600*"} | %{
    $configBefore = $esxcli.nmp.roundrobin.getconfig($_.device)
    $esxcli.nmp.roundrobin.setconfig(0,$_.device,[long]1,"iops",$false)
    $configAfter = $esxcli.nmp.roundrobin.getconfig($_.device)

# Uncomment the following lines if you want to report the settings
#   $configBefore
#   $configAfter
}

>> Source

First, SafeCopy looks a lot like Mozy on the surface.  I immediately see that they are cross platform with both Windows and Mac compatibility, but they also list iPhone.  Digging a bit deeper, I have found that SafeCopy also has a search engine feature for finding and retrieving your data online and thus all your files are from the iPhone or other mobile phone.  I have to say, the Google-style search for my backup is cool.  The online demo shows that you can see versioning of the individual files, also.   Like Mozy, the backups are encrypted.  SafeCopy uses 448-bit Blowfish encryption according to their website.  

Second, their data caps for the yearly rates are really competitive and more realistic in my opinion.  They show two plans on their website a 200GB and a 300GB a year plan available for $50 and $70 a year, respectively.  I’m currently paying about $100 per year for two computers at Mozy, although both computers have unlimited storage (but only for now).  All of my data currently backed up would fit within the 300GB limit, even with versioning (a lot of my data does not change — music and photo libraries, etc.).

Third, its one account for all our home computers – meaning, while I’ve only ever run Mozy on two computers, I could potentially protect more of our home computers.  This is fundamentally different than how Mozy used to work — where Mozy used to charge by the number of computers, SafeCopy charges by data limit and you’re free to use as many computers as you want.  Mozy will limit its users in both number of computers and the amount of storage moving forward.

Beyond these three points, however, SafeCopy has some very appealing features to differentiate it from other providers that I have seen.  The first of these features is something they call “TurboCopy”.  Essentially, “TurboCopy” is deduplication, and this is one area I found Mozy to be lacking.  For instance, I have two iTunes libraries with the same music, but I backed up both on Mozy (yes, blame me for killing unlimited storage plans by wasting space like this).  With SafeCopy, it should only store a single copy of these files and then allow me pointers for both computers to access them.  That is also the best use for your storage capacity.  By only storing the single copy, you’re not consuming your storage allotment with all the same data.

The other major differentiator is real-time backup.  The SafeCopy software is constantly watching for changes on the file system and when it finds one, it moves that to the backup cloud.    There are no schedules to manage making it even simpler to setup and maintain a backup.  But, this feature also worries me.  I am concerned about saturating my internet connection — a problem I experienced with Mozy even though it had trottling.  Fortunately, SafeCopy offers throttling, also.  Saturating my connection has become a problem since I added a Microcell at home.  When saturated, this obviously affects my ability to make and receive phone calls.

So, I will be beginning my SafeCopy trial this weekend and I’ll be posting my additional thoughts and impressions after I begin the process of backing up to their service.  And that brings me to the best part…  SafeCopy offers a free account with 3GB capacity.  I’ll be starting with this initially to test and setup my computers.  Stay tuned.

And, you can sign up for Safecopy below:

I had been worried about losing my digital home movies.  That data is really too large to really push out to a backup service like Mozy and when I load new movies, its usually to the tune of 20 or 30GB at a time, which would take weeks to push up.  In addition to that data, I also have Movies and TV shows that we have purchased through iTunes.

So, I knew that I wanted some sort of external storage with at least mirroring capabilities to protect the movies as best I could.  Back in January, I was close to buying a Western Digital mirrored external hard drive from the Apple store.  I chose instead to get a 1TB, single external drive with Firewire as an upgrade to my 500GB Time Machine which was almost full, and wait for a better solution.

Since it was first released, I have always been in love with the Drobo.  I describe it as an mini-EVA to my co-workers, because it mirrors a lot of the HP Storageworks EVA functionality – like drives auto-leveling and automatically striping data across disks in the disk group.  Maybe, I just like the idea of having something that advanced attached to my home computers…  and I’ll be the first to admit that they don’t compare — they are apples and oranges.

But I do love what the Drobo offers, and so I have been watching them for some time.  Since its introduction, the Drobo had gone from a single device with USB only, to an enhanced version which has USB2 and Firewire (my choice of devices), a version which adds eSATA with USB and Firewire, and several larger devices, the Drobo Elite and the Drobo Pro, which feature iSCSI and NAS functionality targeted towards SMB’s.

As a side note, the Drobo Pro is even VMware certified, and I feel like it is a great solution for small to medium businesses looking for shared storage for an vSphere deployment.  This past week, we attended the Charlotte Regional VMware Users Group meeting, and actually got to see one of the Drobo Elite units on display.

The price has held steady on the Drobo that I have been watching.  At a $399 price point, I couldn’t justify buying it and then having to purchase drives to go inside.  But, as I said before, B&H was offering a great deal – the Drobo device at $100 off its normal MSRP.  It was a good $50 less than I could find it anywhere else, and so I bit — err bought.  I was able to equip it with two Western Digital Caviar Green (my wife would be proud) 1TB drives from NewEgg for a cost of $138.  And so, last night I finished up my transition of data and have everything I wanted protected on the Drobo – and I’m happy…

Our errors look like this:

After several cases with VMware and HP technical support, we are no closer to resolving the issues.  VMware support, for its part, has done a good job of telling us what ESX is reacting to and seeing.  HP support, on the other hand, has been circling around the problem but has made little progress in diagnosing the issue.  We have had an ongoing case for several months and our primary EVA resource at HP has continually examined the EVAperf information and SSSU output that we have sent to HP for analysis.  Those have turned up nothing, and yet the messages continue from VMware.

The errors in the log make sense to me – we are losing a path to a data disk (sometimes even a boot-from-SAN ESX OS disk!) – but why HP cannot see anything in our Brocade switches or within the EVA is beyond me.   Our ESX hosts, whether blade or rack-mounted hardware, are seeing the problems across the board.  The one cluster we waited to upgrade never saw the issues in ESX3.5, but sees them now in ESX4.  And perhaps it is a VMware issue that is just too sensitive in monitoring its storage, but I suspect its something else.   The messages don’t seem to affect operation on the hosts, but it certainly makes investigating problems difficult when trying to determine what is a real problem versus just another failed path message.  Anyone else seeing this?

HP provided us with the best practices document for ESX4 connected to an HP EVA array.  There is a major change in ESX4.  For the first time, ESX is ALUA (Asymmetric Logical Unit Access) aware.  (See this post on Yellow Bricks for more detail about ALUA.)  ALUA allows the array and ESX to determine the optimal path — the path to the managing controller’s ports in the EVA’s case — and use those optimal paths until one isn’t available.  This is important because it prevents flip-flopping on the ESX host.

In previous version of ESX, the desired storage setting was fixed path for the EVA.  In our case, we simultaneously presented the ESX3.5 and ESX4 hosts to the same LUNs, meaning some were fixed and some were set to the ESX4 default, which was MRU.  This caused problems.  After initial issues, we backed away and presented one LUN at a time, performed our VMotions and then unpresented the LUN from the old cluster.  This prevented any flapping issues between controllers.

The best practices document describes two settings for best practice with an EVA that is ALUA aware (EVA 4000/6000/8000 and EVA 4400/6400/8400):

• Set the default path selection policy to round robin (VMW_PSP_RR)
• Set the IOPS Limit to 1

The path selection policy is the bigger deal.   The default path selection policy in ESX4 is most recently used (MRU).  This is still a valid policy and works with the EVA, but it is not their best recommendation because Round Robin allows the ESX host to use two paths or four paths (depending on EVA model) to send traffic to the ESX host which can provide greater performance.

I was not sure what the IOPS setting actually does based on the documentation, but Duncan Epping has since posted about the IOPS=1 setting and about how in a real-world environment this setting may not make a difference.  I agree with his conclusions.

Now, I should not and do not want to take credit for the instructions below… I did not come up with this solution — simply a user.  The esxcli commands came directly from the HP Best Practices Guide, but the scripted IOPS setting changes for the LUNs came from another blog - Virtual IEF.  This solution has worked brilliantly in our environment.  Virutal IEF has a great, in-depth posting about how the EVA LUN ownership works and what makes an optimal path.

Setting Changes for Round Robin PSP
To change the default path selection policy (PSP) to Round Robin. If LUNs are already presented, you will need to reboot to allow the LUNs to rescan and set to Round Robin instead of the default most recently used (MRU) setting.

 esxcli nmp satp setdefaultpsp --satp VMW_SATP_ALUA --psp VMW_PSP_RR

Change the IOPS limit value for the interactive session.

 for i in `ls /vmfs/devices/disks/ | grep naa.600`;
    do esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device $i ;done

Verify the IOPS value is correct, run this script and check values.

 for i in `ls /vmfs/devices/disks/ | grep naa.600`;
    do esxcli nmp roundrobin getconfig --device $i ;done

Your output should look like:

 Device: naa.600508b4001087e400009000012b0000
 I/O Operation Limit: 1
 Limit Type: Iops
 Use Active Unoptimized Paths: false
 Errors:
 Unable to find device with the name naa.600508b4001087e400009000012b0000:1
 Byte Limit: 10485760

Unfortunately, the IOPS value will reset on reboot many times. To ensure that this value is correctly set, you are able to set a value in the post boot script. Add the same for statement to/etc/rc.local to set the IOPS value on each reboot.

 for i in `ls /vmfs/devices/disks/ | grep naa.600`;
    do esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device $i ;done

Background
At work, we use raw disk mappings (RDM) some within our VMware virtual environment.   The last VMworld, I was inundated with folks saying that using an RDM for any reason other than a Microsoft cluster is a bad idea anymore.  None of them really gave concrete reasons why, other than saying that VMDK performance had improved so much that there was really no reason to use them.  Their recommendation was to use them only where required, such as clustering.  

In our datacenter, we have chosen to implement RDM’s for large filesystems, typically database applications.  All of our SQL server VM’s and all of our Oracle VM’s run raw disk mappings for their data storage.   Before we felt comfortable about running these workloads in VM’s and knowing how they’d perform, RDM’s gave us a way to quickly move back to physical server, if needed.  That reasons still stands, though our confidence in our virtual infrastructure has increased considerably.  We have no qualms about moving SQL into a VM, as a matter of fact, we prefer it in most cases for redundancy and disaster recovery reasons.   The only physical Microsoft SQL Server is our multi-instance cluster, something that has redundancy and disaster recovery covered.

Growing, growing, grown
We initially liked RDM’s for Windows VM’s so that we could grow our disks using the SAN array and then extend the partition using diskpart.  This always worked well with our physical database servers, so we continued the practice into the virtual world.  We initally implemented physical mode RDM’s for our SQL servers, but we quickly found out that physical RDM’s prohibit snapshots and other features, so we changed to virtual mode RDM’s. 

With the virtual mode RDM, we found that growing the filesystem and then getting the VM to recognize the new space was more difficult.  Most information I saw indicated that you’d need to reboot the VM to recongize the space, but then I happened upon a great post in the VMware forums.  This post said to rescan the host, VMotion the VM and then rescan the VM and the disk space would be recognized.  This worked great.

To use or not to use
I realize that today, growing a VMDK is an easy task.   ESX 3.5 and 4.0 allow you to grow a VMDK from the Edit Settings… menu, but when we began in the 3.0 days, it wasn’t an easy thing to grow a VMDK.   So, we went this direction and we continue today to keep things consistent.  I think its safe to recommend that others use VMDK’s, since I do agree that the performance hit is small for using a VMDK versus a raw LUN.   That advice is sound, but I wanted to post this for others who might be in our situation, with VM’s that were already setup with virtual RDM’s and trying to figure the way to grow their filesystems.

An EVA normally has two (maybe more, I’m not primarily a storage guy) controllers and those handle all the requests received through the SAN.  For every LUN, one controller is its master.  Both controllers can handle requests for the LUN, but only one actually handles the access.  If the controller on fabric A is the primary but the controller on fabric B is getting more requests, eventually the EVA swaps control for the LUN to fabric B — wherever the majority of requests are coming.

This behavior would only become a problem if you had hosts configured to access the LUN on different fabrics.  ESX4 is ALUA (asymmetric logical unit access) aware, meaning it should automatically determine the optimal path and in the case of an EVA.  The EVA, I’m told by HP support, is supposed to respond an ALUA request for the optimal path by responding with the controller that is the master over the LUN.

If you, like us, have an ESX 3.5 cluster with preferred paths setup, you should proceed with caution.  The ALUA information isn’t apparently shared between clusters.  And if your clusters get different optimal paths, you could end up with controller ping-pong as requests are sent down both fabrics and the volume changes between the two, resulting in more on Fabric A followed by more on Fabric B — forcing the controller to switch masters.

So, while in a migratory state, I think my safest route is to configure the ESX4 hosts to use a preferred path like the ESX3.5 cluster nodes.  I hate to move from the default ESX configuration and this isn’t an official recommendation from HP support, but it certainly makes the most sense to define the paths being used (except in a failure).

I post this because I feel like there have to be other HP Storageworks customers who have the same situation or have experienced something similar.  I would love to hear from you…