So, a little more than a week after its launch, employees were allowed to purchase iPhone 4′s for our personal use.  And I snapped one up.  The 32GB models were in low supply, so I opted for a 16GB, even though I had already filled my previous 16GB iPhone 3G model.  First impressions — I really love the iPhone 4.  The camera is fantastic, the screen is really amazing, and its speedy (which any original iPhone 3G owner can appreciate).   And iOS 4 is a great enhancement when you have the better processors.   I can attest, though, it runs like mud on an iPhone 3G.

That said, I found the attenna issue to be a real thing.  If I touch the left side near the break point, my signal diminishes.  Even with iOS 4.01 update, I still see signal go down.  Apple did discover that they had a software error that caused misrepresentation of signal, but the iOS 4.01 update didn’t resolve all the problems for me.   However, I believe that a case will fix the problem as Consumer Reports and Apple have both reported.  And so, today, I downloaded the iPhone 4 Case Program app from the App Store and selected my free case.  The shipping time shows 3 to 5 weeks to arrive (which is the only bummer), but Apple did a great job of making this program easy to use from a customer’s perspective.

To order your free case from Apple, get the app, launch it and then click the blue button.  You are asked to log into your iTunes account and then the app apparently checks the fulfillment system against your serial and/or iTunes account to see if you have a pending order.  If you have a pending order, it will display a message — otherwise, it displays a list of the available cases.  I was impressed with the selection — cases from Speck, Belkin and other vendors.  I selected my Speck case and placed the order.  The app automatically pulled my address and name information from my iTunes account and so one more tap and the order was complete.  It took only about 4 or 5 taps on screen to complete my order — which is a testament to Apple’s mentality and method of creating easy customer experiences.

That all said, I’m not sure that Apple PR or Steve Jobs handled the situation very gracefully, given the type of experience they strive to create for the customer.  But that is my personal opinion and no one can discount that Apple has made an exceptional device and people want them.  Even with the antenna problem, it is the best iPhone yet.  I can’t wait to see what is up their sleeves next.    That marks one thing off of my Apple to-buy list, but a new iMac and iPad are still on the list…  just waiting until the house comes a little further and we see how all of the finances for it works out.  We are trying to be very budget conscious right now.

http://communities.vmware.com/community/vmug/us-southeast/myrtlebeach

Part of the problem is that documentation tends to get outdated.  Keeping your notes updated as changes are made is tough.  Old documentation is sometimes worse than no documentation…  Its sometimes better to get inside and dig around to see how things are actually working/setup.

After I started at HTC, my co-worker David showed me his Mediawiki setup and running from his personal Linux desktop that he was using for documentation.  He has amassed a pretty respectible collection of documents on the *nix, backup and networking side of the house.  Too bad I was a Windows guy for the company (ouch, did I just type that…  I’m so the anti-Windows Windows guy…).  

After I started the VMware VI 3 deployment, I quickly realized I needed to be keeping notes somewhere for this monster project.  I asked if I would be able to use the wiki he’d established to keep my notes and he agreed.  As we talked on, we quickly realized that everyone in our group was likely in the same boat with their notes.  We had documents strown all over our file share with a hodge podge of information, none of it readily available (searchable) when needed.  

So, after getting VMware deployed and as a test of deploying our first Red Hat VM, we setup something we called NetAdminWiki…  We secretly plotted planned to continue putting our information in this repository and then open it up to our co-workers over time to get them on-board.  I’m glad to say that a year later, we’ve now got all our co-workers with working accounts on the wiki, but we ran into a few issues that we needed to conquer before putting it out before our bosses.  

MediaWiki in the enterprise
MediaWiki is a great solution for documentation, but its meant for everyone to contribute – that’s its focus – a very wide audience.  In our case, we wanted it to only be available and accessible to a select audience.  That initially ment password protecting the Apache website using HTAccess.  Most of our co-workers didn’t like the idea of logging onto the web server and then into MediaWiki…  

We found an extension for MediaWiki that allowed us to do seemless signin using the authenticated account passed from Apache.  That worked great, but then users wanted to use their AD account and password.  That led us to making Apache authenticate against AD’s LDAP and login users that way.  This method was messy and it messed with MediaWiki’s built in account creations method.  

This past week, I finally cleaned all this up as I was asked to add our last co-worker into the wiki.  MediaWiki now has a great LDAP Authentication module with few dependancies.  Once I setup this module, we were logging in against the domain with ease.  We removed the Apache authentication and we now have users logging in through the normal MediaWiki method, but that introduced a new problem.  Anyone could read our content.

The extension library for MediaWiki is expansive, so finding a solution for the new problem was a snap.  With just a few changes to the LocalSettings.php file, we were able to lock down access to the wiki, so that you had to login to ready any content or make changes.  MediaWiki has a great article about preventing access in their online manual.

With these two extensions, we’ve now made a pretty nice documentation solution for our group.  Its possible to take this much farther.  The LDAP Authentication solution also has the ability to synchronize MediaWiki groups with AD groups and you can provide additional permissioning using AD groups.  There are a lot of options here.

Of course, my story has to begin on Friday the 13th…  Sometime around 9:00am, we started getting calls for both our SQL 2005 database cluster and our Exchange cluster.  After investigation, we found that the active nodes were both in the same enclosure and a third ESX host in the same was experiencing problems, too.  The problems were affecting both network and disk IO on the blades.  All of our blades are boot from SAN, so the IO had to be a fiber-channel issue.  

Several hours later, we were finally able to get enough response out of the nodes to be able to force a failover of services for Exchange, shortly followed by SQL 2005.  As I worked with HP support, nothing improved on the affected servers.  We were finally diagnosed with a problem mid-plane on the enclosure.  

 While waiting for the mid-plane to be dispatched to the field service folks, I requested that we go ahead and do a complete power-down on the enclosure and bring it up clean.  This required physically removing power from the enclosure after powering down everything that I could from the onboard administrator.  

After the reboot, everything looked much healthier.  The blades came back to life and everything began operating as expected.  After intense discussions on the HP side, we reseated our OA’s and the sleeve that they plug into on the back side of the enclosure.  Net outcome was the same – everything still operating well.  The OA’s nor the sleeve were loose, so we doubted that was the cause.  

One nugget I learned from HP support (please vett this information on your own), is that the Virtual Connect interconnect modules require communication with the onboard administrators (OA’s).  I’m still not sure I fully understand, but HP support did tell us that if VC lost communication to the OA, its possible that it caused our problems.  If this is so, this smells like very, very bad engineering and design…

Continued investigation on HP’s part has pointed us back to the original diagnosis – a faulty mid-plane.  Only by default did we return to that conculsion, however.  This is the only piece of hardware common to the problems.  Our only other conclusion was that this was a very bad, “hiccup” — which obviously buys us no real peace of mind…  

So, sometime soon, we will be replacing the mid-plane of our enclosure.   I have, of course, lost some faith in the HP blade ecosystem.  We have plans to migrate our corporate VMware cluster onto blades, as well as some Citrix and other servers.  Losing an enclosure like this has un-nerved those plans.  We were fortunate to have drug our feet  to only have 3 blades populated and serving anything at the time this happened.  I will post updates as we move forward…

That’s how my 2009 started… about 13 hours later, I finally left work and resumed my long-interrupted nap.   We had what seems to have been a storage meltdown behind our VMware farm yesterday.  Our file sharing cluster was also affected and so our few employees who were working on New Years Day, well, weren’t working at all.  The short version of the story goes like this.  Our scheduled backup process, using EMC Networker, kicked off VCB backups on the ESX 3.5 hosts around 1:30 am.  By 2:00 am, the process was trying to create snapshots on VMs and this caused some sort of meltdown due to SCSI reservations (found the SCSI reservation problem after VMware analysis).  Turns out the HP Insight Agents loaded on our VMware hosts were causing these SCSI reservation issues.  The agents were checking the disks at a consistent interval and we had not upgraded the agents to the latest revision, which was supported with ESX 3.5 – so not VMware’s fault – they have a great KB article about this issue (see KB 1005009).   As an immediate resolution, one of my co-workers removed the HP agents from our hosts and worked our way through rebooting the entire farm, one host at a time to remove the SCSI reservations.  I cross my fingers on VCB backups working when they kick off in an hour.  Had this been the only issue, we would have been fine.

Unfortunately, at around 4:30 a.m., while I was unaware, our cluster began experiencing troubles, too.  And this is where our detective skills have come up short.  We have been sleuthing to find the cause of some weirdness in both our file sharing and Exchange clusters for several weeks, now.  The file share cluster is, dumbly enough, critical in our environment.  Without it, our users home directories are inaccessible and, since these home directories are defined in Active Directory, it seemingly hoses up our employee’s workstations.  Things that should otherwise be speedy, say opening a program — any program — or browsing to your local hard drive, become unbearably slow.  Even running applications sometimes lock up as it attempts to access some unknown part of Windows during normal operation.  It brings our entire business operation to crawl and that’s unacceptable.  (BTW, if all this sounds familiar, please leave comments or send me an email with suggestions.)

So, what actually happened to our Windows file sharing cluster?  We have an issue where we see the network utilization on the file share cluster drop to nothing, but the cluster nodes still respond to ping and other non-storage related network services – but what we found out later in the process – not to anything which needed IO to respond.  After repeated network sniffs, we were seeing that traffic would come to the cluster, it would be acknowledge, but the node would not start sending data.  The break between the request and data could be as long as 20 or 30 seconds.  And that was consistent with our ‘outage’ periods.  So, I decided to fail over the cluster shares from one node to the node that had been ‘solving’ the problems in the past.  When attempting to fail the share, they locked and never became accessible again.  After waiting for almost an hour, I rebooted a node trying to clear up the locks and let the other node take control.  That was never possible either.  A reboot of the second node only served to cause it to stall during boot, and never provide me a login screen.  Rebooting the other node, same result.  And then, a lengthy phone call with HP support after driving into the office.

The short version of this is that we are running Windows Server 2003 with SP2.  Apparently, therein lies our problems with a) clustering and b) storport.sys.  The StorPort driver issues are pretty well documented and it in combination with several other hotfixes are HP’s recommendation to us.  The hotfixes were released outside of Microsoft’s normal patch schedule to the large number of customers having issues similiar to this.  HP’s recommendation to us was to install the list of suggested hotfixes for post-SP2 (Microsoft KB 935640).  My co-workers successfully completed that on the file share cluster this evening without incident.  (Hallelujia!)  I got the call shortly around 10:15 pm with the all clear.

The file sharing problem has been plauging us for several weeks and we have not been able to deduct the full cause.  We have had theories and as soon as we seem to figure it out or know how it will act, we’re proven wrong.  At least until yesterday, we hope.  The next week will tell for sure.

I also mentioned issues on our Exchange cluster.  We’re not actually sure its having issues.  It may only be showing issues at the same time the file share cluster was having issues.  We believe the above issues on file shares are causing lock ups on client machines and their programs, so we are currently thinking Exchange’s perceived network issues are just the fact that our employee’s Outlook is locked and can’t connect, so we see what looks like Exchange issues.  But, then again, we’re not 100% sure.  We still have some detective work to do here.  Where’s Sherlock Holmes when you need him?

As for VMware, we still have a punch list for a few additional things to do – putting newer HP agents on the ESX hosts for one.  There are some things we may want to customize here based on feedback from VMware.  We may want to disable the storage monitoring agent on these hosts, but more research is required.  Removing these agents all together for now is a our preferred fix.  So, for the next hour, I just need to keep myself awake and hope that VCB backups will go well tonight.

We have trialed on other diskless option – the Pano Logic solution – which we really like.  The most compelling thing to us about these solutions is that there is no OS image to worry with – nothing to ever need patching.  It boots into a BIOS and directs to a desktop solution controlled from the datacenter.  Even with thin-client OS’s, it seems there are vunerabilities and upgrades needed in the future and there is obviously a need to have a management interface for these devices.  That’s another layer to administer.  

The biggest downside to me is price on these units.  With the cheapest quote coming in over $500, that’s a good $200 more than Pano’s solution and that’s without significan’t support.  The support cost differences between the devices almost rules the Dell solution out all together.  

It is nice to see Dell entering this arena and recongizing the viabliity of this market, but their first entry is more towards a fully powered thin-client solution.  And yes, I know I just complained earlier about thin-client being under-performers.  But, at the same time, you want to have a thin-client that isn’t going to cost just as much as a PC…  I dunno, I guess you just can’t please me…  though I do like the Pano Logic solution so far.

Update:  After posting, I thought I should post a link to the FX160.

Beyond that, we are presenting one of the thin client solutions to our CIO tomorrow for review.  It has the potental to save us some money – by saving technician’s time, some on license fees and also the need to roll a truck to some of our remote locations in the county to replace a bum PC.  We are still waiting for our wonderful HP evaluation units…

Our company wants these to be realistic, attainable, measureable goals.  So that includes a time frame for completion.  My only one so far that really meets this criteria and that I am fully in control of is attaining my VCP certification for VMware.   Any thoughts?

And that seems to be the case with many companies.  We are a large HP shop in the datacenter, Dell at the desktop.  Both of them are trying to horizontally and vertically align themselves to meet most everyone’s needs (ok, moreso HP).  Microsoft is trying to conquer rule compete in every area of software.  Anyways, just a quick rant.

I should preface that we have a Microsoft Enterprise Agreement (EA) for our licensing.  This came about several years ago to mitigate auditing and the necessity to keep up with licenses at a granular level within our company.   For a company of our size, this could be one person’s full time job – just to handle license compliance on an ongoing basis.  The EA allows us the ability to add both desktops and servers throughout the year while not needing to purchase anything.  At the end of the year, we perform a true up and pay for the number of new servers or desktops deployed.   It also gives us access to volume license keys which keeps things nice and tidy on the deployment side.  For auditing, we have SMS and other tools to help us know how many devices we’ve added.  

Please note, the information presented here may be slanted for EA customers.  

The take away (as presented to us – your answers may vary by Microsoft rep) was that there are three ways to license for virtual machines in Windows.  With a Windows Server Standard license, you are allowed 1 physical plus 1 VM in Windows 2008.  In Windows 2003, you are allowed on physical server license.  With a Windows Server Enterprise license, you are allowed 1 physical plus 4 VMs in Windows 2008 and 2003.  Microsoft more recently introduced the Datacenter edition of Windows, which is more taylored for virtual environments.  The Windows Server Datacenter edition is licensed on a per-processor model (a la VMware) and allows for an unlimited number of virtual machines to be run on the hardware.  For our existing EA, we can step-up from Standard and Enterprise copies to Datacenter edition.  One copy of Standard or Enterprise equates to a single processor of Datacenter edition in the conversion process.  

Datacenter edition doesn’t have a restriction on how long an instance of Windows must run on the hardware before it can be legally moved.  Microsoft had imposed a restriction on the Enterprise version that an instance of Windows must run on a particular set of hardware for 120 or 180 days before it may be moved, however, we were informed that they have now removed that restriction, also.  

The licensing outlined above is agnostic to the hypervisor, so this information should apply for EA customers using any hypervisor – VMware, Hyper-V or Xenserver.  For our needs, we’re currently planning to continue with Enterprise for some of our VMware clusters and add Datacenter for the servers in at least one of our clusters.  We utilize VMware as a redundancy tool in our burbs and a couple of those clusters are lightly loaded with VMs – less than 4 per host.  So in those locations, Enterprise licenses to account for the virtual machines makes economic sense.  At our price point, the Enterprise license is roughly the same cost as a Datacenter edition one-processor license.  We’d have to purchase two processors for each of our hosts.  

In addition to our ESX licensing, we have been exploring deploying thin clients for our users and moving to virtualized desktops.  Microsoft has created the Vista Enterprise Centralized Desktop (VECD) program to accommodate virtualized desktops.  The VECD is available as part of the Microsoft Desktop Optimization Pack (MDOP) and also directly as VECD licensing for non-qualified desktop products (thin clients, etc. that aren’t capable of running full versions of  Windows XP or Vista).  In the add-on package, VECD allows users to attach to virtual desktops for applications that may not run in their current OS – for instance, applications that won’t run in Vista can be run in an XP virtual desktop that the user connects to.  

One offering that we were introduced to was Kidaro, which is a Microsoft aquistion.  It basically allows a virtual desktop (Virtual PC only, I think) to be packaged for applications that won’t run in the host OS.  It strips away the task bar, start menu and makes it look like a desktop application, even though it runs in a virtual machine.  Basically, the same thing Citrix Presentation Server does for Terminal Services, this does for Virtual PC.  That is compelling, particually if Microsoft can leverage this towards a centralized desktop farm in the datacenter.  

The MDOP includes App-V, which is the rebranding of the Softricity aquisition.  It allows for applications to be packaged and streamed from a centralized server and run on a virtual layer.  This eliminates DLL conflicts and allows otherwise incompatible softwares to run simultaneously on the same desktop.  MDOP packages several other features along with it at a relatively low per-user, per-year price.

VECD in its stand-alone offering allows for what we need in our enterprise.  It allows for non-qualified desktop devices to connect to virtualized desktops run from a datacenter.  The VECD is licensed based on the number of end-point devices.  You may run an unlimited number of virtual desktops for different needs and user groups on the backend infrastructure.  The price point for VECD is lower than purchasing a license of Vista/XP for a qualified desktop device.  VECD seems to provide us with the most flexiblity for running virtual desktops.

One thing not covered during the meeting, but relayed in an earlier call with our sales partner, was the ability to run Vista/XP desktop licenses on virtual desktops – and that’s a no-no.  The Vista/XP license does not allow for things like VMotion and moving virtual machines between physical hosts.  This licensing still has the restrictions that the OS must be tied to particular piece of hardware for 120 or 180 days (can’t remember which).   So, that restriction in play, VECD seems to be the only sanctioned way to run virtualized Windows desktops. 

All in all, the meeting was extemely interesting.  While it was called to answer our licensing issues, the reps took the opportunity to instead make a sales pitch for the entire virtualization product line.  It was a little over the top and it was hard to be thrashed with the same information both at VMworld and again in my own conference room.