Early on, I got a taste of success (particularly for a blogger with NO following) by blogging during VMworld 2008 about the announcements being made from VMware.  It was the first time I had content picked up and linked to from others, and I’ll admit, it felt good.  To this day, my blog still gets some traffic from web searches about VMware’s Fault Tolerence feature.

Those early results did not turn into overnight success.  After VMworld 2008 and 2009, the blog has received relatively low numbers of visits, participation and readership. I allowed it to languish with only a few updates here and there.  The growth has been relatively slow, but recently doors are opening up.  Opportunities, like attending HP Discover and the Blogger Reality Contest, are allowing me to attend great networking and technical events,  to report on great new technology being introduced, and to refine my skills as a blogger.  The future for this blog is looking as bright as ever and I just wanted to to take time to thank those involved for the opportunities, support and encouragement.

That was last year.  My photo preservation project has been pretty idle for the past year, but it hasn’t stopped me from thinking about it from time to time.  I’m thankful to have those photos, to be able to show them with my daughter and future generations and to have digital files to make prints from.

After tackling some of our family’s historical photos, I looked at my file of snapshots from my younger years and even early in my marriage – a time before we had digital cameras.  Two things are evident – we take lots more photos with the digital cameras than back in the film days and, secondly, prints are easy to get very disorganized.

What I am also struggling with is the best way to store and share these scans.  As of today, the most sharing that has happened with the scans is loading them to Facebook or Geni.com.  Unfortunately, neither site stores the photos in high-resolution (as far as I can tell) for sharing with others, so they aren’t print quality.   I don’t have my solution yet for this, but I suspect I’ll use something similar to Gallery, which I have used before for personal photo galleries prior to moving everything to PicasaWeb.

My project so far has left me with a few lessons learned and a few tips to share for you if you’re looking to preserve your family’s historical photos as well as your current snapshots.

Tip One – Organize your photos
The first tip is to start with organized files of photos.  This will greatly aid you once the photos are digitized and I’ll leave it at that.  If you have photos grouped before scanning, you’ll have a much easier time with the digital files once they are scanned.  One thing I’ve also been trying to do is get dates of events with groups of photos (if possible).  This additional information (metadata) will help a lot once you move from scanning to cataloging your photos (in Picasa or iPhoto or whatever photo management software you prefer).

Tip Two – Choose the best scanner for the job…
Before you say “no duh”, there are a few interesting tid-bits.  Scanners come in lots of shapes and sizes.  I picked up a $100 CanoScan a few years ago which is a great portable scanner.  For documents, I use the document feeder and scanner on my HP Officejet.

What I have been missing is a great portable scanner which you can use without a computer and I found a very good deal on one in December.  Radio Shack has previous generation Pandigital Photolink 4″x6″ photo scanners on clearance for only $15 at its company stores (company stores and not franchise stores — I found out in my own experience that franchisees do not honor the clearance price).  The Pandigital scanners are great because they scan directly to an SD card and do not require a computer and they only take a few second per print to scan them, as opposed to putting photos on a flat bed, previewing them, scanning in the images individually and naming them on a computer.  These Pandigital scanners are ultra-portable and come in a variety of sizes from 4″x6″ up to 8.5″x11″ (larger scanners are capable of scanning smaller prints, by the way).

My coworker, Jamie, purchased one of the larger 8.5″x11″ earlier in the year and raved about how good it worked for his snapshots.  On his advice, I began looking at these.  The 8.5″x11″ model is available from Amazon for $99.   I have found Kodak also makes a simliar scanner.

Tip Three – Catalog the scans
When I say catalog the scans, I don’t just mean to import them into iPhoto or Picasa and forget about them.  Cataloging, as any librarian will tell you, is a lot more than just collecting things.  It involves a process of categorizing, organizing and sorting through your items and assembling them in a way that you can search an easily find a particular subject.

Lifehacker recommends that you also spend the time to delete and purge any unwanted digital photos.  As I mentioned earlier, this is a big issue because most of us snap far more photos now with our digital cameras than taking the time to get good photos from back in the film days.  We don’t need ten out of focus photos of weird rock formations.

Tip Four – Archive your photos
Once you have assembled a good catalog of your photos, archiving becomes a big deal.  You have a lot invested in your library now and so you want to protect both the images and their metadata.  There are a lot of ways to go here, so what works for me may not work best for you.

Online backup is my first recommendation here.  If its not your first time on my site, you’ve probably heard me talk about Mozy and other online backup solutions.  These are my preferred backup method for anything you want to survive a fire.  #1 – Online backup moves your data offsite, which is a hallmark of backup strategy.  #2 – Online backup generally provides online download or offline download as part of its service.  #3 – Most services are generally cheap – Mozy, for instance, is $59 per year per computer.  I keep my library on one computer, so it costs me less than $5 per month — less than one Starbucks drink.

External hard drives would be my second recommendation.  These provide quick and easy recovery from a hard drive crash or accidental deletion.  Most external hard drives come with some sort of backup software — and if not — Windows has Windows Backup and the Mac has Time Machine.  I would recommend making it something automatically schedule and not something you must remember to do — because we are all human and forget.

Archival disks would be my third recommendation.  I don’t think that any one backup method is sufficient.  For instance, I use Mozy and Time Machine to backup my Macs at home.  The two protect differently and from different threats.  Mozy is for the most important — cannot part with this if my house burns down — data.  Time Machine is for the oh crap, my hard drive crashed situations (the external hard drive method).  In the same way, I think investing in Archival CD or DVD media is smart.  Once you create collections in your library, move these files onto archival media and then store them with your other valuables – offsite — in a safety deposit box or other form of offsite storage.  Ritz Camera has a nice description of Archival disks and why they are better than regular CD or DVD media.  (They also have a good set of tips for preserving photo prints). I recommend this because these have a greater chance of surviving for future generations.  And of course, keeping your archival media current is crucial.  Whatever replaces DVD’s as a format — you should move your data to that format in the future.  Same with file formats (but that’s a bigger discussion).  And most importantly – label these archival disks!

More Tips:

By no means am I an expert.  These sites have some great advice on the subject and so I want to share these links and articles with you:

• Lifehacker:  Future-Proof Your Digital Photos with Better Archiving Techniques – Jan 5, 2011
• Ritz Camera:  Tips for Photo Archiving & Storage – April, 2010
• Digital Photo Archive – Backup – Help and Tips

A week after our conversation, I began playing with WordPress on my own webserver and started putting together a website mock up with a template from WooThemes and content from their existing site.  It was fairly easy and I like playing around with websites like that.  Wordpress worked pretty well for the basic site.

WordPress also provided them with an easy way to keep the website up-to-date from the road or from their iPhones as they traveled and snapped photos.  Wordpress is supported and updated as security problems were found and should help them stay secure in the future.

Useful Plugins for Musicians
After putting their existing information into the site, I started looking at what else would be good on the site.  I located the fantastic website of developer Dan Coulter, http://blogsforbands.com/.  He had developed several WordPress plug-ins for bands, including a gig calendar and discography.  I added the discography plug-in to a new install of WordPress and began adding songs from the past album that I had.  It worked wondefully — allowing me to post each album, the songs, words to the songs and links to buy the songs from iTunes, Napster or Amazon’s MP3 store. It also allowed a link to buy a physical copy…  but from where?

Enter another plug-in – Tips & Tricks HQ – developers of the WordPress eStore.  I had used WP E-Commerce in the past, but it didn’t want to install on my musician’s web host and it didn’t easily offer digital downloads, but WordPress eStore did…   We added the plugin and began populating it items, like T-Shirts and CD’s.

Our musician friends utilize ReverbNation heavily and so I was able to find their widgets (which our friends were already using on their original site) and place those into text widgets on the sidebar.  The cool thing about this approach is that it allows them to update their music players, gig calendar and mailing list all within ReverbNation and have it feed their website.  Likewise, their website is feeding ReverbNation new mailing list addresses for future mailings and it is collecting stats of who is listening to their music.   ReverbNation is a free service, but offers enhanced and additional pay-for services.

As we were going live and testing everything, I found a couple things – like emails being sent from the eStore were sending from “WordPress”, but Tips & Tricks HQ had another plugin to allow us to customize the friendly name of the sender.

If any other bands or musicians are looking for a solution, I’ll be the first to recommend WordPress!

Part of the problem is that documentation tends to get outdated.  Keeping your notes updated as changes are made is tough.  Old documentation is sometimes worse than no documentation…  Its sometimes better to get inside and dig around to see how things are actually working/setup.

After I started at HTC, my co-worker David showed me his Mediawiki setup and running from his personal Linux desktop that he was using for documentation.  He has amassed a pretty respectible collection of documents on the *nix, backup and networking side of the house.  Too bad I was a Windows guy for the company (ouch, did I just type that…  I’m so the anti-Windows Windows guy…).  

After I started the VMware VI 3 deployment, I quickly realized I needed to be keeping notes somewhere for this monster project.  I asked if I would be able to use the wiki he’d established to keep my notes and he agreed.  As we talked on, we quickly realized that everyone in our group was likely in the same boat with their notes.  We had documents strown all over our file share with a hodge podge of information, none of it readily available (searchable) when needed.  

So, after getting VMware deployed and as a test of deploying our first Red Hat VM, we setup something we called NetAdminWiki…  We secretly plotted planned to continue putting our information in this repository and then open it up to our co-workers over time to get them on-board.  I’m glad to say that a year later, we’ve now got all our co-workers with working accounts on the wiki, but we ran into a few issues that we needed to conquer before putting it out before our bosses.  

MediaWiki in the enterprise
MediaWiki is a great solution for documentation, but its meant for everyone to contribute – that’s its focus – a very wide audience.  In our case, we wanted it to only be available and accessible to a select audience.  That initially ment password protecting the Apache website using HTAccess.  Most of our co-workers didn’t like the idea of logging onto the web server and then into MediaWiki…  

We found an extension for MediaWiki that allowed us to do seemless signin using the authenticated account passed from Apache.  That worked great, but then users wanted to use their AD account and password.  That led us to making Apache authenticate against AD’s LDAP and login users that way.  This method was messy and it messed with MediaWiki’s built in account creations method.  

This past week, I finally cleaned all this up as I was asked to add our last co-worker into the wiki.  MediaWiki now has a great LDAP Authentication module with few dependancies.  Once I setup this module, we were logging in against the domain with ease.  We removed the Apache authentication and we now have users logging in through the normal MediaWiki method, but that introduced a new problem.  Anyone could read our content.

The extension library for MediaWiki is expansive, so finding a solution for the new problem was a snap.  With just a few changes to the LocalSettings.php file, we were able to lock down access to the wiki, so that you had to login to ready any content or make changes.  MediaWiki has a great article about preventing access in their online manual.

With these two extensions, we’ve now made a pretty nice documentation solution for our group.  Its possible to take this much farther.  The LDAP Authentication solution also has the ability to synchronize MediaWiki groups with AD groups and you can provide additional permissioning using AD groups.  There are a lot of options here.

Of course, my story has to begin on Friday the 13th…  Sometime around 9:00am, we started getting calls for both our SQL 2005 database cluster and our Exchange cluster.  After investigation, we found that the active nodes were both in the same enclosure and a third ESX host in the same was experiencing problems, too.  The problems were affecting both network and disk IO on the blades.  All of our blades are boot from SAN, so the IO had to be a fiber-channel issue.  

Several hours later, we were finally able to get enough response out of the nodes to be able to force a failover of services for Exchange, shortly followed by SQL 2005.  As I worked with HP support, nothing improved on the affected servers.  We were finally diagnosed with a problem mid-plane on the enclosure.  

 While waiting for the mid-plane to be dispatched to the field service folks, I requested that we go ahead and do a complete power-down on the enclosure and bring it up clean.  This required physically removing power from the enclosure after powering down everything that I could from the onboard administrator.  

After the reboot, everything looked much healthier.  The blades came back to life and everything began operating as expected.  After intense discussions on the HP side, we reseated our OA’s and the sleeve that they plug into on the back side of the enclosure.  Net outcome was the same – everything still operating well.  The OA’s nor the sleeve were loose, so we doubted that was the cause.  

One nugget I learned from HP support (please vett this information on your own), is that the Virtual Connect interconnect modules require communication with the onboard administrators (OA’s).  I’m still not sure I fully understand, but HP support did tell us that if VC lost communication to the OA, its possible that it caused our problems.  If this is so, this smells like very, very bad engineering and design…

Continued investigation on HP’s part has pointed us back to the original diagnosis – a faulty mid-plane.  Only by default did we return to that conculsion, however.  This is the only piece of hardware common to the problems.  Our only other conclusion was that this was a very bad, “hiccup” — which obviously buys us no real peace of mind…  

So, sometime soon, we will be replacing the mid-plane of our enclosure.   I have, of course, lost some faith in the HP blade ecosystem.  We have plans to migrate our corporate VMware cluster onto blades, as well as some Citrix and other servers.  Losing an enclosure like this has un-nerved those plans.  We were fortunate to have drug our feet  to only have 3 blades populated and serving anything at the time this happened.  I will post updates as we move forward…

My wife and I went into Circuit City yesterday to purchase a new printer.  We found and purchased an Officejet with an attached document feeder for the scanner.  So, yesterday afternoon, I got it all setup and running at home.  I started scanning some of my mortgage documents and trying to store them.  Some quick Google searches later, I’d found how to create searchable PDF’s in Adobe Acrobat Pro, and I was very happy with my results.  I finally had a pretty good solution.   Between Spotlight and my folder structure, things looked pretty good.  Documents were fairly easy to find.  But it wasn’t perfect.  

A few Google searches later, I was demoing DevonThink, which is a great product.  Unfortunately, the Pro version is very pricey and the scan features are currently disabled in this current release, although the very kind error message said they’re working to get that back quickly.  I really liked the software and its very full features, maybe a little too much so for what I was looking for, so the search continued.  

A MacRumors forum post then pointed me to a little app called Yep.  I downloaded and I’ve been demoing it overnight.  Color me impressed.  

This little app is wonderful at what it does – it finds and categories your PDF files.  You can limit it to your home directory, the whole computer, or just your documents folder.  I opted for the latter, since that’s where all my relevant PDF files are at.  

Yep lets you add tags to your PDF files to for quick search.  Much like a blog, you have a tag cloud that shows you which tags are most used in your PDF collection.  You can organize the PDF’s into logical collections of PDF files, too.  The PDF’s all stay in their original file system location (no duplication here) and renaming or updating them has no effect to their record in Yep.  

Yep lets you do a full preview (even full screen view) of your PDF document.  You can open it with your preferred “normal” PDF viewer/editor if you would like to perform touch-ups.  You are able to view documents in your Finder filesystem locations in one view and by collection with just a simple click.  

The downside to all this is its PDF only…  So it won’t be tracking your original documents in Word, iWork or OpenOffice.  But that was kinda my point.  I wasn’t looking for something to track everything (that’s call the filesystem).  I was looking for a nice program to help archive my documents and keep it all straight.  

If something catastrophic were to happen, I need help finding my key documents and getting that information to put things back on the road, and I think this app can do that.  There are some things left to do for me – like finding and backing up its database of tags (or does it attach them to the PDF file??).  All in all, I think I’ve found the perfect solution to my document archiving.

I have found a couple Perl or Python scripts that seem to serve this, but not sure exactly how it all works…

I realize that there are lots of affordable online backup solutions available online, including one that we provide here at HTC.  These are all great solutions.  The killer for all is the initial backup and how it is handled.  Our backup solution here at HTC offers the ability to backup locally to a disk, ship that disk to HTC and then we restore it on our “Vault” back end.  At this point, our software does block-level, incremental backups forever.  And the incremental backups are fast.

From my testing with Mozy and a couple other solutions, its painful to try and backup my 14GB of photos or 100GB of home movies through my internet.  The biggest restriction on broadband is usually the upstream bandwidth.  I could allow this to run for a month or two, saturating my broadband all the time and this might complete, but at a cap of about 50K upstream, its not appealing to me.

The same held true when I setup the backup solution bundled with Norton 360 for a friend of mine.  I was very selective in getting only her documents and email to backup, rather than her iTunes library and everything else.

But enough about the problems with other solutions, what is good about these?  Well, plenty.  First, there is no recurring cost.  Just buy a storage device and find a buddy.  I guess this could be a problem for some of our geek types, but most of us are social enough – at least with other geeks.

CrashPlan is nice in that it offers software for a many operating systems – Mac, Windows and Linux.  You are able to do both on and off-site backups.  It encrypts the data before transmit and includes a feature that it calls “archive tampering protection.”   I hope that is exactly what is sounds like and there is some security on the archive of my social security numbers and bank records that my buddy can’t just break into…  Guess that wouldn’t really make them a buddy, but more of a frenemy, but I digress.    But, the big one for me is that you can start your backup locally and then move it to an off site location and continue backups later.  And this is the differentiating factor for me.  Solutions like Mozy and other online providers don’t offer a way to take a full backup of everything, restore that onto their storage in some way faster than uploading everything over the Internet, and then just adding differentials after that point.  

CrashPlan comes in two versions of software for the home user.  There is a standard version and then a plus version which offers continuous data protection and versioning.  These are two enterprise class features that only seem to exist in these direct to disk backup solutions.

It should also be noted that CrashPlan has an enterprise product as well for larger environments (advertised for 10 or more computers) where it includes a local backup server running CrashPlan Pro Server and the ability to send that data offsite.  One thing that is particularly compelling is that CrashPlan includes a VMware Virtual Appliance of the Pro Server software.  CrashPlan Pro uses 448-bit Blowfish encryption.    Also important to note for medical businesses, the enterprise software is HIPPA compliant.  I do wonder how this solution will do for more enterprise type applications like databases, email server and other more specialized data types.

Cucku is a bit different, but has many of the same features as CrashPlan.  The solution is unfortunately Windows only, which is a major drawback for me.  I have only briefly looked at this solution since it won’t work for me.  But I did want to mention it for others who it might work for.

That’s how my 2009 started… about 13 hours later, I finally left work and resumed my long-interrupted nap.   We had what seems to have been a storage meltdown behind our VMware farm yesterday.  Our file sharing cluster was also affected and so our few employees who were working on New Years Day, well, weren’t working at all.  The short version of the story goes like this.  Our scheduled backup process, using EMC Networker, kicked off VCB backups on the ESX 3.5 hosts around 1:30 am.  By 2:00 am, the process was trying to create snapshots on VMs and this caused some sort of meltdown due to SCSI reservations (found the SCSI reservation problem after VMware analysis).  Turns out the HP Insight Agents loaded on our VMware hosts were causing these SCSI reservation issues.  The agents were checking the disks at a consistent interval and we had not upgraded the agents to the latest revision, which was supported with ESX 3.5 – so not VMware’s fault – they have a great KB article about this issue (see KB 1005009).   As an immediate resolution, one of my co-workers removed the HP agents from our hosts and worked our way through rebooting the entire farm, one host at a time to remove the SCSI reservations.  I cross my fingers on VCB backups working when they kick off in an hour.  Had this been the only issue, we would have been fine.

Unfortunately, at around 4:30 a.m., while I was unaware, our cluster began experiencing troubles, too.  And this is where our detective skills have come up short.  We have been sleuthing to find the cause of some weirdness in both our file sharing and Exchange clusters for several weeks, now.  The file share cluster is, dumbly enough, critical in our environment.  Without it, our users home directories are inaccessible and, since these home directories are defined in Active Directory, it seemingly hoses up our employee’s workstations.  Things that should otherwise be speedy, say opening a program — any program — or browsing to your local hard drive, become unbearably slow.  Even running applications sometimes lock up as it attempts to access some unknown part of Windows during normal operation.  It brings our entire business operation to crawl and that’s unacceptable.  (BTW, if all this sounds familiar, please leave comments or send me an email with suggestions.)

So, what actually happened to our Windows file sharing cluster?  We have an issue where we see the network utilization on the file share cluster drop to nothing, but the cluster nodes still respond to ping and other non-storage related network services – but what we found out later in the process – not to anything which needed IO to respond.  After repeated network sniffs, we were seeing that traffic would come to the cluster, it would be acknowledge, but the node would not start sending data.  The break between the request and data could be as long as 20 or 30 seconds.  And that was consistent with our ‘outage’ periods.  So, I decided to fail over the cluster shares from one node to the node that had been ‘solving’ the problems in the past.  When attempting to fail the share, they locked and never became accessible again.  After waiting for almost an hour, I rebooted a node trying to clear up the locks and let the other node take control.  That was never possible either.  A reboot of the second node only served to cause it to stall during boot, and never provide me a login screen.  Rebooting the other node, same result.  And then, a lengthy phone call with HP support after driving into the office.

The short version of this is that we are running Windows Server 2003 with SP2.  Apparently, therein lies our problems with a) clustering and b) storport.sys.  The StorPort driver issues are pretty well documented and it in combination with several other hotfixes are HP’s recommendation to us.  The hotfixes were released outside of Microsoft’s normal patch schedule to the large number of customers having issues similiar to this.  HP’s recommendation to us was to install the list of suggested hotfixes for post-SP2 (Microsoft KB 935640).  My co-workers successfully completed that on the file share cluster this evening without incident.  (Hallelujia!)  I got the call shortly around 10:15 pm with the all clear.

The file sharing problem has been plauging us for several weeks and we have not been able to deduct the full cause.  We have had theories and as soon as we seem to figure it out or know how it will act, we’re proven wrong.  At least until yesterday, we hope.  The next week will tell for sure.

I also mentioned issues on our Exchange cluster.  We’re not actually sure its having issues.  It may only be showing issues at the same time the file share cluster was having issues.  We believe the above issues on file shares are causing lock ups on client machines and their programs, so we are currently thinking Exchange’s perceived network issues are just the fact that our employee’s Outlook is locked and can’t connect, so we see what looks like Exchange issues.  But, then again, we’re not 100% sure.  We still have some detective work to do here.  Where’s Sherlock Holmes when you need him?

As for VMware, we still have a punch list for a few additional things to do – putting newer HP agents on the ESX hosts for one.  There are some things we may want to customize here based on feedback from VMware.  We may want to disable the storage monitoring agent on these hosts, but more research is required.  Removing these agents all together for now is a our preferred fix.  So, for the next hour, I just need to keep myself awake and hope that VCB backups will go well tonight.

The VMware Technical Forum was the first in the US, I believe, for the Customer Focused Testing group.  This group has specialized in Microsoft Exchange and SQL, Oracle and SAP as product lines but recently added VMware to their testing group, as well.  This group is responsible for testing different scenarios and configurations to provide some best practices and whitepapers for HP users – mimicing real-world environments and configurations.

There were two particularly impressive sessions provided from one of the Paris, France, associates in the CFT group.  Christophe Dubois did a phenominal job covering VMware configurations for Exchange and typical problems of intensive transactional workloads on VMware.   The CFT team specializes in the HP Storageworks products and they have an immense breadth of information on these products.

I have a couple additional topics that were sparked out of this forum that I will be blogging about in a few days.  Check back for more on those.