FacebookTwitterDiggDeliciousStumbleuponGoogle BookmarksRedditNewsvineTechnoratiMixxPinterest
Pin It

I've been creating, programming, maintaining, and managing computers for almost 35 years.  You'd assume I'd know how to be sure *my* data is backed up and easily retrievable in the event of hardware or software problems.  You all know how 'assume' works.


A couple years ago, changed away from some off the shelf NAS devices, and moved to FreeNAS (http://www.freenas.org/) I did all the appropriate reading, about ZFS, best practices, etc.   My take on all that reading is that ZFS is great, fast, has good redundancy -- until it doesn't.   When something fails with ZFS most references say "Wipe it, start over, restore from your backup.  You do make backups right?".   So taking that to heart, I made a backup plan.   An external USB3 drive that would be rsync'd nightly from the ZFS master.

All went quite well (a minor hickup with bad memory about 1 year into its usage -- do other people actually encounter memory failures?).   I found ZFS and FreeNAS to be speedy, and reliable.  

This morning, I sat down at my main work station.. all my shares were unavailable.. A trip downstairs to my rack-o-stuff (which is fairly empty compared to years of old) and I was greeted with a disk error of some kind on the console.   The USB stick (which FreeNAS boots from) was flashing, and the error seemed to refer to the USB stick (/dev/da0), not the ZFS drives.  Ok good.  

FreeNAS keeps its configuration data on the USB stick.   I make somewhat frequent config changes (add/remove jails, etc) so I had planned for this.  Part of my backup script makes an image (using dd) of the raw boot USB stick, and drops it into a ZFS folder, which then of course gets copied to the backup external USB3 drive.

Ok so all i need to do is make a new USB stick from the image.  Yep.. ok. ummm. . how?  Since my computer rack is fairly sparse these days, I actually had no Linux box to plug it into to grab and make the image.  Ok so I googled "extfs3 windows" and found a few solutions.  Downloaded one -- didnt work (unrecognized partition)..  Downloaded another -- bzzzzt  Downloaded a third - nope...   I checked the backup logs (sent in email each day) and everything looked like it should be fine.  

Ok I have a few Raspberry Pi's, so I fired one up, plugged in my backup drive, and tried to mount it  -- "unknown filesystem type ufs"  Ohhhhhhhhhhhhhhhhh. Ding.  Now I remember -- BSD doesn't speak extfs3, it speaks ufs.   Well I found ufstools thru google and it wouldn't mount either  (GPT vs MBR of course, its an old set of tools).   Ok FINE.   

More google searching got me to this:  http://mfsbsd.vx.sk/  Its a Minimal Live CD type FreeBSD image.  And they had USB images ready for download as well.  I dropped that onto and SD card, and pushed that into a SD to USB thingy, plugged that into my NAS Box, and let it boot up.   Upon logging in, it saw my USB3 backup drive device, did a quick mount, and there it was.   Then I stuck a new USB stick into another port, found its device node, and a 15 minute dd session later, I had  new USB stick.

Booting the recovered USB stick was pretty anti-climactic.  It booted, no issues, all was right with the world again.

Now I've taped the SD Card to the NAS machine, with a nice note to myself.   

Even though I had no corruption of the USB stick image file, I've also tweaked the backup routine to keep 4 prior copies of the image, just in case a failure of the USB stick occurs during backup.

Yep backups work.  Recovery time 3 hours.   Next time 15 minutes.