- Details
- Last Updated: 06 March 2016
- Created: 06 March 2016
I've been creating, programming, maintaining, and managing computers for almost 35 years. You'd assume I'd know how to be sure *my* data is backed up and easily retrievable in the event of hardware or software problems. You all know how 'assume' works.
A couple years ago, changed away from some off the shelf NAS devices, and moved to FreeNAS (http://www.freenas.org/) I did all the appropriate reading, about ZFS, best practices, etc. My take on all that reading is that ZFS is great, fast, has good redundancy -- until it doesn't. When something fails with ZFS most references say "Wipe it, start over, restore from your backup. You do make backups right?". So taking that to heart, I made a backup plan. An external USB3 drive that would be rsync'd nightly from the ZFS master.
All went quite well (a minor hickup with bad memory about 1 year into its usage -- do other people actually encounter memory failures?). I found ZFS and FreeNAS to be speedy, and reliable.
This morning, I sat down at my main work station.. all my shares were unavailable.. A trip downstairs to my rack-o-stuff (which is fairly empty compared to years of old) and I was greeted with a disk error of some kind on the console. The USB stick (which FreeNAS boots from) was flashing, and the error seemed to refer to the USB stick (/dev/da0), not the ZFS drives. Ok good.
FreeNAS keeps its configuration data on the USB stick. I make somewhat frequent config changes (add/remove jails, etc) so I had planned for this. Part of my backup script makes an image (using dd) of the raw boot USB stick, and drops it into a ZFS folder, which then of course gets copied to the backup external USB3 drive.
Ok so all i need to do is make a new USB stick from the image. Yep.. ok. ummm. . how? Since my computer rack is fairly sparse these days, I actually had no Linux box to plug it into to grab and make the image. Ok so I googled "extfs3 windows" and found a few solutions. Downloaded one -- didnt work (unrecognized partition).. Downloaded another -- bzzzzt Downloaded a third - nope... I checked the backup logs (sent in email each day) and everything looked like it should be fine.
Ok I have a few Raspberry Pi's, so I fired one up, plugged in my backup drive, and tried to mount it -- "unknown filesystem type ufs" Ohhhhhhhhhhhhhhhhh. Ding. Now I remember -- BSD doesn't speak extfs3, it speaks ufs. Well I found ufstools thru google and it wouldn't mount either (GPT vs MBR of course, its an old set of tools). Ok FINE.
More google searching got me to this: http://mfsbsd.vx.sk/ Its a Minimal Live CD type FreeBSD image. And they had USB images ready for download as well. I dropped that onto and SD card, and pushed that into a SD to USB thingy, plugged that into my NAS Box, and let it boot up. Upon logging in, it saw my USB3 backup drive device, did a quick mount, and there it was. Then I stuck a new USB stick into another port, found its device node, and a 15 minute dd session later, I had new USB stick.
Booting the recovered USB stick was pretty anti-climactic. It booted, no issues, all was right with the world again.
Now I've taped the SD Card to the NAS machine, with a nice note to myself.
Even though I had no corruption of the USB stick image file, I've also tweaked the backup routine to keep 4 prior copies of the image, just in case a failure of the USB stick occurs during backup.
Yep backups work. Recovery time 3 hours. Next time 15 minutes.