I have a load-bearing raspberry pi on my network - it runs a DNS server, zigbee2mqtt, unifi controller, and a restic rest server. This raspberry pi, as is tradition, boots from a microSD card. As we all know, microSD cards suck a little bit and die pretty often; I've personally had this happen not all that long ago.
I'd like to keep a reasonably up-to-date hot spare ready, so when it does give up the ghost I can just swap them out and move on with my life. I can think of a few ways to accomplish this, but I'm not really sure what's the best:
- The simplest is probably cron + dd, but I'm worried about filesystem corruption from imaging a running system and could this also wear out the spare card?
- recreate partition structure, create an fstab with new UUIDs, rsync everything else. Backups are incremental and we won't get filesystem corruption, but we still aren't taking a point-in-time backup which means data files could be inconsistent with each other. (honestly unlikely with the services I'm running.)
- Migrate to BTRFS or ZFS, send/receive snapshots. This would be annoying to set up because I'd need to switch the rpi's filesystem, but once done I think this might be the best option? We get incremental updates, point-in-time backups, and even rollback on the original card if I want it.
I'm thinking out loud a little bit here, but do y'all have any thoughts? I think I'm leaning towards ZFS or BTRFS.
Another option is to use Image File Utilities on the Pi to create an image backup. You can use cron + a bash script to create incremental backups using the tool (e.g., take a 'fresh' backup each month, with daily incremental backups in between). I mount a network 'backup' drive (a local NAS, but you could use anything) to save the image to so I can actually access it. Then, just use balena etcher to flash the backup iso in the event of a failure.
I also use those, but (shame on me) I've never tested the backups. Have you ever restored such an image backup?
Ha yes, once! I just used Balena Etcher and it worked exactly as expected.