So I somehow filled my laptop’s M.2 SSD, my external USB drives are full and possibly failing too, this is a wake up call that I need to do better with my backup plans.
First I had to free up some space just to be able to work on it. I was lucky that I technically did have about 4 GB of free space. (It might have been less, unfortunately I didn’t make a note of it) So the first thing I did was find files I could replace from elsewhere or otherwise no longer needed (for me this was all the Linux ISO’s I keep in ~/ISO/ I just had to make sure I didn’t delete my Windows one too)
Now I booted up a USB Drive with Ubuntu 20.10 on it and then proceeded to mount my drive. (This might be as simple as a mouse click for some, I use disk encryption so for me it was a little more involved.)
I mounted it to /tmp/laptop for the purposes of this post. You may need to change this to reflect your setup.
Now I need to install some tools for all this, along with my GPG public key
sudo apt install s3cmd duplicity python3-boto python3-azure-storage pcscd scdaemon gnupg2 pv gpg --keyserver keyserver.ubuntu.com --recv-keys FC5BFC514D82D7B443C2DCD4A068289733EAAA5F sudo gpg --keyserver keyserver.ubuntu.com --recv-keys FC5BFC514D82D7B443C2DCD4A068289733EAAA5F
I also need to set the trust level for this key, in order to avoid issues later on.
Since I know this is my own key, I’m going to tell it 5 = I trust ultimately.
gpg --edit-key FC5BFC514D82D7B443C2DCD4A068289733EAAA5F gpg> trust Please decide how far you trust this user to correctly verify other users' keys (by looking at passports, checking fingerprints from different sources, etc.) 1 = I don't know or won't say 2 = I do NOT trust 3 = I trust marginally 4 = I trust fully 5 = I trust ultimately m = back to the main menu Your decision? 5
I need to do this again as root (using sudo) for duplicity too.
Now I didn’t start with duplicity, which was likely a mistake on my part.
I started with trying to make a tarball file but storing the file on the same disk, deleting files as I went couldn’t help some large files I had (server backups, virtual machine images and such) I piped it thru pv to give me some idea of progress & then compressed it with xz & encrypted it with GPG (setting it to encrypt it to the key on my yubikey)
sudo tar --remove-files -c /tmp/laptop/ |pv| xz -9 | gpg -r FC5BFC514D82D7B443C2DCD4A068289733EAAA5F -e | s3cmd put - s3://bucket-id/2020-01-25_Laptop.tar.xz.gpg
If we need to decrypt this later on, we’d run.
s3cmd get s3://bucket-id/2020-01-25_Laptop.tar.xz.gpg - | gpg -d | xz -d | tar -xv
At this stage I had everything backed up and I decided to also reinstall Ubuntu 20.10 onto my laptop. I bought a new external drive as well so I’d have a local copy and a remote one. I also installed the same dependencies listed above. I copy the whole system minus some directories we don’t need if we’re doing a recovery. (If you choose to alter this, know that duplicity tends to freak out over the weirdness in /proc )
First is the command to Exoscale & second is to the new external drive.
sudo duplicity --s3-use-new-style --encrypt-key FC5BFC514D82D7B443C2DCD4A068289733EAAA5F --exclude /media --exclude /run --exclude /mnt --exclude /tmp --exclude /proc --exclude /dev --exclude /sys / s3://sos-ch-dk-2.exo.io/bucket-id/MSI-Laptop sudo duplicity --encrypt-key FC5BFC514D82D7B443C2DCD4A068289733EAAA5F --exclude /media --exclude /run --exclude /mnt --exclude /tmp --exclude /proc --exclude /dev --exclude /sys / file:///path/to/external/drive/MSI-Laptop
The best part of doing this right after a re-installation is duplicity supports incremental backups, so as I add my music collection back & configure my vpn & dns settings and work my way to filling my drive again (I hope not) duplicity will make incremental backups every few hours.
Now all that remains is trying to backup the old external drives before they fail and cause data loss (I might already be too late, unfortunately)
Update: Something I overlooked in the decryption section. My GPG keys are on the Yubikey, but when I plugged the yubikey into the live cd to decrypt one of the backups it appeared the private keys were missing. I spent a bit trying to make sense of this, and stumbled into the answer.
I ran this to get some output for debugging, and it happened to also wake up the yubikey and show GPG that it had the private keys we need.