/logs - self-hosting | automation | backups

I wanted to take my self-hosting journey further, because why not?

Wireguard is a highly touted, "simple" self-hosted VPN software. At last, I thought, I could have my own VPN without worry of data theft (let's just ignore the fact that this would be running on a public-cloud VPS :)

How hard could it be to setup, right? Well... I found out the hard way.

I dived right in, before understanding the entire setup, how it worked, whatever else I would need to know, because I believe in learning-by-doing. Turns out, I somehow managed to configure the VPN to route all outgoing packets to 0.0.0.0 which is obviously invalid. What's worse is that this seems to be the equivalent of sending them into a black hole. Apparently it's just being sent to an interface with no end point (r/Wireguard)? I'm not even sure that this was what happened, I still have no idea what happened.

Anyway, the consequence of this was made real once I turned on the VPN: hmm, my shell seems unresponsive, let me exit and retry; huh SSH won't connect now? Ok a bit concerning, let's try connecting through the instance connect facility on the cloud console... and that doesn't work. Well ok, I guess it's time for an instance reboot, but wait! In my eternal wisdom, not only had I turned on the setting to route all traffic through the VPN, I had enabled the systemd service to start the VPN on boot without even testing it first. Talk about idiocy.

Boy am I glad it was a VM. Goodbye sweet summer child, off you go to the terminated pool.

I digress. In any case, I knew such a recovery scenario was inevitable, I had automated data backups in place for just such a thing. But scripts? Setups? Configuration files? Not having dealt with them extensively in a proper enterprise setting, I had not considered backing up those or even documenting the command flows.

Now, keep in mind, I did have instance snapshots to rollback to. This would have been the easy way out. But NO! Being the soldier I am, I took this chance to improve my setup, attain more fault tolerance. This is how I learn and become the best #thegrindneverstops

Jokes aside, I had entirely forgotten about instance snapshots in my panic. It is only at the moment I write this post that my neurons finally fire, saying "Backups? What's another name for that? Snapshot! Now where have I seen that word befo-... oh. OH."

There were so many things I had not considered. Countless bugs I had to workaround, custom modifications that fit my needs, convenience measures: these were all things I had spent days on to solve the first time around. And now I had to do them all over again, but I didn't remember them all that well. Remember kids, document everything.

So I decided I'd automate everything this time: files and bash scripts for a start. I looked into Docker, but it was just too troublesome to deal with: I admired it of course, and I wanted to learn it, but at the moment I was more concerned with getting my applications back up and running. My Actual server would never forgive me otherwise. Finally, after four days of painfully testing my new setup, I managed to hack some setup scripts, config files, backup and restore scripts, cronjobs.

Docker stands in the corner, irritated by my tomfoolery, berating me for not learning it. Well excuse me for being naive and having a ton of other stuff to deal with. And Wireguard can stay in detention until I have more time to deal with it.