Server Move

SpindleyQ's picture

EDIT: It's done! Welcome, Glorious Trainwrecks, to your fourth home. Please please please tell me if _anything_ weird happens so I can look into it ASAP!

Hey all,
Just a quick note to let everyone know that I'll be moving Glorious Trainwrecks to a shiny new server in the next few days. If all goes well, the only things you might notice are:
* https may stop working or point at a bad certificate for hopefully no longer than a few minutes (I'll be generating a new certificate rather than migrating the old one)
* the server will enter "maintenance mode" once or twice and you won't be able to make new posts or potentially read existing ones - again, this should hopefully only happen for a few minutes

If anything weird happens to you during or after this process, don't hesitate to let me know - you can [url=https://www.glorioustrainwrecks.com/privatemsg/msgto/1]PM me on this site, send me an email drectly at if you can't log in, or for a quicker response you can hit me up on discord: SpindleyQ#2039 or uhhh on Mastodon @SpindleyQ@mastodon.social? I'll update this post with a comment once it's complete.

Thank you for continuing to be an incredible community that I'm proud to host <3

SpindleyQ's picture

New server infodump

I kind of want to spam the website with nerd-ass details about the new server, and it's my website, so here it is!!

I had three big reasons to move servers:

  1. I have been paying monthly for 3 dedicated servers for years, simply because of the pain of making sure all my stuff was moved over meant I never quite got everything. I wanted to turn that into 1 server if only to bring my internet bill down.
  2. We're not quite out of space yet - we've got about 6gb of headroom - but I did want the option of expanding if needed. For boring technical reasons this required a fresh OS install built on LVM.
  3. Security updates! Did you know: the _original_ Glorious Trainwrecks server was *still running* serving mirrors.glorioustrainwrecks.com and the Pirate Kart 2 bittorrent tracker, and it was SO out of date that my ssh client literally wouldn't allow me to connect to it? Yikes!!

In the process, I have also done a lot of work to ensure moving and upgrading servers in the future is as easy and trouble-free as I could possibly make it.

The new server runs NixOS, which is a Linux distribution where the entire server configuration is generated from a configuration script, which specifies all packages that should be installed and all their configuration files, written in a special programming language called "Nix". This means I have a small directory on my laptop which completely spells out every server customization and lets me rebuild a new identical server from scratch in minutes. (Do I have copious backups of that directory? Yes I do.) I have a suite of scripts that restores everything from a backup, which means that even if my data center is hit by a meteor, I can have GT back up and running within hours. I can quickly try out any changes on a local VirtualBox VM before putting them live on the internet, rather than accidentally bringing stuff down temporarily while I mess with it.

NixOS is also designed so that many versions of the same program can be installed and coexist at once. Upgrading just means installing new versions of the packages (and configuration files!) alongside the old ones. If something goes wrong, a single command rolls it all back. For years, I have been terrified - TERRIFIED - of upgrading to a new major release of a Linux distro on a server, because if something goes wrong.... GT goes POOF until I sort it out. Not anymore!

Fun fact: Drupal 5 requires PHP 5.2, which is not something any Linux distribution distributes binaries for in 2020. I now have a full recipe for building it from source, with all available security patches, written in Nix. No small amount of pain and suffering to get that sorted out, but now that it's done, it's completely automated - I never have to remember how I did it or where I got patches from or what configure flags I need. I don't know how long I'll realistically be able to keep a Drupal 5 site on the internet, but this is the closest I can reasonably get to making sure I can do so indefinitely. (If I eventually need to pin the compiler & libraries to an old release of NixOS, that's reasonably straightforward to do while continuing to upgrade everything else.)

I can't recommend NixOS for everyone - it's got an extremely steep learning curve, and the documentation is.... not ideal. It took me two weeks to migrate everything from my other servers (I've still got one site left to move) and I definitely had to go digging around in source code for answers to questions I had about how things worked. But for me, for this server, it has brought me _serious_ peace of mind.

pensive-mosquitoes