The problem I have always had when building elaborate home server setups is the "set it and forget it" nature of the systems I've installed bites me in the ass. Since it's not my full-time job to manage these systems, I'm really not familiar with them the way I might be with the systems I manage at work. These systems cruise along for years, and when something finally does go belly-up, I can't remember how I set it up in the first place. Now I have a giant chore looming over me, ruining a perfectly good weekend.
These days, I design everything for home with extreme simplicity coupled with detailed documentation on how I set things up.
Docker has helped tremendously, since you can essentially use an out-of-the-box Linux distro with docker installed, and you don't really have to install anything else on the hardware. Then if at all possible, I use standard docker images provided by the software developer with no modifications (maybe some small tweaks in a docker-compose file to map to local resources).
Anyway, my advice is to keep the number of customizations to a bare minimum, minimize the number of moving parts in your home solutions, document everything you do (starting with installing the OS all the way through configuring your applications), capture as much of the configuration as you can in declarative formats (like docker compose files), back up all your data, and just as importantly, back up every single configuration file.
The author focuses the entire blog post on remote third party services that are alternatives to popular third party services financed by data collection as a "business model". IMO, the single most important component of a home network is not any piece of the hardware/software outside the home that the third parties may control, it is the internet gateway in the home. Routers were the most important computers at the dawn of the internet, and IMO they still are the most important computers today. If the internet gateway in the home is ignored as a point of control,^1 then IMO all bets are off.
A significant amount of data collection by third parties can be eliminated or reduced by retaining control over the internet gateway. Arguably this amount is even greater than what can be affected by simply switching to using carefully selected alternative third parties. IMO, it is a mistake to believe that one can reliably eliminate/reduce data collection simply by choosing the "right" third parties. Whack-A-Mole, cat-and-mouse, whatever the term we use, this is a game the user cannot win. Third parties providing "services" over the internet are outside the user's control. For worse not better, they are subjected to market forces that drive them to collect as much user data as they can get away with.
Regardless of these privacy-destructive market forces, it is still possible to build decent routers from BSD project source code and inexpensive hardware. IMO, this is time well spent.
Most of the data passing it are encrypted: https, SSH.
Cutting off the phone-home requests is best done on respective devices: you can run firewalls on most desktops and laptops, and even phones. Рhones often go online via GSM or LTE, without passing through the home router.
While a proxy like pihole can be helpful sometimes, cutting off tracking and ads is done best by browser extensions and by using open-source clients, where available.
The best the home router should do is to not be vulnerable to exploits, and otherwise up-to-date, and fully under the owner's control That's why my home router runs openwrt.
"... cutting off tracking and ads is best done by browser extensions ..."
What if the browser vendor, who is also a data collector, requires user to log in or otherwise identify herself before she can use extensions.
A home "gateway" is a computer running a kernel with IP forwarding enabled that is being used as the point of egress from the home network to the internet. That is a broad definition and allows for much creativity. That is what I mean by the term "gateway". As such, a gateway can, both in theory and in practice, do anything/nothing that "desktops and laptops, and even phones" can do. Relying solely on pre-configured "limited/special purpose" OS projects as a replacement for DIY and creativity in setting up a gateway was not what I had in mind, but is certainly an option amongst many others.
Don't use such a browser then! Firefox is pretty good.
An on-device firewall can firewall individual processes and applications. An upstream / gateway firewall does not have such fine-grained control. That was my point.
Running stuff on my router is entirely possible, but I limit it to routing and running a wireguard endpoint. I prefer to run my private stuff in the confines of the home LAN.
I use a text-only browser for reading HTML. Many times I do even use a browser for making HTTP requests.
Truly one can use all these strategies, application-based, router-based, gateway-based, if they are available. They are not mutually exclusive. Personally I just would not feel like I can rely on extensions or other solutions tied to some software I do not compile myself. (I do edit and compile the text-only browser.)
All due respect to Firefox, but I have found compiling it is way too time and resource-intensive. It is way beyond what I need for recreational web use. Firefox users seem to rely on Mozilla to do the right things on their behalf. That is not the sort of "control by the user" I am after.
What is behind Mozilla. Online advertising money. Cannot really count on them to do what I want.
For me it has not. But it all depends on what sites one is interested in. Generally I do not tend to find much value in SPAs when I come across them. If there is some data accessed through the page that I really am interested in, I just fnd the endpoint and bypass the Javascript. Most sites posted to HN are not SPAs, do not require Javascript to read, and work perfectly for me. I can glean the information no problem, and fast. However I am interesting reading text not viewing graphics. Not every user has the same sensibilities.
I have always aimed to try them all and I will always try any new one I become aware of. I used Lynx from 1993 through the early 2000's, but since then I prefer Links from Charles University in Prague, with no graphics. It has the best rendering of HTML tables of any text-only browser, IMO. It has been the most stable for me at run-time and it is the source code I am most comfortable with, compiles quickly. There is a recent fork of Elinks 4.x someone has started that I have been watching. (Elinks is a fork of Links.) Not sure if it has been posted to HN yet. Currently it crashes too easily but some of the features he has added are good ideas, IMO.
Okay. How can we fix this? I'm dealing with it right now and this space is so hard -- likely somewhat deliberately so. I'm a 20+ year Linux user trying to get a single home network with multiple ISPs going and it just seems way harder than it ought to be; i.e. -- not that every bit of software needs to be idiot-proof, but this iptables/pfSense/netplan etc etc universe just feels downright hostile to the aspiring home user.
Multi-wan is easier with appliances. I used pfSense over the last 12 years or so with multi-wan on and off (currently off). I've run pfSense in a kvm VM, and you can do multi-wan with this. Though I generally recommend dedicated NICs for the WANs and LAN.
I've looked at the linux based appliances (as late as last week) and only clearos or openwrt supported multi-wan. I could be wrong (I'd like to be as pfSense/OPNsense are FreeBSD based, and that comes with, sadly, huge amounts of baggage, limited hardware support, etc.). I'll likely be looking at that package as a potential replacement for the pfSense system, though if clearos can't handle what I need, OPNsense is like pfSense, but with far less baggage.
If you don't mind tinkering, you might be able to use mwan3[1].
If you prefer OpenWRT, you can look at running it in a VM[2] along with mwan3.
I set up multi-WAN in a few small business contexts (on a Zyxel appliance, and on pfSense) and to be honest it was always a little flakier than I hoped. It was always failing over too quickly, or too slowly, or not reverting back to the primary WAN quickly enough when the failure/congestion cleared up. I think any sort of multi-WAN setup where you aren't actually an AS running BGP is doomed to be somewhat hacky. Of course running an AS is both a ton of work, and basically impossible for a residential customer anyway.
It is uniquely suited to run services without interruption, like file share, messengers, torrent, http server, etc. Smartphone and laptops are mobile devices with spotty connection, powered by battery.
Gateway, not router. I personally do very few and simple things on the gateway. Way less than what an average default-configured router running OpenWRT does.
No worries. Usually they are the same. The terms are probably interchangeable. The point I was trying to make is that the gateway does not necessarily have to be a pre-packaged router with a difficult-to-manage, complex configuration. It can be the computer over which the user feels she has the most control.
No, I'm in the process of moving to OPNsense. Unlike pfsense it's actually open source, not controlled by a company that tries to drive competition out with dirty tricks and they aren't trying to move their customers to a DRM solution.
How hard has it been to move to opnsense? I've been dragging my feet on doing it because my pfsense router is setup just where it needs to be but i really do want to move.
Yep. This will cruise along longer than the parent's solution, but when it breaks, you'll be starting over all of the original services from scratch plus the management system you had built once to manage them.
But it only breaks when all systems fail together; if your router fails, you can rebuild it from the gitlab job. If the VM host fails, you have time to replace it because the rest of your network still functions. If your git host fails, same thing, but where did you put those backups?
My solution is Kubernetes. Everything's configured in YAML files. The solution to all those problems is... change fields in YAML files.
Of course, you need to figure out what you need to change and why, but you'll never not need to do this, if you're rolling your own infra. K8s allows you to roll a lot more of the contextual stuff into the system.
Do you find there to be a good amount of overhead in running your own Kubernetes cluster? I'd think initial setup would be a bit of work, and then keeping the cluster updated and patched would be a good amount of work as well.
Then you've just traded maintaining one system for maintaining another.
Just started this journey myself and while there’s tons to learn, getting something up and running using k3os and flux cd takes no time at all and gets you a working cluster synced to your repo. K3s is pretty light, I know some people running it on pis.
If you use hosted Kubernetes (GKE, EKS, etc) then you don't need to deal with any of that, which is nice. You get the flexibility of Kubernetes without needing to care about any of the underlying infra
Once you learn it it's pretty straightforward. K8s has a very simple underlying architecture. It's intimidating at first, but yields to study and care.
I have also been using Kubernetes for this for years now. My favorite property is that it will run forever, no matter what happens.
The annoying part is that when I do want to do updates (i.e. updating cert-manager from 0.1.x to 1.0.x, etc) it can be a pain. So I save these large updates for once a year or so.
The solution is keeping a local mirror of all images and artifacts, and version pinning for stability (along with a periodic revision of version numbers to the latest stable version).
Oh and don't forget that now maybe you make everything work, but in two years time your setup won't be reproducible, because chances are the original images are not available any more, they got deleted from Docker Hub some months after you used them. Yeah, you should update them anyway for security... but the setup itself is not reproducible, and being forced to use the latest version of something, with the new idiosyncrasies it might bring, is not a nice situation to be in when you just want to hurry up and resolve your downtime.
So I guess that's one more thing to worry about it seems, maintaining your own images repository!
Maybe, but when the original docker image is no longer available on docker hub, chances are there will be something better and even easier to setup. And with docker you don't care about installing / uninstalling apps and figuring out where that obscure setting was hidden - all you need is just a stock distro and a bunch of docker-compose.yml files, plus some mounted directories with the actual data.
But a lot of those unofficial docker images are of unknown quality and could easily contain trojans. It's completely different from installing a package from your distro.
On the plus side, the Dockerfile and the repo with the scripts used to build a container is usually available. If you don't trust it, read through the source and rebuild it. Or just stick to official containers, no matter how terrible they are.
Even if so you're still spending say 50% of the original time investment every year or so just maintaining it. Unfortunately your options seem to be "set up once then never touch it again" or "update everything regularly and be at the mercy of everything changing and breaking at random times".
I mean, you should always have a backup of your dependencies (up to reason).
I develop mobile applications, and use SonarType's Nexus repository storage as my primary dependency resolver. Everytime I fetch a new dependency it gets cached.
A monthly script then takes care of clearing out any cached dependencies which are not listed in any tagged version of my applications.
Agree that documentation is key here. Anything you do that is beyond the vanilla "pave the install and plug it in" should be written down.
It doesn't need to be perfect - I have a onenote notebook that has the customizations that I've done to my router (static IP leases and edits to /etc/config/network), and some helper docs for a local Zabbix install in docker that I have. I recently how to migrate a database from one docker image to another and there is no way I would remember how to do that for the next time, so I wrote everything I learned down.
Just a simple copy/paste and some explanatory text is usually good enough. Anything more complex (e.g., mirroring config files in github) still (IMO) needs enough bootstrap documentation because unless you're working with it daily you're going to forget how your stuff works.
Additionally a part of my brain is worried that if I get hit by a bus my wife/kids will have a hell of a time figuring out what I did to the network. Onenote won't help them there but I haven't figured out the best way of dealing with this.
(I recognize the irony in a "I'll host it myself" post in storing stuff in onedrive with onenote but oh well)
Just to throw more products at the wall, I've been using Bookstack[0] for the same sort of documentation.
Besides being relatively lightweight and simple to setup, out-of-the-box draw.io integration is nice. Makes diagramming networks and other things dead simple. And I know "dead simple" means I'm infinitely more likely to actually do it.
I also started doing something similar via org-files, git, emacs, and Working Copy. It has worked pretty well, though Working Copy (the iOS git client) was buggier than I expected (but they have a great developer and support). My network isn't very good, or I'd just use emacs on iOS via SSH via Blink.
I work on trying to script each install. So if I need to repave, I have a documented, working script, and the source bits to work with.
I've preferred VMs for functional appliances for a while now. I like the isolation compared to containers. Though YMMV.
Right now, the hardest migration I have is my mail system, which makes use of a fairly powerful pipeline of filters in various postfix connected services. Its not fragile, but it is hard to debug.
I host it myself, as the core thesis of the article pointed out, you can be deplatformed, for any reason, with no recourse. And if you lose your mail, you are probably in a world of hurt.
The one thing I am concerned about is long term backup. I need a cold storage capability of a few 10s of TB, that won't blow up my costs too badly. Likely the best route will be a pair of servers at different DCs, running minio or similar behind a VPN that I can rsync to every now and then. Or same servers with zfs and zfs send/recv.
Thinking about this, but still not sure what to do.
I've got the same problem, I have an ubuntu fileserver I set up a few years ago, but have none of the monitoring/alerting that I'd have if I set it up for my day job. And really, I don't want to do my day job at home.
So it's a bit of a catch-22, I want a secure and stable home system, I don't want to spend much time working on it, but I want full flexibility to install and run what I want, and don't want to trust some off the shelf consumer solution that's likely going to be out of support in a couple years.
I just use a docker-compose stack (one yaml file next to a bunch of subdirectories) templated out in Ansible.
Ansible will be around for a while, but even if it's not its (yaml) syntax is incredibly easy to read. Any successor in that area is somewhat likely to have compatibility or at least a migration path.
This together reaps the benefits of Docker (enhanced through Compose), and Ansible is documentation in itself. There's barely any actual comments. The code speaks for itself. Also, I can reproduce my stack with incredible ease.
This right here. I recently lost my home server of 10 years courtesy of the Texas power issues during the winter storm. I rebuilt and started with fresh Linux install. Having a recent backup of /etc made it so much easier than it could have been. I had more trouble with the network driver on the new mb then with all my services, customizations and data.
Yikes, did you have it on UPS? How did a power issue kill your server? A big spike? Most filesystems like XFS should be able to fsck even after a power drop, especially with RAID. (I prefer RAID-10 for speed of rebuilds.)
Surge protector but the spikes were huge. My drives we're actually fine what got killed was the PSU and it was a case integrated PSU, in a system I had been meaning to upgrade for a while anyway. So this gave me the kick in the butt to finally do it. I also went to NVMe for the root drive wow what a difference.
The system that died was not out of date software wise it had been dutifully upgraded with every major Debian release so starting on a fresh install, adding packages and copying conf files from my backups, and adding my data drives really wasn't bad. Couple of days after work to do the hw build and restore.
Because of shortages / cryptomining / covid I couldn't get the HW I really wanted, that was frustrating.
Eventually, whatever platform / tool you use will need to be upgraded. Security vulnerabilities, new features, etc happen and projects like these can get abandoned within a 5-year timeframe. When you have to migrate to either a new or upgraded platform, you have to figure it all out yourself. When the config is broken by an upstream dependency, you’re on the hook too. Who knows if the build tools you used still work on current versions of things.
Like it or not, we’re all kind of stuck on these platforms we don’t control. The alternative is to become fluent in yet another technical stack, but one that will be used infrequently and won’t really translate to anything else unless you’re trying to build your own cloud service on consumer-grade hardware.
Today I've heard fellow SREs discussed whether RRDTool is the best solution for monitoring private things. Its only merit: it stopped evolving. Might or might not outweigh the decades of progress.
RRDTool is great for exactly this reason! I actually still use it to plot time series data on a raspberry pi for various projects. It hasn’t changed in at least a decade and it will run with good performance on any hardware. If it ain’t broke, don’t fix it
Long term platform stability seems to be moving from relying on the OS to higher up in the stack with the advent of Docker. It feels like Docker is being used more and more in cases where I would have considered something like CentOS.
Docker feels like the equivalent of the teenager that doesn't want to clean his room so he just pushes all his mess under the bed.
The complexity is still under the bed. We're all going to have to dig under that bed one day. Or we're just going to end up buying new hockey sticks, football pads, etc. Which is to say, we're going to end up with Linux on top of Docker on top of Linux.
I'm not sure I follow the analogy. I'm guessing the mess represents a messy filesystem/environment. What does the bed represent? A Dockerfile? Dockerfiles are way more transparent than, say, a random server's / directory.
That's just my opinion of course. It's possible to write confusing Dockerfiles but really they're mostly just shell scripts. And the idea of "Linux on top of Docker" seems a bit odd - there's only ever one Linux kernel no matter how many containers you have running. Docker is built on Linux.
It's closer to owning a car factory making disposable cars and getting a new one when it gets dirty. At some point your supplier shuts down and you cannot make a specific part anymore, which prevents you from building another car. You are now stuck with the dirty car.
If your application doesn't rely on hardware details (such as GPU acceleration or networking) that bypasses well-defined OS APIs, it's actually a very good approach.
But when you do Kubernetes or Docker, there’s almost always another stack beneath you that you have to build and maintain. Whether that’s bare metal (yikes), VMWare ($$$) or AWS/Azure (not self-hosting) you still have to deal with upgrades / API changes / hardware refresh there.
Container solutions really only get you out of “doing the work” if you can leverage a prepackaged container management solution from a cloud provider. Self-hosted containers are frequently more trouble than they’re worth since the solutions for managing them are either insanely complex (Kubernetes) or so simplistic you have to write some custom logic to build/deploy them.
Agree with the point about CentOS though; at this point the idea of a Linux “distro” is dead. The way forward is a hypervisor model where the kernel is protected from application code (including all dependencies from the userland) with barebones Docker images like Alpine used as the basis for a declaratively-defined system. The one thing Docker does very well is isolate dependencies which reduces integration complexity and lets you modularuse the whole thing. Incidentally this actually makes it harder for hobbyists to get into as the mental model for it has a lot more complexity and you need more expert knowledge to write the scripts to build and configure your app automatically.
Seems like there's a probably reasonable trend of piling some other tool on top of the stack because dealing with underlying layers is hard. Like electron apps and docker images. Or just web browsers.
Kind of worrisome to abandon lower layers with their problems and build on top of them, but what can you do, but get good at jenga.
I feel the same, however I force myself to keep doing that. Without docker.
Yes, every time a new debian release is out something I fiddled with will break and I have to remember how it works, but I see it as some sort of training. As a dev I don't want to be completely clueless about how things I use every day work, so while changes that break old configs and workflows are annoying I'm forced to learn what changed in the gnu/linux/debian world and maybe even find out why.
Also I get better at documenting things. Years ago I didn't even know what exactly to document since the moment you do something everything just naturally comes together, but after a couple times you kinda get a feeling for what the important bits will be two years down the line.
So about once a year I reserve a perfectly good weekend to upgrade, restructure or otherwise maintain my little home server running debian with things like mariadb, nginx, a filtering bridge for my lan, dnsmasq with a block list, borgbackup, syncthing, cups for printing and a couple other things I don't remember right now.
I encounter this every 6-12 months when I go back to an old project that is 'working' and want to add something/update something and it all just looks foreign to me.
The worst thing is that I have often gone through a lot of effort around making it easy to set up and deploy (docker and whatnot) but even that I have forgotten about. (I came across a docker file in an old project and couldn't get it to work properly until I noticed that there was a docker compose file lying around that I had missed)
How do you keep track of documentation? I guess for a project a README in the git's root is a good start, but what about more complex systems stuff that does not live in a git project? For example, I had to manually edit a bunch of config files on my Proxmox setup to get docker and some other things to work properly. Where would I document such manual steps? I am thinking a text file somewhere in cloud storage but then of course I'd need to remember that...
My doc is split between Notion (for bigger, structured projects) and a bunch of local md files (for general, greppable knowledge).
For my VPS, I have a Notion page where each project (name, url, mapped ports) is a row in a table. Then the project page contains a copy of my docker config and various informations I might need for maintenance/reinstallation.
I don’t try to put the documentation next to the thing being described because then I’ll lose track of it. I set up a simple Gollum wiki and put everything there. That way I get md for useful formatting, version control on the docs without having to create a zillion git repos, and I never have to wonder where the docs are.
Nix helps with this. Or at least it intends to. I still need to track down what's vulnerable and what isn't, but most of my setup is reproducible thanks to Nix.
I've read people admitting that Nix can give you a lot of work from time to time, when something isn't already available for Nix (which happens more often than, say, for Debian), and you want to add it to your system. Is that true in your experience? I may be phrasing it wrong.
I found that the nix learning curve to be steep but short. The first time you need to make something available for nix (particularly a service), you'll probably copy something someone else wrote, make a few changes and then (if it didn't work) stare at your screen for a while wondering why it didn't work[1].
It is very different which can be very off-putting, but is not usually gratuitously different, and once you get used to it, it's pretty straightforward.
After having done it a few times, I find that I can adapt a random project not already in Nixpkgs for nix in under an hour, and it's something I do maybe twice a year or so.
One counterintuitive advantage I found switching to nix from other systems is that since the 4 step "download, configure, make, make install" usually doesn't work, I take the time to make a nix expression. On Gentoo and Arch, I would often just install to /usr/local from source and then forget what I had installed and not know how to upgrade it. If you have more discipline than I do, then it's a bug not a feature, but for me it's super helpful.
1: If the project uses cmake or autotools and has no strange dependencies, then packaging it for nix is trivial. However a surprising number of packages do things like downloading dependencies from the internet at build time, and it's not always immediately obvious how to adapt that to nix. Projects using npm or pip also probably won't work right away just because the long-tail of dependencies means that there will be at least one dependency that isn't already in nixpkgs (haskell should in theory be just as bad, but the strange proclivity for haskellers to use nix means that someone has probably already done the work for you).
Just firewall off all management interfaces and allow via IP as needed. It's still possible your webserver will get become vulnerable, but you'll prob here about it here if it is.
I almost guarantee that when you need to make a change to your home server in 10 years, docker compose will no longer be able to pull any images from docker hub without being upgraded, and upon upgrading you'll find all your config is now invalid and no longer supported...
> when something finally does go belly-up, I can't remember how I set it up in the first place
Why should you need to remember it? Like you wrote later on, you just " document everything you do", as you do it. That's better than any sort of script or version control, since you can describe it how you like, which means quite succinctly. And it's not too difficult to adopt the mindframe of "What would I have needed to know 10 minutes ago to understand what to do".
> Docker has helped tremendously, since you can essentially use an out-of-the-box Linux distro with docker installed, and you don't really have to install anything else on the hardware
This actually illustrates, IMHO, how containers and docker are overused. You're talking about a single machine with a single purpose/set-of-purposes, and no occasional switching of configurations. So - why containerize? Whatever you have on Docker, just have that on your actual system.
> my advice is to keep the number of customizations to a bare minimum
Sound advice based on my own (limited) experience with self-hosted home-servers.
> and just as importantly, back up every single configuration file.
Fine, but don't rely on this too much. It's always a pain, if at all possible, to restore stuff based on the config files. Usually easier to follow your self-instructions for configuration.
I think it's important to document anything, be it at home or for business. I've done good and not so good jobs with this at times, and you feel like a hero when it's well done and you suffer through reinventing things if it's poorly done.
I agree you should have config files and backups, but like with work, I think it would be good to go through a "disaster" where you have to build a version of your config from an unconfigured environment.
A colleague had built such a config for a small office (10 people), but it depended on a specific DNS that ... wasn't set up until 1/2 way through the config. It worked because in testing they were using the network with a DNS that was already working. Small things like that are hard to catch in a working environment.
What I sometimes do is make a bash script that does all the setup. save it in the home folder. You can just copy/paste each line from the script to set things up again and you'll be able to know exactly what you did to the system later on.
> These days, I design everything for home with extreme simplicity coupled with detailed documentation on how I set things up.
Definitely a good strategy, yeah. And on the documentation front, that's exactly what I've started doing differently, too; after having to recreate my mail server one too many times, I eventually decided "you know what, I should probably write down what I'm doing", and now if things go belly-up again I at least have that starting point.
As a bonus, you can also put it online as a tutorial, which is exactly what I did: https://mail.yellowapple.us :)
I am self hosted since forever or I will rather say - since ADSL was available. The server has changed "a bit" during the last 20 years (oh, Mercury mail server :D), but last "few" years (from FreeBSD 8 :D) I am happy camper there. Just using it casually and learn a thing or two by the way. Upgrading it, migrating (2 times until now), adding disks. Nothing special.
Just got into situation where mobo has started failing and I have said it is time to reinstall, bought new hardware and throw it together. Migrated in a week, everything that was customized during ~20 years without taking notes (but with my good ol trusty diffing software) and was migrated from previous server, optimized a bit, removed unnecessary settings, upgraded postfix and dovecot to new server, replaced spamassasin with rspamd, php-fmt,...
The server was mostly operational in 2 days. Everything else was studying new software, doing things that i always wanted to but didnt want to turn around whole configuration (like stuffing everything into jails), customizing netdata, upgrading database/nextcloud/... etc. It would take longer if I would loose the data, but I trust zraid and my LTO drive, they never failed me.
Now I will sniff around it every week, maybe run some update etc. and I will be fine until some disk fails.
Being system administrator is not my occupation, but it helps if you NEVER EVER become a hostage to a cloud provider. The more you go into leisure of someone else doing everything for you - the less chances you will have to learn new things and the more you will be on mercy of someone else. And the longer you are enjoying such situation, the more technology progresses and the bigger the gap between the knowledge you have and the knowledge needed to, in this case, setup a server.
I remember dreaming in 1991, as a kid, that in 20 years everyone will have his own server at home, and how we will transfer files simply. But now everyone is saying they don't have time and buy ready made boxes or pay for the cloud.
The technology came, now it is simpler to use than ever.
But no one has "time" for that now. Or is really the time?
---
Dont take having a home server as a pain. It is a great way to learn things.
I can bake a perfect bread too. "Kicked" all my girlfriends out of kitchen. Brewing home beer. ...
But all this wouldn't happen if I would rather go to bakery. Eat in restaurants. Buying beer in a store. ... Sure I could, but I didnt.
You don’t even need Docker. Another great technology to learn is LXC and driving it from the CLI / a script.
All my hosts run a default OS with a 20 line firewall and a bridge.
The top level host has a zpool backed by a blank file in /tank.device.
The actual work is done by a bunch of LXC hosts all cloned from a standard base installation. Anything persistent goes in a per container zfs filesystem mounted in each container’s root. The only thing I ever backup is the /tank.device file.
Wiping everything and building from scratch is a pleasure.
I personally found Docker and Compose pretty simple to learn, and after years working with them I'm pretty well acquainted, but am interested to learn if there are any tangible benefits of using LXC instead of Docker?
Docker has nice image caching and fetching. It’s all about deploying and containerising in one step, usually a one line shell script alongside the container config.
LXC is more like deploying disposable OS instances and then having to deploy your app yourself, separately. It’s more bare metal but has fewer surprises in terms of supporting IPv6 and not having any inscrutable iptables magic happening on the host OS.
Docker is IKEA. LXC is hiring a joiner. (Not a value judgement.)
If you want stability I would go with a custom OS, eg. no auto updates, only update parts of the system if you need/want to. And apply security/hardening in several layers. Make backups of course. Then you could leave it running, with really nothing to worry about except hardware failure. The problem is that when the hardware do fail, it might be difficult to get a CPU with the same architecture. (like 50 years from now)
I guess this goes back to if you are DIY person or not.
For me i treat that just like I do any service in my home. I am the type that will tear about my Dryer to fix it vs buying new or bring in a repair person.
I repair my own car, appliances, etc. For me the Home server is the same.
I have been through this too way too many times. Every method I found either were as bad or added more problems (like docker, configs in git, etc.). Now I have removed almost all customisation/configuration from services I need and exchanged them with what are basically default setups of containers in proxmox. Everything is backed up and the amount of configuration needed if something did somehow go away is measured in minutes instead of hours. Luckily I don't need remote access which makes this easy.
These days, I design everything for home with extreme simplicity coupled with detailed documentation on how I set things up.
Docker has helped tremendously, since you can essentially use an out-of-the-box Linux distro with docker installed, and you don't really have to install anything else on the hardware. Then if at all possible, I use standard docker images provided by the software developer with no modifications (maybe some small tweaks in a docker-compose file to map to local resources).
Anyway, my advice is to keep the number of customizations to a bare minimum, minimize the number of moving parts in your home solutions, document everything you do (starting with installing the OS all the way through configuring your applications), capture as much of the configuration as you can in declarative formats (like docker compose files), back up all your data, and just as importantly, back up every single configuration file.