I have a weird a problem. I have been doing infrastructure and platform engineering for I don't know how long now. And with it, I have gained practice on all type of orchestration scheduler systems and platforms.
From the linux kernel scheduler like BFQ and CFQ, unit jobs scheduling in systemd, app containers with podman and docker, system containers like LXD, and VM managers too many to count. Or what of the full blown distributed job orchestrations. Like, kubernetes, nomad, docker swarm, and many more. I've touched it all... And that's the problem.
I've dabbled in things. Enough so that I feel comfortable in working with all of these tools. But that's the problem. All of these options have a good place in something. Choosing only one is hard. I can run my bespoke something and totally do. But also have app containers, kubernetes, and have LXD clusters in one way or another somewhere.
Take job orchestration. If you want to run on one machine, you can run on systemd directly. No need for anything but systemd and unit files. But now you handle the job placement and management of jobs. You could use systemd-nspawn and systemd-vmspawn to make those unit jobs contained and controlled. And for certain workloads, that makes a ton of sense. But this in itself is very bespoke. You'll be managing the OS and the orchestrator. Say hello to Ansible, Chef, Salt, and Pyinfra for me. All kind of amazing at what they do. All can make your bespoke something setup both easier and immensely complex. However, you need them to provision the job orchestration even on one machine. Else... good luck with configuration management. That secret boss you find, during the final boss battle, that happens to be even stronger than the final boss...
Say you go a different route. Say you 'just' want to package your app in an app container. Well then, docker and podman are likely your friend. But now you manage the container distribution. You could use quadlet or docker compose. Again, provisioning configuration management plays a vital role here. Distribution of your declared app containers is just as important as the runtime they run.
Then what if you go full in on declared orchestration? You'll now talking nomad, docker swarm, and kubernetes. Where kubernetes happens to be the bike shedding of bike shedding. You will tune and tweak like you have never done before. And you will hate and love yaml engineering. More than you have ever had something in your life. With billions of hours spent throughout industry (and your homelab) to make this kubernetes thing work.
But also... it's billions of hours. Maybe the complexity makes sense, but like... Maybe you don't need make kubernetes your job, for the 45th time, to run your blog.
What about 'just' running a VM? Well, you can go this route. And it can be pretty sweet too. LXD is still good. But again, configuration management is key here. Incus (fork of LXD) even has a whole operating system called IncusOS. Just to focus on running the thing! Which actually does remove the configuration manager of old! For an api configuration manager of new! And it's very much brilliant. You can likely set up an operator for deployment orchestration. Oh. Look. You are back to building a kubernetes.
Unique workloads is where kubernetes kind of creeps in. As the further you go, as the more complicated requirements creep in, that bike shedding starts to feel really nice.
Yet, you can probably get by with ssh, a VM, and duct tape. If you get fancy, you can drop the SSH part all together for deployments by API. Only accessed by some overlay network you constructed cause Linux Unplugged been talking about nebula. You can make the case for any of the options. And likely others not mentioned.
Right now, my set of requirements is I want to run unique workloads that are stateful. Think databases all the way down. That means, maybe system containers are better? nspawn is kind of killer. MicroVMs using qemu is very much doable with little to no extra dependencies. It all boils down to how much of duct tape to write.