HashiCorp employee here. Nomad has an open source core which is very capable. There is an Enterprise edition for those that want support and access to several Enterprise features. Does the open source edition address your non commercial criteria?
I also found that Nomad looked promising for VM orchestration. In my case I was specifically looking to run Firecracker VMs. Unfortunately I found the Firecracker community driver [1] to be difficult to work with when setting up networking beyond a simple "hello world". It does not play well with Nomad's CNI plugin [2] and I could not get a working setup where Nomad, CNI and the guest would agree on an IP.
The firecracker driver is a community one, however, and the offical Qemu driver is most likely more polished. Did not try that one since it did not match my needs.
Does Nomad support typical VM orchestration like OpenStack or vSphere? It seems to me that it is more of just a generic driver that will do basic tasks.
Nomad supports it. I ran Nomad instead of Kubevirt at some point. Functionality was great, but it missed the live migration. My usecase did not require it, but it was later pointed out to me.
I use XCP-Ng for orchestration[0] and Xen-Orchestra[1] for management, both are open-source. Xen-Orchestra is open-core but if you build from sources or use an unofficial docker image, you can get a lot of the commercial offering back for free at your own risk.
With Xen-Orchestra you manage one or more XCP-Ng Servers, combine them in pools, live migrate between hosts, patch your hosts and schedule VM backups to S3. It's a killer combination.
Having tried Kubernetes on Raspberry Pi's once (my homelab is all SBCs, not all Pis but all ARM), there are two major pitfalls:
1. K3s may not be too heavy, but basically anything that runs on it is. For all but the most basic jobs, I was still looking at one, or maybe two, apps per SBC. Couple with 10/100 networking on many of these, and there's a lot of extra time/latency spent just on chattering across the network.
2. It's relatively rare that you're using _just_ the SBC - more likely is that you have some peripherals, or some local storage. That limits where your pods can schedule, but because of #1, you still end up with dedicated nodes for dedicated functions.
I ended up pulling the plug on that project early, and went back to scheduling workloads manually. For instance, I have Transmission and Squid running on the Pi with the big flash drive. InfluxDB and Grafana live side by side, and also live next to the SDR (I take in some metrics by radio). It's all still containerized, just with manual `docker run`'s and versioned config files, instead of with Kubernetes.
System requirements for nomad on Hashicopr's assume a relatively high-load.
Running a single Rpi 3b+ master with a 10-13 clients at home was not resource intensive at all. Master was effectively running on idle temps and opening something like mc or htop took more resources than the server itself.
Orchestration is a tough one because the second you bring it up, it devolves into a bunch of enterprise offerings and datastores and servers and holy wars and evaluations and managed services and horrendously complicated setup. Workflow engines, schedulers, all supporting "standards" but none interop without a million lines of glue code.
I've ended up writing basically this:
- Groovy for a lot of syntactic power, decent performance, ability to make workflow-ish scripts
- GPars for orchestration primitives to do stuff either serially or in parallel.
- a configuration service to have configuration scoped at the global/environment/cluster/datacenter/rack/individual node levels
- an access layer for executing remote commands and getting stdout/stderr back, and for sending, receiving files. It can be SSH, kubectl, AWS SSM, dockerrun, etc.
I've had to rewrite the core framework four times, but it has enabled me to do complicated orchestrations such as spawn clustered databases, spawn fleets of load generators, data loaders, or api servers, tear them down, etc.
I can code distributed backups and restores, and other cluster-wide "applications" with this as the basis.
I can run things from the IDE. I can run locally or from remote servers. I can CRON them or whatever. I can debug them from IDE. I can do adhoc/arbitrary cluster-wide operations/commands using the Groovy Console.
I will probably OSS it at some point, but it's not a finsihed product. But after three years I can't imagine doing what I do with it anywhere else.
It could be called by workflow engines, it can call out to terraform and other tools. It can do multicloud (with enough code :-), it can do mixed access / deployments (with enough code :-)
I guess the other limitation is the basic problem with a single orchestration node: probably won't easily scale to the 1000s of VM/nodes easily, or as efficiently as some other distributed coordinator could, but those things all come with the one-orchestration/integration/workflow-to-rule-them-all products. I don't want to be that.
Using none currently, the following seems to be enough: containers for apps and KVM+vagrant on a laptop for experimenting.
However, for VM orchestration. OpenStack has been my choice for long time, but about to check KubeVirt and (old and underappreciated?) Apache CloudStack.
We use Joyent's Triton in production. It's not something that can be plugged into existing systems easily as you need at least two dedicated server nodes for it, though.
Are you sure Nomad orchestrates VMs? Do you have any relevant links? I thought it was mostly used for Docker containers, but of course you can run QEMU in a Docker container à la kubevirt
Nomad is not a "container orchestrator" but a "task orchestrator", whereas containers can be one type of tasks. Others include Fork/Exec, Java, Podman, QEMU, VMs and more (since it's easily extended). Checkout the full list here: https://www.nomadproject.io/docs/drivers
My personal favorite is using the Firecracker driver for running Linux microvms.
Nomad engineer here; Nomad supports Task Driver[0] plugins enabling you to orchestrate anything you need it to. Nomad ships with a QEMU task driver[1] built in.
hi queue29,
what are the minimum production requirements for nomad ?
what is the cheapest machine i could use from digitalocean for a production task ?
Check out the recommended requirements[0] documentation. In practice it depends greatly on how many client agents you want to manage (i.e. how large is your cluster). I personally run Nomad + Consul + Vault on the $6 AMD DO droplet to run a few website, and that node still has room to spare.
Hey, afaik no, but you can compile nomad to work with OmniOS so i assume you can do the same for SmartOS. Haven't tried that in quite a while, so don't quote me on that. In this fork you can find fixes that add OmniOS support for older version of nomad.
Nomad can orchestrate any process, though it's commonly used with containers (Docker or otherwise). However using to to orchestrate VMs sounds like a poor fit.
thanks for the apt response aligning with the lingo,
yes - nomad usually orchestrates containers. ( and not vms was my belief too, i thought it can replace kubernetes )
On prem Vms setup manually. I cloud provider Vms have their own primitives (aws asg), with something like terraform. Then a kube flavor for all the real orchestration.
What are you actually trying to do? I ask because I’m looking for a good tool to provision VMs on VMWare ESXi. That’s different though, Nomad won’t help me with that.
In my case we use Ansible and Puppet to orchestrate VMs. Puppet for the stuff all VMs need, and Ansible for deploying custom software. Given that we at the most have 8 VMs that are completely identical, Nomad makes very little sense.
In my case it’s a micro services architecture.
We use vms over containers for elevated security.
The goal is to automate production as much as possible.
You could run them all on the same host and use docker with kata containers. The firecracker hypervisor was developed by aws to even run multiple tenants on the same host, so it should be very secure.
Orchestration is more than just configuring individual instances, but rather being able to deploy your entire app which may include instances, load balancers, lambdas, data stores and/or other infrastructure.
Interesting! I haven't used VMs in a while due to being fully sucked into the world of Docker, but isn't VM configuration at the point you can do something similar to a Dockerfile to run? Or is it easier to go from Dockerfile -> VM?
We use Ansible to configure and deploy containers in many cases. Dockerfiles and container images aren’t really doing the same thing as configuration management.
Here comes a long one, but tl;dr: make sure you know your goal before you select your tools.
Orchestration is a somewhat broad concept and can be interpreted at many levels. From a management perspective it is about what processes are acting to the benefit of the organisation to fulfil a business need, regardless of the amount of automation. Having a room full of humans vs. a couple of binaries and event logic doesn't matter if the orchestrated result is the same.
Now, when it comes to your extra information you seem to be using microservices, but deploying them as individual virtual machines instead of just containers. While I applaud the choice of better isolation, virtual machines isn't the only way to do this. Firecracker does this too but much more efficient with the same protections (https://firecracker-microvm.github.io). So before going all-in on virtual machines, think about your goal and check to see if this is really the best way to do it. General business applications really don't have this level of isolation as the main factor in safeguarding applications as there are many other more fragile layers where the actual problems occur. A well-isolated application with no AAA on its interfaces might as well not be isolated at all from a security perspective.
The selection of tools is probably going to be influenced by what else you are going to do. If you have automated infrastructure you'll probably want to tie in to that. Some automations have to trigger at deploy-time to (de-)provision at the right moment, while others have to simply react to the fact that something is needed and autonomously scale with the requirements.
If your (micro)service needs a database, it probably also needs configuration to access it, and the database needs to be configured as well. Perhaps there are some networking components, and storage if you have object or block persistence needs.
If you are running on AWS for example, your 'orchestrator' is EC2, but you still need to 'orchestrate the orchestrator' to start the correct configuration with all its dependencies prepared ahead of time. You'll probably want to prepare the virtual machine image so it actually contains everything it needs during build-time so it can just do the running at run-time. Packer comes to mind for such a job.
Most systems that are 'good' got there because they had a lot of usage, because that is what generates data (feedback) on how well it does the job it needs to do. With that larger amount of usage comes a higher load from stakeholders and with that there tends to be a growth of commercial incentive turning a lot of products into some collection of commercial side-options or main-options.
To get a bit closer to a possible answer, I'll sketch out two scenarios, a managed one and an unmanaged one (from a Layer 1 and 2 perspective, hardware).
Scenario 1: Mostly managed primitives.
Depending on what you already have you are likely to have an API that interfaces with the storage, networking and compute resources that are available to you. Depending on the systems, Terraform and Ansible can configure and prepare almost anything to your liking and make sure it stays in the desired state.
When deploying an application as a virtual machine image, you'll probably have the following events happen:
1. SCM change triggers application build (Git and some CI solution)
2. Successful build triggers next integration step (CI)
3. Integration takes your build output and wraps it in a virtual machine image (Packer)
4. Successful integration triggers deployment step (CI)
5. Deployment step contacts your API of choice to let it know about the new image (Salt/Ansible/Terraform/FaaS/Webhook)
6. API registers the new image and you can start rolling over instances until all the old ones are gone
Scenario 2: You have to DIY everything.
Steps in there are essentially things you have to do before you can start thinking about scenario 1 which is what follows after this.
1. Make your hardware boot from the network
2. Setup a network boot server
3. Prepare self-starting hypervisor images of your choice (KVM-based, Xen-based, Hyper-V-Base, vmWare-based)
4. Make sure your DHCP and/or TFTP servers have logs you can access/parse
5. Once a server boots, it loads a self-starting image and depending on how you configured the images they will start out 'empty' with a well-known key or other credential
6. Use something like Ansible or SaltStack to discover any servers and select the 'empty'/'fresh' ones
7. Depending on your selected hypervisor, you'll now have to connect to a management API, control plane or simply write out a list of nodes somewhere (useful for small-scale SaltStack setups!)
At this point, the 'compute' is done and ready for deployments. A big issue here is that we have many assumptions and the amount of new variables are so large that it's hard to get some more steps in.
Next, you'll need storage which is much too broad to cover here, but it generally ends up with some sort of redundant NFS, iSCSI or more modern setup like Ceph. Those are all configurable via SaltStack or Ansible.
You'll want networking after that, something like a vswitch with VLANs maybe. A very simple way to do this is start an OpenWRT or OPNSense VM with a bunch of pre-loaded virtual interfaces, subnets, and firewall rules. How this exactly works depends very much on what hypervisor and control plane you have. VMWare with NSX is very different from Proxmox with openvswitch (but they do the same thing).
When you now want to 'deploy' a VM, you use your Ansible/Salt/Terraform-enabled infrastructure where you can 'update' the base configuration of a VM-based application and let Ansible/Salt/Terraform create new VMs and destroy the old ones. Configurations might be flat files written out to the network-booted compute nodes, or some configuration database in a controlplane behind an API.
Now, all of this can be partially homelab-automated with things like Proxmox, Xen XCP-ng, VMWare etc. But as soon as you actually need more facilities like actual API-configued load balancers, databases, metrics, discovery you'll start home-cooking a complete OpenStack replacement. At that point, just get OpenStack or realise that the benefit of scale you get with Azure/GCP/AWS lets you do what makes you money instead of doing busywork that doesn't actually matter in the end in many cases.
Hey! Thanks for the awesome response.
Firecracker is awesome, I had a chance to use it a bit. Firecracker is a vm too, I think. It’s KVM under the hood.
You are right that in the general case going for the cloud is the most appropriate choice.
I’ve made a choice to self host and it works for me in so far for a few years. I’m looking to have more automation now.
What I currently have in mind is a nomad/consul (with mesh) architecture over hetzner dedicated nodes. So nomad will oversee a fleet of vms on these nodes.
To use packer or similar tool to create images and deploy using nomad.
This is the base and then I can add load balancers, monitoring, logging etc as needed, cloudflare etc.
While I know the name Hetzner I don't know if their server offering is the same as their 'cloud' offering. They do have an official Terraform provider:
The benefit of this is that you can put all the common items in place (network, volumes, keys, firewalls) ahead of time and then just add instances as needed.
Their official provider also has a lot of Datasources meaning you can automatically get all the data you need in other places and then use terraform to write it out wherever you need it. As an example, Terraform can use the hcloud provider to create or read resources and use the consul provider to then write it out to consul. Then, if your application or orchestration tools need some data they can just get it out of consul.
From the looks of it, you might have the following list of actions:
- Build the software from sources (whatever CI you are using)
- Take the built software and put it in an image using Packer (https://www.packer.io/plugins/builders/hetzner-cloud)
- Use terraform to make sure all the non-deployment-specific stuff is ready
- Use something else (Harvester, Kubevirt, Nomad etc.) to make sure the right amount of servers from the correct image are started/stopped
While you could technically use salt-cloud for everything, In my experience, salt is best at 'runtime' stuff, not as much preparing and desired state configuration. I personally take the approach that Terraform is for everything "outside" virtual machines/containers/FaaS, and CI/Ansible/Salt is for everything "inside" virtual machines/containers/Faas.
Using a ton of different tools might make it really complicated, but trying to do everything with 'one golden tool' is going to give more problems than benefits in general. One benefit of pulling in Kubernetes (but just for the Kubevirt part) is that there is a Kubernetes Autoscaler for Hetzner which means that you can specify "I want 10 virtual machines at all times, and they have to use this specific image" and Kubernetes and the Hetzner Autoscaler will make sure that there are always 10, if one crashes it gets replaced, and when you update the image it first adds 10 replacements and then removes the 10 old ones (or you can configure a different rollover method of course). The downside is that if you don't have Kubernetes experience yet it will be a very steep learning curve and in such a case I really would not recommend this (yet).
I do recommend keeping principles like GitOps (Git is the source of truth and operations have to flow through Git so you always have a record of change and can go back if needed) and disposable/cattle instances (never make an instance special, always be able to just nuke and replace it; this also means that if a system gets compromised you can just nuke/suspend it and know that you can start a new instance [with patches], and if there is an outage you know you'll be able to recover because that recovery is the same as your normal process).
If you already have all your virtualisation covered, it becomes much simpler:
- Find out what APIs/Configuration stores are available
- Put long-lived deployment-static things in Terraform/Salt
- Put show-lived deployment-bound things in CI (and then Ansible/Salt inside there optionally)
The CI-things can trigger OpenStack/XCP/Proxmox/Vmware and tell it to re-deploy your stuff. If you have many 'things', Nomad, Kubevirt (with ArgoCD or Flux), Tekton, Teletraan, Spinnaker are your best options in order of growing magnitude.
For me, if I were to run a small VMWare set for example, I would select Terraform as my orchestrator of choice. Then in my CI (GitLab-CI) I would make a step that once Packer has built a new VM image, I would update a file listing my to-be-deployed images and automatically commit it to a state repository. I would then also have Atlantis running which notices the change and runs terraform for me. It would add/remove all the resources needed for my VM to work with the correct network and storage. I would leave discovery and joining/leaving load balancers etc. to be the job of the VM image, perhaps via Cloud-init or Consul or something like that.
The problem with applications-as-virtual-machines is that those are surrounded by tools and ideas that they live reasonably long lives and consume quite some resources. You won't really see someone run 1000 VMs with 0.15 CPU and 64MB RAM as a microservice 'instance' but you'd see that happen a billion times with containers. Managing the lifecycle of a virtual machine means you have to manage its storage and networking too, and responding to outages of any of the primitives (net, cpu, storage) is much more cumbersome than a container since the container doesn't need to know about any of that, it just receives it from the host.
A secondary problem is that whenever you end up in orchestration land, people tend to think it's complex tooling, or that containers are complex or that Kubernetes is complex. To a degree that is true, but when you start reinventing those wheels you just end up implementing
50% of those existing solutions with 0% documentation and nobody in the world will be able to help you because it contains so much nonstandard stuff. A lot of CI/CD and management processes are just 'state machines' or control loops. This is also what most services in Kubernetes are, and most services in public clouds. They just respond to the state of the world vs. the desired state. Whenever you're not using what already exists, you're essentially just re-implementing those exact same control loops/state machines in a less-robust and untested way.
Thanks! I will check all the names you’ve sent.
I completely agree with you on your last points. In your other comment I added my currently planned architecture. It’s planned to use only standard tools.
Thing is there aren’t so many, therefore this hn post
