One caveat when deciding on an ARM platform to adopt is the fact that someone needs to maintain a custom Linux branch for each individual SoC configuration.
That means you'll want to choose a platform that's very well supported by its parent company if you don't want to get stuck on an obsolete kernel in a year.
Hardkernel, who design and produce popular ARM SoCs, chose to produce x86 based platforms in the future for this reason.
ARM server platforms are moving to adopt standard firmware solutions like UEFI (yes, UEFI sucks in many ways - but A standard is better than what we have now) to start alleviating the device tree madness, just don’t hold your breath on it being mainstream for hobbyist/maker products anytime soon.
While the article talks about not being able to run x86 binaries on Arm, this is the real drawback to the platform.
I really wish we would see a standard Arm kernel instruction set come along and get adopted by every manufacturer who isn't doing something unusual enough to justify a custom architecture.
The ARMv8 server platforms follow a common standard, SBSA. Which is essentially taking the most-hated parts of the x86 platform, ACPI and UEFI, but once you suppress the gag reflex it does get the job done, and you have a mostly x86-like experience that you just boot a generic armv8 distro and it all autodetects stuff on the fly and it all just works.
Haven't heard of any of the single-board platforms adopting it, though.
This is going to be a stupid question...but what does clustering do? Does it increase the amount of "cores" in a system? What about the RAM? Or is the clustering software a task manager that allocates different boards as needed depending on the the task load?
There are no stupid questions. Clustering allows you to do more jobs at once than you could do with a single machine of the same power. It allows for redundancy if you set things up in such a way that if one hardware components fails another takes over. It can allow for better isolation between processes for security reasons than what a process isolated from another process on the same machine can give you. And many other reasons besides, so clustering is a technique that can give you many different benefits.
But...
There is a price to pay. Clustering isn't easy, convenient or without overhead. As soon as you cross the boundaries of what a single machine can give everything suddenly has a price, either in terms of latency, throughput, complexity or loss of control. Clustering software tries to hide some of this but under the hood these will always be there.
Here are some good hints about what limitations you are operating under no matter how good your distributed environment:
> You can't run binaries designed for a normal PC on an ARM board
...doesn't Debian ship some multiarch/QEMU/binfmt plumbing to make this work? Binfmt to taste the binary, see that it's i386/amd64, and then launch QEMU (and associated ARM multiarch libs) with user-mode emulation?
Surely that does not qualify as "running the binaries" if you emulate another arch! I mean if you want to be nitpicky, yes, you're running them, but that is not what most people think when saying that.
Only practical reason I can recommend this would be either you'd like to experiment with cluster tools or connection outages (e.g. seeing how system tolerates when you unplug an ethernet cable).
Also, while OpenStack/K8 could seem to be a reason having something like this, I discovered it is a bit overkill for home needs. For that part, I'd go for a small server instead and manage it manually (no shame).
I was looking for something similar earlier this year and ended up with SOPINE's cluster board. It only supports 7 ARM modules at a time but you can always stack multiple boards. Its also inexpensive. The SOPINE board I have has worked very well and recommend it.
Also check out Rancher. Lite weight K8 for distributed IoT. Scaling to 1000s of ARM nodes, clusters morph into incredibly smart supercomputing grids ;)
Unfortunately, none of the tech referenced in this articled is available yet and should be released end of February. There is a more interesting part that isn't referenced in this article and the is that we'll have two variants of k8s for ARM. One is RKE which is our CNCF certified k8s distribution. The other will be based off k3s (github.com/ibuildthecloud/k3s) which is an optimized k8s distribution for small clusters (smaller memory, easier setup, etc). I expected the second will be far more suited for something like this project.
It's too bad that OpenMosix withered on the vine. Back when it was a thing, I found it really handy to fire up clusterKNOPPIX on a bunch of idle PCs for large compile jobs.
I guess with today's multi-core CPUs, it wasn't thought to be worthwhile, but it could still come in handy for resource-constrained systems like the Pi.
I share your sentiment regarding Mosix. I don't think however, that clustering SBCs is of any value beyond the learning experience. Even a moderately powerful laptop is much faster than a *Pi, e.g. a 2013 Chromebook Pixel is about 16 times faster than a Banana Pi, when measured using `openssl speed`.
It depends when you're trying to do with them, but the network speed of the Pi is definitely a bottleneck for me. If it weren't for that, the throughput could definitely be used for some cool stuff even with modest CPU/RAM.
Unless you are running 3B+ pis, the firmware code for pxe booting has several bugs. Also, Pis have issues with retransmission of DHCP requests on low end network hardware that doesn't set link up fast enough to hear the Pis intial dhcp request. Almost all of these bugs are fixed one way or another at this point though, the github issue tracker for raspberrypi/piserver and friends has lots of info.
If you are using (two) DHCP servers in "proxy" mode as suggested in the official tutorial, it indeed is extremely unreliable. I think the firmware just does not accept the PXE-containing response if it arrives second or something.
Neat! I built something like this long ago (https://clustercompute.com/) to digitize a very large number of CDs for a Dutch broadcasting corporation. If I were to tackle that project today this is probably how I would do it.
Wouldn't doing something like this with FPGAs be ultimately cheaper and more efficient. Sure, it will have a big overhead to learn, build and set up your FPGAs specifically for your problem, but that seems like a constant time overhead?
Given what the article says about k8s not working so well in the RPi's 1GB of RAM, is there another cheap ARM SOC that has more RAM available that could work similarly?
RK3328: Pine64's ROCK64 comes in 2GB and 4GB of RAM
(also the RK3399 based ROCKPro64 is even more awesome, bigger form factor, but two A72 cores for actually good performance! Oh and some other manufacturers even made RK3399 based RPi-sized boards!)
I'm really grateful to Alex Ellis for his gist[0] to install kubernetes on RPis. I built my own 4 nodes cluster following this gist. It's been great to get my hands dirty with kubernetes
That being said, I had to buy an Asus Tinkerboard with 2GB RAM to serve as the cluster master because 1GB RAM was a bit short. My master node went below 100MB of free RAM and pod evictions (of kube-system pods) started to happen. If I were to do it again, I think I would use only Tinkerboards just for the convenience of having the same OS/hardware on each node.
That means you'll want to choose a platform that's very well supported by its parent company if you don't want to get stuck on an obsolete kernel in a year.
Hardkernel, who design and produce popular ARM SoCs, chose to produce x86 based platforms in the future for this reason.