k3s is a “Lightweight Kubernetes”, that’s “built for IoT”. Apparently IoT is no longer about devices with kilobytes of RAM, because there’s no way you can run the whole kubernetes stack that thin.
People kept asking about k3s not being lightweight for a while. The official profiling page sums up the k3s requirements around 1.6Gb RAM and 6% of a CPU core. For tiny VMs with a whooping 2Gb RAM total, k3s alone will eat most of it, so it sounds that the least practical setup is 4Gb RAM nodes. Is that actually so?
Go is aware of the max allocatable memory, so it was always possible to limit k3s’s RAM usage with e.g.
[Service]
MemoryMax=1G
That, though, is a hard limit. If Go didn’t run the GC in time and had to spike over it, the kernel would come and kill k3s. This would cause the control plane downtime, and elevated CPU use for cold restart.
There’s a better mechanism in place now! In Go 1.19, you can set the soft memory limit with a GOMEMLIMIT environment variable. WIth that, Go will run the GC when it gets close to the threshold, but if it shoots over it, the process still runs. If there’s nothing to GC, it still works.
I’ve set my k3s’s GOMEMLIMIT to 800MiB, and it’s an extremely well-behaved citizen, mostly sitting around 750Mb (this particular node is very light on the API calls). Before the change, it was about 1.2Gb. Paired with cilium, promtail, and vmagent, that leaves a bit over a single gigabyte for the workload, which is just enough for my use case. No need for 4Gb RAM nodes!
If you find yourself in a spot where you want your k3s clusters to use less RAM, take a look at GOMEMLIMIT. it’s not a silver bullet, you’re trading CPU for RAM, and there’s only as much it can do, but sometimes it helps to even out the CPU/RAM imbalance.
Be sure to check the guide to Go’s garbage collector for more details and interactive graphs (they are great fun).