As a technical guy, regardless of where I am in my career, I love to keep my hands on the tools... its what makes me able to run technical workshops, talk with engineers, and question statements made that are baseless in fact... call it my unfair advantage.
Well, here at Portainer, I also like to keep my fingers on the pulse of the tech, and so have spent the last couple of weeks rebuilding my homelab (yes, I have one). This is not your regular home setup, where you might run some PLEX or other home automation software, no, this is a homelab for me to keep my knowledge of the market sharp.
What's running in my HomeLab might you ask? A bit of everything..
The hardware I chose to run this on is an old HP Z440 Professional Workstation. It comes with a 28 Core Xeon CPU, 64GB RAM, and a 1TB SSD. I upgraded it with a 2TB NVMe disk, and have just ordered an extra 64GB RAM (because, well OpenShift uses a LOT of resources).
On this "server" I run Proxmox VE v8, which is a simply brilliant piece of open-source software (and I bought the subscription). This Proxmox environment comfortably allows me to run all my VM workloads.. well, once I added the NVMe disk anyway.
I connect to the environment remotely with a really simple OpenVPN Sever (which runs in a container - https://dockovpn.io/) and I use PiHole as a local DNS server + Network privacy shield (for my home network). Other than that, the setup is pretty simple. I will be putting in my favorite reverse proxy, nginxproxymanager. Oh, and I run Seafile for a self-hosted "dropbox" alternate.
So, now that I have this environment, what was the intent of it?
The container space is ever-evolving. Newcomers are arriving into the space, products that flew under the radar are getting "discovered", and even for more mature products, they keep enhancing their offering.
As Portainer (the product) does not include a container runtime, nor a Kubernetes distribution, we are ALWAYS acting as a management overlay for other products. Now, in all honesty, the runtime and distro don't matter to us, as we talk to the native APIs, and as long as the distros are certified, we should have no compatibility issues. Well, that's the theory. In reality, every single distro aims to offer a "value add" over and above the standard Kubernetes capability, be that applied defaults, or just simple configurations.
I am using this environment to 1) learn how to deploy the latest versions of the container runtimes and Kubernetes distros, and 2) make sure that Portainer does actually work well with these distributions (yes, our QA team validates a subset, but I like to experiment myself, chaos engineering style).
A couple of learnings so far
Talos Kubernetes is awesome. I like the fact you can completely deploy and manage the environment from a central location, OS, and Kubernetes, without needing to SSH to the environment at all (well, you can't anyway). It's a pretty lean distro too, only needing 2-3GB of RAM per Kubernetes node. I really like that its "secure by default" too.
OpenShift is VERY involved to deploy and configure, and its a resource-consuming beast. 14GB RAM and 4 CPUs on the control-plane nodes.. just to idle (and 70 "system" namespaces once its up!!). Sure, it's probably got the widest adoption in the industry (as everyone in enterprises buys from Gartner "top right"), and RedHat is known for its awesome support, but wow. Bring your chequebook :). If you are using Portainer to manage OpenShift, then you don't need Openshift ACM (part of the Openshift Platform+ bundle). There is a lot of overlap there.
Rancher hasn't really changed in years.. and I guess, why does it need to? Getting a RKE cluster up and running was pretty easy (would be easier if they had a native Proxmox driver like they do with vSphere). Again, quite a resource-heavy distro, but a fraction of OpenShift. Rancher remains a really good way to spin up and lifecycle Kubernetes Clusters.. however, I prefer Talos.
Docker Swarm feels "old" now, but my goodness is it easy to deploy and use.. I still love it, and remain sad that it never really got to see the light of day at global scale.
K3s and MicroK8s are both pretty good for lightweight distro's, but both really need 1GB of RAM to idle, so 2GB RAM should be considered the absolute minimum to use them for any productive workload. Docker on the other hand uses almost nothing.. 340MB of RAM idle (incl OS). For the far edge, where resources are constrained, Docker (or Podman) still reigns supreme.
What else will this be used for?
I want to start learning more Kubernetes capabilities, as Portainer continues to expand its offering, I need to keep across things. OpenTelemetry, Gateway API, and good old OPA Gatekeeper are the things I have on my agenda to play with over the coming weeks.
I also want to continue my testing to ensure that Portainer central auth / RBAC works flawlessly across all these distro's.
I want to document any learnings as "reference architectures", and generally, I want to be technically able to help answer community tech questions when I can.
This is just the start of my HomeLab v2 Journey... but it's going to be an ongoing project for me... the nerd inside me loves it.
Neil
COMMENTS