When tasked with deploying applications across many thousands of remote Kubernetes clusters connected to the internet, traditional GitOps models encounter challenges that must be carefully navigated. This post explores the ArgoCD approach (including its experimental efforts like the argocd-agent) before contrasting it with how Portainer achieves similar objectives through the native Edge Agent and Edge Stacks. The goal is to provide you with a balanced view of the available strategies, letting you decide which path best fits your operational needs.
The Argo Way
ArgoCD was designed for centralized enterprise environments. Its control plane, typically deployed within a secure data center or cloud environment, continuously monitors a Git repository for the desired application state. It then communicates directly with each Kubernetes cluster’s API server, enforcing consistency across your infrastructure. This approach works seamlessly in scenarios where clusters are part of a private, routable network, eg, Remote/Branch Offices.
However, edge deployments introduce significant challenges. When your clusters are connected to the internet but operate behind CGNAT (with no ability to reliably use port forwarding) the central ArgoCD server cannot reach them directly. Two methods exist to address this limitation.
The first solution is to deploy a dedicated ArgoCD instance on every remote cluster. In this decentralized model, each edge node independently pulls its desired state directly from Git. This approach sidesteps connectivity issues by removing the need for a direct connection from a central control plane. Yet, managing tens of thousands of independent ArgoCD instances introduces substantial operational complexity. You must ensure consistency across versions, enforce uniform security policies, and consolidate monitoring and troubleshooting efforts; a logistical challenge at scale.
The second approach retains a centralized ArgoCD control plane but augments it with networking solutions such as VPNs or API proxies. Each edge cluster runs a lightweight client that establishes an outbound reverse tunnel (e.g., WireGuard or Tailscale) to the central server, effectively simulating direct connectivity. Alternatively, a Kubernetes API proxy could be deployed, with this proxy itself having an agent that runs within each remote cluster, securely exposing the Kubernetes API endpoint via an external identity (eg Sidero Omni, Liqo, InletsPro, Teleport, or even Portainer itself). This preserves a unified GitOps workflow but forces the deployment (and maintenance) of this networking solution capable of reliably handling thousands of secure tunnels amidst intermittent connectivity and potential latency challenges.
The Experimental ArgoCD Agent
For those hoping for an “agent” that natively addresses reverse connectivity challenges, the argocd-agent repository under the argoproj-labs organization might seem promising. This experimental component aims to adapt ArgoCD for outside-in deployments by providing an agent-based approach that could potentially streamline connectivity for CGNAT-bound clusters. However, it’s important to note that the repository has been relatively stagnant for some time and has not yet been integrated into the core ArgoCD offering. While the concept is intriguing, it remains an unproven and unsupported solution for critical, large-scale edge deployments.
The Portainer Way
Portainer approaches edge management from a fundamentally different perspective. Rather than retrofitting a GitOps tool built for direct connectivity, Portainer is purpose-built to handle the constraints of edge environments. With Portainer, every remote cluster is equipped with a lightweight Edge Agent that automatically establishes a secure reverse tunnel (on demand) to a central Portainer instance in your network. This reverse tunnel is initiated from within the edge environment, effectively overcoming the limitations imposed by CGNAT and negating the need for port forwarding. The Portainer agent "checks in" regularly (at a configurable interval) with the central instance. If management actions are pending for it (deploying a new version of the app due to a change in Git), then the tunnel is established, the instructions are retrieved, and the update is performed.
Portainer further simplifies operations with its Edge Stacks. These allow you to define and deploy application configurations centrally. Once declared, these configurations are seamlessly pushed over the secure tunnels to all connected edge clusters within a defined edge group (a logical grouping of edge environments), providing a unified management experience without requiring custom VPN configurations or extensive proxy setups.
Drawing Your Own Conclusions
Both approaches offer viable solutions for managing thousands of remote clusters, but they come with distinct trade-offs. With ArgoCD, you benefit from a robust GitOps engine that excels in centralized environments. However, adapting it to handle CGNAT scenarios forces you to either decentralize by deploying an instance on every edge node or engineer a reverse-tunnel solution via VPNs or proxies. The experimental argocd-agent hints at a native solution, yet its stagnant development means relying on it for production-scale edge deployments carries risks.
In contrast, Portainer’s integrated Edge Agent and Edge Stacks provide a turnkey solution designed specifically for the distributed realities of internet-connected clusters behind CGNAT. The connectivity challenge is addressed natively, reducing the need for custom network engineering while delivering a unified, centrally managed experience.
Ultimately, the choice between these approaches depends on your team’s expertise, your willingness to invest in custom engineering (and the ongoing costs of such), and your environment's specific operational demands. Whether you lean toward adapting ArgoCD’s GitOps-centric model or embrace Portainer’s edge-native design, both paths offer routes to achieving centralized management in even the most challenging, distributed infrastructures.
The decision is yours to make.
COMMENTS