So, you have decided to go into Production with Portainer for the management of remote edge devices, congratulations..
But, your security team have decided to scrutinize the configuration, and rightfully so. They are concerned about exposing Portainer's ports to the internet, and have asked what extra levels of protection can be deployed.
Your first thought is firewall ACLs, except that wont work, as your edge devices have dynamic IP addresses, so there is no way of reliably knowing what IP address they will present as... so what else?
mTLS (MutualTLS) is a mechanism that forces both the client and the server to present valid evidence that they are who they say they are, and that that evidence is issued by a trusted source (certificate authority). This is a great way to add protection to any web service that is exposed to the internet, and that only trusted people/devices should be communicating with. But how do you configure mTLS in Portainer? Let me show you.
First up, you need to configure mTLS when you are deploying Portainer, so this is going to require you to redeploy (dont worry, you can reuse your existing Portainer DB) Portainer.
To keep things simple, i will make the Portainer Server the CA for the purpose of issuing TLS certs to the remote devices, but you should use your corporate CA for this.
First step, is to generate a root CA. SSH to the Portainer server to get started.
Enter the command: openssl req -newkey rsa:8192 -nodes -keyout ca.key -x509 -days 365 -out ca.cert -batch
This will generate you two files, ca.cert and ca.key.
Now we need to generate the server cert.
Enter the commands:
openssl genrsa -out server.key 4096
openssl req -new -key server.key -out server.csr -batch
openssl x509 -req -days 365 -in server.csr -CA ca.cert -CAkey ca.key -CAcreateserial -out server.cert -extfile <(printf "subjectAltName=DNS:<YOURFQDNHERE>\nextendedKeyUsage=serverAuth")
Now we need to generate the client cert, which our edge agents will use (either 1:1 or 1:many).
Enter the commands:
openssl genrsa -out client.key 4096
openssl req -new -key client.key -out client.csr -batch -subj "/CN=client"
openssl x509 -req -days 365 -in client.csr -CA ca.cert -CAkey ca.key -CAcreateserial -out client.cert -extfile <(printf "extendedKeyUsage=clientAuth")
OK, you need to copy the ca.cert, client.cert and client.key files locally, using whatever tools you prefer.
Now lets deploy Portainer.
Enter the folowing (to run under Docker):
docker run --name portainer -d -p 8000:8000 -p 9443:9443 --restart=always \
-v /var/run/docker.sock:/var/run/docker.sock \
-v portainer_data:/data \
-v /root:/certs \
portainer/portainer-ee:latest --sslcert /certs/server.cert --sslkey /certs/server.key --sslcacert /certs/ca.cert
Portainer will now start, using the self-signed cert. Also note i am bind mounting /root as that is where i generated the certs into above.
OK, so onto an agent..
We will first, try to use an agent without the mTLS cert, to show what happens.
For the sake of simplicity, we will use the following command to "bulk-onboard" a device.
PORTAINER_EDGE_ID=$(hostname)
docker run -d \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /var/lib/docker/volumes:/var/lib/docker/volumes \
-v /:/host \
-v portainer_agent_data:/data \
--restart always \
-e EDGE=1 \
-e EDGE_ID=$PORTAINER_EDGE_ID \
-e EDGE_KEY=aHR0cHM6Ly8xMzguNjguNi4xMy5uaXAuaW86OTQ0M3wxMzguNjguNi4xMy5uaXAuaW86ODAwMHxkZTphYzplMzo0Mjo3ZTpjMTo4NTo2YzphNjo2Yzo4ODowNjpkZjo4ODphNjo0NHww \
-e EDGE_INSECURE_POLL=1 \
--name portainer_edge_agent \
portainer/agent:2.14.2
and then watch the logs of the agent.. see that it fails to connect..
and in the Portainer instance, nothing shows up in the edge waiting room.
OK, so stop this agent and remove it.. it will never connect.
now, create a directory called /certs and in that directory copy in the 3x certs saved previously.
Ok, and now we can redeploy the agent using a modified version of the deployment script, as follows:
PORTAINER_EDGE_ID=$(hostname)
docker run -d \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /var/lib/docker/volumes:/var/lib/docker/volumes \
-v /:/host \
-v /certs:/certs \
-v portainer_agent_data:/data \
--restart always \
-e EDGE=1 \
-e EDGE_ID=$PORTAINER_EDGE_ID \
-e EDGE_KEY=aHR0cHM6Ly8xMzguNjguNi4xMy5uaXAuaW86OTQ0M3wxMzguNjguNi4xMy5uaXAuaW86ODAwMHxkZTphYzplMzo0Mjo3ZTpjMTo4NTo2YzphNjo2Yzo4ODowNjpkZjo4ODphNjo0NHww \
-e EDGE_INSECURE_POLL=0 \
--name portainer_edge_agent \
portainer/agent:2.14.2 --sslcert /certs/client.cert --sslkey /certs/client.key --sslcacert /certs/ca.cert
look at the logs of the container, see it connected and has been assigned a placeholder endpointID.
and now in Portainer, go into "edge devices" and "waiting room" to see the device is pending.
Associate it, wait a few seconds, and then see the heartbeat goes green.
You now have a Portainer instance, that is connected to the Internet, ready to accept onboarding requests SOLELY from Edge devices that present the CA issued client cert. Any other requests will be rejected.