Did you know, Docker and Kubernetes are, by default, insecure? It's a confrontational statement i know.. but let me explain what I mean by asking a few simple questions:
- How many of you have allowed your users to access your production Docker environment directly, either by providing TLS keys for Docker API remote access, or by letting them SSH onto the hosts directly?
- How many of you out there have deployed Kubernetes, and think it's "just easier" to share the KubeConfig file created during the deployment (cluster-admin) with your users?
- How many of you are aware of the inherent insecurity of containers and allow users to bind mount local directories, or run a container in privileged mode, or run containers as host PIDs, and what about letting them mount host devices into containers? Are you aware that by allowing these things to happen the containers your users deploy could give them access to the underlying host and all other containers?
- And in Portainer, how many of you have defined all your users as 'Admins'? how many of you use Admin accounts for day to day use? How many are still running with our "Internal Authentication" and allowing users to have simple passwords? how many of you have increased the session timeout so as to not prompt users to login daily?
The inconvenient truth is, 'too many. Engineering leaders are forced to make a conscious decision: do I focus on ease-of-use for my team, or do I focus on security? The reality is that in the majority of cases, ease-of-use wins out. There is, however, often a middle ground.
In this blog, i will quickly describe some of the issues that the above questions spring from, and why you really SHOULDNT do/allow any of these for your users.
Docker
Docker has no notion, whatsoever, of users or roles. With Docker (and Docker Swarm) you either have access to the environment, or you don't. If you have access, you have access to EVERYTHING. You can see every container, every volume, you can stop things, you can delete things, you can even purge (delete) all stopped containers and unused images. Letting all of your users have this level of access in a shared platform (here we are assuming a production environment, not private/local docker instances) is seriously dangerous and borderline negligent. The actions of one user could negatively impact the services managed by another, and that is simply unacceptable. Sure, you can work around this by having multiple clusters and segment your users to their own cluster, but that is hugely inefficient.
Kubernetes
Kubernetes, by default, is not configured with any external authentication, so also has no notion of users. To add users to Kubernetes either means adding them locally per cluster, or by installing an authentication plugin to synchronise user access from 3rd party sources. In addition, once you start adding users, you need to also define what those users can do, which is done with roles and role bindings and is really hard. It's all too common for IT to simply give out the primary cluster admin credentials in order to save the time/complexity associated with configuring users and roles. This, like the above with Docker, is insane.
Privileged Actions
Not many people know, but it would take only take a few seconds to completely compromise a Docker or Kubernetes environment that is not configured with the appropriate security policies. If all your users are Admins, these controls are impossible to enforce, so before you can even consider these risks, you must first address the first two.
If I wanted to gain access to a Container host, I simply need to deploy a container, bind mount the hosts / as /host in the container, and I'm off and running. I have access to the hosts root FS and can access anything and everything. Same goes with allowing users to deploy containers as host PIDs, when you do this, the container can access (and kill) all other processes running on the host, thereby destroying any process isolation. Same with allowing access to devices, there is nothing stopping a user from mapping the device /dev/sd1 into their container, and running fdisk and removing all partitions on that disk. These activities, whilst requiring malicious intent to be dangerous, should be blocked at all times, and only allowed for a specific reason, and under specific approvals.
Portainer Configuration
Portainer is designed to be used from the most simplistic home-lab deployment right through to large enterprise environments, so we leave a lot of configuration options up to the Portainer Admin to set up. If you, as the IT Admin, chose NOT to secure Portainer adequately you will get a platform that allows your users to do things they shouldn’t, either maliciously, or through an accident. That's bad.
So, what should you do?
Portainer has the notion of users, and users can be defined either as 'Admin' or 'Regular Users'. Admins have access to everything, deployed everywhere, and can do anything. Regular users have access only to environments they are authorised to manage, to create and manage deployments they create, or that are shared with them by other users.
As we have determined, you don't want everyone having access to everything, and as such, you really shouldn't have more than one or two Admins. Remember, if all users are Admins, then NONE of the Portainer security controls to limit privileged actions or manage access take effect.
When hackers are trying to gain access to a system, they often look for a way in as "Admin" as this makes their life so much simpler. Having too many users assigned the admin role is a recipe for disaster. Even admins should not login with their admin account on a day to day basis.. Here is a great article (external) that explains why this is important.
Please, please, please don't make all users Admins
If the Portainer CE Admin/User split is not granular enough for you, then we have RBAC capability in Portainer Business (five nodes free or paid) which allows you to have granular access control to who can do what.
For example, you may want to let some of your users have complete control over just one cluster, but not any others (we call this our "endpoint admin" role), or you may want some of your users to be able to see EVERYTHING across a cluster, but be read-only so as not be able to change anything (we call this our "helpdesk" role), maybe you want some users to be able to start/stop running containers, but not be able to create new or delete (thats our "operator" role).
These roles should not be seen as adding complexity, rather they should be seen as a way to correctly assign "principle of least permission" to your environment, because in all honesty, if everyone can do anything, how can you track/audit who does what?
Portainer has its own internal authentication, but its very limited and only designed for non-critical use cases. In critical/business environments, where you need password policies, lockout policies, or to enable multi-factor authentication, you should configure our external authentication, either against LDAP/AD, or by using our OAuth capability to link to your corporate directory. And yes, Portainer CE does let you configure OAuth to ANY provider, and no, we are not "forcing" you to buy Portainer Business for this feature. The "custom" OAuth provider lets you configure against anything. Portainer Business OAuth includes a "click to configure" UI element that makes configuring OAUTH a 2 minute task.
One other thing around authentication, we made a contentious decision to remove a feature we had (no-auth) which as its name implied, allowed you to run Portainer with NO authentication. We did so because many people abused this, and were therefore vulnerable to external compromise. In making this change, we added a new feature to help ease the transition, and that’s a "session lifetime", which defaults to 8 hours. This setting tells Portainer how long a logged in session can remain logged in; some users wanted to make this 1 year, and so we allowed this to be set by the admin, but in all honesty, for a control system like Portainer, we really do recommend against this, and ideally you have your system auto-logout every day (the default).
Always set the Portainer "per host/cluster" security controls (below), so you can remove privileged actions from your users. Unless there is a specific reason for them to use this capabilities, they should be removed.
Its crucial for IT governance and security to know who does what in Portainer, and that’s why, in Portainer Business, we added authentication and activity logs. The logging function records, in a read only journal, who logs in (and who tries but fails) to Portainer, and what activities they perform whilst logged in. This is crucial for IT governance, so for those of you that need this feature, give the Portainer Business free trial a go.
I hope this helps to clarify:
- why you need to secure your environment,
- the danger of all users having "admin" rights, and
- why you should use the security settings we provide in Portainer
Happy containering.
Neil
COMMENTS