Top 5 Kubernetes Security Best Practices

Kubernetes (K8s) is a widely-used, open-source system that automates deployment, scaling, and management of containerized applications. However, due to the high complexity of setting up a Kubernetes cluster and configuring to deploy the applications in it, security is often an afterthought on top of the Kubernetes configuration.

A common misconception is that the cloud is 100% secure by default, however, you have to secure and manage your infrastructure the same way you would with on-premises configurations. In the following article, we will present five best practices to help DevOps engineers properly secure your cloud infrastructure.

Be mindful of how to properly handle users and permissions

First, it is imperative to think about who can access the cluster (authentication) and the permissions they have (authorization), so it is crucial to manage user roles and permissions in Kubernetes and keep privileges as restrictive as possible. This means giving just the required permissions to users only for the K8s resources they need to access.

To manage users and their permissions in K8s, we have Role-Based Access Control (RBAC). These are K8s resources that allow us to regulate access-based roles which enable you to set up specific permissions in a role for specific namespaces. While these roles are attached to users, in K8s there is no actual resource for users since K8s does not manage users natively.

Therefore, no K8s object exists for representing normal user accounts; instead, users are indirectly created in different ways such as importing a list of users into the cluster, or by generating a client certificate for the K8s API server for specific users. The API server handles the authentication of all requests by using one of the methods listed above, after that, you can attach/bind the roles to the specific user.

A cluster role is better suited if the user needs to perform cluster-wide operations across multiple namespaces. Also, if a service needs authorization, a ServiceAccount would be the correct resource for it to get granted permissions which uses tokens to authenticate with the API server. Bear in mind to always apply the rule of least privilege while granting permissions.

Define in detail how pods communicate with each other

By default, each pod can talk to any other pod. However, not every pod needs to talk to all the others so it is important to limit the communication between them by creating network rules in the Kubernetes network layer to determine which pods can talk to which and which pods they can receive traffic from.

For this, we use network policies to define each and every communication rule between all pods. This can prevent an attacker who got ahold of a particular pod from being able to communicate with other pods, therefore limiting the scope of the attack.

Bear in mind that network policies are implemented by a network plugins such as Calico , Weave , etc., and in order to implement configurations on a service level, a service mesh can be used such as Istio which uses proxies to accomplish this task.

Secure secret data

Secrets are a Kubernetes resource used to store sensitive data such as credentials, secret tokens, or private keys. By default, Secrets are not secure, because they are stored unencrypted and only base64 encoded so anyone with permission to view the secrets can read them. One native solution for this issue is to enable encryption using the EncryptionConfiguration resources that K8s provides.

However, you still have to manage the encryption key itself and store it securely somewhere else, for example using 3rd-party tools such as AWS KMS , and Hashicorp Vault among others.

Secrets and all other K8s configuration data are stored in a key-value store called etcd - the K8s backing store for all cluster data. K8s uses this etcd store to keep track and update its configurations and all the resources that make up the cluster and every single update gets saved into the etcd store.

Any change in etcd will lead to changes in the cluster so if an attacker gets access to etcd, they can bypass the API server and make changes directly to the etcd store which will result in K8s resources being updated.

For practical purposes, this is like having unlimited access to the cluster, and in order to mitigate this risk, there are ways to secure the etcd store like putting the etcd behind a firewall and allowing only the API server to access it with only the appropriate authentication. Encrypting the etcd data is also a good idea.

Enforce Image Scanning

K8s security starts with building secure images in the CI/CD pipelines, so always be sure to review what goes inside the image. The libraries, dependencies, and tools that are being used when putting together the application image as code from untrusted registries may include malware, and it is not uncommon that vulnerabilities are found in tools of OS or libraries widely used. One recent example is the log4J exploit (you can read more about it in this blog post on our website).

Therefore, some best practices may include eliminating unnecessary dependencies or libraries, using leaner base images with less unnecessary tools, and taking measures to avoid when an attacker uses a vulnerability in the image to break out of the container and get access to the host (Kubernetes worker node).

If this were to happen, the attacker could theoretically access all the other containers running on that host, read data in the host's volumes, read data from the file system, and read the Kubelet (primary "node agent" that runs on each K8s node) configuration.

To do image scanning, you can use tools such as Snyk or Sysdig that have their own databases of known security vulnerabilities that are updated regularly, or you can add a step in the CI/CD pipeline to build and scan the docker image before pushing it to the registry. These tools can check for vulnerabilities and misconfigurations such as hard-coded secrets.

Also, common registries such as Docker Hub , AWS ECR , or Google Container Registry have features to scan images in the repository itself to check if a newer vulnerability is present in the images hosted in the container so a regular scan of the registry is also recommended.

Never forget to configure security policies

You have set up a good amount of security best practices to protect your infrastructure, but what happens when another developer has to access it? How can you make sure that other developers also apply best practices?

You can define policies to enforce specific configurations and use Policy-as-Code (PoC) to enforce the desired behavior such as preventing pods from running containers with a root user, or defining a network policy for every pod to limit the scope of communication between pods, and so on.

Keep in mind that Security policy resources in K8s are implemented by third-party tools such as Open Policy Agent or Kyevrno . You would create the security policy definitions in K8s with all the rules and hook the security policies into the K8s admission controller component, which decides whether the deployment can go through. This is based on the policies previously defined, so it works as a kind of gatekeeper that decides if a policy has been violated or not. This way you can have automated validations for various security configurations.

Kubernetes has become one of the most popular solutions for container orchestration. Despite its popularity, there is hesitation to implement the solution due to the steep learning curve that it requires, and because of the possible security threats that come with a less than ideal configuration.

If you work with containers, one of your biggest concerns with implementing Kubernetes is likely security, but if you follow the five security practices addressed in this article, you will be on the right path towards a secure Kubernetes implementation. Contact us today at growth@stratusgrid.com if you would like to discuss this concept further!