DevSecOps in Kubernetes

Traditional security processes can often become a roadblock when delivering software via DevOps processes at the rate that today’s business world demands. Today, security is not just the responsibility of the security teams—it is a shared responsibility among all the teams in the applications lifecycle. This integration is known as DevSecOps. DevSecOps is not about slowing progress or sacrificing agility gained through DevOps. Rather, DevSecOps promotes a collaborative approach to security so that it becomes a central part of the application lifecycle.

At the same time, the modernization of infrastructure and applications is driving the rapid growth of containers and container orchestration platforms as a part of DevOps. The most popular orchestration platform for running containers is Kubernetes. Results from the RightScale 2019 State of the Cloud Report show that “Container use is up, and Kubernetes use is skyrocketing. The use of Docker containers continues to grow, with adoption increasing to 57 percent from 49 percent in 2018. Kubernetes achieving even faster growth, increasing from 27 percent to 48 percent adoption.”

As companies start to adopt Kubernetes, it’s critical to incorporate DevSecOps best practices, especially because container applications have multiple layers of abstraction. With container technology, you have an orchestration platform, host OS, container runtime, container images, container registry, and the running containers—all of which need to be secured in order to avoid risks like deficient authorization/authentication, bugs, and misconfiguration.

Let’s look at the risk areas of containers and Kubernetes along with the DevSecOps best practices that help mitigate them.

Securing container images and the container registries

First, let’s address the two most common security risks for containerization: the container images themselves and the container registries.

Securing container images

Risks for container images include clear-text secrets, embedded malware, insecure software or libraries, bugs, outdated/stale images, and the use of untrusted images. There are several ways you can secure images:

• Use an enterprise container security solution like Twistlock, Sysdig Secure, or Aqua, which can scan your images for vulnerabilities. The scanning can be done in the CI/CD pipeline or on existing containers in a registry.
• Instead of using clear-text secrets in container images, use Kubernetes Secrets to store secrets or Key Vault.
• Only use images that are verified via your scanning process.
• Container images are typically stored in container registries. Use a private trusted container registry like Azure Container Registry (ACR).
• Digitally sign your container images and set your orchestration platform, e.g., Azure Kubernetes Service (AKS), to only allow validated images.
• Base images are often updated with new software and security fixes. Every time base images are updated, use automation to build new container images downstream. The process for image updating can be handled by tools like Jenkins or Azure Pipelines (an Azure DevOps service) in continuous integration and continuous deployment (CI/CD).

 

Securing container registries

The second most common security risks in containerization are threats to container registries. These risks include insecure connections to registries, deficient authorization/authentication restrictions, and outdated/stale images. To boost security, you can leverage group isolation through nested namespaces, segmenting image access to specific teams in your organization.

Another isolation technique is to deploy the registry in a dedicated virtual network (VNet). In a dedicated VNet, such as Azure VNet, only users and resources in that VNet can access the registry.

As an example, let’s take a closer look at what Azure Container Registry (ACR) offers. ACR has two authentication methods: individual identity authentication (IIA) and headless/service identity authentication. IIA has an az acr login (via Docker CLI) used by developers to pull and push registry images. Headless authentication uses a service principle used in CI/CD pipelines. Limit the use of individual identity authentication to a small group of users and follow standard user account best practices.

By granting trusted image push permission to select users, you can limit who can push images to your container registry. Again, use a DevOps capability such as Azure DevOps to keep container images up to date. And finally, enable content trust on your container registry so you can enforce using only signed container images.

Authentication and authorization

Authentication and authorization are at the core of securing cloud and DevOps. Authentication is confirming the identity of a user and authorization is the act of verifying and allowing access to resources. Think of these two as AuthN vs AuthZ.

Securing authentication and authorization is important to your container registry and other containerization components like the cluster and CI/CD pipeline. Azure Active Directory (AAD) helps centralize identity and access management to services running in Azure. AAD comes with many features that help with security, including privileged identity management (PIM), multi-factor authentication, service principal accounts, conditional access, and more. AAD extends authentication and access controls with the container registry, the cluster, and even a pipeline in Azure DevOps.

Here are some tips for securing authentication and authorization:

• Key Vault is a service on Azure used for storing secrets such as passwords, certificates, user credentials, and more. Key Vault can be used to store credentials and place a Key Vault string in code, Docker images, or in a CI/CD pipeline vs. the actual user credentials.
• First AAD should be integrated with AKS clusters, next bind AAD users or groups to roles in your Kubernetes cluster. These roles are then used to define permissions to resources in your cluster.
• Enable multi-factor authentication on all accounts that will be accessing container registries, AKS clusters, or Azure DevOps pipelines.
• Utilize AAD user accounts and service principal accounts for accessing your Azure container registries.
• In addition, or in replacement of human accounts, service accounts are commonly used as a best practice for authentication and authorization to resources in DevSecOps. I am referring to service accounts used in automations and in CI/CD pipelines when accessing resources. For example, you may have a service account that is a part of your release pipeline that has access to the lower and production environments that is used when new code is pushed out.
• Use conditional access policies (CAPs) for access to resources within Azure DevOps. For example, a CAP can require that devices be on a corporate network to gain access to resources.

Policy control for pipelines and cluster

Azure Kubernetes Service also enables you to limit access for not only the resources and people, but also the content of the incoming requests. Operators can select from a predefined set of Azure policies and assign them to AKS clusters. This enables deploy-time enforcement of organizational policies. For example, any resources that are currently deployed to AKS can be checked for compliance with the assigned policies and if found to be non-compliant, the violations are reported so appropriate action can be taken to rectify. Similarly, if a user tries to create resources that are not compliant with policy, his or her requests are denied and the user is presented with a message detailing the policy that caused the denial.

Alerting and monitoring

As a part of security, it is important to perform monitoring and have alerting around the state of security in DevOps and cloud. Alerting and monitoring in DevSecOps should include the following: unified pane of glass of security across the DevOps stages, visibility and ability to control the status for enterprise application resources in Azure subscriptions, and actionable alerts for security drift.

• Microsoft provides AKS cluster monitoring via Azure Monitor for containers. This feature gives you monitoring insights into AKS and containers, cluster and container logs, as well as metric-based cluster and pod performance charts.
• Security Center not only has the ability to show security posture across Azure subscriptions, but also can alert you about security that is out of compliance and can assist with remediation of security issues.
• Azure Monitor consists of Log Analytics (infrastructure monitoring) and Application Insights (application performance monitoring). Azure Monitor can assist in the overall visibility and storage of security assessments and issues in Azure subscriptions, as well as advanced alerting and notification when security issues do arise.

Summary

With the changing landscape of modern technologies—including containers, Kubernetes, and DevOps—it’s important to keep security top-of-mind and to shift security left by adopting a DevSecOps mindset. As you embark on your container and Kubernetes journey with AKS, refer to this blog post for things you can do to secure your Kubernetes clusters and container images.

For more information on DevSecOps with Microsoft and AKS, check out the following resources:

• DevSecOps Azure webinar “Help Deliver Applications Securely with DevSecOps” with Brendan Burns, co-creator of Kubernetes
• azsk/DevOpsKit-docs on GitHub
• Secure DevOps Kit for Azure