Kubernetes has become the platform of choice for many enterprises looking to deploy a DevOps platform. Kubernetes is popular because it makes managing and operating applications simple while offering great stability even when dealing with large-scale deployment of hundreds of different services.
Kubernetes is a good tool for managing and running applications, but it is not enough on its own to create a complete DevSecOps platform. You will need several other components, such as infrastructure, application lifecycle management, monitoring, security, and compliance automation.
In this article, we take a look at the essential components needed to make your Kubernetes deployment secure, fast and reliable and answer the question about what is required to build a complete DevSecOps platform on Kubernetes?
What is a DevSecOps Platform?
DevOps's main goal is to produce more reliable software, faster. DevSecOps is an extension of DevOps that incorporates security practice into the development life cycle. When adopting the DevOps approach, we are essentially building a platform where everyone involved in the software can work together to release software.
A DevSecOps Platform comprises toolchains and workflows combined to support the software supply chain from development to production. In other words, the “platform” is the IT business's “backbone” or “core”. These toolchains and workflows play a critical role in enabling self-service capabilities for software engineering organizations.
Toolchains are a set of integrated tools that are used to build, test, and deploy applications. Workflows are the processes that are used to manage these tasks. By providing toolchains and workflows as self-service capabilities, developers easily get the resources they need to be productive. Also, automating the process of testing and deploying changes, ensuring that changes are quickly live into the production environment.
When it comes to implementing a DevSecOps platform, Kubernetes has quickly become the platform of choice for many enterprises. Due to its great flexibility and powerful orchestration capabilities, Kubernetes can support a variety of workloads and tools. Of course, not all platform components are deployed to Kubernetes; in some cases, you will rely on third-party services.
There are many essential components needed for a successful DevSecOps platform. We can divide these components into four categories.
Infrastructure: Compute, DB, Storage, Network, DNS, backup, upgrades, licenses, certificates
Observability: Logs, metrics, trace, analytics, alerts
Application life cycle: CI, CD, Configuration management, secrets Management, artifact management/registry, code quality and gating
Security and compliance: Authentication, Identity, Policies, vulnerability scanning (SAST/DAST)
Let’s deep dive into each category to give you the big picture of the work ahead of you.
What are the essential components needed?
Infrastructure
Assuming we have selected your Kubernetes distribution, we will need to get it ready to host your application. First, Kubernetes requires a lot of computing resources to run the workload. Thus, we must dimension the cluster, meaning selecting which type of VMs to add to your cluster. This activity is known as node sizing and is crucial to optimize your costs.
The next critical part of any Kubernetes installation is the network. We need a good network infrastructure to handle all the traffic, which includes a DNS solution for routing traffic to your cluster. It involves installing components like external-dns to manage DNS configurations as code by adding an annotation to your Ingress and Services. Of course, we will need to set up ingress to enable load balancing. We can use a Cloud provider’s application load-balancer (ALB) solution or install one in the cluster like Nginx ingress or even a combination of both.
With a basic network setup in place, we have a good starting point where applications can connect and communicate with the rest of the world. However, modern production setups use additional tools to make managing microservice communication easier. Including service mesh like Istio to efficiently manage service-to-service communication, API Gateway to securely expose your API to the public like Envoy Gateway, and Oauth2 Proxy to secure your BackOffice and make it only available to staff members.
Great, we have a cluster and networking components in place at that point. Applications most likely need to communicate and connect to some type of storage. It can be a database (Postgres, MongoDB, etc.), an In-memory database (Redis, Couchbase, etc.), or Blob Storage (Amazon S3, Google Cloud Storage, Minio, etc.). For this particular set of services, we have the choice to host them ourselves on Kubernetes or use a cloud provider, but in either case, you will have to set up access and networking for the application to securely manipulate your data.
We are almost done with the infrastructure. The last few points are tightly coupled to the operation of a Kubernetes cluster. Backups are important; we need a good backup solution in case of disaster or data loss. This means evaluating the backup strategy for each tool and considering general strategies like implementing Volume Snapshots for persistent volumes. Upgrades are another important part of a good DevSecOps platform, we need to be able to upgrade the Kubernetes cluster easily and quickly when new features and updates are released or when important security patches are available. Finally, Licences should not be forgotten, they need to be in place for using some Kubernetes Distribution and tools we may install on the clusters.
When building your DevSecOps platform, it's important to start planning and setting up the infrastructure. This includes selecting the right VMs for your cluster size, setting up DNS routing, installing an ingress controller, adding storage, etc. We will also need to set up backups, upgrades, and licenses. With all these components in place, we are not yet ready to deploy your applications onto Kubernetes, we need to put in place the Software delivery life cycle (SDLC), as we will see in the next section.
Application life cycle
Now that we have the infrastructure for your Kubernetes DevSecOps platform, we need to consider the application life cycle. How to release and deploy applications but also how to manage application configurations and secrets.
One of the first things we need to do is decide on the right deployment model for Kubernetes resources. You can use simple Kubernetes manifests, Kustomise or Helm charts. While we can apply changes using CI/CD tools, many companies have adopted the GitOps approach to sync Kubernetes manifest with clusters. GitOps require dedicated continuous deployment (CD) tools like FluxCD or ArgoCD that watch cluster and make sure that all resources defined in the git repository are properly configured.
Once decided on a deployment model and continuous deployment CD tools setup, we need to build and test our application. Tekton is usually a good starting continuous integration (CI) tool. However, many options are available on the market, and it can be hard to decide which is best. In most cases, we would advise considering the solution that your version control provider offers, eg GitHub Actions, Gitlab CI, Azure DevOps or Bitbucket Pipelines. That being said, cloud-native and SCM agnostic solutions like Tekton are very attractive to build a DevSecOps platform; since all configurations are Kubernetes resources, the configuration will be more uniform and easier to learn by developers already familiar with Kubernetes whilst also making the experience of debugging and viewing applications and your pipeline similar and consistent as they are both running in the cluster.
Once the code is built and tested, we need to apply code quality gates before sending applications to production. A code quality gate is a process or tool that checks code for certain criteria. For example, we may want to check that our code follows a coding standard or that all tests have passed. A DevSecOps platform must include tools to perform those checks, including static analysis tools like SonarQube.
After applications are built, tested, and validated via CI pipelines, we need to push them to an artifact repository. In terms of the artifact repository, we have many options. Version control providers GitHub and Gitlab have integrated registries. All major cloud providers AWS, GCP and Azure, offer their solutions, and finally, you can install your own private registry.
Last, secrets management in Kubernetes is critical to keeping your applications and the infrastructure or services they interface with safe and secure. Kubernetes provides a feature to help you securely store and access your secrets, however, native Kubernetes secrets are not so easy to manage in the GitOps context. It is best to use a secret manager from your cloud provider or host your own such as Hashicorp Vault. No matter which secret manager you use sync those secrets as Kubernetes secrets using external-secrets.
Observability
We are halfway through building your complete DevSecOps platform. At this point of the article, we reviewed the infrastructure requirement and added the tooling to build and release our application. The next capability our platform needs is observability, meaning we need to be able to watch what is happening on the platform and respond to eventual problems.
Monitoring is an important part of DevSecOps, and as a result, many providers have specialized in providing Monitoring solutions. In this section, we will ignore such providers and focus on how you could add a monitoring stack on top of Kubernetes.
Applications produce logs to keep records of what is happening. These logs can help troubleshoot problems and understand what is happening in a cluster. While we could use kubectl log command to view the logs for a particular pod or container, this solution is not scalable. Instead, we want to collect, aggregate and display those logs quickly and conveniently. The most popular solution is the ELK stack (Elasticsearch, Logstash and Kibana). Thanks to the Elastic Operator, the ELK stack is easy to install on Kubernetes. This solution is very flexible and powerful.
Metrics are data about the nodes, pods, and services in your cluster. These metrics can help you understand how your applications are performing over time. Some common metrics include CPU usage, memory usage, and disk space usage. By monitoring these metrics, we can identify potential problems with your applications before they cause any downtime. Prometheus is the uncontested solution to collect and store your metrics. However, Prometheus alone has some limitations in terms of how much data it can ingest. Production grad metrics stack will often rely on Thanos to manage a fleet of Prometheus servers. To complete our metrics stack, we can visualize Prometheus metrics via dashboards using Grafana and create metrics base alerts using Alertmanager.
When monitoring applications, we also want to collect traces for all requests made from and to an application. These traces can help you understand how applications are communicating together and follow each request to find the root cause of most problems. Jaeger is a popular solution to collect and store traces and can be used in pairs with Kiali to visualize and search traces.
Finally, we need to be alerted when something goes wrong with our applications. We can use a tool like AlertManager or Elasticsearch to trigger those alerts based on queries respectively on logs or metrics. The monitoring stack ELK + Prometheus is the most extensively used in the industry; all the components are open-source and well-supported by the community.
Security and Compliance
One of the most important aspects of DevSecOps is security. In this last section, we will discuss some ways to secure your DevSecOps platform and your applications.
Good security starts with Authentication. Authentication is the process of verifying the identity of a user or system. We want to leverage Single Sign On (SSO) and Kubernetes RBAC to attribute users the least privilege. Developers should have restricted access to the cluster to prevent security concerns, yet we don’t want to prevent them from debugging and looking at resources in the cluster.
Identity is new to Kubernetes but yet very important, and it can eliminate the use of passwords. Each component in a Kubernetes cluster (node, pod, service, etc.) has an associated identity. This identity can be used to control access to resources. For example, you can use identity-based policies to allow only certain pods to talk to the database or fetch secrets from your Secret management solution. Identity-aware security is still known and often tightly coupled with your cloud provided. OpenID Connect (OIDC) is the popular standard to allow identity-aware permissions.
Security and Compliance Policies are another important aspect of DevSecOps. We can use policies to control what users are allowed to do with our resources. For instance, imposing that securityContext is set and running containers as non-root. OPA Gatekeeper is a popular solution to enforce such policies as it interfaces with Kubernetes Admission Controller and can reject invalid manifests.
Last, new security vulnerabilities are identified every day. So in order to keep up with security threats, our DevSecOps platform needs a vulnerability scanning process. Vulnerability scanning is the process of identifying vulnerabilities in applications and infrastructure. There are two main types of vulnerability scanners: static analysis (SAST) and dynamic analysis (DAST). SAST tools analyze code to identify potential vulnerabilities and will generally be run by CI/CD. DAST tools analyze applications' behavior to identify vulnerabilities installed in a cluster and continuously analyze containers. Trivy Operator is a good DAST solution to consider.
Wrapping up
Building a DevSecOps platform on top of Kubernetes is not something that can be achieved in a day. We first need to configure your environment and have the underlying infrastructure ready to host applications. Then we must choose the right components for releasing, deploying, and monitoring applications. Last, we need to secure applications and infrastructure using security and compliance tools.
While Kubernetes is a powerful platform, there are some challenges that we may face when deploying a DevSecOps platform on it. One of the biggest challenges is knowledge. We may know Kubernetes well, but the process of selecting which tools are best is long and difficult.
This article should give a good idea of the pieces needed to assemble your own platform. If the task seems daunting to you, then it begs the question.
Is building your own platform on Kubernetes the right choice for your organization's needs? It depends on your specific situation. But you may want to opt for a fully managed DevSecOps platform.