Background
Companies care about delivering their products to customers as quickly as possible, with as little cost as possible, so they prefer to choose infrastructure tools that can help achieve these goals. While this is possible when using Kubernetes clusters, it’s common to face resource allocation and security challenges. In this article, we will look at how Kubernetes clusters and, most importantly, how Kubernetes multi-tenancy can help address some of these challenges.
What is multi-tenancy
Multi-tenancy at its core allows tenants (development teams, applications, customers, or projects) to share a single instance of an application, software, or compute resources. Shared resources include CPU, memory, networking, control plane, and others. Having different tenants access different resources or services is not financially scalable, so sharing saves cost, and simplifies administration and operational overhead for most companies.
Multi-tenancy in Kubernetes
Kubernetes inherently implements multi-tenancy by providing constructs to ensure workloads run well side-by-side, and access resources as necessary. A cluster admin looking to achieve this can have a cluster controlled by a logical main node with multiple tenants running one or more workloads in this cluster. Also, it should isolate tenants from accessing each other's workloads to minimize the harm a compromised or malicious tenant can do to the cluster and other tenants.
Multi-tenancy cluster setup by admin
Cluster admins can use the RBAC approach to achieve multi-tenancy, as illustrated in the image below
Image credit Yogesh Dev
Role-Based Access Controls (RBAC)
To set up RBAC per tenant, we need to create a namespace per tenant. The cluster's worker node resources are shared among the namespaces, but the namespace provides an isolated environment for tenants to run workloads. We will later create a service account, a secret for the service account, a role, and role binding for the tenant. Furthermore, we will update the kubeconfig file to access the tenant's Kubernetes resources.
In this exercise, we will use minikube to set up a single node cluster:
1. Set up a cluster
minikube start
2. Create a namespace
kubectl create ns developers
3. Create a service account
kubectl create sa developers-user --namespace developers
4. Create a role
##role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: developers
name: developers-user-full-access
rules:
- apiGroups: ["", "extensions", "apps"]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["batch"]
resources:
- jobs
- cronjobs
verbs: ["*"] kubectl apply -f role.yaml
5. Create role binding
##rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: developers-user-view
namespace: developers
subjects:
- kind: ServiceAccount
name: developers-user
apiGroup: ""
roleRef:
kind: Role
name: developers-user-full-access
apiGroup: rbac.authorization.k8s.iokubectl apply -f rolebinding.yaml
6. Create a secret for the service account
Create a secret for the service account created in Step 3
##secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: developers-user-secret
namespace: developers
annotations:
kubernetes.io/service-account.name: developers-user
type: kubernetes.io/service-account-tokenkubectl apply -f secret.yaml
7. Extract ca.crt from secret
kubectl get secret --namespace developers developers-user-secret -o json | jq -r '.data["ca.crt"]' | base64 -d > "./ca.crt"
8. Get user token from secret
TOKEN=$(echo `kubectl get secret --namespace developers developers-user-secret -o json | jq -r '.data["token"]' | base64 -d`) && export TOKEN
9. Get current context
CURRENT_CONTEXT=$(echo `kubectl config current-context`) && export CURRENT_CONTEXT
10. Get cluster name
CLUSTER_NAME=$(`echo kubectl config get-contexts $CURRENT_CONTEXT -o name`) && export CLUSTER_NAME
11. Get cluster endpoint
CLUSTER_ENDPOINT=$(echo `kubectl config view -o jsonpath="{.clusters[?(@.name == \""${CLUSTER_NAME}"\")].cluster.server}"`) && export CLUSTER_ENDPOINT
12. Setting cluster entry in kubeconfig
kubectl config set-cluster developers-user-developers-minikube --server="${CLUSTER_ENDPOINT}" --certificate-authority="./ca.crt" --embed-certs=true
13. Setting token credentials entry in kubeconfig
Token from step 8
kubectl config set-credentials developers-user-developers-minikube --token="${TOKEN}"
kubectl config set-context developers-user-developers-minikube --cluster=minikube --user=developers-user-developers-minikube --namespace=developers
Now let’s change the context to developers-user-developers-minikube and try to get pods inside the developers' namespace and afterward in all namespaces.
kubectl config use-context developers-user-developers-minikube
kubectl get pods
kubectl get pods -A
Obviously, the tenant is not able to access resources in the other namespace
Quota enforcement
Cluster admins can enforce resource quota in a particular namespace is important to pre-define the allocation of resources for each tenant to avoid unintended resource contention and depletion. ResourceQuota are Kubernetes objects that enable cluster admins to restrict cluster tenants’ resource usage in a certain namespace. The Kubernetes quota admission controller will watch for new objects created in that namespace, monitor and track resource usage.
Create and specify CPU resource quota on a namespace with requests and limits.
##resourcequota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: developers-cpu-quota
namespace: developers
spec:
hard:
requests.cpu: "1"
limits.cpu: "2"kubectl apply -f resourcequota.yaml -n=developers
When applied by the cluster admin, the sum of CPU requests will not exceed 1 core and the sum of CPU limits will not exceed 2 cores for all pods within developers namespace.
Network policies to monitor tenant communications
By default, network communication between namespaces in most Kubernetes deployments is permitted. The cluster admin will need to configure some network policies to add isolation to each namespace. This will also help to support multiple tenants.
Example Network Policy file that will prevent traffic from external namespaces:
##networkpolicy.yaml
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: developers-block-external-namespace-traffic
spec:
podSelector:
matchLabels:
ingress:
- from:
- podSelector: {}Apply the above with the command below
kubectl apply -f networkpolicy.yaml -n=developers
Cost optimization for Kubernetes multi-cluster
As the workloads from tenants or teams scale, it becomes important to optimize the costs of running your Kubernetes cluster. Putting in place the right cost monitoring and optimization setup ensures that cluster admins productively utilize available compute, memory, pods, namespaces, services, controllers, and other resources.
Ensure the correct number of worker nodes: Keeping the correct number of nodes running in a cluster is critical for cost optimization. Therefore, ensuring your team is running the right size, number, and type of node in your Kubernetes cluster is important.
Configure auto-scaling: With Horizontal Pod Autoscaler (HPA), we can dynamically adjust the number of pods in the cluster to handle current workload requirements and the number of pods in deployment to meet tenants’ demand.
Maintaining pod sizes: We can use resource requests and limits to avail enough resources for optimal performance, and avoid wastage. This can help keep our costs in check. Observe pod usage and performance of workloads over time to consider scaling your pods through requests and limits.
Web UI dashboard with limited view
Kubernetes provides a basic general-purpose web UI where users or tenants can view, deploy containerized applications and interact with the cluster. It can be used to troubleshoot or manage existing resources.
The interface shows a high-level overview of applications running on the cluster, and provides information on the current state of the cluster in real-time as well as all the objects that compose it, being able to interact with them so a service can be scaled or a pod can be restarted.
To set up, wen can use the following command as the cluster admin :
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
Accessing the dashboard from a local workstation, we then create a secure channel to our Kubernetes cluster by running the following command:
kubectl proxy
Let's access the dashboard at:
This view provides an interface with token authentication and a dashboard where we have all the main elements of Kubernetes:
Using an authentication token (RBAC)
To log in to the Dashboard, we will be using the bearer token generated for tenants in the developers' namespace in step 8.
Tenants in the developers' namespace cannot view resources in the default namespace as seen in the notification. Deploy a simple hello-node app in the developers' namespace to see some resources on the dashboard
kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.39 -- /agnhost netexec --http-port=8080
Switch from default to developer's namespace by changing the query namespace from default to developers.
The tenant can now view resources in the developers' namespaces.
Multi-tenant cluster management using Argo CD(RBAC configuration for application GitOps)
Apart from providing a great experience of automating day-to-day tasks required to manage and monitor the GitOps continuous delivery tool for Kubernetes, Argo CD also provides the feature to enforce necessary security boundaries with tenants that would like to use Gitops to deploy applications into a Kubernetes cluster to test and verify new features.
Install Argo CD
As the cluster admin, we can install the Argo CD GitOPs instance in the cluster by running the following command in the default namespace.
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
The above command will create a namespace for the Argo CD services and applications to reside. Download the latest Argo CD CLI version from https://github.com/argoproj/argo-cd/releases/latest
Access the Argo CD API server
By default, the Argo CD API server is not exposed to an external IP. For the purposes of this article, we are going to utilize kubectl port-forwarding to connect to the API server without actually exposing the service.
kubectl port-forward svc/argocd-server -n argocd 8080:443
This will make things easier for us as we can now access the API server using localhost:8080.
Note: this method is only for local development or demonstration purposes.
The initial password for the admin account is auto-generated and stored as clear text in the field password in a secret named argocd-initial-admin-secret in your Argo CD installation namespace. You can simply retrieve this password using kubectl:
ADMIN_PASSWORD=$( echo `kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d`) && export ADMIN_PASSWORD
Users and roles in Argo CD
As the cluster admin, we can create new users for different tenants, so we can distribute the GitOps instance with the different tenants and limited permissions. We are going to edit the RBAC ConfigMap directly in this example
kubectl edit configmap argocd-cm -n argocd
Add the data section
data:
accounts.jeff: apiKey,login
…The user, jeff, has two capabilities:
1. apiKey - allows generating authentication tokens for API access
2. login - allows to login using UI
The user has no password set.
To create a password, the cluster admin must use the admin’s password with the following command
argocd account update-password --account jeff --new-password jeff1234 --current-password "${ADMIN_PASSWORD}"
The user can now log in
By default Argo CD has two built-in roles, all new users are using the policy.default role:readonly from the argocd-rbac-cm ConfigMap, which can not create resources or modify Argo CD settings.
The RBAC config map will need to be updated to allow jeff to perform some operations. The cluster admin will run the following to pull up the config map once again:
kubectl get configmap argocd-rbac-cm -n argocd -o yaml > argocd-rbac.yml
Now add the following
data:
policy.csv: |
p, role:developer, applications, *, */*, allow
p, role:developer, clusters, get, *, allow
p, role:developer, repositories, get, *, allow
p, role:developer, repositories, create, *, allow
p, role:developer, repositories, update, *, allow
p, role:developer, repositories, delete, *, allow
g, jeff, role:developer
policy.default: role:''Now apply the RBAC config map by running the following:
kubectl apply -f argocd-rbac.yml
Jeff was able to create an app in the default namespace, but what if the cluster admin wants to allow namespace actions? RBAC in the ConfigMap will not allow this. It is good to separate user access, so the one team/tenant will not affect team/tenants resources. With Argo CD Projects we can achieve this
Projects
Argo CD provides logical application grouping and helps to separate tenants from each other in the multi-tenant Argo CD instance. Cluster Admins can create one project per tenant and use project settings to specify where and what tenants can run their workloads, specifying access for namespaces, repositories, clusters, and so on. And then, we will be able to limit every developer team by their own namespaces only.
The cluster admin creates a project to provide limited access to a tenant:
kubectl apply -n argocd -f - << EOF
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: developers-tenants
spec:
# Deny all cluster-scoped resources from being created.
clusterResourceWhitelist:
- group: '*'
kind: '*'
# Only permit applications to deploy to the developers namespace in the same cluster
destinations:
- namespace: developers
server: https://kubernetes.default.svc
sourceRepos:
# Allow manifests to deploy from any Git repos
- '*'
orphanedResources:
warn: false
EOF
Fields of interest are sourceRepos and destinations:
sourceRepos: Specifies list of repositories that will be allowed for a deployment. We can as well use it to impose usage of only internally hosted Git providers.
destinations: Specifies the Kubernetes clusters and exact namespaces that will be used for deployment. In this exercise, we allow deployment to the developers' namespace.
The members of the developers-tenants group will be able to help themselves and manage all applications of the developers-tenants project. At the same time, they won’t see applications from other tenants and won’t be able to accidentally modify Kubernetes resources that don’t belong to them.
Now, let’s move an existing application guestbook from the argo-test-ns to the new project using the CLI
argocd app set guestbook --project developers-tenants --dest-namespace developers
We have successfully moved the application from the default namespace to the developers' namespace
Now let's try to create a new application in this project, but set its namespace as default:
Let's try to create the app again, in the developers namespace:
Conclusion
The takeaway from this article is that even if multi-tenancy with Kubernetes is crucial for better operations, cost management, and resource efficiency in a company, it is simply not designed as a multi-tenant system; the steps implemented in this article describe how some tools can achieve a good multi-tenant experience, by extending the functionality of Kubernetes and implementing GitOps with Argo CD. We are also working on an operator and product to handle the multi-tenancy use cases with ease. It currently supports OpenShift but soon we will have a version for Kubernetes available. You can read more about the Multi Tenant Operator here.
In future articles, we will be sharing more steps needed to achieve an effective DevSecOps multi-tenant environment.