top of page

Stakater Blog

Follow our blog for the latest updates in the world of DevSecOps, Cloud and Kubernetes

Search

Multi-Tenancy in K8s & OpenShift #12 - Shared Tooling in Multi-Tenant Environments

Updated: 18 hours ago

In a multi-tenant Kubernetes environment, tenants often need access to common tools and services, such as logging, monitoring, backup/restore, GitOps, and secret management. In our previous blog on Resource and Cost Tracking for ShowBack/ChargeBack, we highlighted the importance of fair billing and accountability among tenants.


In this series, we've also explored essential topics, including Namespace-Based Isolation, Network Policies, RBAC, Resource Quotas and LimitRanges, Pod Security Standards (PSS), Storage Isolation, Ingress Control Isolation, Control Plane Robustness, and NodePort and HostPort Restrictions, all aimed at ensuring fair resource and data management in multi-tenant Kubernetes clusters.


However, deploying these tools on a shared cluster without proper isolation can lead to security risks, data exposure, and performance issues. To ensure that tenants receive secure and isolated access to these tools, it's crucial to consider multi-tenancy when configuring shared services in the cluster.


Multi-Tenant Considerations for Shared Tools


Logging Stack Isolation

Logging is essential for visibility, troubleshooting, and auditing. In a shared cluster, a multi-tenant logging stack should:

  • Isolate log data by namespace or tenant to ensure that logs from one tenant cannot be accessed by another.

  • Use tools like Loki with namespace-specific indices or tenant-aware configurations.

  • Implement access controls to restrict visibility in the log viewer (e.g., Loki, Grafana), ensuring each tenant can only view their own logs.


Monitoring Stack Isolation

Monitoring tools collect metrics that help administrators and tenants track resource usage, performance, and health. To maintain tenant isolation:

  • Configure Prometheus with multi-tenant capabilities, such as tenant-specific scraping jobs or federated Prometheus instances.

  • Use Grafana with folder-based permissions or separate dashboards for each tenant, limiting visibility to only their metrics.

  • For larger deployments, consider Thanos or Cortex for multi-tenant Prometheus setups, which offer support for tenant-based access control and scalable metrics storage.


Backup and Restore Isolation (Velero)

Backup and restore functionality is crucial for disaster recovery and data protection. When using tools like Velero:

  • Set up Velero with tenant-specific backup policies, ensuring backups are stored in separate buckets or prefixes in object storage.

  • Use namespace-scoped backups to isolate backup and restore operations per tenant.

  • Consider configuring role-based access control (RBAC) to limit backup and restore permissions to authorized users for each tenant, preventing cross-tenant data access.


GitOps Tool Isolation (ArgoCD)

GitOps tools like ArgoCD automate application deployment and management. For multi-tenant clusters, ensure:

  • Project-based isolation in ArgoCD, where each tenant is assigned to a project that limits access to specific repositories, clusters, and namespaces.

  • RBAC policies enforce permissions per project, ensuring each tenant can only manage applications within their assigned scope.

  • Implement SSO integration with tenant-based group membership, streamlining authentication and access control.


Container Registry Isolation

Many tenants need a secure, isolated registry to store and retrieve container images. In a shared Kubernetes environment:

  • Use multi-tenant container registries like Harbor, which supports project-based isolation, image vulnerability scanning, and access control.

  • Set up namespace-specific repositories or naming conventions to logically separate tenant images within the registry.

  • Enforce network policies that restrict tenant namespaces to accessing only their own images or approved registries.


Secret Management Isolation (Vault)

For secure storage and management of sensitive information, HashiCorp Vault is commonly used. To achieve tenant isolation with Vault:

  • Use namespace-based access by configuring Vault policies that restrict each tenant to their own namespace, enabling separate storage and access control.

  • Integrate Kubernetes authentication with Vault to ensure that each tenant’s pods can only access secrets within their specific namespace or path.

  • If using a cloud provider’s secret management tool, configure isolated access controls based on tenant namespaces or identity provider (IDP) attributes.


By configuring shared tools with multi-tenant isolation in mind, you can provide tenants with the resources they need while maintaining security, compliance, and operational efficiency in your Kubernetes environment. This approach ensures that tenants can collaborate on a shared cluster without compromising data integrity or performance, creating a reliable and secure multi-tenant ecosystem.


Simplifying Multi-Tenancy with Stakater Multi-Tenant Operator

As you’ve seen, implementing multi-tenancy on Kubernetes can be complex. It requires in-depth Kubernetes knowledge and involves configuring namespace isolation, network policies, RBAC, resource quotas, and more. Doing it right takes significant time and effort to ensure security and proper resource management. This is where the Stakater Multi-Tenant Operator (MTO) shines.


The Stakater Multi-Tenant Operator is designed to simplify and accelerate multi-tenancy implementation on Kubernetes clusters. It provides a robust, automated framework for tenant management, enabling organizations to set up secure, isolated, and well-managed environments quickly and efficiently.

Comments

bottom of page