Multi-Tenancy in K8s & OpenShift #10 - Restricting NodePort & HostPort

In this series, we've explored key topics, including Namespace-Based Isolation, Network Policies, RBAC, Resource Quotas and LimitRanges, Pod Security Standards (PSS), Storage Isolation, and Ingress Control Isolation, to ensure fair resource and data management in multi-tenant Kubernetes clusters. In our previous blog, we discussed Control Plane Robustness to Safeguard Shared Kubernetes Resources, highlighting the importance of maintaining stability in multi-tenant setups.

In Kubernetes, NodePort and HostPort services allow applications to expose their services to external traffic by using the underlying nodes’ network interfaces. While this can be helpful in certain scenarios, it poses significant security risks in a multi-tenant environment. Restricting NodePort and HostPort usage is crucial for ensuring network security and preventing tenants from bypassing internal security controls, which could lead to unauthorized access or unintended exposure of services.

Multi-Tenancy in Kubernetes & Openshift: A Comprehensive Guide

Part 1: Use Cases & Implementations

Part 2: Namespace-Based Isolation for Workload Separation

Part 3: Network Policies for Network Isolation

Part 4: Role-Based Access Control (RBAC) for Authorization

Part 5: Resource Quotas and LimitRanges for Resource Control

Part 6: Pod Security Standards (PSS) for Workload Security

Part 7: Storage Isolation for Persistent Volume Security

Part 8: Ingress Control Isolation for External Access Segregation

Part 9: Control Plane Robustness to Safeguard shared Kubernetes Resources

Part 10: NodePort and HostPort Restrictions for Enhanced Network Security

Part 11: Resource and Cost Tracking for ShowBack/ChargeBack

Part 12: Multi-Tenant Considerations for Shared Tools

NodePort and HostPort Restrictions for Enhanced Network Security

What Are NodePort and HostPort Services?

• NodePort: A type of Kubernetes service that opens a specific port on every node in the cluster, allowing external access to a service. It forwards traffic from the external port to the pod running the service.

• HostPort: This lets a pod bind directly to a port on the host node’s network interface, bypassing Kubernetes' internal networking and using the host network. HostPort essentially gives the pod direct access to the host's network stack.

While these services can be helpful for debugging or simple external access, they pose security risks in multi-tenant environments because:

• They bypass NetworkPolicies, allowing direct access to host-level resources.

• They expose services on all nodes, increasing the attack surface.

• Multiple tenants might accidentally or maliciously expose services on the same port, leading to port conflicts or unintended access to critical applications.

Why Disable NodePort and HostPort for Multi-Tenancy?

In a multi-tenant environment, restricting the use of NodePort and HostPort is essential for several reasons:

• Prevent Cross-Tenant Access: Tenants might accidentally expose their services on easily discoverable ports across all nodes in the cluster, allowing other tenants to access these services without permission.

• Avoid NetworkPolicy Bypass: NodePort and HostPort bypass the cluster’s internal network policies, exposing services directly to the outside world, regardless of existing network restrictions.

• Limit External Attack Surface: By exposing services directly on nodes, the attack surface increases, as each node becomes a potential entry point for unauthorized access.

• Port Conflicts: Multiple tenants might inadvertently expose services using the same port, leading to port conflicts or unintentional access between tenants.

Disabling these service types enforces best security practices by ensuring that services stay contained within the cluster or are accessed through controlled and secure mechanisms like Ingress or LoadBalancer services.

How to Disable NodePort and HostPort

There are several strategies to prevent or limit the usage of NodePort and HostPort services in Kubernetes, ensuring tenants can’t use these features to inadvertently expose their workloads.

• Use Admission Controllers to Restrict NodePort and HostPort: Admission controllers can block the creation of services or pods that specify NodePort or HostPort. Gatekeeper, combined with Open Policy Agent (OPA), allows you to define and enforce policies that restrict the use of these service types.

• Implement a Custom Admission Webhook: A custom admission webhook can intercept requests to the Kubernetes API server and block any attempts to create NodePort services or configure HostPort on pods. Deploying this webhook prevents resources from being created with insecure configurations before they are applied.

• Deploy a Custom Controller: Create a custom controller using the Kubernetes client-go library to monitor and manage services and pods. Configure it to watch for NodePort and HostPort configurations and automatically delete, flag, or alert on any non-compliant resources after they are created.

• Enable Monitoring and Alerting for NodePort and HostPort: Use logging and monitoring tools like Prometheus or Kubernetes API server audit logs to detect and alert administrators when NodePort or HostPort configurations are used. This approach can provide visibility without enforcing immediate deletion or denial.

• Promote Secure Alternatives: Encourage tenants to use Ingress or LoadBalancer services instead of NodePort and HostPort for external access. These alternatives offer more secure and controlled options for exposing applications, reducing the risk of bypassing network policies or unintended exposure.

Best Practices for NodePort and HostPort Restrictions

1. Enforce via Admission Controllers: Use tools like OPA with Gatekeeper to enforce strict policies that block NodePort and HostPort usage. This ensures tenants can’t expose their services in insecure ways.

2. Promote Secure Alternatives: Encourage tenants to use Ingress or LoadBalancer services for external access. These options provide more granular control and security, such as TLS termination and hostname-based routing.

3. Monitor Service and Pod Configurations: Continuously audit existing services and pods in the cluster to identify any that are using NodePort or HostPort. Tools like Kubeaudit or Kubescape can help detect these configurations and ensure compliance with security policies.

4. Use NetworkPolicies to Complement Restrictions: Even when using secure services like Ingress or LoadBalancer, enforce NetworkPolicies that restrict access to only the intended users or services.

Conclusion

NodePort and HostPort Restrictions are critical for maintaining network security in a multi-tenant Kubernetes environment. By enforcing policies that prevent tenants from using NodePort and HostPort, you reduce the attack surface, avoid bypassing internal security controls, and prevent tenants from unintentionally exposing services. Encouraging the use of more secure alternatives like Ingress or LoadBalancer services further enhances the overall security of the cluster while maintaining external access when necessary.

Simplifying Multi-Tenancy with Stakater Multi-Tenant Operator

Implementing multi-tenancy in Kubernetes comes with a unique set of challenges, especially around network security and isolation. Configuring elements like namespace isolation, network policies, RBAC, and resource quotas can be complex and time-consuming, demanding strong Kubernetes expertise to ensure secure and properly managed resources. This is where the Stakater Multi-Tenant Operator (MTO) makes a difference.

The Stakater Multi-Tenant Operator is built to streamline and accelerate the deployment of multi-tenancy in Kubernetes clusters. By offering a comprehensive, automated framework for tenant management, it helps organizations swiftly establish secure, isolated, and well-regulated environments. MTO also supports network security by guiding tenants to use secure alternatives like Ingress or LoadBalancer for external access—reducing the risks associated with NodePort and HostPort services, which can expose critical resources and increase the attack surface in a shared environment.

In the next blog, we will discuss Resource and Cost Tracking for ShowBack/ChargeBack, exploring how to manage and monitor resource usage effectively in a multi-tenant setup.