Secure GKE clusters with Custom Organization Policies in GCP

By Kishore Jagannath．Oct 11, 2022

Enterprises are moving towards deploying their applications as containers and kubernetes is a popular orchestration tool for managing and deploying container based workloads. Google Cloud provides Google Kubernetes Engine a.k.a GKE, a platform creating and managing kubernetes clusters. From a security perspective, a kubernetes based application can be divided into four dimensions: 1. Code, 2. Containers 3. Cluster and 4. Cloud .

Below figure shows the different Security planes of a Kubernetes application running in a public cloud environment. In this post we will deep-dive into enforcing the security of a Kubernetes cluster(Cluster Plane)in GKE via “Custom Organization Policies feature in GKE”.

Introduction

Organization policies provide centralised control and enforcement of secure practices in GCP environment. There are 70+ in built organisational policies currently available. GCP plans to extend this with Custom Organisation Policies enabling administrators to extend the default policies. “Custom Organisational Policies for GKE” enable administrators to enforce security policies and best practices on GKE cluster resources. This feature which is currently in public preview, enables organisations in securing the “Cluster Plane”, and enforcing best practices in creating and managing GKE clusters. With this feature, you can create a custom policy to restrict the attributes of GKE clusters and apply the policy constraint at an Organisational or Folder or Project level.

Custom Organisational Policies for GKE

Organisational policies for GKE enables an Organisational Policy administrator to specify constraints on attributes of a GKE cluster resource. These constraints can enforce best practices and policies on GKE cluster, and further strengthen the security posture from a GKE cluster perspective. Some examples of constraints could be

1. GKE cluster should always be private and expose private endpoints only.
2. Maintenance window should be defined for GKE cluster.
3. Auto upgrade of clusters cannot be disabled.
4. Enable Workload Identity Pool.

GKE Organisational Policy Example

Let’s consider the use case where an Organisational Policy Admin wants to ensure that all GKE clusters created in the organisation should be Private. This essentially means that any VM worker nodes created for the cluster should not expose public IP address.

To enforce this policy, the Organisation admin has to ensure that the private endpoint attribute is always set to true within a GKE cluster. To find out the exact attribute names of a GKE Cluster, an administrator can view the GKE cluster object by looking at the Resource definition of GKE Cluster REST APIs link.

The PrivateClusterConfig entity contains the attribute “enablePrivateNodes” to ensure that any nodes created from this cluster contains only a private Ip and a public IP for nodes are not created.

This means that the constraint enforced on the GKE cluster should check the attribute “PrivateClusterConfig.enablePrivateNodes” in the condition, when a cluster is created or updated.

The constraint for a GKE cluster can be created as follows

The above yaml file creates an org policy constraint and evaluates the “condition” whether enablePrivateNodes is set to false. The action_type is set to DENY, so if a GKE cluster is created with enablePrivateNode attribute set to false, it will be Denied. Alternatively we could have also evaluated that the policy should ALLOW(action_type) only when enablePrivateNode is set to true in the condition, else it should fail. The method_types specifies that the policy should be enforced both during creation and modification of GKE Clusters. The conditions in the above constraint are based on Common Expression Language

The Org policy constraint can be created by the below command.

gcloud org-policies set-custom-constraint constraint-priv-node.yaml

To enforce the constraint create a policy yaml as below.

In the yaml above we have specified that the policy should be enforced at the organisation level, hence it will apply to all GKE clusters created across different projects for the organisation. Alternatively we could also specify projects/<Project_ID> in the policy name, which will enforce this policy only to GKE clusters at the specified project.

Apply the above policy by entering the below command.

gcloud org-policies set-policy policy-priv-cluster.yaml

Testing the Policy Constraint

Create a GKE cluster with the below command. By default enablePrivateNodes attribute of a cluster is false, hence in the below command the cluster worker nodes will expose public ip which is prohibited by the Organization Policy Constraint we created above

gcloud container clusters create gke-test 
--project=<PROJECT_ID> 
--zone=<COMPUTE_ZONE> 
--num-nodes=1

The below error is thrown when a cluster is created with public nodes

Create a private cluster by explicitly enabling private nodes and specifying CIDR range for Kubernetes master plane communication. Since private nodes is set to true the Organization policy constraint is satisfied and cluster is successfully created.

Conclusion

Organisation policies enable administrators to enforce security controls and best practices across the resources created in GCP environment. With “Custom Organisational Policies for GKE” administrators can now enforce security and best practices on GKE clusters. These policies when rightly configured will dramatically help secure the Cluster Plane of Kubernetes resources.

References

• Custom Organisational Policies for GKE
• Organisational Policies in GCP
• GKE Cluster Object and attributes

---

The original article published on Medium.