Security Analyst – Jeff Soehner CISSP, GXPN,GPYC,GMOB,GCIH

Kubernetes has become the defacto way to run software these days and many people are simply not aware of just how much it relies on cryptography. All the nodes use certificates for almost all the connections, the commands between them, Identity of services, and encryption at rest and in transit, and well… every part of the operating model. Functioning as its own Public Key Infrastructure, the security of the cluster is directly related to the safe and secure methods associated with all this key management. 

I wanted to raise the awareness for most of my readers, of many organizations use of Kubernetes and introduce the practices of Certificate Lifecycle Management as they apply to Kubernetes.

If you have used KinD (Kubernetes in Docker) before, you may notice that when you use the latest version of the kind-control-plane (from this image)

kindest/node     v1.29.2   09c50567d34e   4 weeks ago     956MB

You should notice that the Certificate Authority certificate is *still* being created to last 10 years.

This helps to demonstrate that the primary Certificate authority that is used to stand up your kind cluster, has a lifecycle of 10 years. This seems overly permissive but for any root CA, it is still well under the acceptable limit. Why don’t we try to determine what our production GKE clusters are using by executing this command…

gcloud container clusters describe --zone <zone> <clustername> --project <projectname> --format="value(masterAuth.clusterCaCertificate)" | base64 --decode | openssl x509 -startdate -enddate -noout

When you execute this command on your tenant, what do you think we should see as the output? Would we see that you used your own root-CA to create an Intermediary CA for Google to use in your clusters? 

Ideally, a mature organization could be using a private CA (either external or now, with help as a Google service offering) to secure all certificates being created and verified inside each GKE cluster or nodes? (This might be a little too much to ask – we must walk before we learn to run)

What about letting Google, who is probably creating the cluster for you, create the Kubernetes Root CA and perhaps they would be following the best practice for root certificates by limiting it to 15 years? (Note: Google is *also* proposing a maximum term limit of 7 years for any publicly accessible root CA certificate – https://www.chromium.org/Home/chromium-security/root-ca-policy/moving-forward-together/).  

30 years!

So now that we know that all certificates, for *any* purpose, within your GKE clusters, (at least this one) will be using a default lifetime of 30 years, you may be asking, is there anything we can do about that? I mean, what if the key becomes compromised, we should trust ANY activity, no service mesh, no identity, not even any nodes trying to communicate with the etcd (Kubernetes database) …. well surprisingly…. YES you can – you can rotate them yourself.

In this reference from the GKE help pages, we learn that we can perform a credential rotation as a way to mitigate the loss or compromise of your root CA keys! Great, but how hard is it?

You can use the same version of each cluster to force each node to request and use the newly created certificates that will be signed by the new Kubernetes root CA certificate. Once you are sure that all your nodes have been rotated and are using new credentials, you can complete the synchronization.

gcloud container clusters update CLUSTER_NAME --region REGION_NAME --complete-credential-rotation

So why don’t more organizations do this if the process is so easy to recover? Well, if you are deploying your clusters using defaults, you probably aren’t aware of this little fact. Security people have learned that it is better to be safe than to be sorry. Mature organizations become aware of these gotchas early on in their usage of Kubernetes (over 10 years now) and first learn how to correct bad outcomes. Later, they learn how to avoid these bad outcomes by managing all parts of the Kubernetes platform to help assure it.

Kubernetes *may* be the greatest invention since Virtualization, but with great power comes great responsibility. Don’t let an old hacker (like me) get the better of your organization!