Archive
Forget SPAM, why not backdoor software instead
Open Source software is used by so many people, that it has become the target for a more sophisticated attacker. One that is so pernicious, it is known, in the Common Weakness and Exposure chart as CWE-506.
Let’s face it, over the last few decades, as the world was being eaten by software* we became too reliant on other people’s work. We ‘trusted’ what others were doing and relied too much on ‘good people doing great things’ so we could focus more on what we wanted.
The strength of open source (where so many eyes would review the code instead of just a handful of coders in a cave), makes us dependent on those same coders so if they don’t catch the malicious additions, who will?
Luckily, in such a short time, a few posts by a couple of curious sysadmins on Thursday, managed to catch this ‘feature’ being added to the source, which quickly produced this CVE (above) so patch your stuff before you too, become backdoored.
For my readers, I can only say that, if you use software in your business (that should be *all* of you), then you probably use open source so ‘Verify then Trust’ because this happens more often than you realize. Invest in your Security program which should include a threat analysis channel. We are at War with the hacker community and they only need one to win!
Old tech for an old Techy
How many of you remember the days of the Pentium processors? What about the 386 when we used 30 pin SIMMs (1mb shown here) and this 5 pin DIN adapter for the PS2 keyboard?
I think that piece of memory actually cost me $25 in 1980s dollars! Computers have come a long way since the early days.
Cloud makes it easier to use but remember, it also makes it easier for the hackers 😜.
Kubernetes and Certificates
Kubernetes has become the defacto way to run software these days and many people are simply not aware of just how much it relies on cryptography. All the nodes use certificates for almost all the connections, the commands between them, Identity of services, and encryption at rest and in transit, and well… every part of the operating model. Functioning as its own Public Key Infrastructure, the security of the cluster is directly related to the safe and secure methods associated with all this key management.
I wanted to raise the awareness for most of my readers, of many organizations use of Kubernetes and introduce the practices of Certificate Lifecycle Management as they apply to Kubernetes.
If you have used KinD (Kubernetes in Docker) before, you may notice that when you use the latest version of the kind-control-plane (from this image)
kindest/node v1.29.2 09c50567d34e 4 weeks ago 956MB
You should notice that the Certificate Authority certificate is *still* being created to last 10 years.
docker exec -t kind-control-plane openssl x509 -startdate -enddate -noout -in /etc/kubernetes/pki/ca.crt
notBefore=Mar 12 17:27:26 2024 GMT
notAfter=Mar 10 17:32:26 2034 GMT
This helps to demonstrate that the primary Certificate authority that is used to stand up your kind cluster, has a lifecycle of 10 years. This seems overly permissive but for any root CA, it is still well under the acceptable limit. Why don’t we try to determine what our production GKE clusters are using by executing this command…
gcloud container clusters describe --zone <zone> <clustername> --project <projectname> --format="value(masterAuth.clusterCaCertificate)" | base64 --decode | openssl x509 -startdate -enddate -noout
When you execute this command on your tenant, what do you think we should see as the output? Would we see that you used your own root-CA to create an Intermediary CA for Google to use in your clusters?
Ideally, a mature organization could be using a private CA (either external or now, with help as a Google service offering) to secure all certificates being created and verified inside each GKE cluster or nodes? (This might be a little too much to ask – we must walk before we learn to run)
What about letting Google, who is probably creating the cluster for you, create the Kubernetes Root CA and perhaps they would be following the best practice for root certificates by limiting it to 15 years? (Note: Google is *also* proposing a maximum term limit of 7 years for any publicly accessible root CA certificate – https://www.chromium.org/Home/chromium-security/root-ca-policy/moving-forward-together/).
Well, sadly, they continue to create the default term for their GKE Kubernetes clusters of…
notBefore=Mar 17 15:43:17 2022 GMT
notAfter=Mar 9 16:43:17 2052 GMT
30 years!
So now that we know that all certificates, for *any* purpose, within your GKE clusters, (at least this one) will be using a default lifetime of 30 years, you may be asking, is there anything we can do about that? I mean, what if the key becomes compromised, we should trust ANY activity, no service mesh, no identity, not even any nodes trying to communicate with the etcd (Kubernetes database) …. well surprisingly…. YES you can – you can rotate them yourself.
In this reference from the GKE help pages, we learn that we can perform a credential rotation as a way to mitigate the loss or compromise of your root CA keys! Great, but how hard is it?
gcloud container clusters update CLUSTER_NAME --region REGION_NAME --start-credential-rotation
gcloud container clusters upgrade CLUSTER_NAME --location=LOCATION --cluster-version=VERSION
You can use the same version of each cluster to force each node to request and use the newly created certificates that will be signed by the new Kubernetes root CA certificate. Once you are sure that all your nodes have been rotated and are using new credentials, you can complete the synchronization.
gcloud container clusters update CLUSTER_NAME --region REGION_NAME --complete-credential-rotation
So why don’t more organizations do this if the process is so easy to recover? Well, if you are deploying your clusters using defaults, you probably aren’t aware of this little fact. Security people have learned that it is better to be safe than to be sorry. Mature organizations become aware of these gotchas early on in their usage of Kubernetes (over 10 years now) and first learn how to correct bad outcomes. Later, they learn how to avoid these bad outcomes by managing all parts of the Kubernetes platform to help assure it.
Kubernetes *may* be the greatest invention since Virtualization, but with great power comes great responsibility. Don’t let an old hacker (like me) get the better of your organization!
Certificate Management may be hard, but you don’t have much choice any longer.
Ever since the 1990s when Netscape₁ first introduced “Secure Sockets”, we have turned this thing called “The Internet” into an ecommerce engine worth over 3 trillion USD today. Statistics show that its growth is expected to top 5 trillion USD by 2029₂. Efforts to secure the Internet have been going on for three decades since then so why should be alarmed now? Well, it involves two of the most popular subjects in our modern era, Artificial Intelligence and Quantum Computing.
AI has proven to be highly effective at finding defects in software₃, something that humans continue to create and Quantum Computers will speed up computational power by a factor of 10x. Think of a hacker who never sleeps, has no preconceived notions about ‘if’ something can be accomplished, and just sets itself on a target of guessing your password or even breaking your encryption keys for your secure session with your bank? Is there any doubt that it will succeed…eventually, now that it is 10x faster? Does this sound like a George Orwell book, well it should, that time has arrived!
Traditional certificates relied on factorization of prime numbers. That is just a fancy way of saying 3 times 5 equals 15 (although this is an oversimplification). When you use factors that are thousands of digits long, computers were needed to solve these equations and reversing those equations would take years or even centuries. Now enter the Quantum computer that performs these calculations at dizzying speeds, and you are no longer safe. The only answer to help treat those risks is to replace those equations more often that one or twice every few years.
The scope of the problem becomes apparent when you see how prevalent traditional certificates are in our electronic world. Major use cases are not just limited to SSL/TLS certificates to protect your ecommerce or banking sites. They are used to provide integrity verification used in encryption for proof of ownership or tampering. They are also used for Identity (like secure shell or tokens) and systems that rely on trust. With AI and quantum wildly in use today, these systems are at risk if you do not replace these on a regular basis.
Google wants to shorten the lifecycle of certificates₄, to help manage the risk associated with SSL/TLs certificate usage on the Internet. By replacing the secrets more often, it makes it harder to guess them. Let’s Encrypt has be successful since the last decade, at generating 90-day certificates. There are many client implementations₅ that support the ACME standard that helps accomplish this.
This begs the question, “How do we manage hundreds of thousands of certificates at speeds that would take an army to accomplish?”
Automation is the key! Maybe you can ask your friendly AI prompt to help you accomplish this before someone uses it to crack your password and empty your bank account? 😊
Security Analyst - Jeff Soehner CISSP, GXPN,GPYC,GMOB,GCIH 