Posts Tagged ‘tools’

Build/Maintain your own golden container base images

August 3, 2022 Leave a comment

Containers have become essential in the optimization of software delivery these days for any business. They can support the principals of least privilege and least access by removing most of the attack surface associated with exposing services for public consumption. They are the smallest unit that make up the 4Cs’ (CNCF uses this term to describe Cloud, Cluster, Container and Code) and have become an important part of Kubernetes management. Stripping away the complexity and isolation benefits makes them portable and it almost seems as though they have no downside right? Containers (and Kubernetes) are ephemeral and support the idea of a fully automated workload but we don’t patch them like we used to. So how do we ensure that the inevitable vulnerabilities that arise (daily if not weekly) can also be mitigated or even remediated? You start over (and over) again and again by using the ‘latest and greatest’ base images. To understand this process, we need to compare the strategy of traditional software deployment strategies and see how they differ.

First there was the base OS build where we deployed a operation System and struggled to keep it updated. We applied patches to the OS to replace any software components that needed to be replaced. Many organizations struggled with patching cadence when the fleet of systems grew to large to manage. The speed of patching needed to increase as more and more vulnerabilities were found which presented a challenge for larger organizations.

Containers start with a very small base image to provide some of the libraries that are necessary for the code that was deployed with the image. Developers need to actively minimize the components that are necessary for some core capabilities (like openssl for https, glibc for os file and device options, etc.) Failure to minimize the base image used can results in adding more and more of the libraries needed rather than relying on the benefits of a shared kernel. Best practices require the understanding of the OS being used so that the image size can be smaller and the attack surface can be reduced. This results in less vulnerabilities introduced at the container level which can result in a longer runtime using that container image.

In support of this model, it is suggested that we consider how to maintain an approved (secure) base image for any container development so our deployment strategy can make use of secure (known NOT vulnerable) images to start from. The OS manufacturers are always releasing patched versions of their base image file-systems complete with the updated components. If we consider how to turn those updated base OS images into approved secure base images, the benefits provided can increase our productivity while reducing our attack surfaces.

The process proposed here can help us obtain and build base images that have a unique hash associated with them. Since container filesystems (ausfs, overlay) can be fingerprinted, we can validate the base image hash through the entire release life-cycle. This provides an added layer of detection against rogue container use and can provide an early warning detection mechanism for both development as well as operations teams. Detecting who is using a known vulnerable base image can provide notification to be sent to application owners until those vulnerable images are removed from all of our systems.

Let me show you how this can be accomplished for any of the base images that should be approved for consumption. We start by using the ‘current build’ for any of the base OS images that we want to use. (Remember, whether your nodes run RedHat, Debian, Ubuntu, Oracle, etc. to gain the best performance and to make the best use of resources, your choice of base OS should match your node runtime version. Lets grab the latest version of the Jammy base OS for amd64 – I will use podman to build my OCI compatible image but we can also do this with docker.)

Step 1 (we should repeat this whenever there is a major change in this release. The vendor will update this daily)

root : podman import jsi-jammy-08-22

Downloading from ""
Getting image source signatures
Copying blob 911f3e234304 skipped: already exists
Copying config 4d667a55fb done
Writing manifest to image destination
Storing signatures

(Notice the hash of the base container image above. MY image was already downloaded)

Step 2 (we save the image archive now as a container to be tested)

root : podman save -o jsi-08-22.tar –format oci-archive jsi-jammy-08-22

Copying blob bb2923fbc64c done
Copying config 4d667a55fb done
Writing manifest to image destination
Storing signatures

(we are using the name of the image and the date [mm/yy] to identify it. You may also use image tags but it is best practice to use unique naming)

Step 3 (lets save some space and compress it)

root : gzip -9 jsi-jammy-08-22.tar – ( results in the image named jsi-jammy-08-22.tar.gz)

Final step is to run it though a security scan to ensure there are no high or critical vulnerabilities contained in this base image.

C:\image>snyk container test oci-archive:jsi-jammy-08-22.tar.gz

Testing oci-archive:jsi-jammy-08-22.tar.gz…

✗ Low severity vulnerability found in tar
Description: NULL Pointer Dereference
Introduced through: meta-common-packages@meta
From: meta-common-packages@meta > tar@1.34+dfsg-1build3

✗ Low severity vulnerability found in shadow/passwd
Description: Time-of-check Time-of-use (TOCTOU)
Introduced through: shadow/passwd@1:4.8.1-2ubuntu2, adduser@3.118ubuntu5, shadow/login@1:4.8.1-2ubuntu2
From: shadow/passwd@1:4.8.1-2ubuntu2
From: adduser@3.118ubuntu5 > shadow/passwd@1:4.8.1-2ubuntu2
From: shadow/login@1:4.8.1-2ubuntu2

✗ Low severity vulnerability found in pcre3/libpcre3
Description: Uncontrolled Recursion
Introduced through: pcre3/libpcre3@2:8.39-13ubuntu0.22.04.1, grep@3.7-1build1
From: pcre3/libpcre3@2:8.39-13ubuntu0.22.04.1
From: grep@3.7-1build1 > pcre3/libpcre3@2:8.39-13ubuntu0.22.04.1

✗ Low severity vulnerability found in pcre2/libpcre2-8-0
Description: Out-of-bounds Read
Introduced through: meta-common-packages@meta
From: meta-common-packages@meta > pcre2/libpcre2-8-0@10.39-3build1

✗ Low severity vulnerability found in pcre2/libpcre2-8-0
Description: Out-of-bounds Read
Introduced through: meta-common-packages@meta
From: meta-common-packages@meta > pcre2/libpcre2-8-0@10.39-3build1

✗ Low severity vulnerability found in ncurses/libtinfo6
Description: Out-of-bounds Read
Introduced through: ncurses/libtinfo6@6.3-2, bash@5.1-6ubuntu1, ncurses/libncurses6@6.3-2, ncurses/libncursesw6@6.3-2, ncurses/ncurses-bin@6.3-2, procps@2:3.3.17-6ubuntu2, util-linux@2.37.2-4ubuntu3, ncurses/ncurses-base@6.3-2
From: ncurses/libtinfo6@6.3-2
From: bash@5.1-6ubuntu1 > ncurses/libtinfo6@6.3-2
From: ncurses/libncurses6@6.3-2 > ncurses/libtinfo6@6.3-2
and 10 more...

✗ Low severity vulnerability found in krb5/libkrb5support0
Description: Integer Overflow or Wraparound
Introduced through: krb5/libkrb5support0@1.19.2-2, adduser@3.118ubuntu5, krb5/libk5crypto3@1.19.2-2, krb5/libkrb5-3@1.19.2-2, krb5/libgssapi-krb5-2@1.19.2-2
From: krb5/libkrb5support0@1.19.2-2
From: adduser@3.118ubuntu5 > shadow/passwd@1:4.8.1-2ubuntu2 > pam/libpam-modules@1.4.0-11ubuntu2 > libnsl/libnsl2@1.3.0-2build2 > libtirpc/libtirpc3@1.3.2-2ubuntu0.1 > krb5/libgssapi-krb5-2@1.19.2-2 > krb5/libkrb5support0@1.19.2-2
From: adduser@3.118ubuntu5 > shadow/passwd@1:4.8.1-2ubuntu2 > pam/libpam-modules@1.4.0-11ubuntu2 > libnsl/libnsl2@1.3.0-2build2 > libtirpc/libtirpc3@1.3.2-2ubuntu0.1 > krb5/libgssapi-krb5-2@1.19.2-2 > krb5/libk5crypto3@1.19.2-2 > krb5/libkrb5support0@1.19.2-2
and 8 more...

✗ Low severity vulnerability found in gmp/libgmp10
Description: Integer Overflow or Wraparound
Introduced through: gmp/libgmp10@2:6.2.1+dfsg-3ubuntu1, coreutils@8.32-4.1ubuntu1, apt@2.4.6
From: gmp/libgmp10@2:6.2.1+dfsg-3ubuntu1
From: coreutils@8.32-4.1ubuntu1 > gmp/libgmp10@2:6.2.1+dfsg-3ubuntu1
From: apt@2.4.6 > gnutls28/libgnutls30@3.7.3-4ubuntu1 > gmp/libgmp10@2:6.2.1+dfsg-3ubuntu1
and 1 more...

✗ Low severity vulnerability found in glibc/libc-bin
Description: Allocation of Resources Without Limits or Throttling
Introduced through: glibc/libc-bin@2.35-0ubuntu3.1, meta-common-packages@meta
From: glibc/libc-bin@2.35-0ubuntu3.1
From: meta-common-packages@meta > glibc/libc6@2.35-0ubuntu3.1

✗ Low severity vulnerability found in coreutils
Description: Improper Input Validation
Introduced through: coreutils@8.32-4.1ubuntu1
From: coreutils@8.32-4.1ubuntu1

✗ Medium severity vulnerability found in perl/perl-base
Description: Improper Verification of Cryptographic Signature
Introduced through: meta-common-packages@meta
From: meta-common-packages@meta > perl/perl-base@5.34.0-3ubuntu1
Tested 102 dependencies for known issues, found 11 issues.


(Look Ma, no high or critical findings!)

Now we have a base OS image ready to be used with any new/existing container build process. Best practices include the ability to digitally sign these images so that build pipelines can verify that any images being included are tested and approved. We can remove the previous version of the base OS image and provide a notice to current/future users that vulnerabilities have been found in the previous version. Dev teams can bump the version in any code they have and begin to test if there are any breaking changes that would require refactoring. Even if there is no change in the code, they must release their containers using these new base OS images to mitigate any vulnerabilities that are introduced.

Categories: security, Work related Tags: ,

After the breach…

July 26, 2022 Leave a comment

Accidents happen and in the security field, they are usually called an ‘0-day’.

There are (at least) three questions you may be asked by your board, about your AppSec program…

  • Was all the software tested using all of our controls & capabilities that were applicable?
  • Did all the findings that were produced measure below our acceptable risk ratings?
  • Were any/all of the vulnerabilities being fixed according to our accepted remediation timelines?

Lets unpack that for everyone in an attempt to understand the motivations of some of our brightest ‘captains’. (If I was a board member…)

Misinformation – Does this event signal a lack of efficacy of our overall Appsec program? Do the controls work according to known practices? Perhaps, this is an anomaly, an edge case that now requires additional investment? What guarantees do we have that any correction strategy will be effective? If changes are warranted, which part should we focus on, People, Process or Technology?

Jeff says – changing the program can take a large investment for any/all of these. Get back to the basics and start with some metrics to see if you have effective coverage first. Prioritize making policy/configuration visible for each implementation of your security tools and aim for all of your results in one tool.

Liability – Is our security assessment program effective enough? Does this blind spot show us the inability to understand/avoid these threats at scale? Does this event indicate a systemic failure to detect/prevent this type of threat in the future?

Jeff says – Push results from Pentesting/Red Team/Security Ops back into the threat model and show if/how any improvement can be effective. Moving at the speed of DevOps means running more tests, more often, and correlating the findings to show value through velocity by catching and fixing them quickly.

Profit and Loss – Do we have a software quality problem that may require us to consider an alternative resource pool? If digitization is increasing in cost due to loss, maybe we need to improve our control capabilities to detect/prevent bad software from reaching production? Maybe we should take additional steps to ensure we have the right development teams to avoid mistakes?

Jeff says – to stop the bleeding, you might consider a different source of secure code? You might also consider an adjustment to your secure training programs? Maybe your security analysts are having their own quality issues? Consider raising the threshold of approved tools to be considered? Broker communication for your dev teams to take on more of the security responsibility.

For any leadership who is dealing with CyberSecurity these days, these are all very good questions. Security is Hard, Application Security, Cloud Security, Data Security – they are ALL hard individually so how does any one person/team understand them entirely?

I began to ask myself that question almost a decade ago during my mobile penetration testing period. When Facebook had created React which involved more than one software language in the same project. I found a cross site scripting flaw in the mobile client during testing which I felt pretty confident was NOT a false positive. I decided to check the static code findings to see if this could be correlated. (We can save the rest of that story for another blog post).

A light went off in my head, ‘correlation between two or more security tools in a single pane of glass’. What an idea – you need something that can pull in all of the datasets (finding reports) and provide some deduplication (so we don’t give dev teams multiple findings from multiple tools), just the fact that we are confident of the viability of the finding. I investigated some of the tool vendors and worked with them for a few years while the capability began to mature in the industry.

Today, Gartner calls this space Application Security Orchestration and Correlation, a contraction of ‘security orchestration’ (where you apply policy as code) and correlating, deduping the results. When done successfully, it also provides a single pane of glass for the operations team or any other orchestration or reporting software in use in your org. Think of it as the one endpoint with all the answers; a way to abstract away the API schema and various life-cycle changes that are associated with new and existing tool-sets.

Whether you wish to interconnect all of your existing orchestration tooling for your pipelines & other infrastructure or perhaps you want to build out your security governance capabilities by conducting all of your own security testing, ASOC tools are capable of providing security at the speed of DevOps.

There really is no other way to accomplish it at scale!

Categories: security, Work related Tags: ,

Zero-Day Exploitation of Atlassian Confluence | Volexity

June 3, 2022 Leave a comment

There is another 0-day for Atlassian, they are having a tough time with RCEs

Categories: General Tags: ,

Want to recover some space on your ESXi server–read on

February 5, 2012 Leave a comment

I wanted to tell my readers about a great tool I found from those creative people at Quest Software called vOptimizer. This tool is an essential part of any virtual administrators toolbox.

You start by connecting to your ESXi server and create a Project. This project will use a boot ISO uploaded by the windows client wizard during initialization and it will allow you to run a disk optimization that can do things like removing old temp files all the way to resizing your VM disk!


We started by connecting to a Windows 2003 VM on our ESXi 5 server that was grossly oversized at 40G. This project was setup to remove every excess file (including removing the swap file) and once we had successfully ran the disk optimizer we were left with a running server that was only about 11G in size. The entire process only took about 40 minutes of downtime.


We checked the ESXi summary before it was completed (left) and after (right).VM-disk-summary-beforeVM-disk-summary-after

Now I used to run a gparted iso and change the size of the disk and then expand the NTFS partition to make a disk larger but this trick allows you to make it bigger or smaller.

Unfortunately is does not work for LVM disks and with most of the Linux distros moving towards this default installation we are still looking to resolve this issue (gparted still doesn’t support it either).

(BTW this tool cannot optimize a VM if there is a snapshot so you will need to back it up first and then remove the snapshot before you can run the optimizer. This is true for windows vms as well).

After 30 days this tool will run as a free tool and only optimize 2 VMs but it is still a handy tool to have in your toolbox.

Categories: Work related Tags: , ,