Scan Container Images with Clair V4

In this video, I've showed 3 different container image tools - one for building container images, one for checking their efficiency and another one for manipulating and promoting them.

One critical pipeline stage that was missing is a security scan.

We need to make sure that our image does not have any known vulnerabilities.

One way to do this is to use the container registry with a built-in security scan.

Most of the modern registries have such feature built in.

For example, AWS ECR allows you to automatically scan each image right after you push it.

Another example is Quay, which provides the security scan result for any image, including the public ones.

There is a common component that both ECR and Quay use for the security scanning: its name is Clair.

Clair is an open source tool that provides Vulnerability Static Analysis for Containers.

The way it works is by regularly updating internal database from multiple sources of known vulnerability metadata.

Clair supports most of the popular Linux distributions, but it also has scanning of programming language-specific packages on the roadmap.

In addition to the database, Clair also exposes an API. You can send your image to this API and get a security report in return.

It's very important to continuously scan your container registry to detect any unpatched images. This is how Quay is using Clair in the background.

But what is also important is ... to make security scans a part of your Continuous Delivery pipeline. This way you can get an instant feedback and prevent the promotion of the vulnerable container image before it's too late.

This is exactly what we will implement today.

I am going to use Clair Version 4.

The reason for this is that Clair Version 2 is now in the maintenance mode and won't get any new features and.. Clair Version 3 was never released.

We will add another stage to the same pipeline we've built in the video from the beginning, so check that one out if you are confused about what's happening here.

I am going to start with deploying a Clair instance on top of the Kubernetes.

Clair needs a PostgreSQL database, so first I will deploy the most basic PostgreSQL instance.

Here I have a Deployment and a Service. It's nothing close to a production database, but it's good for quick experiments and demos like this.

Let me apply this yaml file.

Clair needs a configuration file to be able to connect to the database. Clair version 4 is split into a couple of microservices, like indexer and matcher that you see in this config file.

I am going to create a new secret with the contents of this config file.

kubectl create secret generic clair-config-secret --from-file=config.yaml=./config.yaml

kubectl create is a very convenient subcommand, check our article with tips for passing Certified Kubernetes Administrator exam to learn about different time savers like this.

Finally, let's look at the Clair objects.

There is a deployment, that points to the latest release of Clair version 4 and it also mounts the secret as a file inside the pod.

There is also a service in front of Clair pods and an Ingress to make it available externally.

Let me apply clair.yaml as well.

If we check the logs of clair pod, we will see that clair is updating it's database with the latest security metadata. This process can take few minutes before we can actually scan any images.

Now let's head back to the language tool image repository. I am going to add a new script scan.sh

First of all, I need to add a docker authentication file. Sadly at the moment this is the only way to login to private registry.

Clair is using google container registry Go library, that defaults to certain locations of registry authentication files:

authn package - github.com/google/go-containerregistry/pkg/authn - Go Packages


go-containerregistry/pkg/authn at main · google/go-containerregistry


#!/bin/sh
HOME=/tmp
IMAGE_NAME=$REGISTRY/build-tools:$CI_PIPELINE_ID
mkdir /tmp/.docker
echo $DOCKER_AUTH_CONFIG > /tmp/.docker/config.json
clairctl report --host http://paas.mkdev.me:80 $IMAGE_NAME > report
cves=$(cat report | grep " found " | wc -l)
if [ $cves -gt 0 ]
then
  cat report
  exit 1
fi

To scan the image I will use clairctl, which is included in the clair image. I will save the result of the scan to a text file.

Then, I will do some beautiful shell scripting to check if there are any issues found.

If there are, the script will fail the build.

Now let's add a new stage to the Gitlab CI config file.

scan:
  image: Quay
  stage: test
  script:


./bin/scan.sh

Scan is going to be the part of test stage, so that it runs in parallel with the dive analysis.

Now let me push the image and time travel so we can see the result.

The scan failed, due to many CVEs discovered in the image.

Luckily, if we check the last column, we see that all of those CVEs are fixed already. If we update packages, we will get a green build.

Let's head to the Containerfile and switch from the old UBI version to the latest one.

Now let me commit the change and time travel again.

This time the scan phase is green, image is promoted successfully, and we have increased confidence in our image.

You should always make security scans part of your continuous delivery pipeline, in addition to regular scans of your whole registry. Clair is one the most popular tools to achieve this.

Here's the same article in video form for your convenience: