Skip to main content

Container Image Signing

Container image signing is a feature that we provide automatically for all cloud providers. This adds a digital fingerprint to the containers. The kubernetes cluster will check the fingerprint before it starts the container. A container without a valid signature is rejected.

Each ODJ Application has its own digital signature that is used to sign and validate the containers.

Signature key pair setup​

When ODJ is setting up the infrastructure in the cloud provider it also creates the private and public key pair to generate the signatures. The private key pair is stored in the k8s cluster to validate the signatures. The first version of the signature public key is created and shared with the pipeline. The private signature key will then be used to sign the images built from the pipeline.

Signature public key version and rotation​

ODJ is capable of rotating the signature public keys. Each time a new public key is created it will replace the latest public key in the pipeline. Only container images which are signed with last 3 versions of the signature public keys are allowed to be started in the k8s cluster.

Example: When you have 3 public keys created, all images are allowed to be started on the cluster. If you start the rotation process again, a fourth key will be generated and the first key will become invalid. All container images signed with that key are not allowed to be started anymore on the cluster.

Signature creation and usage​

Container Signing Flow

  1. An action (e.g. a code change) triggers the pipeline to start a build to create and deploy a new container image
  2. The pipeline builds the container. The container is pushed to the container registry. After that the container image signature is created with the newest version of signature public key and pushed to the container registry.
  3. The deployment of the new container is triggered
  4. The k8s cluster is pulling the new container image to start a new pod
  5. Before the new pod can start, "kyverno" validates if the container image signature of the new container is valid and matches the signature private key. When the signature is valid, the pod starts. If not the pod will be blocked and does not start.

Kyverno​

Kyverno is an operator installed into the k8s cluster and is used for container image signature validation

ODJ internal container images and signature validation​

Depending on the cloud provider there are slightly differences in the container image signing process.

The following containers are excluded from the signature validation process:

  • eu.gcr.io/sit-odj/*
  • eu.gcr.io/sit-odj/debug/*
  • reg.infra.ske.eu01.stackit.cloud/*
  • kbudde/rabbitmq-exporter:*
  • bitnami/postgres-exporter:*
  • prom/mysqld-exporter:*
  • percona/mongodb_exporter:*
  • google/apigee-envoy-adapter:*
  • prom/prometheus:*
  • grafana/promtail:*
  • docker.io/busybox:*
  • quay.io/jetstack/*
  • ghcr.io/kyverno/*
  • k8s.gcr.io/ingress-nginx/*
  • registry.k8s.io/ingress-nginx/*
  • registry.k8s.io/kube-state-metrics/kube-state-metrics:*
info

This list will be reduced in the future with additional signature checks

info

We will streamline the used technologies for container image validation in the future