Lab 33 - Image scanning, encryption at rest, audit log

Lab 33 - Image scanning, encryption at rest, audit log

Goal: find vulnerabilities before they run, make Secrets unreadable in etcd, and record who does what. Chapter: Images, secrets and audit

Part 1 - Trivy (support01)

# 1 - install from the vendor's apt repository
sudo apt-get install -y wget gnupg
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key \
  | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" \
  | sudo tee /etc/apt/sources.list.d/trivy.list
sudo apt-get update
apt-cache policy trivy | head -5
sudo apt-get install -y trivy
trivy --version
Warning

Check the version: v0.69.4 was a malicious release (March 2026 supply-chain attack). Your trainer tells you the version to pin (sudo apt-get install trivy=<version>).

# 2 - scan images
trivy image --severity HIGH,CRITICAL nginx:1.27
trivy image --severity HIGH,CRITICAL nginx:1.27-alpine
trivy image --severity HIGH,CRITICAL nginxinc/nginx-unprivileged:1.27-alpine

Compare the counts. The base image decides most of your CVEs.

# 3 - scan manifests for misconfigurations
trivy config ~/hardened.yaml
kubectl create deploy risky --image nginx --dry-run=client -o yaml > ~/risky.yaml
trivy config ~/risky.yaml

risky.yaml collects findings (root, no limits, writable root filesystem, latest…); hardened.yaml from Lab 29 is almost clean.

Part 2 - Secrets at rest (control01)

# 4 - a Secret, and how it looks inside etcd
kubectl create secret generic db-pass -n default --from-literal=password=S3cret-before
ssh student@control01
sudo crictl exec $(sudo crictl ps --name etcd -q) etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/db-pass | strings | grep -i s3cret

S3cret-before - in plain text, in every etcd backup.

Part 3 - configure the API server (still on control01)

Both features need files on the control plane and flags on kube-apiserver - one static Pod manifest edit for both.

# 5 - encryption config with a random key
sudo mkdir -p /etc/kubernetes/enc /etc/kubernetes/audit /var/log/kubernetes/audit
KEY=$(head -c 32 /dev/urandom | base64)
sudo tee /etc/kubernetes/enc/enc.yaml >/dev/null <<YAML
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: ["secrets"]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: $KEY
      - identity: {}
YAML
sudo chmod 600 /etc/kubernetes/enc/enc.yaml
# 6 - audit policy
sudo tee /etc/kubernetes/audit/policy.yaml >/dev/null <<'YAML'
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages: ["RequestReceived"]
rules:
  - level: None
    users: ["system:kube-proxy", "system:apiserver"]
  - level: None
    verbs: ["get", "list", "watch"]
    userGroups: ["system:nodes", "system:serviceaccounts:kube-system"]
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  - level: RequestResponse
    resources:
      - group: "rbac.authorization.k8s.io"
  - level: Metadata
YAML
# 7 - back up the manifest, then add flags and mounts
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak
sudo python3 - <<'PY'
p = "/etc/kubernetes/manifests/kube-apiserver.yaml"
s = open(p).read()
s = s.replace("    - kube-apiserver\n", """    - kube-apiserver
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
    - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=1
    - --audit-log-maxbackup=2
    - --audit-log-maxsize=50
""", 1)
s = s.replace("    volumeMounts:\n", """    volumeMounts:
    - mountPath: /etc/kubernetes/enc
      name: enc
      readOnly: true
    - mountPath: /etc/kubernetes/audit
      name: audit-policy
      readOnly: true
    - mountPath: /var/log/kubernetes/audit
      name: audit-log
""", 1)
s = s.replace("  volumes:\n", """  volumes:
  - hostPath:
      path: /etc/kubernetes/enc
      type: Directory
    name: enc
  - hostPath:
      path: /etc/kubernetes/audit
      type: Directory
    name: audit-policy
  - hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-log
""", 1)
open(p, "w").write(s)
PY
sudo grep -c -E -- '--(encryption-provider|audit-)' /etc/kubernetes/manifests/kube-apiserver.yaml   # 6 flags

The kubelet sees the changed file and restarts the API server - about a minute without API.

# 8 - wait for it
watch -n 5 'sudo crictl ps --name kube-apiserver'     # new container, Running; Ctrl-C
exit                                                   # back to support01
kubectl get nodes                                      # API is back
Tip

API does not come back after 3 minutes? On control01: sudo cp /root/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml and check sudo crictl ps -a --name kube-apiserver / sudo crictl logs <id>.

Part 4 - verify

# 9 - new and rewritten Secrets are encrypted
kubectl create secret generic db-pass2 --from-literal=password=S3cret-after
kubectl get secrets -A -o json | kubectl replace -f - >/dev/null   # rewrite all
ssh student@control01 'sudo crictl exec $(sudo crictl ps --name etcd -q) etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/db-pass | strings | head -3'
kubectl get secret db-pass -o jsonpath='{.data.password}' | base64 -d; echo

etcd shows k8s:enc:aescbc:v1:key1 and binary noise; the API still returns S3cret-before - encryption is transparent to users.

# 10 - the audit log
kubectl create rolebinding audit-test --clusterrole=view --user=eve -n default
ssh student@control01 'sudo grep audit-test /var/log/kubernetes/audit/audit.log | tail -1' \
  | python3 -m json.tool | grep -E '"(verb|username|sourceIPs|name|level)"'

Who (kubernetes-admin), what (create rolebinding), from where (support01’s IP) - with the full object, because RBAC is logged at RequestResponse.

Discovery

Infodiscover

Task A. Find in the audit log every request Jane made in Labs 26-27.

ssh student@control01 'sudo grep "\"username\":\"jane\"" /var/log/kubernetes/audit/audit.log | wc -l'
kubectl --context jane get pods -n dev
ssh student@control01 'sudo grep "\"username\":\"jane\"" /var/log/kubernetes/audit/audit.log | tail -1'

Only requests after step 8 are logged - audit is not retroactive.

Task B. Why is a Secret get logged only at Metadata level?

At Request/RequestResponse the audit log would contain the Secret values in plain text - moving the problem from etcd to the log files and the SIEM.

Clean up

kubectl delete secret db-pass db-pass2
kubectl delete rolebinding audit-test -n default

Encryption and audit logging stay on for the rest of the course.