Labs on Eriks training corner

Lab 01 - Workstation setup

Mon, 01 Jan 0001 00:00:00 +0000

Lab 01 - Workstation setup

Goal: turn support01 into your kubectl workstation. Chapter: Architecture · Cheat sheet: kubectl

support01 has Docker and the NFS exports, but deliberately no kubectl and no kubeconfig. Building your own client is the first thing you do at any customer, so you may as well do it here.

Before you start: network on support01

support01 does not have its addresses yet, so SSH will not work. Fix it once from the Oracle VirtualBox console (double-click the support01 VM, log in as student / linux).

Lab 02 - First look at the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Lab 02 - First look at the cluster

Goal: map every box of the architecture diagram to something running. Chapter: Architecture

# 1 - where am I connected, and as whom?
kubectl cluster-info
kubectl config get-contexts
kubectl config view --minify | grep server

Check config get-contexts before anything destructive. Most production accidents are the right command in the wrong cluster.

The Pod network

# 2 - which CNI runs here, and which address range do Pods get?
kubectl get pods -n calico-system -o wide # one calico-node per node
kubectl get cm -n kube-system kubeadm-config -o yaml | grep -E 'podSubnet|serviceSubnet'
kubectl get pods -A -o wide | grep 10.244 | head -5

Calico was installed with the cluster. Every Pod gets an address from 10.244.0.0/16, Services from 10.96.0.0/12. You work with Calico itself on Day 4.

Lab 03 - The Kubernetes API and kubectl

Mon, 01 Jan 0001 00:00:00 +0000

Lab 03 - The Kubernetes API and kubectl

Infodiscover

Goal: answer any Kubernetes question from the terminal - and see the HTTP behind kubectl. Chapter: The Kubernetes API and kubectl

Try each task first; the box holds one working answer.

Task 1. List every kind the cluster knows. Find the short name and API group of deployments, and whether it is namespaced.

Show the command

kubectl api-resources | grep -i deploy

deployments deploy apps/v1 true Deployment

Lab 04 - Imperative vs. declarative objects

Mon, 01 Jan 0001 00:00:00 +0000

Lab 04 - Imperative vs. declarative objects

Goal: feel why apply is the only thing you put in a pipeline. Chapter: Declarative vs. imperative

Imperative

# 1 - an action, executed once
kubectl create deploy web --image nginx:1.27
kubectl create deploy web --image nginx:1.27 # AlreadyExists - not idempotent
kubectl scale deploy web --replicas 2
kubectl delete deploy web

Declarative

# 2 - generate the manifest instead of writing it
cd ~
kubectl create deploy web --image nginx:1.27 --replicas 2 \
 --dry-run=client -o yaml > web.yaml
cat web.yaml

# 3 - validate against the API without storing anything
kubectl apply -f web.yaml --dry-run=server

client only builds the object locally; server runs full validation and admission, then throws the result away.

Lab 05 - Namespaces

Mon, 01 Jan 0001 00:00:00 +0000

Lab 05 - Namespaces

Goal: scope your work and stop typing -n. Chapter: Namespaces

# 1 - what exists now
kubectl get ns

Four namespaces come with every cluster. kube-system holds the control plane Pods - look but do not deploy there.

# 2 - declarative (recommended)
cd ~
cat > ns-dev.yaml <<'YAML'
apiVersion: v1
kind: Namespace
metadata:
 name: dev
YAML
kubectl apply -f ns-dev.yaml

# 3 - imperative, for the throwaway one
kubectl create ns test

# 4 - the same name in both, with no conflict
kubectl run web --image nginx:1.27 -n dev
kubectl run web --image nginx:1.27 -n test
kubectl get pods -n dev
kubectl get pods -A | grep ' web '

Names are unique per namespace per kind. That is the whole point: dev and prod can both have a web.

Lab 06 - Labels, selectors and annotations

Mon, 01 Jan 0001 00:00:00 +0000

Lab 06 - Labels, selectors and annotations

Infodiscover

Goal: slice a pile of Pods any way you need. Chapter: Labels, selectors and annotations

Set-up - six Pods with different labels:

cd ~
kubectl create ns shop
for p in "web-1 app=web,env=prod,tier=frontend" "web-2 app=web,env=dev,tier=frontend" \
 "api-1 app=api,env=prod,tier=backend" "api-2 app=api,env=test,tier=backend" \
 "db-1 app=db,env=prod" "cache-1 app=cache,env=dev"; do
 set -- $p
 kubectl run $1 -n shop --image nginx:1.27 --labels $2
done
kubectl config set-context --current --namespace=shop

Task 1. List all Pods with their labels. Then show only app and env as columns.

Lab 07 - Pods

Mon, 01 Jan 0001 00:00:00 +0000

Lab 07 - Pods

Goal: generate a Pod manifest, add resources, and see what a Pod really is. Chapter: Pods

# 1 - imperative: create and throw away
kubectl run bbox --image busybox --restart=Never -- sleep 3600
kubectl get pods
kubectl delete pod bbox

--restart=Never is what makes kubectl run produce a bare Pod instead of something managed. -- sleep 3600 overrides the image’s command, because busybox would otherwise exit immediately.

# 2 - generate the manifest instead
cd ~
kubectl run bbox --image busybox --restart=Never --dry-run=client -o yaml \
 -- sleep 3600 > bbox.yaml
cat bbox.yaml

Edit bbox.yaml and add resources to the container:

Lab 08 - ReplicaSets and Deployments

Mon, 01 Jan 0001 00:00:00 +0000

Lab 08 - ReplicaSets and Deployments

Goal: watch the reconciliation loop, see the selector do the binding, and run your first Deployment. Chapter: ReplicaSets and Deployments

ReplicaSet

# 1 - a ReplicaSet by hand (the only time you will write one)
cd ~
cat > web-rs.yaml <<'YAML'
apiVersion: apps/v1
kind: ReplicaSet
metadata:
 name: web-rs
spec:
 replicas: 3
 selector:
 matchLabels:
 app: web
 template:
 metadata:
 labels:
 app: web
 spec:
 containers:
 - name: web
 image: nginx:1.27
YAML
kubectl apply -f web-rs.yaml
kubectl get rs,pods -l app=web -o wide

# 2 - self-healing
kubectl delete pod "$(kubectl get pods -l app=web -o name | head -1)"
kubectl get pods -l app=web

The same loop that created the Pods counted two, wanted three, created one.

Lab 09 - DaemonSets

Mon, 01 Jan 0001 00:00:00 +0000

Lab 09 - DaemonSets

Goal: one Pod per node, and the two surprises that come with it. Chapter: DaemonSets

# 1 - the cluster already runs several
kubectl get ds -A
kubectl get pods -A -o wide | grep -E 'kube-proxy|calico-node'

kube-proxy and the CNI agent are DaemonSets: they must exist on every node, including nodes that join tomorrow. There is no replicas field anywhere - the node list is the count.

# 2 - your own
cd ~
cat > logger-ds.yaml <<'YAML'
apiVersion: apps/v1
kind: DaemonSet
metadata:
 name: node-logger
spec:
 selector:
 matchLabels:
 app: node-logger
 template:
 metadata:
 labels:
 app: node-logger
 spec:
 containers:
 - name: logger
 image: busybox:1.36
 command: ["sh", "-c", "while true; do echo \"$(date) $(hostname)\"; sleep 30; done"]
 resources:
 requests: { cpu: "10m", memory: "16Mi" }
YAML
kubectl apply -f logger-ds.yaml
kubectl get ds node-logger
kubectl get pods -l app=node-logger -o wide

Three Pods, one per worker - and none on control01.

Lab 10 - StatefulSets

Mon, 01 Jan 0001 00:00:00 +0000

Lab 10 - StatefulSets

Goal: stable names, per-Pod storage, ordered start - and the clean-up surprise. Chapter: StatefulSets

Storage first

Each replica needs its own volume. This cluster has no StorageClass yet (Day 5), so create three PersistentVolumes on the NFS exports of support01:

# 0 -
cd ~
for i in 1 2 3; do cat <<YAML
---
apiVersion: v1
kind: PersistentVolume
metadata:
 name: pv-storage$i
spec:
 capacity:
 storage: 1Gi
 accessModes: ["ReadWriteMany"]
 persistentVolumeReclaimPolicy: Retain
 storageClassName: ""
 nfs:
 server: support01
 path: /storage$i
YAML
done > pvs.yaml
kubectl apply -f pvs.yaml
kubectl get pv # three, STATUS Available

Headless Service first

# 1 -
cd ~
cat > web-headless.yaml <<'YAML'
apiVersion: v1
kind: Service
metadata:
 name: web-headless
spec:
 clusterIP: None # headless - no virtual IP
 selector:
 app: web-sts
 ports:
 - port: 80
YAML
kubectl apply -f web-headless.yaml
kubectl get svc web-headless # CLUSTER-IP is "None"

A normal Service hides the Pods behind one IP. A headless Service does the opposite: DNS returns the individual Pod addresses, so each replica is addressable by name. A StatefulSet requires one.

Lab 11 - Jobs and CronJobs

Mon, 01 Jan 0001 00:00:00 +0000

Lab 11 - Jobs and CronJobs

Goal: run work that finishes, in parallel, on a schedule - and see how a failing Job gives up. Chapter: Jobs and CronJobs

# 1 - a one-off Job
cd ~
kubectl create job hello --image busybox:1.36 -- sh -c 'echo "hello from $(hostname)"; sleep 5'
kubectl get job,pods -l job-name=hello
kubectl wait --for=condition=complete job/hello --timeout=60s
kubectl logs job/hello

COMPLETIONS 1/1 and the Pod in Completed. The Pod is kept so you can read its log.

Lab 12 - ConfigMaps and Secrets

Mon, 01 Jan 0001 00:00:00 +0000

Lab 12 - ConfigMaps and Secrets

Goal: same image, different configuration - and learn which changes need a restart. Chapter: ConfigMaps and Secrets

# 1 - from literals and from a file
cd ~
printf 'color=blue\nretries=3\n' > app.properties
kubectl create cm app-cfg \
 --from-literal=DATACENTER=dc01 \
 --from-file=app.properties
kubectl get cm app-cfg -o yaml

--from-literal makes one key; --from-file makes a key named after the file whose value is the whole file. Both live in the same data map.

Lab 13 - Services and DNS

Mon, 01 Jan 0001 00:00:00 +0000

Lab 13 - Services and DNS

Goal: reach Pods by a stable name, from inside and from outside. Chapter: Networking, CNI, Services and DNS

# 1 - something to expose
cd ~
kubectl create deploy web --image nginx:1.27 --replicas 3
kubectl get pods -l app=web -o wide # note the Pod IPs

# 2 - ClusterIP, the default
kubectl expose deploy web --port 80 --name web-svc
kubectl get svc web-svc
kubectl get endpointslices -l kubernetes.io/service-name=web-svc

The EndpointSlice is the interesting object: it holds the three Pod IPs. The Service is just a stable front; the EndpointSlice is the live membership list, maintained by a controller as Pods become ready.

Lab 14 - Helm: install Traefik, releases and charts

Mon, 01 Jan 0001 00:00:00 +0000

Lab 14 - Helm: install Traefik, releases and charts

Goal: install Helm, then use it on a real chart: Traefik, the entry point for Labs 15 and 16. Inspect, install, upgrade, roll back. Chapter: Helm

Install Helm

# 1 -
cd ~
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-4
chmod 700 get_helm.sh && ./get_helm.sh
helm version

Look before you install

# 2 - the repository
helm repo add traefik https://traefik.github.io/charts
helm repo update
helm search repo traefik/traefik
helm search repo traefik/traefik --versions | head -5

# 3 - the chart's interface
helm show chart traefik/traefik | head -12
helm show values traefik/traefik | grep -n -A6 -E '^(service|gateway|providers):'

show values is the public interface of a chart. A setting that is not there does not exist - --set on an unknown key is silently ignored.

Lab 15 - Ingress with Traefik (legacy)

Mon, 01 Jan 0001 00:00:00 +0000

Lab 15 - Ingress with Traefik (legacy)

Goal: publish two applications by host name with the old API - enough to read the Ingress objects you will meet in existing clusters. Chapter: Ingress (legacy)

Requires Traefik from Lab 14.

# 1 - two applications, ClusterIP only
cd ~
kubectl create ns web
kubectl create deploy shop --image nginx:1.27 -n web
kubectl create deploy wiki --image httpd:2.4 -n web
kubectl expose deploy shop --port 80 -n web
kubectl expose deploy wiki --port 80 -n web

# 2 - one Ingress, two hosts
kubectl create ingress web -n web --class traefik \
 --rule='shop.k8s.lab/*=shop:80' \
 --rule='wiki.k8s.lab/*=wiki:80' \
 --dry-run=client -o yaml > ingress.yaml
cat ingress.yaml
kubectl apply -f ingress.yaml
kubectl get ingress -n web

# 3 - test through Traefik's NodePort
curl -s -H 'Host: shop.k8s.lab' http://worker01:30080 | grep -o '<title>.*</title>'
curl -s -H 'Host: wiki.k8s.lab' http://worker01:30080
curl -s -o /dev/null -w '%{http_code}\n' -H 'Host: other.k8s.lab' http://worker01:30080

One port, two applications, and a 404 from Traefik for an unknown host.

Lab 16 - Gateway API with Traefik

Mon, 01 Jan 0001 00:00:00 +0000

Lab 16 - Gateway API with Traefik

Goal: route by host, path and header through one entry point, split traffic for a canary - as the application team, on a Gateway owned by the platform team. Chapter: Gateway API

Requires Traefik and the Gateway API CRDs from Lab 14 and the shop / wiki apps in namespace web from Lab 15.

The platform side

# 1 - what the platform team gives you
kubectl get gatewayclass
kubectl get gateway -n traefik
kubectl get gateway traefik-gateway -n traefik -o yaml | grep -A12 'listeners:'

GatewayClass traefik - the implementation.
Gateway traefik-gateway in namespace traefik, listener web on port 8000 - Traefik’s entry point, exposed on NodePort 30080.
allowedRoutes.namespaces.from: All - routes from any namespace may attach.

# 2 - PROGRAMMED must be True
kubectl get gateway traefik-gateway -n traefik

The application side

# 3 - one HTTPRoute per application, in the application's namespace
cd ~
cat > routes.yaml <<'YAML'
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: shop
 namespace: web
spec:
 parentRefs:
 - name: traefik-gateway
 namespace: traefik
 hostnames: ["shop.k8s.lab"]
 rules:
 - backendRefs:
 - name: shop
 port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: wiki
 namespace: web
spec:
 parentRefs:
 - name: traefik-gateway
 namespace: traefik
 hostnames: ["wiki.k8s.lab"]
 rules:
 - backendRefs:
 - name: wiki
 port: 80
YAML
kubectl apply -f routes.yaml
kubectl get httproute -n web

# 4 - did the Gateway accept them?
kubectl describe httproute shop -n web | grep -A20 'Status:'

Look for Accepted: True and ResolvedRefs: True under the parent traefik-gateway. The route tells you if something is wrong - Ingress never did.

Lab 17 - Kustomize overlays

Mon, 01 Jan 0001 00:00:00 +0000

Lab 17 - Kustomize overlays

Goal: one base, two environments, no copy-paste. Chapter: Kustomize

The base

# 1 -
mkdir -p ~/kust/base ~/kust/overlays/dev ~/kust/overlays/prod
cd ~/kust
kubectl create deploy web --image nginx:1.27 --dry-run=client -o yaml > base/deployment.yaml
kubectl create svc clusterip web --tcp 80:80 --dry-run=client -o yaml > base/service.yaml
cat > base/kustomization.yaml <<'YAML'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
 - deployment.yaml
 - service.yaml
YAML
kubectl kustomize base

kubectl kustomize only renders - nothing is sent to the cluster.

Lab Extra - 01 - External access from your PC

Mon, 01 Jan 0001 00:00:00 +0000

Lab Extra - 01 - External access from your PC

Goal: open http://red.app.internal, http://blue.app.internal and http://green.app.internal in the browser on your Windows PC and land on three different applications in the cluster - through an edge proxy on support01, a MetalLB load-balancer address and the Gateway API. Chapters: Services · Helm · Gateway API

Info

Homework. This lab does not fit into the course day - plan 2-3 hours. It needs Traefik, the Gateway API CRDs and the Gateway traefik-gateway from Lab 14 and Lab 16, and nothing from later days. Later labs do not depend on it: NodePort 30080 keeps working whatever you do here.

Lab 18 - Rolling updates and rollbacks

Mon, 01 Jan 0001 00:00:00 +0000

Lab 18 - Rolling updates and rollbacks

Goal: roll forward, break it, roll back - and choose a strategy. Chapter: Rolling updates and rollbacks

# 1 - a Deployment with 4 replicas
cd ~
kubectl create deploy web --image nginx:1.27 --replicas 4
kubectl annotate deploy web kubernetes.io/change-cause="initial 1.27"
kubectl rollout status deploy/web

# 2 - roll forward, watch it happen (second terminal: kubectl get pods -l app=web -w)
kubectl set image deploy/web nginx=nginx:1.28
kubectl annotate deploy web kubernetes.io/change-cause="upgrade to 1.28" --overwrite
kubectl rollout status deploy/web
kubectl get rs -l app=web
kubectl rollout history deploy/web

Two ReplicaSets: old at 0, new at 4. CHANGE-CAUSE makes the history readable.

Lab 19 - Requests, limits and quotas

Mon, 01 Jan 0001 00:00:00 +0000

Lab 19 - Requests, limits and quotas

Goal: install metrics-server, see throttling and OOM kills, and put guard rails on a namespace. Chapter: Requests, limits and quotas

Install metrics-server

# 1 -
kubectl top nodes # error: Metrics API not available
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.0/components.yaml
kubectl patch deploy metrics-server -n kube-system --type=json \
 -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl rollout status deploy/metrics-server -n kube-system
sleep 30; kubectl top nodes

--kubelet-insecure-tls: kubeadm kubelets use self-signed serving certificates. Fine in a lab; in production sign them (serverTLSBootstrap).

Lab 20 - Autoscaling with HPA

Mon, 01 Jan 0001 00:00:00 +0000

Lab 20 - Autoscaling with HPA

Goal: put load on an application and watch it scale out - and back in. Chapter: Autoscaling with HPA

Requires metrics-server from Lab 19.

# 1 - a CPU-hungry app WITH a CPU request
cd ~
kubectl create deploy php-apache --image registry.k8s.io/hpa-example
kubectl set resources deploy php-apache --requests=cpu=200m --limits=cpu=500m
kubectl expose deploy php-apache --port 80
kubectl rollout status deploy/php-apache

# 2 - the HPA: target 50 % of the request, 1-6 replicas, fast scale-down for the lab
cat > hpa.yaml <<'YAML'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
 name: php-apache
spec:
 scaleTargetRef:
 apiVersion: apps/v1
 kind: Deployment
 name: php-apache
 minReplicas: 1
 maxReplicas: 6
 metrics:
 - type: Resource
 resource:
 name: cpu
 target:
 type: Utilization
 averageUtilization: 50
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 30
YAML
kubectl apply -f hpa.yaml
kubectl get hpa php-apache # TARGETS cpu: 0%/50% (may show <unknown> for ~30 s)

# 3 - load (in a second terminal, leave it running)
kubectl run load --image busybox:1.36 --restart=Never -- \
 sh -c 'while true; do wget -q -O- http://php-apache; done'

# 4 - watch it scale out
kubectl get hpa php-apache -w # Ctrl-C after a few minutes
kubectl get deploy php-apache
kubectl describe hpa php-apache | tail -8

TARGETS jumps to 200-300 %, REPLICAS climbs until the average is back near 50 %. The events show each decision: New size: 4; reason: cpu resource utilization (percentage of request) above target.

Lab 21 - Scheduling and node maintenance

Mon, 01 Jan 0001 00:00:00 +0000

Lab 21 - Scheduling and node maintenance

Goal: reserve a node for a workload, require specific hardware, and take a node out for maintenance without downtime. Chapter: Scheduling and node operations

Tolerations permit

# 1 - reserve worker03 for "database" workloads
kubectl taint node worker03 workload=db:NoSchedule

cd ~
cat > tolerant.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
 name: tolerant
spec:
 tolerations:
 - key: workload
 operator: Equal
 value: db
 effect: NoSchedule
 containers:
 - name: box
 image: busybox:1.36
 command: ["sleep", "3600"]
YAML
kubectl apply -f tolerant.yaml
kubectl get pod tolerant -o wide

Look at which node it picked. It probably is not worker03. The toleration said “I can live with that taint”, not “put me there” - the scheduler was free to choose any node and the empty one was not necessarily the best fit.

Lab 22 - Health checks

Mon, 01 Jan 0001 00:00:00 +0000

Lab 22 - Health checks

Goal: see what each probe does when it fails - and how readiness protects a rollout. Chapter: Startup, readiness and liveness probes

Readiness: no traffic, no restart

# 1 - three replicas behind a Service
cd ~
kubectl create deploy web --image nginx:1.27 --replicas 3
kubectl expose deploy web --port 80
kubectl rollout status deploy/web

# 2 - add a readiness probe on a file we control
kubectl patch deploy web -p '{"spec":{"template":{"spec":{"containers":[{"name":"nginx","readinessProbe":{"httpGet":{"path":"/ready.html","port":80},"periodSeconds":3}}]}}}}'
kubectl get pods -l app=web -w # Ctrl-C: new Pods Running but 0/1 READY

The rollout is stuck, safely: new Pods never become ready, so the old ones keep serving.

Lab 23 - Logs, events and troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Lab 23 - Logs, events and troubleshooting

Infodiscovery

Goal: read logs and events fluently, then diagnose five broken Pods from cluster output alone. Chapters: Logs and events · Troubleshooting

Warm-up: logs and events

# setup - a chatty Deployment with two containers per Pod
cd ~
cat > chatty.yaml <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
 name: chatty
spec:
 replicas: 2
 selector:
 matchLabels: { app: chatty }
 template:
 metadata:
 labels: { app: chatty }
 spec:
 containers:
 - name: app
 image: busybox:1.36
 command: ["sh", "-c", "i=0; while true; do i=$((i+1)); echo \"app line $i\"; sleep 2; done"]
 - name: sidecar
 image: busybox:1.36
 command: ["sh", "-c", "while true; do echo sidecar alive; sleep 5; done"]
YAML
kubectl apply -f chatty.yaml

Task L1. Show the last 5 lines of the app container of all chatty Pods, with the Pod name in front of each line.

Lab 24 - Ephemeral containers and kubectl debug

Mon, 01 Jan 0001 00:00:00 +0000

Lab 24 - Ephemeral containers and kubectl debug

Goal: debug a Pod that has no shell, a Pod that crashes at start, and a node. Chapter: Troubleshooting and kubectl debug

A Pod without tools

# 1 - a minimal image: no shell, nothing
kubectl run minimal --image registry.k8s.io/pause:3.10
kubectl exec -it minimal -- sh # fails: no sh in the image

# 2 - bring your own tools as an ephemeral container
kubectl debug -it minimal --image busybox:1.36 --target minimal

Inside the debug container:

Lab 25 - Troubleshooting challenge

Mon, 01 Jan 0001 00:00:00 +0000

Lab 25 - Troubleshooting challenge

Infodiscover

Goal: a “production” namespace with several faults. Make it work using only what you learned in Days 1-3. Chapter: Troubleshooting and kubectl debug

The scenario

The shop team deployed to namespace challenge and went home. Nothing works. Your definition of done:

frontend Deployment: 3/3 ready.
worker Deployment: 1/1 ready.
curl -H 'Host: shop.k8s.lab' http://worker01:30080 returns the nginx page (Traefik + Gateway from Day 2).
kubectl get events -n challenge --types=Warning shows nothing new.

Set it up - do not read the manifest, that is cheating:

Lab 26 - Users, certificates and kubeconfig

Mon, 01 Jan 0001 00:00:00 +0000

Lab 26 - Users, certificates and kubeconfig

Goal: create user jane (group dev-team) with a certificate signed by the cluster, and log in as her. Chapter: Authentication and authorization

# 1 - who am I now?
kubectl auth whoami

kubernetes-admin, group kubeadm:cluster-admins - the certificate in your kubeconfig.

Jane asks for a certificate

# 2 - private key and CSR: CN = user name, O = group
mkdir -p ~/jane && cd ~/jane
openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -out jane.csr -subj "/CN=jane/O=dev-team"

# 3 - send it to the cluster
cat > csr.yaml <<YAML
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
 name: jane
spec:
 request: $(base64 -w0 jane.csr)
 signerName: kubernetes.io/kube-apiserver-client
 expirationSeconds: 86400
 usages: ["client auth"]
YAML
kubectl apply -f csr.yaml
kubectl get csr jane # Pending

The admin approves

# 4 -
kubectl certificate approve jane
kubectl get csr jane # Approved,Issued
kubectl get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt
openssl x509 -in jane.crt -noout -subject -issuer -enddate

Issuer = the cluster CA, valid 24 h - because we asked for expirationSeconds: 86400.

Lab 27 - RBAC

Mon, 01 Jan 0001 00:00:00 +0000

Lab 27 - RBAC

Goal: give Jane exactly what she needs in dev - and nothing else. Chapter: RBAC and least privilege

Requires user jane from Lab 26.

# 1 - something to look at
kubectl create deploy web --image nginx:1.27 --replicas 2 -n dev
kubectl --context jane get pods # Forbidden

# 2 - a Role: read Pods and their logs
kubectl create role pod-reader -n dev \
 --verb=get,list,watch --resource=pods,pods/log
kubectl create rolebinding jane-pod-reader -n dev \
 --role=pod-reader --user=jane
kubectl --context jane get pods # works
kubectl --context jane logs deploy/web --tail=2
kubectl --context jane delete pod --all # Forbidden
kubectl --context jane get deploy # Forbidden - only pods

# 3 - the whole list, from the admin side
kubectl auth can-i --list -n dev --as jane

Groups and built-in roles

# 4 - dev-team may edit in dev (built-in ClusterRole "edit", scoped by RoleBinding)
kubectl create rolebinding dev-team-edit -n dev --clusterrole=edit --group=dev-team
kubectl --context jane get deploy # works now - via her group
kubectl --context jane scale deploy web --replicas 3
kubectl --context jane get secrets # works - edit includes secrets!
kubectl --context jane get pods -n default # Forbidden - other namespace

edit includes reading Secrets and pods/exec. Convenient - and a lot of power. Is that what the team needs?

Lab 28 - Service accounts

Mon, 01 Jan 0001 00:00:00 +0000

Lab 28 - Service accounts

Goal: see the token every Pod carries, call the API with it, and give an application exactly the rights it needs. Chapter: Service accounts

# 1 - a Pod with curl, running as the default ServiceAccount
kubectl run api-client -n dev --image curlimages/curl:8.10.1 --command -- sleep 3600
kubectl get pod api-client -n dev -o jsonpath='{.spec.serviceAccountName}{"\n"}' # default
kubectl exec -n dev api-client -- ls /var/run/secrets/kubernetes.io/serviceaccount

# 2 - what is in the token? (a JWT: header.payload.signature)
kubectl exec -n dev api-client -- cat /var/run/secrets/kubernetes.io/serviceaccount/token \
 | cut -d. -f2 | base64 -d 2>/dev/null; echo

Find sub (system:serviceaccount:dev:default), aud, exp (about 1 hour) and the Pod name - the token is bound to this Pod.

Lab 29 - Workload hardening

Mon, 01 Jan 0001 00:00:00 +0000

Lab 29 - Workload hardening

Goal: take a default container apart, then run the same web server hardened: non-root, read-only, no capabilities. Chapter: Security context and container hardening

What a default container can do

# 1 -
kubectl create ns hard
kubectl run plain -n hard --image nginx:1.27
kubectl exec -n hard plain -- id # uid=0(root)
kubectl exec -n hard plain -- sh -c 'echo hacked > /usr/share/nginx/html/index.html; echo written'
kubectl exec -n hard plain -- grep Cap /proc/1/status

Root, writable filesystem, a set of capabilities (CapEff not zero). If an attacker gets code execution, they can rewrite your website.

Lab 30 - Pod Security Admission

Mon, 01 Jan 0001 00:00:00 +0000

Lab 30 - Pod Security Admission

Goal: roll out Pod Security Standards on a namespace the safe way: warn first, enforce later. Chapter: Pod Security Standards and Admission

Uses namespace hard and hardened.yaml from Lab 29.

# 1 - which namespaces have a level today?
kubectl get ns -L pod-security.kubernetes.io/enforce,pod-security.kubernetes.io/warn

Nothing - every namespace allows privileged Pods.

# 2 - a privileged Pod, allowed today
kubectl run root-shell -n dev --image busybox:1.36 --restart=Never \
 --overrides='{"spec":{"hostPID":true,"containers":[{"name":"root-shell","image":"busybox:1.36","command":["sleep","3600"],"securityContext":{"privileged":true}}]}}'
kubectl get pod root-shell -n dev
kubectl delete pod root-shell -n dev

hostPID + privileged = full control over the node. Nothing stopped it.

Lab 31 - Admission policies

Mon, 01 Jan 0001 00:00:00 +0000

Lab 31 - Admission policies

Goal: enforce your own rules with a built-in ValidatingAdmissionPolicy - no extra software. Chapter: Admission policies

Rule 1: no `latest`

# 1 -
cd ~
cat > vap-latest.yaml <<'YAML'
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
 name: no-latest-tag
spec:
 failurePolicy: Fail
 matchConstraints:
 resourceRules:
 - apiGroups: ["apps"]
 apiVersions: ["v1"]
 operations: ["CREATE", "UPDATE"]
 resources: ["deployments"]
 validations:
 - expression: >-
 object.spec.template.spec.containers.all(c,
 c.image.contains(':') && !c.image.endsWith(':latest'))
 message: "images must have an explicit tag other than latest"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
 name: no-latest-tag
spec:
 policyName: no-latest-tag
 validationActions: ["Warn"]
 matchResources:
 namespaceSelector:
 matchLabels:
 policy: enforced
YAML
kubectl apply -f vap-latest.yaml
kubectl create ns policy-demo
kubectl label ns policy-demo policy=enforced

# 2 - Warn mode
kubectl create deploy a -n policy-demo --image nginx
kubectl create deploy b -n policy-demo --image nginx:1.27

a is created with a warning; b without. Warn first - see who would be hit.

Lab 32 - Network policies with Calico

Mon, 01 Jan 0001 00:00:00 +0000

Lab 32 - Network policies with Calico

Goal: get to know the cluster’s CNI, then segment a three-tier app: only frontend -> backend -> db, nothing else. Chapter: Network policies and segmentation

Calico, up close

# 1 - the operator's view
kubectl get tigerastatus
kubectl get installation default -o yaml | grep -A8 calicoNetwork
kubectl get pods -n calico-system -o wide

# 2 - Calico resources, through the Calico API server
kubectl api-resources --api-group=projectcalico.org
kubectl get ippools -o wide

One IP pool 10.244.0.0/16, VXLAN encapsulation - the overlay you saw on Day 2.

Lab 33 - Image scanning, encryption at rest, audit log

Mon, 01 Jan 0001 00:00:00 +0000

Lab 33 - Image scanning, encryption at rest, audit log

Goal: find vulnerabilities before they run, make Secrets unreadable in etcd, and record who does what. Chapter: Images, secrets and audit

Part 1 - Trivy (support01)

# 1 - install from the vendor's apt repository
sudo apt-get install -y wget gnupg
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key \
 | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" \
 | sudo tee /etc/apt/sources.list.d/trivy.list
sudo apt-get update
apt-cache policy trivy | head -5
sudo apt-get install -y trivy
trivy --version

Warning

Check the version: v0.69.4 was a malicious release (March 2026 supply-chain attack). Your trainer tells you the version to pin (sudo apt-get install trivy=<version>).

Lab 34 - Security review challenge

Mon, 01 Jan 0001 00:00:00 +0000

Lab 34 - Security review challenge

Infodiscover

Goal: a team asks for a security review before go-live. Find the problems, fix them, and prove each fix. Chapters: Day 4 - all of them.

The application

cd ~
cat > payments.yaml <<'YAML'
apiVersion: v1
kind: Namespace
metadata: { name: payments }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: payments-app, namespace: payments }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: { name: payments-app-admin }
subjects: [{ kind: ServiceAccount, name: payments-app, namespace: payments }]
roleRef: { kind: ClusterRole, name: cluster-admin, apiGroup: rbac.authorization.k8s.io }
---
apiVersion: v1
kind: ConfigMap
metadata: { name: payments-cfg, namespace: payments }
data:
 DB_HOST: db
 DB_PASSWORD: "Summer2026!"
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: api, namespace: payments }
spec:
 replicas: 2
 selector: { matchLabels: { app: api } }
 template:
 metadata: { labels: { app: api } }
 spec:
 serviceAccountName: payments-app
 containers:
 - name: api
 image: nginx:latest
 envFrom: [{ configMapRef: { name: payments-cfg } }]
 securityContext: { privileged: true }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: db, namespace: payments }
spec:
 replicas: 1
 selector: { matchLabels: { app: db } }
 template:
 metadata: { labels: { app: db } }
 spec:
 containers:
 - name: db
 image: nginx:1.27
---
apiVersion: v1
kind: Service
metadata: { name: db, namespace: payments }
spec:
 selector: { app: db }
 ports: [{ port: 80 }]
YAML
kubectl apply -f payments.yaml

(db is a stand-in - nginx on port 80 plays the database.)

Lab 35 - Static and dynamic storage

Mon, 01 Jan 0001 00:00:00 +0000

Lab 35 - Static and dynamic storage

Goal: replace hand-written PersistentVolumes with self-service: a CSI driver and a StorageClass. Chapter: PV, PVC and StorageClasses

On Day 2 (Lab 10) you wrote three static NFS PVs by hand. Check they are still there:

kubectl get pv # pv-storage1..3

Dynamic provisioning with the NFS CSI driver

Everything so far was static: you wrote a PV for every share, by hand. That does not scale - ten teams asking for volumes means ten PVs and a ticket each. A CSI driver plus a StorageClass turns that into self-service: the PVC asks, the driver creates.

Lab 36 - Running a stateful application

Mon, 01 Jan 0001 00:00:00 +0000

Lab 36 - Running a stateful application

Goal: run Redis with persistent storage the production way - dynamic volumes, spread replicas, a disruption budget - then grow its disk and survive a node drain. Chapter: Stateful workloads

Requires the nfs-csi StorageClass from Lab 35.

# 1 -
kubectl create ns state
kubectl config set-context --current --namespace=state
cd ~
cat > redis.yaml <<'YAML'
apiVersion: v1
kind: Service
metadata:
 name: redis
spec:
 clusterIP: None
 selector: { app: redis }
 ports: [{ port: 6379 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: redis
spec:
 serviceName: redis
 replicas: 2
 selector:
 matchLabels: { app: redis }
 template:
 metadata:
 labels: { app: redis }
 spec:
 topologySpreadConstraints:
 - maxSkew: 1
 topologyKey: kubernetes.io/hostname
 whenUnsatisfiable: DoNotSchedule
 labelSelector:
 matchLabels: { app: redis }
 containers:
 - name: redis
 image: redis:7.4
 args: ["--appendonly", "yes"]
 ports: [{ containerPort: 6379 }]
 readinessProbe:
 exec: { command: ["redis-cli", "ping"] }
 resources:
 requests: { cpu: "50m", memory: "64Mi" }
 limits: { memory: "256Mi" }
 volumeMounts:
 - { name: data, mountPath: /data }
 volumeClaimTemplates:
 - metadata:
 name: data
 spec:
 accessModes: ["ReadWriteOnce"]
 storageClassName: nfs-csi
 resources:
 requests:
 storage: 200Mi
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
 name: redis
spec:
 maxUnavailable: 1
 selector:
 matchLabels: { app: redis }
YAML
kubectl apply -f redis.yaml
kubectl rollout status sts/redis
kubectl get pods,pvc -o wide

Two Pods on two different nodes, two PVCs created by the driver - nobody wrote a PV.

Lab 37 - Prometheus and Grafana

Mon, 01 Jan 0001 00:00:00 +0000

Lab 37 - Prometheus and Grafana

Goal: install the monitoring stack, read the cluster in Grafana, scrape your own application and write an alert. Chapter: Monitoring with Prometheus and Grafana

Install kube-prometheus-stack

# 1 -
cd ~
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
cat > kps-values.yaml <<'YAML'
prometheus:
 prometheusSpec:
 retention: 2d
 serviceMonitorSelectorNilUsesHelmValues: false # watch ALL ServiceMonitors
 ruleSelectorNilUsesHelmValues: false # and ALL PrometheusRules
# kubeadm binds these to localhost - skip them in the lab
kubeEtcd: { enabled: false }
kubeScheduler: { enabled: false }
kubeControllerManager: { enabled: false }
YAML
helm install kps prometheus-community/kube-prometheus-stack -n monitoring \
 --create-namespace -f kps-values.yaml --wait --timeout 10m
kubectl get pods -n monitoring

# 2 - what arrived?
kubectl get crd | grep monitoring.coreos.com
kubectl get ds,deploy,sts -n monitoring
kubectl get servicemonitors -n monitoring

Prometheus and Alertmanager (StatefulSets, run by the Prometheus Operator), Grafana, kube-state-metrics, node-exporter (DaemonSet) - and a dozen ServiceMonitors describing what to scrape.

Lab 38 - Production incident analysis

Mon, 01 Jan 0001 00:00:00 +0000

Lab 38 - Production incident analysis

Infodiscover

Goal: three tickets from users. Find the root cause with metrics, events and logs, mitigate, and write a short postmortem. Chapter: Monitoring with Prometheus and Grafana

Requires the monitoring stack from Lab 37.

Set the scene

cd ~
cat > incidents.yaml <<'YAML'
apiVersion: v1
kind: Namespace
metadata: { name: prod }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: checkout, namespace: prod }
spec:
 replicas: 2
 selector: { matchLabels: { app: checkout } }
 template:
 metadata: { labels: { app: checkout } }
 spec:
 containers:
 - name: checkout
 image: registry.k8s.io/hpa-example
 resources:
 requests: { cpu: "50m" }
 limits: { cpu: "50m" }
---
apiVersion: v1
kind: Service
metadata: { name: checkout, namespace: prod }
spec:
 selector: { app: checkout }
 ports: [{ port: 80 }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: reports, namespace: prod }
spec:
 replicas: 1
 selector: { matchLabels: { app: reports } }
 template:
 metadata: { labels: { app: reports } }
 spec:
 containers:
 - name: reports
 image: busybox:1.36
 command: ["sh", "-c", "x=; while true; do x=\"$x$(head -c 1000000 /dev/zero | tr '\\0' a)\"; sleep 2; done"]
 resources:
 limits: { memory: "64Mi" }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: catalog, namespace: prod }
spec:
 replicas: 2
 selector: { matchLabels: { app: catalog, track: stable } }
 template:
 metadata: { labels: { app: catalog, track: stable } }
 spec:
 containers: [{ name: web, image: "nginx:1.27" }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: catalog-test, namespace: prod }
spec:
 replicas: 1
 selector: { matchLabels: { app: catalog, track: test } }
 template:
 metadata: { labels: { app: catalog, track: test } }
 spec:
 containers:
 - name: web
 image: nginx:1.27
 command: ["sh", "-c", "rm /usr/share/nginx/html/index.html; nginx -g 'daemon off;'"]
---
apiVersion: v1
kind: Service
metadata: { name: catalog, namespace: prod }
spec:
 selector: { app: catalog }
 ports: [{ port: 80 }]
---
apiVersion: v1
kind: Pod
metadata: { name: traffic, namespace: prod }
spec:
 containers:
 - name: traffic
 image: busybox:1.36
 command: ["sh", "-c", "while true; do wget -qO- http://checkout >/dev/null; wget -qO- http://catalog >/dev/null 2>&1; done"]
YAML
kubectl apply -f incidents.yaml

Wait 5 minutes for the metrics to build up. Do not read the manifest - work from the symptoms.

Lab 39 - Backup and restore with Velero

Mon, 01 Jan 0001 00:00:00 +0000

Lab 39 - Backup and restore with Velero

Goal: back up a namespace with its volume data to S3 object storage, destroy it, and restore it. Chapter: Backup and disaster recovery

Object storage: Garage on support01

MinIO’s free edition stopped shipping images in 2025, so we use Garage - a small S3-compatible server, one container.

# 1 - config and credentials
mkdir -p ~/garage/meta ~/garage/data && cd ~/garage
cat > garage.toml <<TOML
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
replication_factor = 1
rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "$(openssl rand -hex 32)"
[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
TOML
export GARAGE_DEFAULT_ACCESS_KEY="GK$(openssl rand -hex 16)"
export GARAGE_DEFAULT_SECRET_KEY="$(openssl rand -hex 32)"
export GARAGE_DEFAULT_BUCKET=velero
printf '[default]\naws_access_key_id=%s\naws_secret_access_key=%s\n' \
 "$GARAGE_DEFAULT_ACCESS_KEY" "$GARAGE_DEFAULT_SECRET_KEY" > ~/credentials-velero

# 2 - run it
docker run -d --name garage --restart unless-stopped \
 -p 3900:3900 -p 3901:3901 \
 -v ~/garage/garage.toml:/etc/garage.toml \
 -v ~/garage/meta:/var/lib/garage/meta -v ~/garage/data:/var/lib/garage/data \
 -e GARAGE_DEFAULT_ACCESS_KEY -e GARAGE_DEFAULT_SECRET_KEY -e GARAGE_DEFAULT_BUCKET \
 dxflrs/garage:v2.3.0 /garage server --single-node --default-bucket
sleep 5
docker exec garage /garage status
docker exec garage /garage bucket list # velero

Install Velero

# 3 - the CLI (pin the version your trainer gives you; plugin 1.14.x = Velero 1.18.x)
VELERO=v1.18.4
curl -fsSL https://github.com/vmware-tanzu/velero/releases/download/$VELERO/velero-$VELERO-linux-amd64.tar.gz \
 | tar xz -C /tmp
sudo mv /tmp/velero-$VELERO-linux-amd64/velero /usr/local/bin/
velero version --client-only

# 4 - the server side (Pods reach Garage by IP - support01 is not in cluster DNS)
S3_IP=$(hostname -I | tr ' ' '\n' | grep '^192.168.128.')
echo $S3_IP
velero install \
 --provider aws \
 --plugins velero/velero-plugin-for-aws:v1.14.0 \
 --bucket velero \
 --secret-file ~/credentials-velero \
 --backup-location-config region=garage,s3ForcePathStyle=true,s3Url=http://$S3_IP:3900 \
 --use-volume-snapshots=false \
 --use-node-agent \
 --default-volumes-to-fs-backup
kubectl get pods -n velero -w # velero + one node-agent per node; Ctrl-C
velero backup-location get # PHASE Available

--use-node-agent + --default-volumes-to-fs-backup: volume data is copied file by file (Kopia) - works with any storage, including NFS.

Lab 40 - etcd snapshot and restore

Mon, 01 Jan 0001 00:00:00 +0000

Lab 40 - etcd snapshot and restore

Goal: save the complete cluster state, make a “fatal mistake”, and roll the whole cluster back to the snapshot. Chapter: Backup and disaster recovery

Warning

This lab rewinds the entire cluster. Everything created after the snapshot disappears. Trainer: take a VM snapshot of control01 before the lab.

Tools on control01

# 1 - which etcd version runs? (on support01)
kubectl get pod -n kube-system etcd-control01 -o jsonpath='{.spec.containers[0].image}{"\n"}'

# 2 - matching etcdctl / etcdutl on control01
ssh student@control01
ETCD_VER=v3.6.5 # <- the version from step 1, with "v", without "-0"
curl -fsSL https://github.com/etcd-io/etcd/releases/download/$ETCD_VER/etcd-$ETCD_VER-linux-amd64.tar.gz \
 | tar xz -C /tmp
sudo mv /tmp/etcd-$ETCD_VER-linux-amd64/etcdctl /tmp/etcd-$ETCD_VER-linux-amd64/etcdutl /usr/local/bin/
etcdctl version
alias ectl='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
ectl endpoint health
ectl member list -w table
exit

State worth saving

# 3 - on support01
kubectl create ns precious
kubectl create deploy vault -n precious --image nginx:1.27 --replicas 2
kubectl create cm before-snapshot --from-literal=state=old

Snapshot

# 4 - on control01
ssh student@control01
alias ectl='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
ectl snapshot save /root/etcd-snap.db
sudo etcdutl snapshot status /root/etcd-snap.db -w table
sudo cp /root/etcd-snap.db /tmp/ && sudo chmod 644 /tmp/etcd-snap.db
exit
scp student@control01:/tmp/etcd-snap.db ~/etcd-snap-$(date +%F).db # off the node!

The file holds every object of the cluster, including all Secrets - store it like a password.

Lab 41 - An operator: PostgreSQL

Mon, 01 Jan 0001 00:00:00 +0000

Lab 41 - An operator: PostgreSQL

Goal: install a real operator, create a database cluster with one small manifest, and watch the operator repair it. Chapter: CRDs and operators

You will use CloudNativePG, a PostgreSQL operator. Everything you have learned so far applies: what the operator creates is StatefulSet-like Pods, Services, Secrets and PVCs - objects you can already read.

Storage first - and it must be dynamic

PostgreSQL is the first workload in this course that will not run on the static NFS PVs you wrote by hand. nginx ran as root and wrote to the export root quite happily; the CloudNativePG image runs as the unprivileged postgres user (UID 26), and it cannot write into a root-owned NFS directory. The Pod starts, initdb fails on permissions, and the Pod crash-loops.

Lab 42 - End-to-end production scenario

Mon, 01 Jan 0001 00:00:00 +0000

Lab 42 - End-to-end production scenario

Infodiscover

Goal: take an application to production with everything from the five days - then survive a game day. Chapter: Production operations and best practices

Work in pairs. Build everything in one file ~/prod-shop.yaml (plus the Velero commands) so it can go to git.

The application

podinfo ghcr.io/stefanprodan/podinfo:6.7.0 - port 9898, health /healthz, readiness /readyz, metrics /metrics, message from env PODINFO_UI_MESSAGE.