Labs on Eriks training corner

Lab 01 - Workstation setup

Mon, 01 Jan 0001 00:00:00 +0000

Lab 01 - Workstation setup

Goal: turn support01 into your kubectl workstation. Cheat sheet: kubectl

support01 has Docker and the NFS exports, but deliberately no kubectl and no kubeconfig. Building your own client is the first thing you do at any customer, so you may as well do it here.

# 1 - log in to the jump host (from your PC)
ssh student@192.168.56.101 # password: linux

# 2 - who am I, and can I reach the cluster nodes?
hostname -f
ip -brief address # note your 192.168.128.0/24 address
for n in control01 worker01 worker02 worker03; do getent hosts $n; done

If a name does not resolve, note its IP now - you will need it in step 6.

Lab 02 - Docker containers

Mon, 01 Jan 0001 00:00:00 +0000

Lab 02 - Docker containers

Goal: the container lifecycle, and why the writable layer matters. Cheat sheet: Docker

# 1 - the shortest possible container
sudo docker run hello-world
sudo docker ps # nothing - it already exited
sudo docker ps -a # there it is, Exited (0)

A container lives exactly as long as its main process. hello-world prints and exits, so the container is finished before docker ps can see it. This is the same reason a Kubernetes Pod with no long-running process ends up in CrashLoopBackOff.

Lab 03 - Docker images and Dockerfile

Mon, 01 Jan 0001 00:00:00 +0000

Lab 03 - Docker images and Dockerfile

Goal: understand layers by watching the build cache work. Cheat sheet: Docker

# 1 - pull and inspect
sudo docker pull nginx:1.27
sudo docker images
sudo docker history nginx:1.27 # one line per layer, with sizes

history is the image’s build recipe read backwards. Most layers are 0 B - metadata only. The big one is the base filesystem.

# 2 - layer sharing
sudo docker pull nginx:1.27-alpine
sudo docker images | grep nginx

Compare the sizes. Alpine images are small because the base is small, not because nginx is different.

Lab 04 - Docker networking and volumes

Mon, 01 Jan 0001 00:00:00 +0000

Lab 04 - Docker networking and volumes

Goal: name resolution between containers, and data that survives. Cheat sheet: Docker

Networking

# 1 - the default networks
sudo docker network ls

# 2 - on the default bridge, name resolution does NOT work
sudo docker run -d --name db busybox sleep 3600
sudo docker run --rm busybox ping -c1 db # bad address 'db'

# 3 - a user-defined network gives you DNS by container name
sudo docker network create appnet
sudo docker run -d --name db2 --network appnet busybox sleep 3600
sudo docker run --rm --network appnet busybox ping -c1 db2 # works

This is the single most useful thing Docker networking does, and it is the concept a Kubernetes Service generalises: reach a moving target by a stable name. The difference is that Docker’s name points at one container, while a Service’s name points at whichever Pods are currently ready.

Lab 05 - First look at the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Lab 05 - First look at the cluster

Goal: see what the cluster is made of, and the difference between imperative and declarative. Cheat sheet: kubectl

# 1 - where am I connected, and as whom?
kubectl cluster-info
kubectl config get-contexts
kubectl config view --minify | grep server

Get into the habit of config get-contexts before anything destructive. Most production accidents are a command typed against the right object in the wrong cluster.

Lab 06 - Finding your way around kubectl

Mon, 01 Jan 0001 00:00:00 +0000

Lab 06 - Finding your way around kubectl

Infodiscovery

Goal: never need the internet to answer a Kubernetes question again. Chapter: Finding your way

Each task states what you should end up seeing. Try it yourself first; the box holds one working answer.

Task 1. List every kind of object this cluster knows about, and find out whether ingresses is namespaced and what its short name is.

Lab 07 - Pods

Mon, 01 Jan 0001 00:00:00 +0000

Lab 07 - Pods

Goal: generate a Pod manifest, add resources, and see what a Pod really is. Chapter: Pods

# 1 - imperative: create and throw away
kubectl run bbox --image busybox --restart=Never -- sleep 3600
kubectl get pods
kubectl delete pod bbox

--restart=Never is what makes kubectl run produce a bare Pod instead of something managed. -- sleep 3600 overrides the image’s command, because busybox would otherwise exit immediately.

# 2 - generate the manifest instead
cd ~
kubectl run bbox --image busybox --restart=Never --dry-run=client -o yaml \
 -- sleep 3600 > bbox.yaml
cat bbox.yaml

Edit bbox.yaml and add resources to the container:

Lab 08 - Pod troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Lab 08 - Pod troubleshooting

Infodiscovery

Goal: diagnose four broken Pods from cluster output alone. Chapter: Troubleshooting routine

For each case: create the mess with the command given, then answer the question before you open the box. The routine is always get -> describe -> logs.

Case 1 - it will not pull

kubectl run broken1 --image=nginx:doesnotexist
kubectl get pod broken1

Question. What state is it in, and where exactly does the cluster tell you why?

Lab 09 - Namespaces

Mon, 01 Jan 0001 00:00:00 +0000

Lab 09 - Namespaces

Goal: scope your work and stop typing -n. Chapter: Namespaces

# 1 - what exists now
kubectl get ns

Four namespaces come with every cluster. kube-system holds the control plane Pods - look but do not deploy there.

# 2 - declarative (recommended)
cd ~
cat > ns-dev.yaml <<'YAML'
apiVersion: v1
kind: Namespace
metadata:
 name: dev
YAML
kubectl apply -f ns-dev.yaml

# 3 - imperative, for the throwaway one
kubectl create ns test

# 4 - the same name in both, with no conflict
kubectl run web --image nginx:1.27 -n dev
kubectl run web --image nginx:1.27 -n test
kubectl get pods -n dev
kubectl get pods -A | grep ' web '

Names are unique per namespace per kind. That is the whole point: dev and prod can both have a web.

Lab 10 - ReplicaSets

Mon, 01 Jan 0001 00:00:00 +0000

Lab 10 - ReplicaSets

Goal: watch the reconciliation loop, and see the selector do the binding. Chapter: ReplicaSets

# 1 - a ReplicaSet by hand (the only time you will write one)
cd ~
cat > web-rs.yaml <<'YAML'
apiVersion: apps/v1
kind: ReplicaSet
metadata:
 name: web-rs
spec:
 replicas: 3
 selector:
 matchLabels:
 app: web
 template:
 metadata:
 labels:
 app: web
 spec:
 containers:
 - name: web
 image: nginx:1.27
 resources:
 requests: { cpu: "20m", memory: "32Mi" }
YAML
kubectl apply -f web-rs.yaml
kubectl get rs,pods -l app=web -o wide

Note the Pod names: web-rs- plus a hash of the template plus a random suffix. The controller owns them, which kubectl get pod <name> -o yaml | grep -A5 ownerReferences will confirm.

Lab 11 - Deployments and probes

Mon, 01 Jan 0001 00:00:00 +0000

Lab 11 - Deployments and probes

Goal: roll forward, break it, roll back - and see why the break was harmless. Chapter: Deployments

# 1 - generate, do not hand-write
cd ~
kubectl create deploy web --image nginx:1.27 --replicas 3 \
 --dry-run=client -o yaml > web-deploy.yaml
kubectl apply -f web-deploy.yaml
kubectl get deploy,rs,pods -l app=web

Three objects for one command: the Deployment you asked for, the ReplicaSet it created, and the Pods that ReplicaSet created.

Lab 12 - DaemonSets

Mon, 01 Jan 0001 00:00:00 +0000

Lab 12 - DaemonSets

Goal: one Pod per node, and the two surprises that come with it. Chapter: DaemonSets

# 1 - the cluster already runs several
kubectl get ds -A
kubectl get pods -n kube-system -o wide | grep -i -E 'proxy|calico|cilium|flannel'

kube-proxy and the CNI agent are DaemonSets: they must exist on every node, including nodes that join tomorrow. There is no replicas field anywhere - the node list is the count.

Lab 13 - Taints, tolerations and affinity

Mon, 01 Jan 0001 00:00:00 +0000

Lab 13 - Taints, tolerations and affinity

Goal: control where Pods land - and see the difference between permitted and attracted. Chapter: Scheduling

Taints repel

# 1 - a baseline: 6 pods spread over the three workers
kubectl create deploy filler --image nginx:1.27 --replicas 6
kubectl get pods -o wide | sort -k7

# 2 - taint worker03 and watch new pods avoid it
kubectl taint node worker03 disk=ssd:NoSchedule
kubectl describe node worker03 | grep -i -A2 taint
kubectl scale deploy filler --replicas 12
kubectl get pods -o wide | grep -c worker03 # unchanged

The Pods already on worker03 are still there. NoSchedule only affects placement; it does not evict anything.

Lab 14 - Services

Mon, 01 Jan 0001 00:00:00 +0000

Lab 14 - Services

Goal: reach Pods by a stable name, from inside and from outside. Chapter: Services

# 1 - something to expose
cd ~
kubectl create deploy web --image nginx:1.27 --replicas 3
kubectl get pods -l app=web -o wide # note the Pod IPs

# 2 - ClusterIP, the default
kubectl expose deploy web --port 80 --name web-svc
kubectl get svc web-svc
kubectl get endpointslices -l kubernetes.io/service-name=web-svc

The EndpointSlice is the interesting object: it holds the three Pod IPs. The Service is just a stable front; the EndpointSlice is the live membership list, maintained by a controller as Pods become ready.

Lab 15 - Gateway API

Mon, 01 Jan 0001 00:00:00 +0000

Lab 15 - Gateway API

Goal: install a Gateway controller and route two applications by host name through one port - then split traffic between two versions. Chapter: Gateway API

Gateway API resources are only data. Two things must exist first: the CRDs (they are not part of Kubernetes) and a controller that reads them.

Install the CRDs

# 1 - the standard channel: GatewayClass, Gateway, HTTPRoute
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.3.0/standard-install.yaml
kubectl get crd | grep gateway.networking
kubectl api-resources --api-group=gateway.networking.k8s.io

You have just extended the API server with new kinds - exactly the CRD mechanism covered on Day 3, met here in the wild. Check the releases page if a newer version is out.

Lab 16 - Node operations

Mon, 01 Jan 0001 00:00:00 +0000

Lab 16 - Node operations

Infodiscovery

Goal: take a node out for maintenance and put it back, safely. Chapter: Nodes

Everything here is taints underneath - see Lab 13 if you want to see the same effects expressed by hand.

# setup - something to move around
kubectl create deploy filler --image nginx:1.27 --replicas 6
kubectl get pods -o wide | sort -k7

Task 1. Stop new Pods from being scheduled on worker03, without disturbing anything already running there. Expected: kubectl get nodes shows Ready,SchedulingDisabled for worker03, and the Pods on it are untouched.

Lab 17 - ConfigMaps and Secrets

Mon, 01 Jan 0001 00:00:00 +0000

Lab 17 - ConfigMaps and Secrets

Goal: same image, different configuration - and learn which changes need a restart. Chapter: ConfigMaps and Secrets

# 1 - from literals and from a file
cd ~
printf 'color=blue\nretries=3\n' > app.properties
kubectl create cm app-cfg \
 --from-literal=DATACENTER=dc01 \
 --from-file=app.properties
kubectl get cm app-cfg -o yaml

--from-literal makes one key; --from-file makes a key named after the file whose value is the whole file. Both live in the same data map.

Lab 18 - Persistent storage on NFS

Mon, 01 Jan 0001 00:00:00 +0000

Lab 18 - Persistent storage on NFS

Goal: give a Pod storage that survives it - with no StorageClass and no CSI driver, the way an on-premise cluster really looks. Chapter: Persistent storage

support01 exports /storage1, /storage2 and /storage3 over NFS. There is no dynamic provisioning in this cluster, so you write the PersistentVolume.

Prepare the nodes

# 1 - every node that mounts NFS needs the client tools
for n in worker01 worker02 worker03; do
 ssh student@$n 'dpkg -l nfs-common >/dev/null 2>&1 && echo "$(hostname): ok" || sudo apt-get install -y nfs-common'
done

# 2 - prove the export works from a node, outside Kubernetes first
ssh student@worker01 'sudo mkdir -p /mnt/t && sudo mount -t nfs support01:/storage1 /mnt/t && df -h /mnt/t && sudo umount /mnt/t'

Always test the mount by hand before blaming Kubernetes. A Pod stuck in ContainerCreating with an NFS error is usually a missing nfs-common, a wrong export path, or export permissions.

Lab 19 - StatefulSets

Mon, 01 Jan 0001 00:00:00 +0000

Lab 19 - StatefulSets

Goal: stable names, per-Pod storage, ordered start - and the clean-up surprise. Chapter: StatefulSets

This lab needs the three PVs from Lab 18. Check they are Available:

kubectl get pv
kubectl patch pv pv-storage1 -p '{"spec":{"claimRef":null}}' 2>/dev/null
kubectl get pv

Headless Service first

# 1 -
cd ~
cat > web-headless.yaml <<'YAML'
apiVersion: v1
kind: Service
metadata:
 name: web-headless
spec:
 clusterIP: None # headless - no virtual IP
 selector:
 app: web-sts
 ports:
 - port: 80
YAML
kubectl apply -f web-headless.yaml
kubectl get svc web-headless # CLUSTER-IP is "None"

A normal Service hides the Pods behind one IP. A headless Service does the opposite: DNS returns the individual Pod addresses, so each replica is addressable by name. A StatefulSet requires one.

Lab 20 - Helm releases

Mon, 01 Jan 0001 00:00:00 +0000

Lab 20 - Helm releases

Goal: install, configure, upgrade and roll back somebody else’s chart. Chapter: Helm basics

Helm should already be installed from Lab 14. If not:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod +x get_helm.sh && sudo ./get_helm.sh && helm version

# 1 - a repository
helm repo add podinfo https://stefanprodan.github.io/podinfo
helm repo update
helm repo list
helm search repo podinfo
helm search repo podinfo --versions | head -5

A repository is just a web server with an index.yaml. helm repo update refetches that index - Helm does not go online for anything else until you install.

Lab 21 - Writing your own chart

Mon, 01 Jan 0001 00:00:00 +0000

Lab 21 - Writing your own chart

Goal: package an application so it can be installed many times with different values. Chapter: Helm charts

# 1 - scaffold and read what you got
cd ~
helm create shop
find shop -type f | sort
cat shop/Chart.yaml
head -30 shop/values.yaml

helm create gives you a working nginx chart with far more machinery than you need. Reading it is worthwhile: it shows the conventions every public chart follows.

Lab 22 - An operator: PostgreSQL

Mon, 01 Jan 0001 00:00:00 +0000

Lab 22 - An operator: PostgreSQL

Goal: install a real operator, create a database cluster with one small manifest, and watch the operator repair it. Chapter: CRDs and operators

You will use CloudNativePG, a PostgreSQL operator. Everything you have learned so far applies: what the operator creates is StatefulSet-like Pods, Services, Secrets and PVCs - objects you can already read.

Storage first - and it must be dynamic

PostgreSQL is the first workload in this course that will not run on the static NFS PVs you wrote by hand. nginx ran as root and wrote to the export root quite happily; the CloudNativePG image runs as the unprivileged postgres user (UID 26), and it cannot write into a root-owned NFS directory. The Pod starts, initdb fails on permissions, and the Pod crash-loops.

Lab 23 - Application end to end

Mon, 01 Jan 0001 00:00:00 +0000

Lab 23 - Application end to end

Infodiscovery optional / if time permits

Goal: build, operate, break and fix a complete application using everything from the three days.

This lab gives you what, not how. Every command is in the cheat sheets, in kubectl explain, or behind --help.

Tip

Each stage ends with an answer key box. Try the stage first; open the box if you get stuck or if you want to check your solution against one that works. Each box is self-contained, so you can rejoin the lab at the start of any stage without having done the previous one your own way. If you are completely lost, jump to Catch up in one paste at the bottom.

Extended Lab A - Deploy KubeQuiz

Mon, 01 Jan 0001 00:00:00 +0000

Extended Lab A - Deploy KubeQuiz

Infochallenge optional / after the course

Chapters: CRDs and operators, Gateway API, Persistent storage Cheat sheet: kubectl

This lab lives in a git repository, together with the application it deploys and a script that grades your work. Everything runs on support01:

sudo apt-get update && sudo apt-get install -y git
git clone https://github.com/ecaha/kubequiz.git ~/kubequiz
cd ~/kubequiz
./labs/verify.sh a0

Tip

A challenge lab gives you the task and the result, not the commands. Each stage ends with a Show one answer box - open it after you have tried, not before. ./labs/verify.sh <stage> tells you PASS or FAIL per criterion, so you never have to wait for anybody.

Extended Lab B - Ship a change

Mon, 01 Jan 0001 00:00:00 +0000

Extended Lab B - Ship a change

Infochallenge optional / after the course

Prerequisite: Extended Lab A finished and still running. You also need: a GitHub account and a personal access token.

Goal: change one line of C#, and follow it all the way into the running cluster without ever typing kubectl apply.

Starting point: Extended Lab A finished, with the app running from ghcr.io/ecaha/... and outbound internet from support01 and the cluster nodes.