Extended Lab A - Deploy KubeQuiz

Extended Lab A - Deploy KubeQuiz

Infochallenge optional / after the course

Chapters: CRDs and operators, Gateway API, Persistent storage Cheat sheet: kubectl

This lab lives in a git repository, together with the application it deploys and a script that grades your work. Everything runs on support01:

sudo apt-get update && sudo apt-get install -y git
git clone https://github.com/ecaha/kubequiz.git ~/kubequiz
cd ~/kubequiz
./labs/verify.sh a0
Tip

A challenge lab gives you the task and the result, not the commands. Each stage ends with a Show one answer box - open it after you have tried, not before. ./labs/verify.sh <stage> tells you PASS or FAIL per criterion, so you never have to wait for anybody.

Goal: put a stateful, highly available application on Kubernetes, then attack it and keep it running.

You are on your own cluster and you work alone. Nothing here depends on anybody else, and you can stop and resume at any stage.

This lab assumes nothing. If you skipped earlier labs, or your environment was rebuilt, Stages 0 and 1 put everything back. If you did those labs, both stages take two minutes to confirm and you move on.

Check your work at any time - ./labs/verify.sh a0 for one stage, ./labs/verify.sh a for all of them.

What you are building:

browser ─▶ support01:30080 ─▶ worker01:30080 ─▶ Gateway data plane
                                                     │
                              ┌──────────────────────┴────────────┐
                              ▼                                   ▼
                      kubequiz-web (nginx)                 kubequiz-api
                      Blazor WebAssembly                   ASP.NET Core
                                                                  │
                                                          kubequiz-db-rw
                                                                  │
                                                  CloudNativePG: 3 PostgreSQL
                                                  instances, 1 primary

Stage 0 - Your workstation

Everything in this lab runs from support01. Your laptop only ever holds an SSH session and a browser.

Task. Get onto support01 with a working kubectl, helm and git, and put this repository in ~/kubequiz.

Done when

  • ssh student@192.168.56.101 works (password linux)
  • kubectl get nodes lists 4 nodes, all Ready
  • helm version and git --version answer
  • ~/kubequiz/labs/verify.sh exists

Check: ./labs/verify.sh a0

ssh student@192.168.56.101          # password: linux

If kubectl is missing you skipped Lab 01. Do it properly, or take the shortcut - but know that building a client and fetching a kubeconfig is a thing you will do at every customer:

sudo apt-get update && sudo apt-get install -y git
git clone https://github.com/ecaha/kubequiz.git ~/kubequiz
cd ~/kubequiz
./tools/bootstrap-workstation.sh     # kubectl, helm, kubeconfig from control01

If kubectl already works, you only need the clone:

sudo apt-get install -y git
git clone https://github.com/ecaha/kubequiz.git ~/kubequiz
cd ~/kubequiz
kubectl get nodes

Think about it. ~/.kube/config contains a client certificate that is effectively cluster-admin. What are the consequences of it sitting in a home directory on a shared jump host?

Stage 1 - Cluster prerequisites

KubeQuiz needs three things this cluster does not have out of the box: a way in from outside (Gateway API plus a controller that implements it), somewhere to store data (PersistentVolumes - there is no dynamic provisioning here), and CPU metrics for the autoscaling stage.

Task. Make all three available, and create the namespace you will work in.

Done when

  • kubectl get gatewayclass lists nginx
  • the NGINX Gateway Fabric controller is running in nginx-gateway
  • nothing else already owns NodePort 30080
  • three PersistentVolumes exist and are Available
  • kubectl top nodes returns numbers
  • namespace kubequiz exists and is your default

Check: ./labs/verify.sh a1

./tools/bootstrap-cluster.sh        # Gateway API + NGF, the PVs, metrics-server

kubectl create namespace kubequiz
kubectl config set-context --current --namespace=kubequiz

The PVs are NFS exports from support01. The worker nodes need the client side, or Pods will hang in ContainerCreating with a mount error:

for w in worker01 worker02 worker03; do
  ssh student@$w 'sudo apt-get install -y nfs-common'
done

If you did Lab 18, your PVs already exist but are probably still bound to a deleted claim. Release them:

for p in pv-storage1 pv-storage2 pv-storage3; do
  kubectl patch pv $p -p '{"spec":{"claimRef":null}}'
done
kubectl get pv

Think about it. A PersistentVolume that is Released rather than Available will never be reused, even though nothing is using it. Why does Kubernetes refuse to recycle it automatically?

Stage 2 - The operator and the database

Task. Install CloudNativePG, then create a three-instance PostgreSQL cluster named kubequiz-db from gitops/base/db/cluster.yaml. It must commit synchronously to at least one replica.

Done when

  • the CloudNativePG controller is Running in cnpg-system
  • kubectl api-resources --api-group=postgresql.cnpg.io lists clusters
  • kubectl get cluster kubequiz-db reports 3 ready instances
  • exactly one Pod carries role=primary
  • three PVCs are Bound
  • Services kubequiz-db-rw, -ro and -r exist
  • select version(); runs inside the primary

Check: ./labs/verify.sh a2

helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update
helm install cnpg cnpg/cloudnative-pg -n cnpg-system --create-namespace
kubectl -n cnpg-system rollout status deploy/cnpg-cloudnative-pg

kubectl apply -f gitops/base/db/cluster.yaml
kubectl get cluster kubequiz-db -w          # Ctrl-C when it says healthy

kubectl get pods -l cnpg.io/cluster=kubequiz-db -L role
kubectl get pvc
kubectl get svc | grep kubequiz-db
kubectl exec -it kubequiz-db-1 -- psql -c 'select version();'

If the Pods stay Pending, the PVCs found no PV - go back to Stage 1. If they stay ContainerCreating, the workers are missing nfs-common.

Think about it. You installed a Deployment and some CRDs, then wrote thirty lines of YAML. Count the objects that appeared. Which of them would you have got wrong writing them by hand?

Stage 3 - Credentials

Task. The application expects the database connection as five environment variables: PGHOST, PGPORT, PGDATABASE, PGUSER, PGPASSWORD. The operator has already generated a password - find it and build a Secret named kubequiz-db-env from it.

Do not invent a password, do not read one from a file in this repository, and do not commit one anywhere.

Done when

  • Secret kubequiz-db-env has all five keys
  • PGPASSWORD equals the password in the operator’s kubequiz-db-app Secret
  • PGHOST points at the read-write Service, not at a Pod

Check: ./labs/verify.sh a3

kubectl get secret kubequiz-db-app -o jsonpath='{.data.password}' | base64 -d; echo

kubectl create secret generic kubequiz-db-env \
  --from-literal=PGHOST=kubequiz-db-rw \
  --from-literal=PGPORT=5432 \
  --from-literal=PGDATABASE=kubequiz \
  --from-literal=PGUSER=kubequiz \
  --from-literal=PGPASSWORD="$(kubectl get secret kubequiz-db-app -o jsonpath='{.data.password}' | base64 -d)"

Think about it. Why kubequiz-db-rw and not kubequiz-db-1? What breaks in Stage 7 if you get this wrong - and would you notice before then?

Stage 4 - Deploy the application

Task. Deploy the API and the frontend from gitops/overlays/lab. The database schema must be created and seeded before the API starts serving.

The images are already published at ghcr.io/ecaha/kubequiz-api and ghcr.io/ecaha/kubequiz-web, so there is nothing to build. Your nodes need to be able to reach ghcr.io.

Done when

  • Job kubequiz-db-init is Completed
  • Deployments kubequiz-api and kubequiz-web are Ready
  • GET /api/whoami returns 200 through the Service and names the Pod
  • GET /api/questions/active returns a question with four options

Check: ./labs/verify.sh a4

kubectl apply -k gitops/overlays/lab

kubectl get jobs
kubectl rollout status deploy/kubequiz-api
kubectl rollout status deploy/kubequiz-web

kubectl get --raw \
  /api/v1/namespaces/kubequiz/services/kubequiz-api:http/proxy/api/whoami; echo
Warning

If it does not come up: kubectl describe pod, then kubectl logs. The two most common failures here are an image that cannot be pulled, and a Secret key spelled differently from what the container expects.

Stage 5 - Reach it from your browser

Your laptop can route to 192.168.56.101 and nothing else. The Gateway’s data plane listens on NodePort 30080 on the cluster nodes, which are on 192.168.128.0/24. Bridging that gap is your job.

Task. Publish the app at kubequiz.k8s.lab, make support01 relay TCP 30080 to a cluster node, and load the page in the browser on your laptop. Vote once.

Done when

  • Gateway kubequiz-gw reports PROGRAMMED=True
  • an HTTPRoute sends /api and /hub to kubequiz-api and / to kubequiz-web
  • on support01: curl -H 'Host: kubequiz.k8s.lab' http://worker01:30080/api/about works
  • the same request against http://192.168.56.101:30080 also works
  • http://kubequiz.k8s.lab:30080 renders in your laptop’s browser, and a vote registers on the Results page

Check: ./labs/verify.sh a5

kubectl get gateway kubequiz-gw              # PROGRAMMED must be True
kubectl get httproute kubequiz -o wide

# NGF builds one data plane per Gateway, in the Gateway's namespace
kubectl get pods,svc -l gateway.networking.k8s.io/gateway-name=kubequiz-gw

./tools/setup-relay.sh                       # socat as a systemd unit
systemctl status k8s-relay-30080 --no-pager

If the Gateway never becomes Programmed, or its Service gets a random port instead of 30080, something else already owns that port - almost always the web-gw Gateway left over from the Gateway API lab:

kubectl get svc -A | grep 30080
kubectl delete gateway web-gw -n default

Then on your laptop, add to the hosts file (C:\Windows\System32\drivers\etc\hosts, Notepad as Administrator):

192.168.56.101  kubequiz.k8s.lab
192.168.56.101  argocd.k8s.lab

and browse to http://kubequiz.k8s.lab:30080.

Cannot edit the hosts file on a locked-down laptop? Then from your laptop’s terminal use an SSH tunnel instead, and browse http://localhost:30080:

ssh -L 30080:worker01:30080 student@192.168.56.101

Think about it. The relay points at worker01. Does the site break if the Gateway’s data-plane Pod happens to run on worker03? Explain it in terms of what a NodePort actually is.

Stage 6 - Probes

The API can be told to fail either health endpoint on demand: touch /tmp/unready makes readiness fail, touch /tmp/unhealthy makes liveness fail.

Task. Scale the API to 3 replicas. Using only those marker files, first take one Pod out of service without restarting it, then cause a different Pod to be restarted.

Done when

  • you can show a Pod that is Running, 0 restarts, and absent from kubectl get endpoints kubequiz-api
  • you can show a Pod whose restart count went up, and quote the event that explains why
  • all Pods are healthy and back in Endpoints at the end

Check: ./labs/verify.sh a6

kubectl scale deploy/kubequiz-api --replicas=3
kubectl get endpoints kubequiz-api

P=$(kubectl get pod -l app.kubernetes.io/component=api -o name | head -1)
kubectl exec $P -- touch /tmp/unready
sleep 15
kubectl get endpoints kubequiz-api            # one address fewer
kubectl get $P                                # Running, 0 restarts
kubectl describe $P | grep -A2 Readiness
kubectl exec $P -- rm /tmp/unready

P2=$(kubectl get pod -l app.kubernetes.io/component=api -o name | tail -1)
kubectl exec $P2 -- touch /tmp/unhealthy
kubectl get pods -w                           # restart count climbs

The restart wipes the container filesystem, so the marker disappears with it.

Think about it. A Pod that is slow to start and a Pod that has hung look identical to a liveness probe. Which third probe exists to tell them apart, and what would happen here without it?

Stage 7 - Survive a database failover

One person cannot fill a room, so the API generates its own load: POST /api/simulate?votes=5 casts five random votes.

Task. With votes arriving continuously, delete the primary PostgreSQL Pod. Report three numbers: how long writes failed, how many votes were lost, and which instance is primary afterwards.

Done when

  • tools/voters.sh is running and printing 200
  • you deleted the Pod labelled role=primary
  • a replica was promoted and the old primary rejoined as a replica
  • the vote total after the failover is greater than or equal to the total before it - nothing was lost
  • you can state the outage window in seconds from the voter output

Check: ./labs/verify.sh a7

# terminal 1
./tools/voters.sh

# terminal 2
kubectl get pods -l cnpg.io/cluster=kubequiz-db -L role
kubectl delete pod "$(kubectl get pods -l cnpg.io/cluster=kubequiz-db,role=primary -o name)"
kubectl get pods -l cnpg.io/cluster=kubequiz-db -L role -w
kubectl describe cluster kubequiz-db | tail -20

In terminal 1 you will see a handful of non-200 lines, then 200s again. That gap is your answer.

Do it again, this time deleting a replica. The service does not notice. Why is that the correct behaviour, and what did the operator do in the background?

Optional. Remove spec.postgresql.synchronous from the Cluster and repeat the first experiment. The outage gets shorter. What did you trade away for it?

Stage 8 - Scale out, and find the bug

Task. Make the API scale automatically under load. Then open the Results page in two browser tabs, vote, and watch carefully.

Done when

  • the HPA scales kubequiz-api beyond its minimum while /api/load is hit
  • replicas come back down afterwards
  • with more than one API replica you can demonstrate that one of the two Results tabs stops updating live
  • you can explain the cause, and fix it with gitops/overlays/prod

Check: ./labs/verify.sh a8

kubectl get hpa kubequiz-api
kubectl top pods

kubectl run load --rm -it --image=curlimages/curl --restart=Never -- sh -c \
  'while true; do curl -s -o /dev/null http://kubequiz-api/api/load?ms=400; done'
kubectl get hpa kubequiz-api -w

The live-results bug: SignalR pushes only to the browsers connected to that replica. The WebSocket connections were load-balanced across Pods, so the fan-out is incomplete. A shared backplane fixes it:

kubectl apply -k gitops/overlays/prod     # adds Redis, sets REDIS_HOST

Think about it. Kubernetes scaled a broken application perfectly happily. Whose responsibility was the bug, and what does that tell you about “just add replicas”?

Stage 9 - Survive node maintenance

Task. Take a worker node out for maintenance without the application ever becoming unreachable.

Done when

  • a request loop against the Gateway runs throughout with no failures
  • kubectl drain completes on the node hosting most of the API Pods
  • you can show the PodDisruptionBudget that made the eviction wait
  • the node is back in service and Pods have rescheduled onto it

Check: ./labs/verify.sh a9

kubectl get pods -o wide
kubectl get pdb kubequiz-api

# terminal 1
while true; do curl -s -o /dev/null -w '%{http_code} ' \
  -H 'Host: kubequiz.k8s.lab' http://worker01:30080/api/about; sleep .3; done

# terminal 2
kubectl drain worker02 --ignore-daemonsets --delete-emptydir-data
kubectl get pods -o wide
kubectl uncordon worker02

Draining the node your PostgreSQL primary sits on is also worth watching - the operator fails over rather than letting the eviction hang.

Then: scale the API to 1 replica and drain its node. What does the PDB do now, and is that the behaviour you want at 3am?

Stage 10 - Lock it down

Task. Only the API may reach PostgreSQL. Prove it both ways.

Done when

  • a throwaway Pod in the same namespace cannot open TCP 5432 to kubequiz-db-rw
  • the API Pod still can, and the site still works
  • you can name the object that enforces this and the field that selects its targets

Check: ./labs/verify.sh a10

kubectl apply -f gitops/base/api/networkpolicy.yaml
kubectl get networkpolicy

kubectl run probe --rm -it --image=busybox:1.36 --restart=Never -- \
  sh -c 'nc -z -w3 kubequiz-db-rw 5432; echo rc=$?'      # should fail

kubectl exec deploy/kubequiz-api -- sh -c 'timeout 3 nc -zv kubequiz-db-rw 5432'

If the throwaway Pod connects anyway, your CNI is not enforcing NetworkPolicy. Find out which one is installed - that answer matters more than the lab does.

Finished

./labs/verify.sh a

Leave the cluster running. Extended Lab B picks up exactly here and replaces ghcr.io/ecaha/... with images you built yourself.