Lab 23 - Application end to end

Lab 23 - Application end to end

Infodiscovery optional / if time permits

Goal: build, operate, break and fix a complete application using everything from the three days.

This lab gives you what, not how. Every command is in the cheat sheets, in kubectl explain, or behind --help.

Tip

Each stage ends with an answer key box. Try the stage first; open the box if you get stuck or if you want to check your solution against one that works. Each box is self-contained, so you can rejoin the lab at the start of any stage without having done the previous one your own way. If you are completely lost, jump to Catch up in one paste at the bottom.

Build it

  1. Create a namespace shop and make it your default.
  2. Create a ConfigMap web-content holding an index.html that says Shop v1.
  3. Create a Deployment shop-web, image nginx:1.27, 3 replicas, with:
    • CPU request 50m, memory request 64Mi, memory limit 128Mi
    • the ConfigMap mounted at /usr/share/nginx/html
    • a readiness probe on /
  4. Expose it with a ClusterIP Service.
  5. Publish it through a Gateway with an HTTPRoute for shop.k8s.lab. (Remember the data plane Service lives in the Gateway’s namespace.)
  6. Verify from support01 - curl -H 'Host: shop.k8s.lab' http://worker01:<gateway-nodeport> returns Shop v1.
# 1 - namespace, and make it the default for this context
cd ~
kubectl create namespace shop
kubectl config set-context --current --namespace=shop

# 2 - the content, from a real file so the key is named index.html
echo '<h1>Shop v1</h1>' > index.html
kubectl create configmap web-content --from-file=index.html

# 3 - generate the Deployment skeleton, then edit it
kubectl create deploy shop-web --image nginx:1.27 --replicas 3 \
  --dry-run=client -o yaml > shop-web.yaml

Replace the spec.template.spec block in shop-web.yaml so the container has resources, the probe and the mount - the whole file should look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-web
  labels:
    app: shop-web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: shop-web
  template:
    metadata:
      labels:
        app: shop-web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: "50m"
              memory: "64Mi"
            limits:
              memory: "128Mi"
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5
          volumeMounts:
            - name: content
              mountPath: /usr/share/nginx/html
      volumes:
        - name: content
          configMap:
            name: web-content

kubectl apply -f shop-web.yaml
kubectl get pods -l app=shop-web        # wait for 3/3 READY

# 4 - ClusterIP service
kubectl expose deploy shop-web --port 80 --name shop-svc
kubectl get endpointslices -l kubernetes.io/service-name=shop-svc   # 3 addresses

# 5 - Gateway in THIS namespace, plus a route
cat > gateway.yaml <<'YAML'
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shop-gw
spec:
  gatewayClassName: nginx
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
spec:
  parentRefs:
    - name: shop-gw
  hostnames: ["shop.k8s.lab"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: / }
      backendRefs:
        - name: shop-svc
          port: 80
YAML
kubectl apply -f gateway.yaml
kubectl get gateway shop-gw             # PROGRAMMED should become True

# 6 - the data plane service is in THIS namespace, named after the Gateway
kubectl get svc -n shop -l gateway.networking.k8s.io/gateway-name=shop-gw
PORT=$(kubectl get svc -n shop -l gateway.networking.k8s.io/gateway-name=shop-gw \
  -o jsonpath='{.items[0].spec.ports[?(@.port==80)].nodePort}')
echo "gateway node port: $PORT"
curl -s -H 'Host: shop.k8s.lab' http://worker01:$PORT

Expected: <h1>Shop v1</h1>.

If it does not work, in this order: kubectl get pods (are they Ready? a failing readiness probe keeps them out of the Service), kubectl get endpointslices -l kubernetes.io/service-name=shop-svc (empty = label mismatch), kubectl describe httproute shop (look for ResolvedRefs).

Operate it

  1. Scale to 5 replicas and confirm they are spread across the workers.
  2. Change the content to Shop v2 and make it visible without deleting any Pod by hand.
  3. Roll the image forward to nginx:1.28 with zero downtime. Prove it by running a request loop during the rollout.
  4. Roll back to the previous revision and show the history.
# 7 - scale and check placement
kubectl scale deploy shop-web --replicas=5
kubectl get pods -l app=shop-web -o wide | sort -k7

# 8 - update the ConfigMap; the MOUNTED FILE refreshes on its own
echo '<h1>Shop v2</h1>' > index.html
kubectl create configmap web-content --from-file=index.html \
  --dry-run=client -o yaml | kubectl apply -f -

sleep 70                                 # kubelet sync period, up to ~1 minute
curl -s -H 'Host: shop.k8s.lab' http://worker01:$PORT     # Shop v2

This is the point of the exercise: the ConfigMap is mounted as a volume, so the file updates by itself and no restart is needed. If you had injected the content as an environment variable, nothing would have changed and you would have needed kubectl rollout restart deploy/shop-web - which is also an acceptable answer here, just a bigger hammer.

# 9 - in a SECOND terminal, start the loop first
while true; do
  curl -s --max-time 1 -H 'Host: shop.k8s.lab' http://worker01:$PORT \
    | grep -o 'Shop v[0-9]' || echo FAIL
  sleep 0.2
done

# back in the first terminal
kubectl set image deploy/shop-web nginx=nginx:1.28
kubectl rollout status deploy/shop-web
kubectl get rs -l app=shop-web           # old at 0, new at 5

The loop should never print FAIL. Stop it with Ctrl-C.

# 10 - roll back
kubectl rollout undo deploy/shop-web
kubectl rollout status deploy/shop-web
kubectl rollout history deploy/shop-web
kubectl get pods -l app=shop-web -o jsonpath='{.items[0].spec.containers[0].image}{"\n"}'

Add state

  1. Give the application a PersistentVolumeClaim mounted at /data in every Pod. It must be shared by all replicas.
  2. Write a file into /data from one Pod and read it from another.

Five replicas on three nodes means the volume must be ReadWriteMany. Two routes - use whichever matches what you built in Lab 18.

With the CSI StorageClass (simplest, if you installed the driver):

cat > shop-pvc.yaml <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shop-data
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 300Mi
  storageClassName: nfs-csi
YAML
kubectl apply -f shop-pvc.yaml
kubectl get pvc shop-data                # Bound

With a static NFS PV (no driver): claim one of pv-storage1/2 instead - they are ReadWriteMany and have no class.

kubectl get pv                           # find an Available one
cat > shop-pvc.yaml <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: shop-data
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 300Mi
  storageClassName: ""
YAML
kubectl apply -f shop-pvc.yaml
kubectl get pvc shop-data

If it stays Pending, kubectl describe pvc shop-data tells you why - usually a leftover claimRef on the PV: kubectl patch pv <pv> -p '{"spec":{"claimRef":null}}'.

Now mount it in every Pod:

kubectl patch deploy shop-web --type merge -p '{
  "spec":{"template":{"spec":{
    "containers":[{"name":"nginx","volumeMounts":[
      {"name":"content","mountPath":"/usr/share/nginx/html"},
      {"name":"data","mountPath":"/data"}]}],
    "volumes":[
      {"name":"content","configMap":{"name":"web-content"}},
      {"name":"data","persistentVolumeClaim":{"claimName":"shop-data"}}]}}}}'
kubectl rollout status deploy/shop-web

A strategic merge patch replaces whole lists, which is why both volumes and both mounts are repeated - leaving out content would silently unmount your website.

# 12 - write from one pod, read from another
PODS=($(kubectl get pods -l app=shop-web -o name))
kubectl exec ${PODS[0]} -- sh -c 'echo "written by $(hostname)" > /data/shared.txt'
kubectl exec ${PODS[1]} -- cat /data/shared.txt
kubectl get pods -l app=shop-web -o wide | head -3    # confirm different nodes

Break it and fix it

  1. Set the image to nginx:nosuchtag. Answer from kubectl output alone: is the site still up, and why? Which ReplicaSet is stuck, and what does its event say?
  2. Recover.
  3. Change the Service selector so it matches nothing. Prove the breakage using endpoints, then fix it.
  4. Cordon the node running most of the Pods, drain it, and show the application stayed available throughout.
# 13 - break the image
kubectl set image deploy/shop-web nginx=nginx:nosuchtag
kubectl rollout status deploy/shop-web --timeout=30s     # never completes
kubectl get pods -l app=shop-web                         # 5 Running + new ones failing
kubectl get rs -l app=shop-web
curl -s -H 'Host: shop.k8s.lab' http://worker01:$PORT    # still Shop v2

The answers: the site is up because maxUnavailable (default 25 %) forbids removing healthy Pods until replacements are Ready, and they never are. The new ReplicaSet - the one with non-zero DESIRED but zero READY - is stuck:

kubectl describe rs -l app=shop-web | grep -A5 Events
kubectl describe pod -l app=shop-web | grep -A5 'Failed'

Failed to pull image ... manifest unknown.

# 14 - recover
kubectl rollout undo deploy/shop-web
kubectl rollout status deploy/shop-web
kubectl get pods -l app=shop-web

# 15 - break the selector
kubectl patch svc shop-svc -p '{"spec":{"selector":{"app":"typo"}}}'
kubectl get endpointslices -l kubernetes.io/service-name=shop-svc    # no addresses
curl -s --max-time 3 -H 'Host: shop.k8s.lab' http://worker01:$PORT   # 502/timeout
kubectl describe svc shop-svc | grep -i selector

# fix it
kubectl patch svc shop-svc -p '{"spec":{"selector":{"app":"shop-web"}}}'
kubectl get endpointslices -l kubernetes.io/service-name=shop-svc    # addresses back
curl -s -H 'Host: shop.k8s.lab' http://worker01:$PORT

# 16 - drain the busiest node, with the request loop running in terminal 2
kubectl get pods -l app=shop-web -o wide | sort -k7        # pick the busiest
NODE=worker01                                              # adjust to what you saw
kubectl cordon $NODE
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data
kubectl get pods -l app=shop-web -o wide                   # rescheduled elsewhere
kubectl uncordon $NODE
kubectl rollout restart deploy/shop-web                    # rebalance

The loop should not print FAIL - five replicas across three nodes means draining one never removes them all. If it did fail, you likely drained the node running the Gateway’s data plane Pod: that has one replica, so the entry point itself moved. Worth noticing - highly available applications need a highly available ingress path too.

Package it

  1. Turn the whole thing into a Helm chart with replicaCount, the content message, and the hostname as values.
  2. Install it twice, under two release names, on two hostnames.
cd ~
helm create shopchart
cd shopchart
rm -f templates/hpa.yaml templates/ingress.yaml templates/serviceaccount.yaml
rm -rf templates/tests

values.yaml:

replicaCount: 3

image:
  repository: nginx
  tag: "1.27"
  pullPolicy: IfNotPresent

service:
  port: 80

content:
  message: "Shop from a chart"

host: shop.k8s.lab

serviceAccount:
  create: false

templates/configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "shopchart.fullname" . }}-content
data:
  index.html: |
    <h1>{{ .Values.content.message }}</h1>

templates/httproute.yaml:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: {{ include "shopchart.fullname" . }}
spec:
  parentRefs:
    - name: shop-gw
  hostnames: ["{{ .Values.host }}"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: / }
      backendRefs:
        - name: {{ include "shopchart.fullname" . }}
          port: {{ .Values.service.port }}

In templates/deployment.yaml, mount the ConfigMap at /usr/share/nginx/html exactly as you did by hand, using {{ include "shopchart.fullname" . }}-content as the ConfigMap name.

helm lint .
helm template test . | less              # read it before installing

# 18 - two releases, two hostnames, one chart
cd ~
helm install shop1 ./shopchart -n shop --set host=shop1.k8s.lab
helm install shop2 ./shopchart -n shop \
  --set host=shop2.k8s.lab \
  --set content.message="Second shop" \
  --set replicaCount=1

helm list -n shop
kubectl get deploy,svc,httproute -n shop
curl -s -H 'Host: shop1.k8s.lab' http://worker01:$PORT
curl -s -H 'Host: shop2.k8s.lab' http://worker01:$PORT

Both releases exist side by side because every object is named after the release - that is what include "shopchart.fullname" buys you.

Clean up

  1. Remove both releases, the namespace, the PVC, and release the PV.
helm uninstall shop1 shop2 -n shop
kubectl delete -f ~/gateway.yaml -n shop --ignore-not-found
kubectl delete namespace shop            # takes everything else with it

# the PV is cluster-scoped and survives the namespace
kubectl get pv
kubectl patch pv <pv-name> -p '{"spec":{"claimRef":null}}'   # if Released
kubectl config set-context --current --namespace=default
rm -rf ~/shopchart ~/shop-web.yaml ~/gateway.yaml ~/shop-pvc.yaml ~/index.html

Catch up in one paste

Lost the thread completely, or want to skip ahead to a later stage? This rebuilds everything up to the end of Operate it (steps 1-10) in one go. Paste it, wait, then continue from Add state.

cd ~
kubectl create namespace shop --dry-run=client -o yaml | kubectl apply -f -
kubectl config set-context --current --namespace=shop
echo '<h1>Shop v2</h1>' > index.html
kubectl create configmap web-content --from-file=index.html \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl apply -f - <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-web
  labels: { app: shop-web }
spec:
  replicas: 5
  selector:
    matchLabels: { app: shop-web }
  template:
    metadata:
      labels: { app: shop-web }
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          ports:
            - containerPort: 80
          resources:
            requests: { cpu: "50m", memory: "64Mi" }
            limits:   { memory: "128Mi" }
          readinessProbe:
            httpGet: { path: /, port: 80 }
            initialDelaySeconds: 3
            periodSeconds: 5
          volumeMounts:
            - name: content
              mountPath: /usr/share/nginx/html
      volumes:
        - name: content
          configMap: { name: web-content }
---
apiVersion: v1
kind: Service
metadata:
  name: shop-svc
spec:
  selector: { app: shop-web }
  ports:
    - port: 80
      targetPort: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shop-gw
spec:
  gatewayClassName: nginx
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces: { from: All }
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
spec:
  parentRefs:
    - name: shop-gw
  hostnames: ["shop.k8s.lab"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: / }
      backendRefs:
        - name: shop-svc
          port: 80
YAML

kubectl rollout status deploy/shop-web
PORT=$(kubectl get svc -n shop -l gateway.networking.k8s.io/gateway-name=shop-gw \
  -o jsonpath='{.items[0].spec.ports[?(@.port==80)].nodePort}')
echo "gateway node port: $PORT"
curl -s -H 'Host: shop.k8s.lab' http://worker01:$PORT
Tip

If you get stuck anywhere, the answer is almost always: kubectl describe the object and read the Events.