Lab 36 - Running a stateful application

Lab 36 - Running a stateful application

Goal: run Redis with persistent storage the production way - dynamic volumes, spread replicas, a disruption budget - then grow its disk and survive a node drain. Chapter: Stateful workloads

Requires the nfs-csi StorageClass from Lab 35.

# 1 -
kubectl create ns state
kubectl config set-context --current --namespace=state
cd ~
cat > redis.yaml <<'YAML'
apiVersion: v1
kind: Service
metadata:
  name: redis
spec:
  clusterIP: None
  selector: { app: redis }
  ports: [{ port: 6379 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  serviceName: redis
  replicas: 2
  selector:
    matchLabels: { app: redis }
  template:
    metadata:
      labels: { app: redis }
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels: { app: redis }
      containers:
        - name: redis
          image: redis:7.4
          args: ["--appendonly", "yes"]
          ports: [{ containerPort: 6379 }]
          readinessProbe:
            exec: { command: ["redis-cli", "ping"] }
          resources:
            requests: { cpu: "50m", memory: "64Mi" }
            limits:   { memory: "256Mi" }
          volumeMounts:
            - { name: data, mountPath: /data }
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: nfs-csi
        resources:
          requests:
            storage: 200Mi
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: redis
spec:
  maxUnavailable: 1
  selector:
    matchLabels: { app: redis }
YAML
kubectl apply -f redis.yaml
kubectl rollout status sts/redis
kubectl get pods,pvc -o wide

Two Pods on two different nodes, two PVCs created by the driver - nobody wrote a PV.

# 2 - write data, kill the Pod, read it back
kubectl exec redis-0 -- redis-cli set course kubernetes-gt
kubectl delete pod redis-0
kubectl wait --for=condition=Ready pod/redis-0 --timeout=90s
kubectl exec redis-0 -- redis-cli get course          # kubernetes-gt

--appendonly yes writes every change to /data, the PVC survives the Pod.

# 3 - the disk is full-ish: grow it
kubectl patch pvc data-redis-0 -p '{"spec":{"resources":{"requests":{"storage":"500Mi"}}}}'
kubectl get pvc data-redis-0 -w                       # CAPACITY 500Mi; Ctrl-C

Works because the class has allowVolumeExpansion: true. (On NFS the size is bookkeeping - on block storage the driver really grows the disk.)

# 4 - node maintenance with state
NODE=$(kubectl get pod redis-0 -o jsonpath='{.spec.nodeName}')
kubectl drain $NODE --ignore-daemonsets --delete-emptydir-data
kubectl get pods -o wide                              # redis-0 on another node
kubectl exec redis-0 -- redis-cli get course          # still there
kubectl uncordon $NODE

The PDB let drain evict one replica at a time; the volume followed the Pod to the new node because it is network storage.

Discovery

Infodiscover

Task A. Scale to 4 replicas. How many are Running, and why?

kubectl scale sts redis --replicas 4
kubectl get pods -o wide
kubectl describe pod redis-3 | tail -3

All 4 run: three workers, maxSkew: 1 allows 2/1/1. Spread constraints balance, they do not cap Pods per node. For “never two on one node” use required pod anti-affinity - then the 4th replica stays Pending.

Task B. Scale back to 2. What happened to data-redis-2 and data-redis-3?

kubectl scale sts redis --replicas 2
kubectl get pvc

Still there. Scale up again and the replicas get their old data back.

Clean up

kubectl config set-context --current --namespace=default
kubectl delete ns state
kubectl get pv | grep Released        # Retain - see Lab 35 for cleanup