Lab 25 - Troubleshooting challenge

Lab 25 - Troubleshooting challenge

Infodiscover

Goal: a “production” namespace with several faults. Make it work using only what you learned in Days 1-3. Chapter: Troubleshooting and kubectl debug

The scenario

The shop team deployed to namespace challenge and went home. Nothing works. Your definition of done:

  1. frontend Deployment: 3/3 ready.
  2. worker Deployment: 1/1 ready.
  3. curl -H 'Host: shop.k8s.lab' http://worker01:30080 returns the nginx page (Traefik + Gateway from Day 2).
  4. kubectl get events -n challenge --types=Warning shows nothing new.

Set it up - do not read the manifest, that is cheating:

cd ~
cat > challenge.yaml <<'YAML'
apiVersion: v1
kind: Namespace
metadata: { name: challenge }
---
apiVersion: v1
kind: ResourceQuota
metadata: { name: team, namespace: challenge }
spec:
  hard: { pods: "3" }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: frontend, namespace: challenge }
spec:
  replicas: 3
  selector: { matchLabels: { app: frontend } }
  template:
    metadata: { labels: { app: frontend } }
    spec:
      containers:
        - name: nginx
          image: nginx:1.27
          envFrom: [{ configMapRef: { name: frontend-cfg } }]
          readinessProbe:
            httpGet: { path: /, port: 8080 }
            periodSeconds: 3
---
apiVersion: v1
kind: Service
metadata: { name: frontend, namespace: challenge }
spec:
  selector: { app: front-end }
  ports: [{ port: 80, targetPort: 80 }]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: worker, namespace: challenge }
spec:
  replicas: 1
  selector: { matchLabels: { app: worker } }
  template:
    metadata: { labels: { app: worker } }
    spec:
      nodeSelector: { disktype: nvme }
      containers:
        - name: worker
          image: busybox:1.36
          command: ["sh", "-c", "while true; do echo working; sleep 10; done"]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: frontend, namespace: challenge }
spec:
  parentRefs: [{ name: traefik-gateway, namespace: traefik }]
  hostnames: ["shop.k8s.lab"]
  rules:
    - backendRefs: [{ name: frontend, port: 80 }]
YAML
kubectl apply -f challenge.yaml
kubectl config set-context --current --namespace=challenge

Work the routine: get -> describe / events -> logs. Fix faults with kubectl commands (edit, patch, create, scale) - one at a time, and re-check after each fix; new problems appear when old ones are solved.

Hints

Open a hint only when stuck for 5 minutes.

kubectl get all,quota,httproute and kubectl events --types=Warning. How many Pods exist, compared to how many should?

Look at the state of the frontend Pods and the event behind it. Something they reference does not exist.

Which port does nginx listen on, and which port does the probe check?

Two separate reasons: first the ReplicaSet (kubectl describe rs), then the scheduler.

kubectl get endpointslices - and compare the Service selector with the Pod labels. Then check the HTTPRoute status.

Solution

#FaultFound withFix
1ConfigMap frontend-cfg missing -> CreateContainerConfigErrordescribe podkubectl create cm frontend-cfg --from-literal=MODE=prod
2Readiness probe on port 8080, nginx listens on 80describe pod -> Readiness probe failedkubectl patch deploy frontend --type=json -p '[{"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/httpGet/port","value":80}]'
3Quota allows 3 Pods, 4 needed; during rollouts even moredescribe rs -> exceeded quotakubectl patch quota team -p '{"spec":{"hard":{"pods":"6"}}}'
4worker requires a node label disktype=nvme that no node hasdescribe pod -> FailedSchedulingkubectl patch deploy worker --type=json -p '[{"op":"remove","path":"/spec/template/spec/nodeSelector"}]'
5Service selects app=front-end, Pods are app=frontendempty EndpointSlicekubectl patch svc frontend -p '{"spec":{"selector":{"app":"frontend"}}}'
kubectl get deploy
curl -s -H 'Host: shop.k8s.lab' http://worker01:30080 | grep -o '<title>.*</title>'

Discuss: which faults would a pipeline with kubectl rollout status have caught? Which would only a monitoring check have caught?

Clean up

kubectl config set-context --current --namespace=default
kubectl delete ns challenge