Lab 42 - End-to-end production scenario

Lab 42 - End-to-end production scenario

Infodiscover

Goal: take an application to production with everything from the five days - then survive a game day. Chapter: Production operations and best practices

Work in pairs. Build everything in one file ~/prod-shop.yaml (plus the Velero commands) so it can go to git.

The application

podinfo ghcr.io/stefanprodan/podinfo:6.7.0 - port 9898, health /healthz, readiness /readyz, metrics /metrics, message from env PODINFO_UI_MESSAGE.

Requirements

#AreaRequirement
1Namespaceprod-shop, Pod Security restricted enforced; ResourceQuota (max 10 Pods, 2 CPU requests); LimitRange with defaults
2Identityown ServiceAccount shop, no API token mounted
3ConfigConfigMap with PODINFO_UI_MESSAGE="Hello from GT"; Secret api-key injected as env API_KEY
4WorkloadDeployment, 3 replicas, non-root, read-only root FS, drop ALL, requests + memory limit, readiness + liveness
5Availabilityspread over nodes, PDB minAvailable: 2, HPA 3-6 replicas at 60 % CPU
6TrafficService + HTTPRoute shop.prod.k8s.lab on traefik-gateway
7Networkdefault-deny ingress; allow 9898 only from namespaces traefik and monitoring
8MonitoringServiceMonitor; PrometheusRule alert when fewer than 2 replicas are available for 1 min
9BackupVelero schedule every 6 h, 3 days retention; one manual backup

Check yourself:

kubectl get all,pdb,hpa,netpol,httproute,servicemonitor,prometheusrule -n prod-shop
curl -s -H 'Host: shop.prod.k8s.lab' http://worker01:30080 | grep -o '"message": "[^"]*"'
trivy config ~/prod-shop.yaml
velero backup get

Game day

Run a request loop in a second terminal the whole time and watch for errors:

while true; do curl -s -o /dev/null -w '%{http_code} ' -H 'Host: shop.prod.k8s.lab' http://worker01:30080; sleep 0.2; done
#ExerciseExpected
G1upgrade to podinfo:6.7.1only 200s during the rollout
G2drain the node with most shop Podsonly 200s; PDB respected; uncordon + rebalance afterwards
G3load test: kubectl run load -n prod-shop ... hammering the ServiceHPA scales out, alert stays quiet
G4the trainer deletes namespace prod-shoprestore from Velero; measure your RTO from “404” to “200”
G5postmortem of G4timeline, RTO, what would make it faster (GitOps?)

G3 load test inside the namespace needs a NetworkPolicy exception - or run the load Pod in traefik… think about which is acceptable in production.

Reference solution

apiVersion: v1
kind: Namespace
metadata:
  name: prod-shop
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
---
apiVersion: v1
kind: ResourceQuota
metadata: { name: shop, namespace: prod-shop }
spec:
  hard: { pods: "10", requests.cpu: "2" }
---
apiVersion: v1
kind: LimitRange
metadata: { name: defaults, namespace: prod-shop }
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: "50m", memory: "32Mi" }
      default: { memory: "128Mi" }
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: shop, namespace: prod-shop }
automountServiceAccountToken: false
---
apiVersion: v1
kind: ConfigMap
metadata: { name: shop, namespace: prod-shop }
data: { PODINFO_UI_MESSAGE: "Hello from GT" }
---
apiVersion: v1
kind: Secret
metadata: { name: api-key, namespace: prod-shop }
stringData: { API_KEY: "change-me" }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: shop, namespace: prod-shop, labels: { app: shop } }
spec:
  replicas: 3
  selector: { matchLabels: { app: shop } }
  strategy: { rollingUpdate: { maxUnavailable: 0, maxSurge: 1 } }
  template:
    metadata: { labels: { app: shop } }
    spec:
      serviceAccountName: shop
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile: { type: RuntimeDefault }
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: { matchLabels: { app: shop } }
      containers:
        - name: podinfo
          image: ghcr.io/stefanprodan/podinfo:6.7.0
          ports: [{ name: http, containerPort: 9898 }]
          envFrom: [{ configMapRef: { name: shop } }]
          env:
            - name: API_KEY
              valueFrom: { secretKeyRef: { name: api-key, key: API_KEY } }
          readinessProbe: { httpGet: { path: /readyz, port: 9898 } }
          livenessProbe:  { httpGet: { path: /healthz, port: 9898 }, periodSeconds: 10 }
          resources:
            requests: { cpu: "100m", memory: "64Mi" }
            limits:   { memory: "128Mi" }
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities: { drop: ["ALL"] }
          volumeMounts: [{ name: tmp, mountPath: /tmp }]
      volumes: [{ name: tmp, emptyDir: {} }]
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: shop, namespace: prod-shop }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: shop } }
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: shop, namespace: prod-shop }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: shop }
  minReplicas: 3
  maxReplicas: 6
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }
---
apiVersion: v1
kind: Service
metadata: { name: shop, namespace: prod-shop, labels: { app: shop } }
spec:
  selector: { app: shop }
  ports: [{ name: http, port: 80, targetPort: 9898 }]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: shop, namespace: prod-shop }
spec:
  parentRefs: [{ name: traefik-gateway, namespace: traefik }]
  hostnames: ["shop.prod.k8s.lab"]
  rules: [{ backendRefs: [{ name: shop, port: 80 }] }]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny-ingress, namespace: prod-shop }
spec:
  podSelector: {}
  policyTypes: ["Ingress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: shop-from-gateway-and-monitoring, namespace: prod-shop }
spec:
  podSelector: { matchLabels: { app: shop } }
  ingress:
    - from:
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: traefik } }
        - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: monitoring } }
      ports: [{ protocol: TCP, port: 9898 }]
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: { name: shop, namespace: prod-shop }
spec:
  selector: { matchLabels: { app: shop } }
  endpoints: [{ port: http, path: /metrics }]
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: { name: shop, namespace: prod-shop }
spec:
  groups:
    - name: shop
      rules:
        - alert: ShopReplicasLow
          expr: kube_deployment_status_replicas_available{namespace="prod-shop",deployment="shop"} < 2
          for: 1m
          labels: { severity: critical }
          annotations: { summary: "shop has fewer than 2 available replicas" }

kubectl apply -f ~/prod-shop.yaml
velero schedule create prod-shop --schedule "0 */6 * * *" --include-namespaces prod-shop --ttl 72h
velero backup create prod-shop-manual --include-namespaces prod-shop --wait

G1: kubectl set image deploy/shop -n prod-shop podinfo=ghcr.io/stefanprodan/podinfo:6.7.1 (then update the file - no drift!). G4: velero restore create --from-backup prod-shop-manual --wait.

Wrap-up

What would you add before real production? Typical answers: GitOps (Argo CD) instead of kubectl apply, secrets from a vault, OIDC logins, image signing, a second environment, tested runbooks for every alert.

Clean up

kubectl delete ns prod-shop
velero schedule delete prod-shop --confirm