Lab 19 - Requests, limits and quotas

Lab 19 - Requests, limits and quotas

Goal: install metrics-server, see throttling and OOM kills, and put guard rails on a namespace. Chapter: Requests, limits and quotas

Install metrics-server

# 1 -
kubectl top nodes                                  # error: Metrics API not available
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.9.0/components.yaml
kubectl patch deploy metrics-server -n kube-system --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl rollout status deploy/metrics-server -n kube-system
sleep 30; kubectl top nodes

--kubelet-insecure-tls: kubeadm kubelets use self-signed serving certificates. Fine in a lab; in production sign them (serverTLSBootstrap).

# 2 - usage vs requests
kubectl top pods -A --sort-by=cpu | head
kubectl describe node worker01 | grep -A6 'Allocated resources'

QoS classes

# 3 -
cd ~
kubectl create ns res
kubectl config set-context --current --namespace=res
kubectl run besteffort --image nginx:1.27
kubectl run burstable  --image nginx:1.27 --overrides='{"spec":{"containers":[{"name":"burstable","image":"nginx:1.27","resources":{"requests":{"memory":"32Mi"}}}]}}'
kubectl run guaranteed --image nginx:1.27 --overrides='{"spec":{"containers":[{"name":"guaranteed","image":"nginx:1.27","resources":{"requests":{"cpu":"100m","memory":"64Mi"},"limits":{"cpu":"100m","memory":"64Mi"}}}]}}'
kubectl get pods -o custom-columns=NAME:.metadata.name,QOS:.status.qosClass

CPU limit = throttling

# 4 - burn CPU with a 200m limit
kubectl run burner --image busybox:1.36 \
  --overrides='{"spec":{"containers":[{"name":"burner","image":"busybox:1.36","command":["sh","-c","while true; do :; done"],"resources":{"limits":{"cpu":"200m","memory":"32Mi"}}}]}}'
sleep 60; kubectl top pod burner

About 200m - never more, and never killed. Just slow.

Memory limit = OOMKilled

# 5 - eat memory with a 50Mi limit
kubectl run hog --image busybox:1.36 \
  --overrides='{"spec":{"containers":[{"name":"hog","image":"busybox:1.36","command":["sh","-c","tail /dev/zero"],"resources":{"limits":{"memory":"50Mi"}}}]}}'
sleep 20; kubectl get pod hog
kubectl describe pod hog | grep -A5 'Last State'

OOMKilled, exit code 137, then CrashLoopBackOff.

Guard rails: LimitRange and ResourceQuota

# 6 - defaults for everything without resources
cat > limits.yaml <<'YAML'
apiVersion: v1
kind: LimitRange
metadata:
  name: defaults
spec:
  limits:
    - type: Container
      defaultRequest: { cpu: "50m", memory: "32Mi" }
      default:        { cpu: "200m", memory: "128Mi" }
YAML
kubectl apply -f limits.yaml
kubectl run defaulted --image nginx:1.27
kubectl get pod defaulted -o jsonpath='{.spec.containers[0].resources}{"\n"}'

The LimitRange filled in requests and limits at admission.

# 7 - a quota for the namespace
cat > quota.yaml <<'YAML'
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team
spec:
  hard:
    pods: "8"
    requests.cpu: "500m"
    limits.memory: 1Gi
YAML
kubectl apply -f quota.yaml
kubectl describe quota team

# 8 - exceed it
kubectl create deploy big --image nginx:1.27 --replicas 6
kubectl get deploy big                    # READY stops below 6
kubectl describe rs -l app=big | grep -i -A2 'exceeded quota' | head -6

The Deployment looks fine, the Pods simply never appear. The reason is on the ReplicaSet (FailedCreate), not on a Pod.

Discovery

Infodiscover

Task A. Which resource of the quota is exhausted, and how much is used?

kubectl get quota team -o jsonpath='{.status.used}{"\n"}{.status.hard}{"\n"}'

Task B. Which Pod in the whole cluster uses the most memory right now?

kubectl top pods -A --sort-by=memory | head -3

Clean up

kubectl config set-context --current --namespace=default
kubectl delete ns res

Keep metrics-server - Lab 20 needs it.