Lab 13 - Taints, tolerations and affinity

Lab 13 - Taints, tolerations and affinity

Goal: control where Pods land - and see the difference between permitted and attracted. Chapter: Scheduling

Taints repel

# 1 - a baseline: 6 pods spread over the three workers
kubectl create deploy filler --image nginx:1.27 --replicas 6
kubectl get pods -o wide | sort -k7

# 2 - taint worker03 and watch new pods avoid it
kubectl taint node worker03 disk=ssd:NoSchedule
kubectl describe node worker03 | grep -i -A2 taint
kubectl scale deploy filler --replicas 12
kubectl get pods -o wide | grep -c worker03      # unchanged

The Pods already on worker03 are still there. NoSchedule only affects placement; it does not evict anything.

# 3 - NoExecute does evict
kubectl taint node worker03 disk=ssd:NoSchedule-           # remove the first one
kubectl taint node worker03 maint=true:NoExecute
kubectl get pods -o wide | grep worker03                   # emptied within seconds
kubectl get events --sort-by=.lastTimestamp | tail -5

That is what kubectl drain does under the hood, politely, one Pod at a time.

# 4 - clean up the taint
kubectl taint node worker03 maint=true:NoExecute-
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Tolerations permit

# 5 - reserve worker03 for "database" workloads
kubectl taint node worker03 workload=db:NoSchedule

cd ~
cat > tolerant.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: tolerant
spec:
  tolerations:
    - key: workload
      operator: Equal
      value: db
      effect: NoSchedule
  containers:
    - name: box
      image: busybox
      command: ["sleep", "3600"]
YAML
kubectl apply -f tolerant.yaml
kubectl get pod tolerant -o wide

Look at which node it picked. It probably is not worker03. The toleration said “I can live with that taint”, not “put me there” - the scheduler was free to choose any node and the empty one was not necessarily the best fit.

This is the single most misunderstood thing about taints. Repel and attract are separate decisions.

# 6 - now attract it as well
kubectl label node worker03 workload=db
kubectl delete -f tolerant.yaml
cat > pinned.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: pinned
spec:
  tolerations:
    - key: workload
      operator: Equal
      value: db
      effect: NoSchedule
  nodeSelector:
    workload: db
  containers:
    - name: box
      image: busybox
      command: ["sleep", "3600"]
YAML
kubectl apply -f pinned.yaml
kubectl get pod pinned -o wide          # worker03, guaranteed

Taint + toleration + selector is the complete “reserve these nodes” recipe: the taint keeps everyone else out, the toleration lets this Pod in, the selector puts it there.

Node affinity

# 7 - label two nodes and require one of them
kubectl label node worker01 disktype=ssd
kubectl label node worker02 disktype=hdd
kubectl get nodes -L disktype

cat > affinity.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: needs-ssd
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: disktype
                operator: In
                values: ["ssd"]
  containers:
    - name: box
      image: busybox
      command: ["sleep", "3600"]
YAML
kubectl apply -f affinity.yaml
kubectl get pod needs-ssd -o wide       # worker01

# 8 - what happens when nothing matches?
kubectl label node worker01 disktype=hdd --overwrite
kubectl delete pod needs-ssd
kubectl apply -f affinity.yaml
kubectl get pod needs-ssd               # Pending
kubectl describe pod needs-ssd | tail -6

didn't match Pod's node affinity/selector. A required rule that matches nothing means the Pod never runs - it does not fall back.

# 9 - the soft variant always schedules
kubectl delete pod needs-ssd
cat > prefer.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: prefers-ssd
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - key: disktype
                operator: In
                values: ["ssd"]
  containers:
    - name: box
      image: busybox
      command: ["sleep", "3600"]
YAML
kubectl apply -f prefer.yaml
kubectl get pod prefers-ssd -o wide     # runs, even though no node has ssd

# 10 - IgnoredDuringExecution, demonstrated
kubectl label node worker02 disktype=ssd --overwrite
kubectl get pod prefers-ssd -o wide     # it did NOT move

Affinity is evaluated once, at scheduling time. Changing labels afterwards changes nothing for running Pods.

Discovery

Infodiscovery

Task A. Make the filler Deployment spread its replicas so that no two of its Pods sit on the same node if it can be avoided. Expected: with 3 replicas you get one Pod per worker.

kubectl scale deploy filler --replicas=3
kubectl patch deploy filler -p '{"spec":{"template":{"spec":{"affinity":{"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"weight":100,"podAffinityTerm":{"labelSelector":{"matchLabels":{"app":"filler"}},"topologyKey":"kubernetes.io/hostname"}}]}}}}}}'
kubectl get pods -l app=filler -o wide | sort -k7

The modern alternative, which balances instead of merely repelling:

kubectl patch deploy filler -p '{"spec":{"template":{"spec":{"topologySpreadConstraints":[{"maxSkew":1,"topologyKey":"kubernetes.io/hostname","whenUnsatisfiable":"ScheduleAnyway","labelSelector":{"matchLabels":{"app":"filler"}}}]}}}}'

Task B. Find every taint in the cluster in one command, including the ones Kubernetes set itself.

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
kubectl describe nodes | grep -i -B3 'Taints:'

control01 carries the control-plane taint - that is why your DaemonSet skipped it in Lab 12.

Task C. Without using kubectl drain, make worker02 refuse new Pods and push its existing Pods elsewhere. Then undo it.

kubectl taint node worker02 evacuate=yes:NoExecute
kubectl get pods -o wide | grep -c worker02      # 0
kubectl taint node worker02 evacuate=yes:NoExecute-

Drain is friendlier - it respects PodDisruptionBudgets and evicts gradually - but the underlying idea is this taint.

Task D. Explain why kubectl cordon worker01 does not need a taint you write yourself.

kubectl cordon worker01
kubectl get node worker01 -o jsonpath='{.spec.taints}{"\n"}'
kubectl uncordon worker01

Cordon adds node.kubernetes.io/unschedulable:NoSchedule. Every node command from Lab 16 is taints underneath.

Clean up

kubectl delete pod pinned needs-ssd prefers-ssd --ignore-not-found
kubectl delete deploy filler
kubectl taint node worker03 workload=db:NoSchedule-
kubectl label node worker01 disktype- ; kubectl label node worker02 disktype-
kubectl label node worker03 workload-
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints