Scheduling and node operations

Scheduling and node operations

How the scheduler decides

graph LR P[new Pod] --> F["Filter<br/>requests fit? taints tolerated?<br/>selector / affinity match?"] F --> S["Score<br/>spread, preferred affinity,<br/>least allocated"] S --> B[bind to best node] F -->|no node left| PE[Pending + FailedScheduling event]

The tools

MechanismOnSays
nodeSelectorPodrun only on nodes with these labels
node affinity required...Podsame, with operators (In, NotIn, Exists)
node affinity preferred...Podtry these nodes, else anywhere
pod anti-affinityPodnot next to Pods like these
topologySpreadConstraintsPodspread evenly over nodes / zones
taintNodekeep Pods away…
tolerationPod…unless they tolerate it

A toleration permits, it does not attract. To reserve nodes for a workload: taint the nodes + toleration + selector on the Pod.

kubectl taint node worker03 workload=db:NoSchedule     # add
kubectl taint node worker03 workload=db:NoSchedule-    # remove
kubectl label node worker03 workload=db

spec:
  tolerations:
    - { key: workload, operator: Equal, value: db, effect: NoSchedule }
  nodeSelector:
    workload: db
Taint effectNew PodsRunning Pods
NoScheduleblockedstay
PreferNoScheduleavoidedstay
NoExecuteblockedevicted

IgnoredDuringExecution: affinity is checked at scheduling time only. Changing labels later does not move running Pods.

Node maintenance

stateDiagram-v2 [*] --> Ready Ready --> SchedulingDisabled: cordon SchedulingDisabled --> Drained: drain Drained --> Ready: uncordon SchedulingDisabled --> Ready: uncordon
kubectl cordon worker03
kubectl drain worker03 --ignore-daemonsets --delete-emptydir-data
# ... patch, reboot ...
kubectl uncordon worker03
  • Drain evicts Pods; controllers recreate them elsewhere.
  • DaemonSet Pods stay (--ignore-daemonsets); bare Pods block drain until --force - and are then gone for good.
  • PodDisruptionBudget (minAvailable / maxUnavailable) makes drain wait instead of taking too many replicas down.
  • uncordon does not rebalance - use kubectl rollout restart.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: web }

References