Lab 16 - Node operations

Lab 16 - Node operations

Infodiscovery

Goal: take a node out for maintenance and put it back, safely. Chapter: Nodes

Everything here is taints underneath - see Lab 13 if you want to see the same effects expressed by hand.

# setup - something to move around
kubectl create deploy filler --image nginx:1.27 --replicas 6
kubectl get pods -o wide | sort -k7

Task 1. Stop new Pods from being scheduled on worker03, without disturbing anything already running there. Expected: kubectl get nodes shows Ready,SchedulingDisabled for worker03, and the Pods on it are untouched.

kubectl cordon worker03
kubectl get nodes
kubectl get pods -o wide | grep worker03

Task 2. Prove the cordon works: scale the Deployment to 9 and show that no new Pod landed on worker03.

kubectl scale deploy filler --replicas 9
kubectl get pods -o wide | grep -c worker03      # same count as before

Task 3. Now actually empty worker03 so it can be rebooted. Expected: no workload Pods left on it, all of them running elsewhere.

kubectl drain worker03 --ignore-daemonsets --delete-emptydir-data
kubectl get pods -o wide | grep worker03         # only DaemonSet pods, if any
kubectl get pods -o wide | sort -k7

Drain evicts Pod by Pod, and the ReplicaSet recreates each one elsewhere - that is why the application never goes down, as long as it has more than one replica.

Task 4. Explain why the command needed --ignore-daemonsets. Show which Pods it refers to.

kubectl get ds -A
kubectl get pods -A -o wide --field-selector spec.nodeName=worker03

DaemonSet Pods belong to the node itself. Evicting them would be pointless - the DaemonSet controller would immediately recreate them on the same node.

Task 5. Bring worker03 back into service and get the workload spread across all three workers again. Expected: after your commands, kubectl get pods -o wide shows Pods on worker03.

kubectl uncordon worker03
kubectl get pods -o wide | sort -k7      # still nothing on worker03!
kubectl rollout restart deploy/filler
kubectl get pods -o wide | sort -k7

This is the part people get wrong: uncordon does not rebalance. Kubernetes never moves a running Pod. Rebalancing means replacing Pods, which is what rollout restart does.

Task 6. Create a Pod that is not managed by any controller, pinned to worker03, then try to drain the node again. Expected: drain refuses, and tells you why.

kubectl run orphan --image nginx:1.27 --overrides='{"spec":{"nodeName":"worker03"}}'
kubectl drain worker03 --ignore-daemonsets
# error: cannot delete Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet
kubectl drain worker03 --ignore-daemonsets --force
kubectl get pod orphan                    # gone - not rescheduled anywhere
kubectl uncordon worker03

--force does not move the Pod. It deletes it. A bare Pod has nothing to recreate it, which is the whole argument against bare Pods in production.

Task 7. Find out how much CPU and memory is currently requested on worker01, and how that compares with actual usage.

kubectl describe node worker01 | grep -A6 'Allocated resources'
kubectl top node worker01

The scheduler only looks at the first number. A node can be almost idle and still reject Pods.

Clean up

kubectl delete deploy filler
kubectl get nodes                          # all Ready, none SchedulingDisabled
Warning

kubectl delete node worker03 removes only the API object. The kubelet keeps running and re-registers. A real removal is drain -> delete node -> kubeadm reset on the machine.