Production operations and best practices
Production operations and best practices
Everything in this course, as one checklist.
Platform
| Area | Practice |
|---|---|
| Control plane | 3 control plane nodes (odd etcd quorum), or a managed service |
| Upgrades | one minor version at a time, every ~4 months; control plane first, then nodes (drain, upgrade kubelet, uncordon) |
| Certificates | kubeadm certs expire after 1 year: kubeadm certs check-expiration |
| Backups | etcd snapshots + Velero, tested restores |
| Capacity | headroom for one node failure + rolling updates |
| Access | OIDC for people, no shared admin kubeconfig, audit log to SIEM |
| Policies | PSA baseline everywhere, restricted where possible; admission policies |
| Network | default-deny NetworkPolicies per namespace |
| Observability | metrics, logs, alerts on symptoms, runbooks per alert |
Workloads
| Area | Practice |
|---|---|
| Images | pinned tags/digests, scanned, non-root, minimal |
| Resources | requests on everything, memory limits, quotas per namespace |
| Health | readiness + liveness (+ startup), no dependency checks in liveness |
| Availability | ≥ 2 replicas, anti-affinity / spread, PDB |
| Config | ConfigMaps/Secrets, nothing baked into images; secrets from a vault |
| Identity | own ServiceAccount, no token unless needed, least-privilege RBAC |
| Delivery | GitOps (Argo CD / Flux), Helm or Kustomize, no kubectl edit in prod |
| Shutdown | handle SIGTERM, terminationGracePeriodSeconds fits the app |
Incidents
graph LR
D[detect<br/>alert] --> T[triage<br/>impact, severity] --> M[mitigate<br/>rollback, scale, failover] --> R[resolve<br/>root cause fix] --> P[postmortem<br/>blameless, actions]
Mitigate first, investigate second: kubectl rollout undo restores service
while you look for the cause. Write the postmortem: timeline, impact, root
cause, what went well, action items with owners.
Upgrade in one screen (kubeadm)
# control plane
sudo apt-get install -y kubeadm=1.36.x-* # next minor only
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.36.x
# each node, one at a time
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
sudo apt-get install -y kubelet=1.36.x-* kubectl=1.36.x-* && sudo systemctl restart kubelet
kubectl uncordon <node>