Lab 40 - etcd snapshot and restore

Lab 40 - etcd snapshot and restore

Goal: save the complete cluster state, make a “fatal mistake”, and roll the whole cluster back to the snapshot. Chapter: Backup and disaster recovery

Warning

This lab rewinds the entire cluster. Everything created after the snapshot disappears. Trainer: take a VM snapshot of control01 before the lab.

Tools on control01

# 1 - which etcd version runs? (on support01)
kubectl get pod -n kube-system etcd-control01 -o jsonpath='{.spec.containers[0].image}{"\n"}'

# 2 - matching etcdctl / etcdutl on control01
ssh student@control01
ETCD_VER=v3.6.5        # <- the version from step 1, with "v", without "-0"
curl -fsSL https://github.com/etcd-io/etcd/releases/download/$ETCD_VER/etcd-$ETCD_VER-linux-amd64.tar.gz \
  | tar xz -C /tmp
sudo mv /tmp/etcd-$ETCD_VER-linux-amd64/etcdctl /tmp/etcd-$ETCD_VER-linux-amd64/etcdutl /usr/local/bin/
etcdctl version
alias ectl='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
ectl endpoint health
ectl member list -w table
exit

State worth saving

# 3 - on support01
kubectl create ns precious
kubectl create deploy vault -n precious --image nginx:1.27 --replicas 2
kubectl create cm before-snapshot --from-literal=state=old

Snapshot

# 4 - on control01
ssh student@control01
alias ectl='sudo etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key'
ectl snapshot save /root/etcd-snap.db
sudo etcdutl snapshot status /root/etcd-snap.db -w table
sudo cp /root/etcd-snap.db /tmp/ && sudo chmod 644 /tmp/etcd-snap.db
exit
scp student@control01:/tmp/etcd-snap.db ~/etcd-snap-$(date +%F).db    # off the node!

The file holds every object of the cluster, including all Secrets - store it like a password.

The fatal mistake

# 5 - on support01
kubectl delete ns precious
kubectl delete cm before-snapshot
kubectl create cm after-snapshot --from-literal=state=new
kubectl get ns precious                           # NotFound

Restore

# 6 - on control01: restore into a NEW data directory
ssh student@control01
sudo etcdutl snapshot restore /root/etcd-snap.db --data-dir /var/lib/etcd-restore
sudo ls /var/lib/etcd-restore                      # member

# 7 - point the etcd static Pod at it (back up the manifest first)
sudo cp /etc/kubernetes/manifests/etcd.yaml /root/etcd.yaml.bak
sudo sed -i 's#path: /var/lib/etcd$#path: /var/lib/etcd-restore#' /etc/kubernetes/manifests/etcd.yaml
sudo grep -n -A2 'hostPath' /etc/kubernetes/manifests/etcd.yaml | grep etcd

The kubelet restarts etcd on the restored data; the API server reconnects. Give it one to two minutes.

# 8 - wait, then back to support01
watch -n 5 'sudo crictl ps --name "etcd|kube-apiserver"'      # both Running; Ctrl-C
exit
kubectl get ns precious                            # back
kubectl get deploy -n precious
kubectl get cm before-snapshot after-snapshot      # before: yes, after: NotFound

The cluster is exactly where it was at step 4 - including things you did not want to lose after step 4.

Tip

Not coming back? On control01: sudo cp /root/etcd.yaml.bak /etc/kubernetes/manifests/etcd.yaml returns to the old data. sudo crictl logs $(sudo crictl ps -a --name etcd -q | head -1) shows why etcd refused to start.

Discovery

Infodiscover

Task A. Make the snapshot a nightly job on control01 that keeps 7 days.

sudo tee /etc/cron.d/etcd-snapshot >/dev/null <<'CRON'
30 1 * * * root etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /var/backups/etcd-$(date +\%F).db && find /var/backups -name 'etcd-*.db' -mtime +7 -delete
CRON

Then copy the files off the node (rsync, object storage) - a backup on the machine it protects is not a backup.

Task B. Which is the better tool to bring back the deleted namespace precious alone, without rewinding everything else - and why?

Velero (Lab 39): it restores one namespace, keeps the rest of the cluster current and can include volume data. etcd restore is for losing the control plane or cluster-wide corruption - all or nothing.

Clean up

kubectl delete ns precious; kubectl delete cm before-snapshot