Backup and disaster recovery
Backup and disaster recovery
What exactly can you lose?
| Layer | Contains | Backed up by |
|---|---|---|
| etcd | every Kubernetes object of the cluster | etcdctl snapshot save |
| Resources per namespace / app | Deployments, Secrets, PVC definitions… | Velero, git (GitOps) |
| Volumes | the data | Velero (file system backup or CSI snapshots), application backups |
| Outside the cluster | DNS, certificates, registry, cloud resources | their own processes |
graph LR
subgraph cluster
E[(etcd)]
NS[namespace shop<br/>objects + PVCs]
end
E -->|etcdctl snapshot| F[snapshot file<br/>off the node!]
NS -->|Velero backup| OS[(S3 object storage<br/>Garage)]
G[git repo] -->|GitOps re-apply| NS
| etcd snapshot | Velero | |
|---|---|---|
| Scope | whole cluster, all or nothing | namespaces, labels, resource types |
| Volumes | no | yes |
| Restore to another cluster | no (same cluster identity) | yes - migrations |
| Typical use | control plane loss, fatal mistakes | app restore, namespace deleted, migration |
Velero
A controller in the cluster plus a CLI. Backups go to object storage
(BackupStorageLocation); volume data via CSI snapshots or the node agent
(file-system backup with Kopia).
velero backup create shop-1 --include-namespaces shop
velero schedule create shop-daily --schedule "0 2 * * *" --include-namespaces shop --ttl 168h
velero restore create --from-backup shop-1
RPO and RTO
- RPO - how much data may be lost: the backup interval.
- RTO - how long may recovery take: measured only by restoring.
A backup you have never restored is a hope, not a backup. Practise restores on a schedule - that is what Labs 39 and 40 do.
Warning
An etcd snapshot holds all Secrets of the cluster. Encrypt it, store it off the node, restrict who can read it. With encryption at rest (Lab 33) you also need the encryption key to use it.