Lab 39 - Backup and restore with Velero
Lab 39 - Backup and restore with Velero
Goal: back up a namespace with its volume data to S3 object storage, destroy it, and restore it. Chapter: Backup and disaster recovery
Object storage: Garage on support01
MinIO’s free edition stopped shipping images in 2025, so we use Garage - a small S3-compatible server, one container.
# 1 - config and credentials
mkdir -p ~/garage/meta ~/garage/data && cd ~/garage
cat > garage.toml <<TOML
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
replication_factor = 1
rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "$(openssl rand -hex 32)"
[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
TOML
export GARAGE_DEFAULT_ACCESS_KEY="GK$(openssl rand -hex 16)"
export GARAGE_DEFAULT_SECRET_KEY="$(openssl rand -hex 32)"
export GARAGE_DEFAULT_BUCKET=velero
printf '[default]\naws_access_key_id=%s\naws_secret_access_key=%s\n' \
"$GARAGE_DEFAULT_ACCESS_KEY" "$GARAGE_DEFAULT_SECRET_KEY" > ~/credentials-velero
# 2 - run it
docker run -d --name garage --restart unless-stopped \
-p 3900:3900 -p 3901:3901 \
-v ~/garage/garage.toml:/etc/garage.toml \
-v ~/garage/meta:/var/lib/garage/meta -v ~/garage/data:/var/lib/garage/data \
-e GARAGE_DEFAULT_ACCESS_KEY -e GARAGE_DEFAULT_SECRET_KEY -e GARAGE_DEFAULT_BUCKET \
dxflrs/garage:v2.3.0 /garage server --single-node --default-bucket
sleep 5
docker exec garage /garage status
docker exec garage /garage bucket list # velero
Install Velero
# 3 - the CLI (pin the version your trainer gives you; plugin 1.14.x = Velero 1.18.x)
VELERO=v1.18.4
curl -fsSL https://github.com/vmware-tanzu/velero/releases/download/$VELERO/velero-$VELERO-linux-amd64.tar.gz \
| tar xz -C /tmp
sudo mv /tmp/velero-$VELERO-linux-amd64/velero /usr/local/bin/
velero version --client-only
# 4 - the server side (Pods reach Garage by IP - support01 is not in cluster DNS)
S3_IP=$(hostname -I | tr ' ' '\n' | grep '^192.168.128.')
echo $S3_IP
velero install \
--provider aws \
--plugins velero/velero-plugin-for-aws:v1.14.0 \
--bucket velero \
--secret-file ~/credentials-velero \
--backup-location-config region=garage,s3ForcePathStyle=true,s3Url=http://$S3_IP:3900 \
--use-volume-snapshots=false \
--use-node-agent \
--default-volumes-to-fs-backup
kubectl get pods -n velero -w # velero + one node-agent per node; Ctrl-C
velero backup-location get # PHASE Available
--use-node-agent + --default-volumes-to-fs-backup: volume data is copied
file by file (Kopia) - works with any storage, including NFS.
Something worth saving
# 5 - an app with data on a PVC (nfs-csi from Lab 35)
kubectl create ns shop2
cat > ~/shop2.yaml <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: orders, namespace: shop2 }
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: nfs-csi
resources: { requests: { storage: 100Mi } }
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: orders, namespace: shop2 }
spec:
replicas: 1
selector: { matchLabels: { app: orders } }
template:
metadata: { labels: { app: orders } }
spec:
containers:
- name: app
image: busybox:1.36
command: ["sh", "-c", "while true; do date >> /data/orders.log; sleep 10; done"]
volumeMounts: [{ name: data, mountPath: /data }]
volumes:
- name: data
persistentVolumeClaim: { claimName: orders }
YAML
kubectl apply -f ~/shop2.yaml
kubectl create secret generic payment-key -n shop2 --from-literal=key=abc123
sleep 30; kubectl exec -n shop2 deploy/orders -- wc -l /data/orders.log
Back up
# 6 -
velero backup create shop2-1 --include-namespaces shop2 --wait
velero backup describe shop2-1 --details | grep -A5 -E 'Resource List|Pod Volume Backups' | head -20
docker exec garage /garage bucket info velero # objects arrived
Disaster
# 7 - someone deletes the namespace
kubectl delete ns shop2
kubectl get ns shop2 # NotFound
Restore
# 8 -
velero restore create --from-backup shop2-1 --wait
velero restore describe $(velero restore get -o name | tail -1 | cut -d/ -f2) | head -15
kubectl get all,pvc,secret -n shop2
kubectl exec -n shop2 deploy/orders -- head -3 /data/orders.log # old lines are back
Objects, Secret and the log lines from before the backup - restored into a new PVC.
Discovery
InfodiscoverTask A. Schedule a backup of shop2 every hour, keep each for 24 hours.
Task B. Restore shop2 into a new namespace shop2-copy next to the
original - e.g. to inspect old data.
Clean up
kubectl delete ns shop2-copy --ignore-not-found
velero schedule delete shop2-hourly --confirm 2>/dev/null
Keep Velero and Garage - the final scenario (Lab 42) uses them.