Lab 18 - Persistent storage on NFS

Lab 18 - Persistent storage on NFS

Goal: give a Pod storage that survives it - with no StorageClass and no CSI driver, the way an on-premise cluster really looks. Chapter: Persistent storage

support01 exports /storage1, /storage2 and /storage3 over NFS. There is no dynamic provisioning in this cluster, so you write the PersistentVolume.

Prepare the nodes

# 1 - every node that mounts NFS needs the client tools
for n in worker01 worker02 worker03; do
  ssh student@$n 'dpkg -l nfs-common >/dev/null 2>&1 && echo "$(hostname): ok" || sudo apt-get install -y nfs-common'
done

# 2 - prove the export works from a node, outside Kubernetes first
ssh student@worker01 'sudo mkdir -p /mnt/t && sudo mount -t nfs support01:/storage1 /mnt/t && df -h /mnt/t && sudo umount /mnt/t'

Always test the mount by hand before blaming Kubernetes. A Pod stuck in ContainerCreating with an NFS error is usually a missing nfs-common, a wrong export path, or export permissions.

The PersistentVolume

# 3 - one PV per export
cd ~
cat > pvs.yaml <<'YAML'
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-storage1
spec:
  capacity:
    storage: 1Gi
  accessModes: ["ReadWriteMany"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  nfs:
    server: support01
    path: /storage1
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-storage2
spec:
  capacity:
    storage: 1Gi
  accessModes: ["ReadWriteMany"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  nfs:
    server: support01
    path: /storage2
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-storage3
spec:
  capacity:
    storage: 1Gi
  accessModes: ["ReadWriteMany"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  nfs:
    server: support01
    path: /storage3
YAML
kubectl apply -f pvs.yaml
kubectl get pv

All three are Available - they exist but belong to nobody. PVs are cluster-scoped: there is no -n for them.

storageClassName: "" means “no class”. It matters: a PVC that also asks for no class will bind to these, and a PVC asking for a class will not.

The claim

# 4 -
cat > pvc.yaml <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: web-data
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 500Mi
  storageClassName: ""
YAML
kubectl apply -f pvc.yaml
kubectl get pvc,pv

The claim asked for 500Mi and got a whole 1Gi volume. Binding is exclusive and never splits a PV - the extra capacity is simply unused. Note which PV it picked; the control plane chose, not you.

Use it

# 5 - a Pod that writes to the volume
cat > pvpod.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: writer
spec:
  containers:
    - name: box
      image: busybox
      command: ["sh", "-c", "echo \"written by $(hostname) at $(date)\" >> /data/log.txt; sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: web-data
YAML
kubectl apply -f pvpod.yaml
kubectl get pod writer -o wide
kubectl exec writer -- cat /data/log.txt

The Pod references the claim, never the volume. That indirection is what makes the manifest portable: the same Pod spec works on NFS here and on EBS in a cloud.

# 6 - the data outlives the Pod
kubectl delete pod writer
kubectl apply -f pvpod.yaml
kubectl exec writer -- cat /data/log.txt      # two lines now

# 7 - and it is genuinely shared (ReadWriteMany)
kubectl run reader --image busybox --restart=Never --overrides='{"spec":{"containers":[{"name":"reader","image":"busybox","command":["sleep","3600"],"volumeMounts":[{"name":"d","mountPath":"/data"}]}],"volumes":[{"name":"d","persistentVolumeClaim":{"claimName":"web-data"}}]}}'
kubectl get pod reader -o wide                # possibly a different node
kubectl exec reader -- cat /data/log.txt      # same file

Two Pods, possibly two nodes, one file. That is what RWX means, and it is something block storage cannot do.

Discovery

Infodiscovery

Task A. Create a PVC that asks for 5Gi. Expected: it stays Pending. Find out - from the cluster, not from this page - exactly why.

kubectl create -f - <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: toobig
spec:
  accessModes: ["ReadWriteMany"]
  resources: { requests: { storage: 5Gi } }
  storageClassName: ""
YAML
kubectl get pvc toobig
kubectl describe pvc toobig | tail -5

no persistent volumes available for this claim and no storage class is set. With a StorageClass the cluster would have created one; here it cannot.

Task B. Show which PV each PVC is bound to, and which claim owns each PV, in one line each.

kubectl get pvc -o custom-columns=PVC:.metadata.name,VOLUME:.spec.volumeName,SIZE:.status.capacity.storage
kubectl get pv -o custom-columns=PV:.metadata.name,STATUS:.status.phase,CLAIM:.spec.claimRef.name

Task C. Delete the web-data PVC and predict the PV’s state. Then explain how you would make that PV usable again.

kubectl delete pod writer reader
kubectl delete pvc web-data
kubectl get pv

The PV goes to Released, not Available: the reclaim policy is Retain, so the data is kept and the volume will not be handed to anyone else. To reuse it you clear the stale binding:

kubectl patch pv pv-storage1 -p '{"spec":{"claimRef":null}}'
kubectl get pv

With persistentVolumeReclaimPolicy: Delete the PV would have been removed instead - along with the data.

Dynamic provisioning with the NFS CSI driver

Everything so far was static: you wrote a PV for every share, by hand. That does not scale - ten teams asking for volumes means ten PVs and a ticket each. A CSI driver plus a StorageClass turns that into self-service: the PVC asks, the driver creates.

Note

A word on names. The nfs: volume type you used above is the in-tree NFS plugin - compiled into Kubernetes itself. In-tree storage plugins are frozen and being removed; every driver now lives out of tree and speaks CSI. So there is no such thing as an “in-tree CSI driver”: what you install below, csi-driver-nfs, is the out-of-tree replacement for the in-tree plugin. Same NFS server, same exports - different plumbing, and it is the one with a future.

# 8 - install the driver (a DaemonSet + a controller Deployment, nothing exotic)
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm repo update
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs \
  --namespace kube-system --version 4.12.0
kubectl get pods -n kube-system -l app.kubernetes.io/name=csi-driver-nfs -w   # Ctrl-C when Running

# 9 - what did that add?
kubectl get csidrivers
kubectl get ds,deploy -n kube-system | grep nfs

Look at the shape of it: a DaemonSet (the node plugin, which does the actual mounting - so it must be on every node, exactly as in Lab 12) and a Deployment (the controller, which creates and deletes volumes). The nodes still need nfs-common, which you installed in step 1 - CSI does not replace the kernel mount, it drives it.

The StorageClass

# 10 -
cd ~
cat > sc-nfs.yaml <<'YAML'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi
provisioner: nfs.csi.k8s.io
parameters:
  server: support01
  share: /storage3
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
  - nfsvers=4.1
YAML
kubectl apply -f sc-nfs.yaml
kubectl get sc

Read it as a recipe the cluster can follow without you:

FieldMeaning
provisionerwhich CSI driver handles claims for this class
parametersdriver-specific: the NFS server and the parent share
reclaimPolicywhat happens to the PV when its PVC is deleted - Retain here
volumeBindingModeImmediate provisions at once; WaitForFirstConsumer waits for a Pod

The reclaim policy is set on the class, and every PV it creates inherits it. With Delete - the usual default - deleting a PVC deletes the data. Retain is what you want for anything you would be sad to lose.

Tip

Free the export first: /storage3 is about to become the parent directory the driver creates subdirectories in, so it must not be claimed by the static PV. kubectl delete pv pv-storage3 if it is still Available.

Self-service: a PVC with no PV

# 11 - remove the static PV for /storage3, then just ask
kubectl delete pv pv-storage3 --ignore-not-found

cat > pvc-dyn.yaml <<'YAML'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dyn-data
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 300Mi
  storageClassName: nfs-csi
YAML
kubectl apply -f pvc-dyn.yaml
kubectl get pvc dyn-data
kubectl get pv

No PV existed and now one does. Its name is pvc-<uuid>, its RECLAIM POLICY is Retain, and it was created by the driver - not by you. That is the whole difference between the first half of this lab and this one.

# 12 - where did it actually put the data?
PV=$(kubectl get pvc dyn-data -o jsonpath='{.spec.volumeName}')
echo $PV
kubectl get pv $PV -o jsonpath='{.spec.csi.volumeAttributes}{"\n"}'
ssh student@support01 'ls -l /storage3'

The driver created a subdirectory inside /storage3 named after the volume. One export, many volumes - that is how a single NFS share serves a whole cluster.

# 13 - use it
cat > dynpod.yaml <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: dyn-writer
spec:
  containers:
    - name: box
      image: busybox
      command: ["sh", "-c", "echo \"dynamic volume, $(date)\" >> /data/log.txt; sleep 3600"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: dyn-data
YAML
kubectl apply -f dynpod.yaml
kubectl exec dyn-writer -- cat /data/log.txt

The Pod spec is identical to the static one. Applications never know or care whether the volume was provisioned by hand or by a driver - they reference a claim, and that is the point of the abstraction.

Retain in action: the data outlives the claim

# 14 - throw away the Pod *and* the claim
kubectl delete pod dyn-writer
kubectl delete pvc dyn-data
kubectl get pv

The PV is Released, not gone. With reclaimPolicy: Delete the driver would have deleted the subdirectory on the NFS server by now; with Retain it left everything alone and simply refuses to hand the volume to anybody else.

# 15 - prove the data is still on the server
ssh student@support01 'ls -l /storage3; find /storage3 -name log.txt -exec cat {} \;'

# 16 - hand the same volume to a new claim
kubectl patch pv $PV -p '{"spec":{"claimRef":null}}'
kubectl get pv $PV                       # Available again

cat > pvc-reuse.yaml <<YAML
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: dyn-data-v2
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 300Mi
  storageClassName: nfs-csi
  volumeName: $PV
YAML
kubectl apply -f pvc-reuse.yaml
kubectl get pvc dyn-data-v2

volumeName is what stops the provisioner from creating a new volume: the claim names the PV it wants, so this is a static binding to a dynamically created volume. Clearing claimRef is the manual step Retain deliberately forces on you - it is the moment a human confirms the old data may be reused.

# 17 - a new pod, the old data
sed 's/dyn-writer/dyn-reader/; s/claimName: dyn-data$/claimName: dyn-data-v2/' dynpod.yaml \
  | kubectl apply -f -
kubectl exec dyn-reader -- cat /data/log.txt      # the line from step 13 is there

Two lines now: one written before you deleted the claim, one after. The Pod is new, the PVC is new, the data survived both.

Static or dynamic?

Static PVStorageClass + CSI
Who creates the volumeyou, per volumethe driver, on demand
Scales to many teamsnoyes
Needs a drivernoyes
Good fora fixed appliance, an existing shareeverything else

Production clusters run both: a default StorageClass for self-service, plus a handful of hand-written PVs for storage that already exists and must be mounted exactly as it is.

Clean up (CSI part)

kubectl delete pod dyn-reader --ignore-not-found
kubectl delete pvc dyn-data-v2 --ignore-not-found
kubectl patch pv $PV -p '{"spec":{"claimRef":null}}' 2>/dev/null
kubectl delete pv $PV --ignore-not-found      # Retain means this is your job
ssh student@support01 'sudo rm -rf /storage3/pvc-*'
kubectl get pv,pvc
Warning

Retain means nobody cleans up after you. Released PVs and their data sit there until someone deletes them - which is the safety you asked for and the tidiness bill that comes with it.

Clean up (whole lab)

kubectl delete pod writer reader --ignore-not-found
kubectl delete pvc web-data toobig --ignore-not-found

# put pv-storage3 back - Lab 19 needs three static PVs
kubectl apply -f pvs.yaml
for p in pv-storage1 pv-storage2 pv-storage3; do
  kubectl patch pv $p -p '{"spec":{"claimRef":null}}' 2>/dev/null
done
kubectl get pv                      # three, all Available

Leave the CSI driver and the nfs-csi StorageClass installed - Lab 22 can use them instead of hand-written PVs if you would rather see the operator do dynamic provisioning.