Lab 22 - An operator: PostgreSQL

Goal: install a real operator, create a database cluster with one small manifest, and watch the operator repair it. Chapter: CRDs and operators

You will use CloudNativePG, a PostgreSQL operator. Everything you have learned so far applies: what the operator creates is StatefulSet-like Pods, Services, Secrets and PVCs - objects you can already read.

Storage first - and it must be dynamic

PostgreSQL is the first workload in this course that will not run on the static NFS PVs you wrote by hand. nginx ran as root and wrote to the export root quite happily; the CloudNativePG image runs as the unprivileged postgres user (UID 26), and it cannot write into a root-owned NFS directory. The Pod starts, initdb fails on permissions, and the Pod crash-loops.

Dynamic provisioning fixes this properly: the CSI driver creates a separate subdirectory per volume and can chmod it, so each database instance gets a directory it actually owns.

# 1 - is the CSI driver already here? (you installed it in Lab 18)
kubectl get csidrivers
kubectl get sc
kubectl get pods -n kube-system -l app.kubernetes.io/name=csi-driver-nfs

If nfs.csi.k8s.io is listed, skip to step 3. If not, install it now:

# 2 - only if the driver is missing
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm repo update
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs \
  --namespace kube-system --version 4.12.0
kubectl get pods -n kube-system -l app.kubernetes.io/name=csi-driver-nfs -w   # Ctrl-C when Running
kubectl get csidrivers

A StorageClass for databases

Rather than reuse the general-purpose class from Lab 18, create a second one tuned for databases. A cluster normally has several classes - fast/slow, retained/disposable - and this is what that looks like.

# 3 -
cd ~
cat > sc-nfs-db.yaml <<'YAML'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-db
provisioner: nfs.csi.k8s.io
parameters:
  server: support01
  share: /storage2
  subDir: ${pvc.metadata.namespace}-${pvc.metadata.name}
  mountPermissions: "0777"
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
  - nfsvers=4.1
  - hard
YAML
kubectl apply -f sc-nfs-db.yaml
kubectl get sc

The two parameters that matter here, and the reason this lab needs the driver:

Setting	Why
`mountPermissions: "0777"`	the driver `chmod`s the new directory after creating it, so the `postgres` user (UID 26) can write. This is the fix for the permission failure.
`subDir: ${pvc.metadata.namespace}-${pvc.metadata.name}`	one directory per claim, named readably on the NFS server instead of a random volume ID
`reclaimPolicy: Retain`	a deleted database claim leaves its data on disk. For a database this is not optional
`mountOptions: hard`	an NFS client that retries forever rather than returning I/O errors - what a database needs

# 4 - the static PVs are not used from here on; remove the one that would
#     otherwise share /storage2 with this class
kubectl delete pv pv-storage2 --ignore-not-found
kubectl get pv

Note

Leaving pv-storage1 and pv-storage3 alone is fine - they simply sit Available and unused. Only pv-storage2 is removed because the new class provisions subdirectories inside the same /storage2 export, and having a static PV pointed at the export root as well is asking for confusion later.

Install the operator

# 5 -
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm repo update
helm install cnpg cnpg/cloudnative-pg -n cnpg-system --create-namespace
kubectl get pods -n cnpg-system -w        # Ctrl-C when Running

# 6 - what did the operator add to your cluster?
kubectl get crd | grep cnpg
kubectl api-resources --api-group=postgresql.cnpg.io
kubectl get deploy -n cnpg-system

Two things arrived: new kinds (clusters, backups, scheduledbackups, poolers) and one controller Deployment that watches them. That is the entire operator pattern - a CRD teaches the API server a noun, a controller gives it a verb.

# 7 - the new kinds behave exactly like built-in ones
kubectl explain cluster.spec | head -30
kubectl explain cluster.spec.storage

kubectl explain works on a CRD because the schema was registered with the API server. So do RBAC, describe, labels, and everything else.

Create a database

# 8 - the whole request
cd ~
cat > pgcluster.yaml <<'YAML'
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: pgtest
spec:
  instances: 2
  storage:
    size: 500Mi
    storageClass: nfs-db        # the class you just created
  postgresql:
    parameters:
      shared_buffers: "32MB"
YAML
kubectl apply -f pgcluster.yaml
kubectl get cluster pgtest -w             # Ctrl-C when it reports healthy
kubectl get pvc                           # created by the operator, via the class

Watch the PVCs appear on their own. Nobody wrote a PersistentVolume: the operator created a claim per instance, the class provisioned a directory per claim, and the driver set the permissions. If a Pod is stuck in ContainerCreating or crash-looping, check the events - kubectl describe pod <name> - and see the troubleshooting note at the end of this lab.

Fifteen lines. Now look at what the operator built from them:

# 9 -
kubectl get pods -l cnpg.io/cluster=pgtest -o wide
kubectl get pvc
kubectl get svc | grep pgtest
kubectl get secret | grep pgtest
kubectl describe cluster pgtest | tail -25

one Pod per instance, with a role label: one primary, one replica
one PVC per instance, bound to your NFS PVs
three Services: -rw (primary), -ro (replicas), -r (any)
generated Secrets with the superuser and application credentials

Writing all of that by hand is an afternoon. More importantly, the operator will keep it correct.

# 10 - connect and use it
kubectl exec -it pgtest-1 -- psql -c 'select version();'
kubectl exec -it pgtest-1 -- psql -c 'create table demo(id int); insert into demo values (1),(2);'
kubectl exec -it pgtest-1 -- psql -c 'select count(*) from demo;'

Watch it operate

# 11 - which Pod is the primary?
kubectl get pods -l cnpg.io/cluster=pgtest -L role

# 12 - kill the primary and watch the failover
kubectl delete pod "$(kubectl get pods -l cnpg.io/cluster=pgtest,role=primary -o name)"
kubectl get pods -l cnpg.io/cluster=pgtest -L role -w    # Ctrl-C after the switch
kubectl describe cluster pgtest | tail -15

The replica was promoted, the -rw Service now points at it, and the old primary rejoins as a replica. No Kubernetes controller could do this - it needs PostgreSQL knowledge. That knowledge is what an operator packages.

# 13 - the data survived
kubectl exec -it "$(kubectl get pods -l cnpg.io/cluster=pgtest,role=primary -o name | head -1 | cut -d/ -f2)" -- psql -c 'select count(*) from demo;'

Discovery

Infodiscovery

Task A. Find the password of the application user that the operator generated, without reading any file you created.

Show one answer

kubectl get secret | grep pgtest
kubectl get secret pgtest-app -o jsonpath='{.data.password}' | base64 -d; echo
kubectl get secret pgtest-app -o jsonpath='{.data.username}' | base64 -d; echo

Task B. Scale the database to three instances by editing nothing but the custom resource. Expected: a third Pod and a third PVC appear.

Show one answer

kubectl patch cluster pgtest --type merge -p '{"spec":{"instances":3}}'
kubectl get pods -l cnpg.io/cluster=pgtest -w
kubectl get pvc

You edited one field of one object. The operator did the rest - that is the difference between a chart and an operator.

Task C. Show every event the operator produced for this cluster, newest last.

Show one answer

kubectl get events --sort-by=.lastTimestamp | grep -i pgtest
kubectl describe cluster pgtest | tail -30
kubectl logs -n cnpg-system deploy/cnpg-cloudnative-pg --tail=30

Task D. Delete the Cluster object. Predict first: what happens to the Pods, and what happens to the data?

Show the answer

kubectl delete cluster pgtest
kubectl get pods -l cnpg.io/cluster=pgtest
kubectl get pvc

The Pods and Services go, because the operator owns them. The PVCs remain - same principle as a StatefulSet. Your data is never collateral damage.

Clean up

kubectl delete cluster pgtest --ignore-not-found
kubectl get pvc                                  # still there - Retain
kubectl delete pvc --all
kubectl get pv                                   # Released, not deleted
helm uninstall cnpg -n cnpg-system
kubectl delete ns cnpg-system

With reclaimPolicy: Retain the PVs and the directories on the NFS server survive on purpose. To reclaim the space:

kubectl delete pv --field-selector status.phase=Released
ssh student@support01 'sudo ls /storage2; sudo rm -rf /storage2/default-*'

Leave the nfs-db StorageClass and the CSI driver installed.

If the database will not start

The symptom is almost always storage permissions, and the events say so.

Symptom	Cause
PVC `Pending`, event `failed to provision`	the driver is missing or the class has a typo - `kubectl describe pvc <name>`
Pod `ContainerCreating`, mount errors	`nfs-common` missing on the node, or the export path is wrong
Pod crash-loops, log says `could not create directory` or `Permission denied`	`mountPermissions` missing from the StorageClass - this is the static-PV failure this lab exists to avoid
`initdb` complains the directory is not empty	a reused directory from an earlier attempt; delete the PVC and the subdirectory on the server

kubectl describe pvc -l cnpg.io/cluster=pgtest
kubectl logs -l cnpg.io/cluster=pgtest --tail=30 --prefix
kubectl logs -n cnpg-system deploy/cnpg-cloudnative-pg --tail=30
ssh student@support01 'sudo ls -la /storage2'

Note

Deleting the operator does not delete your Cluster objects or their data - the CRDs and custom resources survive, they simply stop being reconciled. That is worth knowing before you upgrade one.