Day 5 on Eriks training corner

Persistent storage

Mon, 01 Jan 0001 00:00:00 +0000

Persistent storage

A container’s writable layer dies with the container, and emptyDir dies with the Pod. Anything that must outlive a reschedule has to come from outside the cluster.

Three objects

graph LR PV[PersistentVolume the actual storage cluster-scoped] --- B((bind)) PVC[PersistentVolumeClaim a request for storage namespaced] --- B POD[Pod mounts the claim] --> PVC SC[StorageClass creates PVs on demand] -.optional.-> PV

PersistentVolume (PV) - a piece of storage that exists. Cluster-scoped.
PersistentVolumeClaim (PVC) - “I need 1 GiB, ReadWriteOnce”. Namespaced. The Pod references the claim, never the volume.
StorageClass - a factory that creates PVs automatically (dynamic provisioning).

This cluster ships with no StorageClass and no CSI driver, so at first nothing is created automatically: you write the PV yourself, pointing at one of the NFS exports on support01 (/storage1, /storage2, /storage3). That is the on-premise reality a cloud cluster hides from you - and Lab 35 fixes it by installing a driver.

Stateful workloads

Mon, 01 Jan 0001 00:00:00 +0000

Stateful workloads

Stateless apps are easy: kill a Pod, start another. State changes the rules.

Should it run in Kubernetes at all?

Run it in the cluster when	Keep it outside when
an operator handles failover, backup, upgrades	nobody on the team can operate the database
storage is solid (CSI, snapshots, tested restores)	storage is “an NFS share somebody set up”
you want the same tooling for everything	a managed service (Azure SQL, RDS) fits the need

The building blocks

graph TB SS[StatefulSet or operator CR] --> P0[pod-0] & P1[pod-1] P0 --> V0[(PVC data-0)] P1 --> V1[(PVC data-1)] V0 & V1 --> SC[StorageClass -> CSI] PDB[PodDisruptionBudget] -.protects.-> P0 & P1 AA[anti-affinity / topology spread] -.spreads.-> P0 & P1

Concern	Tool
stable identity, one volume per replica	StatefulSet `volumeClaimTemplates`
replicas on different nodes / zones	pod anti-affinity, `topologySpreadConstraints`
maintenance without losing quorum	PodDisruptionBudget
slow start (recovery, replay)	startupProbe
more space	`allowVolumeExpansion: true` + edit the PVC
point-in-time copy	VolumeSnapshot (CSI) / application backup
failover, replication, upgrades	an operator

Access modes decide the design

RWO (block storage, most cloud disks): one node at a time - one PVC per replica, replication done by the application.
RWX (NFS, CephFS, Azure Files): shared - convenient, but two writers on one database file is corruption, not high availability.

Backups are part of the design

A replica is not a backup: a DROP TABLE is replicated in milliseconds. Plan application-consistent backups (database dump, operator backups to object storage) and test the restore - Lab 39.

Monitoring with Prometheus and Grafana

Mon, 01 Jan 0001 00:00:00 +0000

Monitoring with Prometheus and Grafana

Three signals

Signal	Answers	Typical tools
Metrics	how much, how fast, how often - over time	Prometheus, Grafana
Logs	what exactly happened	Loki, Elasticsearch
Traces	where did the time go across services	OpenTelemetry, Tempo, Jaeger

Prometheus in Kubernetes

graph LR subgraph targets KS[kube-state-metrics object state] NE[node-exporter DaemonSet: CPU, disk, net] KL[kubelet / cAdvisor container usage] APP[your app /metrics] end P[Prometheus] -->|scrape every 30 s| KS & NE & KL & APP P -->|rules fire| AM[Alertmanager] --> N[mail / Teams / PagerDuty] G[Grafana] -->|PromQL| P

Prometheus pulls metrics over HTTP and stores them as time series: metric_name{label="value"} number.

Backup and disaster recovery

Mon, 01 Jan 0001 00:00:00 +0000

Backup and disaster recovery

What exactly can you lose?

Layer	Contains	Backed up by
etcd	every Kubernetes object of the cluster	`etcdctl snapshot save`
Resources per namespace / app	Deployments, Secrets, PVC definitions…	Velero, git (GitOps)
Volumes	the data	Velero (file system backup or CSI snapshots), application backups
Outside the cluster	DNS, certificates, registry, cloud resources	their own processes

	etcd snapshot	Velero
Scope	whole cluster, all or nothing	namespaces, labels, resource types
Volumes	no	yes
Restore to another cluster	no (same cluster identity)	yes - migrations
Typical use	control plane loss, fatal mistakes	app restore, namespace deleted, migration

Velero

A controller in the cluster plus a CLI. Backups go to object storage (BackupStorageLocation); volume data via CSI snapshots or the node agent (file-system backup with Kopia).

CRDs and operators

Mon, 01 Jan 0001 00:00:00 +0000

CRDs and operators

Custom Resource Definitions

A CRD teaches the API server a new kind. After you apply one, kubectl get, describe, explain, RBAC and every other tool work on it exactly as they do on Pods - because to the API server there is no difference.

kubectl get crd
kubectl api-resources | grep -v -E 'v1$|apps/v1' # what is not core Kubernetes
kubectl explain cluster.spec # works on CRDs too

A CRD alone stores data and does nothing with it. It is a table with validation.

Production operations and best practices

Mon, 01 Jan 0001 00:00:00 +0000

Production operations and best practices

Everything in this course, as one checklist.

Platform

Area	Practice
Control plane	3 control plane nodes (odd etcd quorum), or a managed service
Upgrades	one minor version at a time, every ~4 months; control plane first, then nodes (drain, upgrade kubelet, uncordon)
Certificates	kubeadm certs expire after 1 year: `kubeadm certs check-expiration`
Backups	etcd snapshots + Velero, tested restores
Capacity	headroom for one node failure + rolling updates
Access	OIDC for people, no shared admin kubeconfig, audit log to SIEM
Policies	PSA baseline everywhere, restricted where possible; admission policies
Network	default-deny NetworkPolicies per namespace
Observability	metrics, logs, alerts on symptoms, runbooks per alert

Workloads

Area	Practice
Images	pinned tags/digests, scanned, non-root, minimal
Resources	requests on everything, memory limits, quotas per namespace
Health	readiness + liveness (+ startup), no dependency checks in liveness
Availability	≥ 2 replicas, anti-affinity / spread, PDB
Config	ConfigMaps/Secrets, nothing baked into images; secrets from a vault
Identity	own ServiceAccount, no token unless needed, least-privilege RBAC
Delivery	GitOps (Argo CD / Flux), Helm or Kustomize, no `kubectl edit` in prod
Shutdown	handle SIGTERM, `terminationGracePeriodSeconds` fits the app

Incidents

graph LR D[detect alert] --> T[triage impact, severity] --> M[mitigate rollback, scale, failover] --> R[resolve root cause fix] --> P[postmortem blameless, actions]

Mitigate first, investigate second: kubectl rollout undo restores service while you look for the cause. Write the postmortem: timeline, impact, root cause, what went well, action items with owners.