Extended Lab B - Ship a change

Extended Lab B - Ship a change

Infochallenge optional / after the course

Prerequisite: Extended Lab A finished and still running. You also need: a GitHub account and a personal access token.

Goal: change one line of C#, and follow it all the way into the running cluster without ever typing kubectl apply.

Starting point: Extended Lab A finished, with the app running from ghcr.io/ecaha/... and outbound internet from support01 and the cluster nodes.

Note

This lab changes a running system. There is nothing here that stands on its own - do Extended Lab A first.

The pipeline you are building:

  you edit source
        │  git push
        ▼
  GitHub Actions ──build──▶ ghcr.io/<you>/kubequiz-api:<tag>
        │                                    ▲
        │ commits the new tag into           │ pulled by the kubelet
        │ gitops/base/kustomization.yaml     │
        ▼                                    │
  git repository ◀──watches── Argo CD ──applies──▶ your cluster

Nothing in the cluster needs a GitHub credential, and nothing in GitHub needs a kubeconfig. Argo CD pulls. Keep that in mind when you get to Stage 5.

Check your work: ./labs/verify.sh b2, or ./labs/verify.sh b for all of it.

Stage 1 - Your own images

Task. Fork this repository, let its pipeline run, and end up with two container images in your own GitHub Container Registry that the cluster is allowed to pull anonymously.

Done when

  • your fork’s build-and-publish workflow has a green run
  • ghcr.io/<you>/kubequiz-api and ghcr.io/<you>/kubequiz-web exist as public packages
  • both are tagged latest and with a dated tag like 20260806-a1b2c3d

Check: none - this stage lives on GitHub. Stage 2 proves it worked.

  1. Fork https://github.com/ecaha/kubequiz on github.com (button, top right).
  2. On support01, point your existing clone at your fork and keep the original as upstream:
cd ~/kubequiz
git remote rename origin upstream
git remote add origin https://github.com/<you>/kubequiz.git
git remote -v

# GitHub wants a token, not your password. Settings -> Developer settings ->
# Personal access tokens -> Fine-grained -> repo contents: read and write.
git config credential.helper 'cache --timeout=36000'
  1. On the fork: Actions tab → I understand my workflows, enable them.
  2. Trigger a run - an empty commit is enough:
git commit --allow-empty -m 'trigger ci'
git push
  1. When it is green: Packages → each package → Package settings → Change visibility → Public. Private packages need an imagePullSecret, which is a different lesson.

Think about it. The workflow authenticates to GHCR with secrets.GITHUB_TOKEN, which you never created. Where did it come from, how long is it valid, and what can it do? permissions: at the top of the job is the answer.

Stage 2 - Run your build

Task. Make the cluster run your images instead of your instructor’s. Change it in git, not with kubectl set image.

Done when

  • kubectl get deploy kubequiz-api -o jsonpath='{..image}' contains your GitHub username
  • both Deployments are Ready and the site still works in your browser
  • the change is committed and pushed

Check: ./labs/verify.sh b2

Only the newName: lines change - the name: keys stay ghcr.io/ecaha/... because they identify which image in the manifests is being replaced, not where it comes from.

sed -i 's|newName: ghcr.io/ecaha/|newName: ghcr.io/<you>/|g' \
  gitops/overlays/lab/kustomization.yaml
git diff
git commit -am 'use my own images' && git push

kubectl apply -k gitops/overlays/lab      # the last time you do this by hand
kubectl rollout status deploy/kubequiz-api
kubectl get deploy kubequiz-api -o jsonpath='{..image}'; echo

Stage 3 - Hand the cluster to Argo CD

Task. Install Argo CD, reach its UI from your laptop’s browser through the relay and Gateway you already have, and give it ownership of the kubequiz namespace from gitops/overlays/lab with automated sync and self-healing.

Done when

  • kubectl -n argocd get app kubequiz reports Synced and Healthy
  • the UI opens at http://argocd.k8s.lab:30080 and you are logged in
  • kubectl scale deploy/kubequiz-api --replicas=7 is reverted within a minute, and the Argo CD UI shows why

Check: ./labs/verify.sh b3

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd rollout status deploy/argocd-server

# Argo CD serves HTTPS itself; run it insecure behind the Gateway instead of
# terminating TLS twice
kubectl -n argocd patch deploy argocd-server --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--insecure"}]'
kubectl -n argocd rollout status deploy/argocd-server

kubectl apply -f gitops/argocd/httproute.yaml
# add "192.168.56.101 argocd.k8s.lab" to your LAPTOP's hosts file

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo      # user: admin

# point the Application at YOUR fork first - Argo CD must watch the repository
# your CI pushes to, not the instructor's
sed -i 's|https://github.com/ecaha/kubequiz.git|https://github.com/<you>/kubequiz.git|' \
  gitops/argocd/application.yaml
git commit -am 'argocd tracks my fork' && git push
kubectl apply -f gitops/argocd/application.yaml
kubectl -n argocd get app kubequiz -w

kubectl scale deploy/kubequiz-api --replicas=7
kubectl get deploy kubequiz-api -w        # selfHeal puts it back

Think about it. Your kubectl scale was undone within seconds. Is that helpful or infuriating? Both answers are defensible - what makes the difference is whether the cluster or the repository is meant to be the source of truth, and whether everyone on the team agrees which it is.

Stage 4 - The whole loop

Task. Put your own name into the running application by editing source code and pushing it. You may not use kubectl apply, kubectl set image, or the Argo CD Sync button - the pipeline does all of it.

Two strings to change, one in each image, so that you prove both halves of the pipeline:

  • src/KubeQuiz.Api/Program.cs → the About constant
  • src/KubeQuiz.Web/wwwroot/index.html → the <title>

Done when

  • curl -H 'Host: kubequiz.k8s.lab' http://192.168.56.101:30080/api/about returns your text
  • the browser tab title shows your text
  • /api/whoami reports a version equal to the new image tag, not dev
  • the running image tag matches the tag your workflow just published
  • you did the whole thing with git push and waiting

Check: ./labs/verify.sh b4

sed -i 's|KubeQuiz - edit me in Program.cs|KubeQuiz - Erik was here|' \
  src/KubeQuiz.Api/Program.cs
sed -i 's|<title>KubeQuiz</title>|<title>KubeQuiz - Erik</title>|' \
  src/KubeQuiz.Web/wwwroot/index.html

git commit -am 'my name in the app' && git push

Then watch it travel:

# 1. the build            -> github.com/<you>/kubequiz/actions
# 2. the commit CI made   -> git pull && git log --oneline -3
# 3. Argo CD noticing     -> kubectl -n argocd get app kubequiz -w
# 4. the rollout          -> kubectl rollout status deploy/kubequiz-api
# 5. the proof
curl -s -H 'Host: kubequiz.k8s.lab' http://192.168.56.101:30080/api/about
curl -s -H 'Host: kubequiz.k8s.lab' http://192.168.56.101:30080/api/whoami

Time it. From git push to the new text being live. Most of that number is one specific step - which, and what would you do about it in a real project?

Stage 5 - Break it, then get out of it

Task. Ship something broken, watch what Kubernetes does with it, and recover - using git, not kubectl.

Do both:

  1. A bad image. Commit an image tag that does not exist in the registry.
  2. A bad application. Make the API fail its readiness probe on startup - for example, point PGHOST at a database that is not there.

Done when

  • for each case you can answer: was the site still up? - and say why
  • you can name the ReplicaSet that is stuck and quote the event that explains it
  • both are fixed by a git revert that Argo CD picks up on its own
  • kubectl rollout history deploy/kubequiz-api shows the whole story

Check: ./labs/verify.sh b5

sed -i 's|newTag: .*|newTag: nosuchtag|' gitops/overlays/lab/kustomization.yaml
git commit -am 'break it' && git push
kubectl get pods                       # ImagePullBackOff on the NEW pods only
kubectl get rs
kubectl describe rs <the new one> | tail -20
curl -s -H 'Host: kubequiz.k8s.lab' http://192.168.56.101:30080/api/about   # still fine

git revert --no-edit HEAD && git push
kubectl -n argocd get app kubequiz -w
kubectl rollout history deploy/kubequiz-api

The site stayed up because maxUnavailable: 0 means the old ReplicaSet is not scaled down until the new Pods are Ready - and they never were. The rollout is stuck, which is exactly what you want it to be.

Think about it. The bad image never took the site down. The bad configuration might have, depending on your probes. Which of the two would your monitoring have told you about, and how long would the other have sat there unnoticed?

Stage 6 - Optional: make the pipeline mean something

Pick any of these and implement it:

  • Fail the build on HIGH or CRITICAL findings (exit-code: '1' on the Trivy step). Then fix or justify what it finds.
  • Protect main so changes have to arrive by pull request, and confirm the CI-committed tag bump still works.
  • Add a real test project and make the build fail without it passing. Note that the workflow currently ends the test step with || true - find it, and decide whether that is honest.
  • Give the db-init Job an Argo CD PreSync hook (it already has one) and prove the ordering by making it fail.

Finished

./labs/verify.sh b

You changed a line of C# and it reached three PostgreSQL replicas’ worth of running application without you touching the cluster once. That is the whole argument for GitOps, and you have now done it rather than been told about it.