Kubernetes Incident Lab

Real Kubernetes failures. Fixed on your own laptop.

Every incident is a ticket and a broken workload. You find the cause with your own kubectl, then a validator checks the live cluster. Hide the symptom and it says no.

  • CKA and CKAD practice
  • Interview prep
  • macOS, Linux, Windows with WSL2
  • Offline after setup

Every ticket arrives with a theory. Some are wrong.

These are real openings from the incidents. Each one sounds reasonable, and each one points somewhere the cause is not.

Incident 04, from the team lead

"The dashboard shows the worker nodes at low CPU utilisation, so we have plenty of room. I don't understand why Kubernetes won't place it."

Pod stuck in Pending
Incident 17, from SRE on-call

"Exit code 137, so it must be running out of memory. Somebody please double the memory limit."

Restarting every 20 seconds
Incident 31, from the release train

"Every deploy takes more than five minutes. We suspect the node is unhealthy or the container runtime is slow."

Deploys that take forever
Incident 41, from finance

"The nightly report job has not produced a report for days. Is the CronJob controller broken?"

A job that quietly stopped
Incident 46, from the rendering team

"There is no CrashLoop and no OOMKill, and the node has plenty of disk. Somebody wants to cordon the node."

Pods that keep disappearing
Incident 45, from platform security

"A teammate edited the rules twice, but the change keeps disappearing. Someone wants to bind cluster-admin to stop the tickets."

A role that will not stay fixed

Treat the ticket as a theory, not a fact. Collecting the evidence is the exercise.

Practise the skills exams and interviews test

Reading about Kubernetes is not the same as fixing it while something is on fire. This is the second kind of practice.

Interviews

Debugging questions are about how you think. Each incident trains you to collect evidence, rule out the wrong theory and explain the root cause out loud. Every walkthrough shows the evidence trail the way you would narrate it, and Pro adds a 150-question bank with model-answer outlines.

CKA and CKAD

Both exams are hands-on: a terminal, a live cluster and a clock. Here you work the same way, with kubectl against a broken workload and a check at the end. Troubleshooting is a major part of the CKA. This is practice for that skill, not a complete exam course.

How Kubernetes really works

Learn what the scheduler, controllers, admission, probes and RBAC actually do by watching them cause a failure. Later incidents have layered causes, so fixing one problem reveals the next.

How an incident runs

Short, hands-on and repeatable. Ten to forty-five minutes each, and a reset command puts everything back.

  1. Start it./lab.sh start 12 breaks something in its own namespace.
  2. InvestigateUse your own kubectl in a shell that can only reach the lab cluster.
  3. Validate./lab.sh validate 12 inspects the running cluster, not just the manifest you applied.
  4. LearnStuck? Three progressive hints. Passed? You get the production lesson, and with Core and above a full walkthrough.

Shortcuts it rejects

  • Deleting the probe that is failing
  • Raising the quota instead of finding what uses it
  • Granting cluster-admin to make a permission error stop
  • Creating an empty object so the error message changes

What it accepts

  • The earliest cause found and fixed
  • The running container showing the right configuration
  • A rollout that finishes with the replicas it should have
  • Later causes found after the first fix reveals them

Fifty incidents across fifteen areas

From a first outage to hard failures with several causes. Each one is a different way production goes wrong, so you meet more than the usual suspects. The chart shows how many incidents sit at each difficulty level.

7 Security6 Rollouts5 Scheduling4 Networking4 Storage4 Configuration3 Workloads3 Health checks3 Batch3 Resources3 Stateful workloads2 Admission1 Images1 Availability1 Lifecycle

The five free incidents

  • The Workload Won't Start
  • The Service Is Unreachable
  • A New Deployment Won't Start
  • The Pod Is Stuck Pending
  • An API Request Is Forbidden

Run them with the same tooling as every paid version. If they click, the other forty-five are the same lab with harder problems.

Start free. Go as deep as you need.

Every version is a download of the same lab, so moving up never means starting over.

Free

Five incidents to try the method

Free

No payment

  • Incidents 01 to 05
  • Three hints per incident
  • Production lesson after each pass
  • Notes template and completion summary
Recommended

Core

Thirty incidents with full solutions

See price

One payment

  • Incidents 01 to 30
  • Full walkthrough for every incident
  • Learning paths
  • Completion summary to show a manager
  • 12 months of maintenance updates

Pro

All fifty, plus interview and post-mortem material

See price

One payment

  • Everything in Core
  • The 20 hardest multi-cause incidents
  • 150-question interview bank
  • Post-mortem template and worked examples
  • Lifetime maintenance updates

Team

Pro for ten people, with a game-day kit

See price

One organisation, up to 10 named people

  • Everything in Pro
  • 90-minute game-day guide
  • Scoring sheet and manager summary
  • Script that merges everyone's results
  • Lifetime maintenance updates

Maintenance updates are fixes and compatibility updates for new Kubernetes and kind releases.

Is this for you?

Good fit

You know what a Pod and a Deployment are, and you want practice fixing things when they break, not another tutorial. Useful for on-call prep, CKA and CKAD study, and interviews.

Not a fit

You have never installed Docker or kubectl, you want a video course, or you need cloud-specific content for EKS, GKE or AKS.

What you need

Docker running with 4 GB of free memory, kubectl and kind, on macOS, Linux, or Windows with WSL2. ./lab.sh doctor checks all of it for you.

Questions

Will it touch my real clusters?

No. The lab creates its own kind cluster and its own kubeconfig, and its shell can only reach that cluster. One command resets an incident, and another removes the whole cluster.

Does it need the internet?

Once, during ./lab.sh setup, which downloads the cluster image and the small container images the incidents use. After that it works offline.

Does it run on Windows?

Yes, through WSL2 with Ubuntu and Docker Desktop. The download includes a step-by-step guide. Run the lab from your Linux home folder, not from a Windows drive.

Can I just read the answers?

Core, Pro and Team include a full walkthrough for every incident. Run ./lab.sh solution 12 before you have passed and it asks you to confirm first, so you meet the problem before the answer. Hints come in three steps, from a nudge to nearly the answer. The Free sampler has hints and the production lesson, but no walkthroughs.

What are maintenance updates?

Fixes and compatibility updates for new Kubernetes and kind releases. Core includes twelve months. Pro and Team include them for the life of the product.

Can my team use it?

The Team version is licensed for one organisation and up to ten named people, and comes with a facilitator kit for a 90-minute game day.

Can I get a refund?

All sales are final, which is why the free sampler exists. It uses the same tooling as every paid version, so you can install it, run real incidents and see the validator work before you pay. If the incident list matches what you want to practise, buy with confidence. The exceptions are a duplicate charge or a damaged download that cannot be fixed: contact the seller through your receipt and it will be put right.

How do I know it will run on my machine?

Install the free sampler first. It uses the same lab tooling as every paid version, and ./lab.sh doctor checks Docker, kubectl, kind and memory and tells you what to fix. If the sampler runs, so do the others.

Will it help with CKA, CKAD or interviews?

It practises the skill those both lean on: finding the cause of a failure in a live cluster with kubectl. The exams are hands-on and so is this, and each walkthrough lays out the evidence trail, root cause and fix the way you would explain it in an interview. It is not a complete exam course and does not cover every exam domain.

The next outage will not come with the answer attached.

Practise on broken clusters that cannot hurt anyone, and know it works because the validator says so.