Last post ended with me promising a real Kubernetes cluster. I built one. Then I destroyed it by closing VMware.

Here's what happened, why it wasn't recoverable, and the thing I learned about etcd that no course had bothered to tell me.

The setup

Three weeks ago I started studying for the CKA. Every guide says the same thing: don't use managed Kubernetes, don't use k3s, build it by hand with kubeadm. So I did — two Ubuntu VMs on VMware Workstation, one control plane, one worker, Calico as the CNI.

It took me a full day and about five different failures to get it standing. Swap that turned itself back on. containerd with the CRI plugin disabled by default. A pod network CIDR that overlapped my LAN. Each one taught me something, which was the point.

By the end of the day I had a healthy two-node cluster and I was pleased with myself.

Then I closed the VMware window with the VMs still running.

The symptom

Next morning, kubelet was screaming:

kubelet.go:3516] "Unable to register mirror pod because node is not registered yet"

Over and over, a few times per second. And kubectl was dead:

The connection to the server 192.168.188.131:6443 was refused

That second error is the useful one. 6443 means kubectl found my kubeconfig, resolved the API server address, tried to connect, and got nothing. This is not a client problem — the API server is genuinely down.

(If it had said localhost:8080, that's the opposite diagnosis: kubectl has no kubeconfig at all and fell back to its ancient default. Same-looking error, completely different fix. The port tells you which one you're dealing with.)

Diagnosing without kubectl

Here's the thing about a broken control plane: your main tool is gone. kubectl talks to the API server, and the API server is what's broken. So you drop a level, to the container runtime itself.

bash

sudo crictl ps -a
CONTAINER      STATE     NAME                      ATTEMPT
babd4b7288f55  Exited    etcd                      12
c87b60d1c1d52  Exited    kube-apiserver            11
0a39e01aaabcf  Running   kube-scheduler            6
0e36064fa5be6  Running   kube-controller-manager   3

Two things jump out. etcd and the API server are both dead, and the ATTEMPT column is in double digits — they're in a crash loop, dying and restarting over and over.

And the order matters. The API server depends on etcd. If etcd can't start, the API server has nothing to talk to and dies too. So the API server isn't the problem, it's a symptom. Same for all the Calico pods further down the list. You chase the thing at the top of the dependency chain, not the loudest error.

That's etcd.

Reading the logs

systemctl status tells you that something failed. It never tells you why. For that you need the actual logs, and since etcd runs as a container, that's crictl:

bash

sudo crictl logs --tail 30 babd4b7288f55

The first fifty lines looked completely healthy. etcd found its data directory, loaded its certificates, configured its listeners, reported "member-initialized":true. Everything normal.

Then, at the very end:

panic: freepages: failed to get all reachable pages
(page 164: multiple references (stack: [1216 379 115 23 164]))

goroutine 112 [running]:
go.etcd.io/bbolt.(*DB).freepages.func2()

That's the whole story right there.

What that error actually means

etcd stores its data in a file at /var/lib/etcd/member/snap/db, using an embedded key-value store called bbolt. bbolt organizes that file into pages arranged in a B+ tree, and every page should be referenced exactly once.

"page 164: multiple references" means one page is referenced twice. In a healthy database that's structurally impossible. It means the file is corrupted at the storage-engine level.

etcd writes with fsync specifically to survive crashes — that's the whole design. But fsync guarantees data reached the OS and the OS said it reached the disk. When a hypervisor kills a VM mid-write, that chain gets cut somewhere in the middle, and you can end up with a file that's half-updated in a way no amount of replay can fix.

There is no fsck for etcd. You don't repair this. You restore from a snapshot, or you rebuild.

Rebuilding, and the interesting part

I had no etcd snapshot. (Lesson filed.) So I took the ugly option: move the data directory aside and let etcd start from nothing.

bash

sudo mv /var/lib/etcd /var/lib/etcd.corrupted
sudo mkdir -p /var/lib/etcd
sudo chmod 700 /var/lib/etcd
sudo systemctl restart kubelet

etcd came up. The API server came up behind it. And then this:

Error from server (Forbidden): nodes is forbidden: 
User "kubernetes-admin" cannot list resource "nodes" at the cluster scope

I stared at that for a minute before it clicked, and it turned out to be the most useful thing I learned all week.

"Forbidden" is not "connection refused." The API server is up. It received my request. It authenticated me — it knows I'm kubernetes-admin. It just won't let me do anything.

That's because authentication and authorization live in completely different places:

  • Who you are comes from your certificate, on disk, in /etc/kubernetes/pki/. Untouched by the etcd wipe.
  • What you can do comes from RBAC objects — ClusterRoles and ClusterRoleBindings — which are stored in etcd. Which I had just emptied.

Valid identity, zero permissions.

The same split explained everything else I saw. The static pods came back on their own, because kubelet reads them from /etc/kubernetes/manifests/ on disk and doesn't need etcd to know about them. But CoreDNS, kube-proxy and all of Calico were simply gone — those were objects in etcd, and etcd was empty.

I ended up doing a full kubeadm reset and init rather than patching the pieces back. Not because the phases wouldn't have worked, but because it was my third time building this cluster and I wanted the reps.

What I'd do differently

Snapshot etcd on a schedule. It's one command and it would have turned a lost day into fifteen minutes:

bash

ETCDCTL_API=3 etcdctl snapshot save /opt/backup/etcd.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

Those three certificate flags aren't optional and they're not memorizable — but you never need to memorize them, because they're sitting right there in /etc/kubernetes/manifests/etcd.yaml as arguments to the container.

Snapshot the VMs too. VMware snapshots take two minutes and cover the whole class of problems where you break something at the OS level.

Shut VMs down properly. sudo shutdown -h now, not the X button. This entire post exists because I closed a window.

The takeaways

  • Read the port in a connection error. localhost:8080 is a missing kubeconfig. <ip>:6443 is a dead API server. Two completely different investigations.
  • systemctl status gives you the symptom; logs give you the cause. For containerized control plane components, that's crictl logs, not journalctl.
  • Chase dependencies upward. Four things were broken. Only one was the actual problem.
  • Some failures don't have a fix. Corrupted bbolt pages aren't a puzzle to solve, they're a restore-or-rebuild decision. Recognizing that fast is worth more than clever debugging.
  • Authentication and authorization live in different places. Certificates on disk, RBAC in etcd. That distinction explains a whole category of confusing errors.

What's next

I passed the CKA a few days ago. The cluster survived — it's now running on Omarchy with my second Linux machine joined as a worker node, on real hardware instead of VMs.

Next up: moving it to AWS. EKS, everything defined in Terraform, torn down at the end of every session so it costs a few dollars instead of a few hundred. Then real SLOs, alerting on burn rate instead of CPU, and deliberately breaking things to see whether the alerts actually fire.

Which, given how this post went, I have some practice at.

— Leo RH

Last Update: August 31, 2026