Kubernetes service account token rotation: how to actually revoke the old token

You rotate a Kubernetes service account token. The new one works. You update the consumer, close the ticket, and feel good.

Then someone asks: "Is the old token dead?"

It isn't. A token minted through the TokenRequest API stays valid until its exp claim, whether that is in one hour or in seven days. "Rotation" that only creates a new credential is key issuance. The leak window you were trying to close is still open.

This post explains the API feature that makes revocation possible, the extra machinery you need around it, and what we measured when we built it into zenxPAM, Zenxsys's agentless, just-in-time privileged access platform.

What we found when we ran the naive rotation

Before changing anything we ran the existing rotation path against a real single-node k3s 1.30.5 cluster. It worked, and it had gaps:

• A rotation never invalidated the previous token. Depending on the requested lifetime, that is up to a 24 hour to 7 day window where a copied token still authenticates.
• No preflight, so a missing RBAC verb surfaced as a failed rotation halfway through.
• No check that the new token actually authenticates before the old one is dropped.
• No grace window for consumers that pick up the new token a few minutes late.
• No way back if the new token turns out to be wrong.

Rotation without verification, grace and rollback is how you get a 2 a.m. page.

The primitive: bound-object tokens

POST /api/v1/namespaces/{ns}/serviceaccounts/{sa}/token accepts spec.boundObjectRef. If you bind the token to an object, the API server also checks that the object still exists and has the same UID whenever the token is presented. Delete the object and the token stops working.

Bind to a Pod and the token lives and dies with the Pod. That is what projected volumes do. For a token that a CI system or an external workload holds, there is no Pod to bind to. A Secret works fine as the anchor. We call it a marker Secret: type Opaque, no data, only a name and a UID.

We proved it on the lab cluster before building on it: request a token bound to a Secret, delete the Secret, poll with the token. In our lab test the API server refused the token about 10 seconds later (9.2 to 11.0 seconds across runs). The delay is the API server's authentication cache. A design that promises instant revocation would be lying.

Two slots, A/B, per identity

Each managed identity gets two marker slots in the target namespace:

zxp-<first 12 hex of identity id>-a
zxp-<first 12 hex of identity id>-b

Both carry the labels app.kubernetes.io/managed-by=zenxpam and zenxpam.io/nhi=<id>. The current token is bound to one slot. The next rotation uses the other. That is what lets the old token keep working during a grace window while the new one is already live.

The rotation, in order

 resolve admin credentials ──► validate target ──► read current token (claims only)
            │
            ▼
 pick the free slot ──► create marker Secret ──► TokenRequest bound to it (uid from create response)
            │
            ▼
 verify: SelfSubjectReview with the NEW token must say system:serviceaccount:<ns>:<sa>
            │
            ▼
 keep previous token in the vault ──► write new token ──► grace window? ─┬─ yes: old token stays valid
                                                                         └─ no:  delete old marker Secret

Details that matter:

1. The new token is proven before anything switches. A SelfSubjectReview with the new token must return the expected service account. A 401 fails the rotation. A 403 or 404 (a hardened or old cluster that does not allow the check) is not a failure, but the step is marked skipped with a warning rather than silently passing.
2. Failure after the marker Secret is created always compensates. Delete the new marker, restore the vault, delete the retained copy. No orphaned Secrets, no half-rotated state.
3. The grace window is capped at the old token's remaining lifetime. An expired token needs no revoking.
4. Finalize deletes the old marker Secret and then polls the old token for up to 30 seconds. Because of that 10 second auth cache, "refused" is a pass and "still accepted after 30 s" is a warning with evidence, not a hidden assumption.
5. Roll back first proves the previous token still authenticates, then restores it and deletes the new marker. If the old token is already dead, nothing is changed.

Least privilege for the rotator itself

A rotator with cluster-admin would just move the problem. The admin service account needs, in the target namespace only:

Permission	Scope
create on serviceaccounts/token	resourceNames: the target service account
create on secrets	namespace wide (Kubernetes cannot restrict create by name)
get, delete on secrets	resourceNames: the two slot names only

No cluster-wide rights. The honest footnote: because create cannot be name-limited, the rotator can create Secrets in that namespace, but it can only read or delete its own two markers.

The part nobody expects: the rotator is a confused-deputy risk

A system that takes an API URL and an admin token from configuration and then calls the URL is a gadget for exfiltrating whatever token it is given. Three rules in zenxPAM close that:

• The cluster address and CA come only from the cloud-account connector, never from the identity being rotated.
• The admin token and CA vault keys must start with K8S_. The server refuses any other name, so a connector cannot be pointed at an unrelated vault secret and a cluster address you control.
• Private or internal API hosts are refused unless an operator has put them on an allowlist.

Honest limits

• Verified on a real k3s 1.30.5 cluster, not on EKS, AKS or GKE. Managed clusters may cap token lifetime or restrict SelfSubjectReview; the preflight reports the lifetime the API server grants.
• If you turn binding off (KubeBindToSecret=false), the old token cannot be revoked and stays valid until expiry. The result says so, with the end time.
• Revocation lags about 10 seconds. Plan the grace window with that in mind.

Why this belongs in a PAM platform and not a cron job

You can script all of this with kubectl. What you cannot easily script is the evidence: which step ran, in which order, who approved the rotation, what the verification returned, whether the old token was confirmed refused. In zenxPAM every preflight, rotation, roll back and cleanup writes an evidence record with timestamped steps, the outcome and credential fingerprints (last four characters only, never a value), and the action is audited. The answer to "is the old token dead?" is a record, not a recollection.

Next up in this series: how the same pattern (verify new, keep old, revoke on a timer, roll back on demand) works for cloud keys that have no equivalent of a bound object.

See how zenxPAM handles non-human identities: https://zenxsys.com