Auditing Kubernetes long-lived token Secrets without reading a single Secret

Since Kubernetes 1.24 a service account no longer gets a token Secret automatically. That does not mean your cluster has none. Anyone who ran kubectl create secret with type: kubernetes.io/service-account-token, and every cluster upgraded from an older release, can still have tokens that never expire, sitting in a namespace nobody looks at.

They are the classic leaked credential: long-lived, bearer, and usually forgotten. Finding them sounds like a one-liner. It isn't, and the obvious way to do it is a security problem of its own.

The problem with "just list the Secrets"

kubectl get secrets -A --field-selector type=kubernetes.io/service-account-token -o json

That returns each Secret with its data, meaning the token itself. Your audit tool now holds every long-lived credential in the cluster. If the tool is compromised, logs the response, or caches it, you have turned an inventory into a credential store.

Worse, there is a subtlety in the RBAC model that surprises people: the list verb on secrets is enough to read their values (a list returns full objects). Granting "just list" to an audit account is not a read-only-metadata grant. It is a grant to read every secret in scope.

Ask the API server for metadata only

The API server can return partial objects. Send this header on the list request:

Accept: application/json;as=PartialObjectMetadataList;g=meta.k8s.io;v=v1, application/json

You get names, uids, labels, annotations and timestamps back, and no data. In our lab test on a real k3s 1.30.5 cluster, the same request without that header returned data, and with it returned none. So the header is what keeps values out, not the verb.

We did not stop at trusting the header:

• The scanner parses the response with a reader that extracts only allow-listed fields. It never materialises data or stringData.
• If a server ignores the header and sends values anyway, the property is skipped unread, nothing is kept, logged or returned, and the scan records a warning: "The API server returned Secret values; they were discarded."
• The scan only uses GET. No get secret on individual objects, no watch.

Two lists per scope: one with fieldSelector=type=kubernetes.io/service-account-token for the legacy tokens, one with a label selector for the Secrets zenxPAM itself creates as revocation markers (see the previous post in this series).

What a finding tells you

For each long-lived token Secret we record the namespace, the service account, the Secret's uid, its age, and when it was last used. The last one is the useful part, and Kubernetes gives it to you for free: the API server writes a kubernetes.io/legacy-token-last-used label on the Secret.

• On Kubernetes 1.28 and later, no label means never used since tracking started. That is the "delete this" candidate.
• On 1.26 and 1.27 the information is partial; below that it is unknown. The scan says so rather than guessing.
• In our lab test the API server wrote the label 0.3 to 0.5 seconds after a token was used.

Four traps we hit (and fixed) building it

1. A page cap that retries once is a denial-of-service. Anyone who can create Secrets can attach up to 256 KiB of annotations. With a 1 MB page cap and one retry, a single planted object can stall the scan forever. The page size now keeps shrinking: 100, 25, 5, then 1, and it only fails if a single object is still over the cap.
2. Labels are attacker-copyable. A Secret carrying app.kubernetes.io/managed-by=zenxpam is not necessarily ours. A Secret is shown as managed only when its name is also the deterministic one the rotator uses.
3. A resume cursor with only a position skips namespaces. If someone edits the namespace list between pages, a position-only cursor resumes at a shifted index and the scan still ends "complete". The cursor now carries a fingerprint of the scope list, and a mismatch fails the scan instead of silently under-reporting.
4. The save validator and the scanner must accept the same config. If they disagree on what a valid namespace list looks like, a saved setting fails every scan.

None of these show up in a happy-path demo.

Permissions: use a separate account

The scan needs one verb, list on secrets, in the namespaces you choose (a Role per namespace), or cluster-wide with a ClusterRole. Because of the list-reads-values point above, do not reuse the rotation admin token for this. Create a separate, read-only cloud account limited to the namespaces you scan:

kubectl -n NS create role zenx-inventory --verb=list --resource=secrets
kubectl -n NS create rolebinding zenx-inventory --role=zenx-inventory --serviceaccount=NS:SERVICE_ACCOUNT

The scan itself never reads a single Secret, so get is not granted, and in our test the cluster refused get for that account while the scan still succeeded.

What happens after you find them

Findings are stale, never-used, or without an owner. From a finding you can open the rotation flow with the namespace and service account filled in. Rotation issues new, revocable tokens. It does not delete the old long-lived Secret: that stays until you remove it (kubectl -n NS delete secret NAME), because deleting a credential some forgotten consumer still uses is a decision for a person.

Honest limits

• Verified on a real single-node k3s 1.30.5 cluster through the full UI and services, with a separate read-only account. Not verified on EKS, AKS or GKE, against a large number of Secrets, or against a legacy token the cluster has marked invalid.
• Bound (projected) tokens are not stored anywhere, so they cannot be listed. Service accounts without a token Secret are principals, not credentials, and are not findings.
• The scan is read-only. It never changes the cluster.

See how zenxPAM inventories and rotates non-human credentials: https://zenxsys.com