← All cheat sheets

KUBERNETES & CONTAINER INCIDENT RESPONSE

Plain-text reference · 17 KB. Read it, search it (Ctrl-F) or print it.

A triage reference for a suspected compromise of a Kubernetes cluster or a
container host: what to preserve first, where to look and how to inspect
each place read-only. It is for investigation and response, NOT for
attacking clusters. kubectl lines assume an admin kubeconfig on a trusted
workstation; node lines assume root on the node with containerd and
crictl; Docker hosts use the docker lines.

Placeholders: <ns> namespace, <pod> pod, <node> node, <sa> service
account, <cid> container ID, <image> image reference. Everything below is
read-only except the containment lines in section 1, which change the
cluster on purpose.

1. FIRST MOVES AND EVIDENCE PRESERVATION#

Do NOT delete the pod or the node first. A deleted pod takes its writable
layer, emptyDir volumes and container logs with it, and its Deployment
simply starts a clean copy. Isolate it, then collect.
Contain (these lines write to the cluster):
  kubectl cordon <node>                   # no new pods; running ones stay
  kubectl -n <ns> label pod <pod> quarantine=true --overwrite
  kubectl apply -f quarantine.yaml        # deny-all policy below
quarantine.yaml (no rules listed = all ingress and egress denied):
  apiVersion: networking.k8s.io/v1
  kind: NetworkPolicy
  metadata: {name: quarantine, namespace: <ns>}
  spec:
    podSelector: {matchLabels: {quarantine: "true"}}
    policyTypes: [Ingress, Egress]
Preserve:
  kubectl -n <ns> get pod <pod> -o yaml > <pod>.yaml
  kubectl -n <ns> describe pod <pod> > <pod>.describe.txt
  kubectl -n <ns> logs <pod> --all-containers --timestamps > <pod>.log
  kubectl -n <ns> logs <pod> -c <container> --previous --timestamps
                                          # last crashed run (error if
                                          # the container never restarted)
Flags: a NetworkPolicy only works if the network plugin enforces it, and
its effect on hostNetwork pods is undefined (a plugin may ignore them),
so test that the pod is really cut off. Traffic between the pod and its
own node is always allowed. Scaling a Deployment to 0 deletes its pods:
collect first.

2. WHAT CHANGED RECENTLY (ATT&CK T1610)#

Inspect:
  kubectl get events -A --sort-by=.lastTimestamp
  kubectl -n <ns> get events --field-selector involvedObject.name=<pod>
  kubectl -n <ns> rollout history deploy/<name>
  kubectl -n <ns> rollout history deploy/<name> --revision=2
                                          # pod template of revision 2
  kubectl get ns --sort-by=.metadata.creationTimestamp      # newest last
  kubectl get clusterrolebindings --sort-by=.metadata.creationTimestamp
  kubectl get clusterroles --sort-by=.metadata.creationTimestamp
API audit log (only if kube-apiserver runs with --audit-log-path; one
JSON event per line; managed clusters: read the provider's audit logs):
  jq -r 'select(.stage=="ResponseComplete" and
    (.verb|IN("create","update","patch","delete")))
    | [.requestReceivedTimestamp,.user.username,.verb,
       .objectRef.resource,.objectRef.namespace,.objectRef.name]
    | @tsv' audit.log
Flags: events are kept for 1 hour by default (kube-apiserver --event-ttl),
so save them early. A rollout nobody announced (CHANGE-CAUSE is often
<none>, which is normal); a namespace, ClusterRole or ClusterRoleBinding
created in the incident window; writes by a service account or from a
source IP outside your admin ranges.

3. WORKLOAD TRIAGE (ATT&CK T1610, T1611)#

Inspect:
  kubectl get pods -A -o wide             # pod IP = node IP: hostNetwork
  kubectl -n <ns> describe pod <pod>
  # the image digest that actually ran, per pod:
  kubectl get pods -A -o json | jq -r '.items[] | .metadata.namespace+"/"+
    .metadata.name+" "+([.status.containerStatuses[]?.imageID]|join(" "))'
  # pods with host namespaces, hostPath volumes or privileged containers:
  kubectl get pods -A -o json | jq -r '.items[] | select(
    .spec.hostPID or .spec.hostNetwork or .spec.hostIPC
    or any(.spec.volumes[]?; .hostPath)
    or any(.spec.containers[], .spec.initContainers[]?;
           .securityContext.privileged == true))
    | .metadata.namespace+"/"+.metadata.name'
  # service account and added capabilities per pod:
  kubectl get pods -A -o json | jq -r '.items[] | .metadata.namespace+"/"+
    .metadata.name+" sa="+.spec.serviceAccountName+" "+
    ([.spec.containers[].securityContext.capabilities.add[]?]|join(","))'
Flags: privileged: true; hostPID, hostNetwork or hostIPC; hostPath of /,
/etc, /var/run/docker.sock or /run/containerd/containerd.sock; added
SYS_ADMIN, SYS_PTRACE or NET_ADMIN; an image from a registry you do not
use. Control-plane static pods, kube-proxy and the CNI pods normally use
hostNetwork and hostPath, so compare with a known-good cluster.

4. RBAC REVIEW (ATT&CK T1098.006, T1078.001)#

Inspect:
  kubectl auth can-i --list --as=system:serviceaccount:<ns>:<sa>
  kubectl auth can-i create pods/exec -n <ns> \
    --as=system:serviceaccount:<ns>:<sa>
  # who holds cluster-admin:
  kubectl get clusterrolebindings -o json | jq -r '.items[]
    | select(.roleRef.name=="cluster-admin") | .metadata.name+" "+
    ([.subjects[]? | .kind+":"+(.namespace // "")+"/"+.name]|join(","))'
  # bindings to anonymous or unauthenticated users:
  kubectl get clusterrolebindings,rolebindings -A -o json | jq -r '.items[]
    | select(any(.subjects[]?; .name|IN("system:anonymous",
      "system:unauthenticated"))) | .metadata.name'
  # roles with a wildcard verb or resource:
  kubectl get clusterroles,roles -A -o json | jq -r '.items[]
    | select(any(.rules[]?; any(.verbs[]?, .resources[]?; . == "*")))
    | .metadata.name'
  # roles that can escalate, bind, impersonate, exec or reach nodes:
  kubectl get clusterroles,roles -A -o json | jq -r '.items[]
    | select(any(.rules[]?; any(.verbs[]?;
        IN("escalate","bind","impersonate")) or any(.resources[]?;
        IN("pods/exec","pods/attach","nodes/proxy",
           "serviceaccounts/token"))))
    | .metadata.name'
Flags: a service account or user bound to cluster-admin besides the
system:masters and kubeadm defaults; any binding to system:anonymous or
system:unauthenticated other than system:public-info-viewer (and the
kubeadm cluster-info role in kube-public); wildcards outside cluster-admin.
The built-in admin, edit and system:aggregate-to-edit roles contain exec
and impersonate by design: check who is bound to them. "kubectl auth
can-i --as=system:anonymous" fails because anonymous cannot ask the API;
read the bindings instead.

5. SECRETS AND SERVICE ACCOUNT TOKENS (ATT&CK T1552.007, T1528)#

Inspect (names and types only; never paste secret values into tickets):
  kubectl get secrets -A --field-selector \
    type!=kubernetes.io/service-account-token
  kubectl get secrets -A --field-selector \
    type=kubernetes.io/service-account-token  # long-lived legacy tokens
  # pods that read a secret through env, envFrom or a volume:
  kubectl get pods -A -o json | jq -r '.items[] | select(
    any(.spec.containers[]; any(.env[]?; .valueFrom.secretKeyRef)
      or any(.envFrom[]?; .secretRef))
    or any(.spec.volumes[]?; .secret)) | .metadata.namespace+"/"+
    .metadata.name'
  # pods that really got an API token mounted:
  kubectl get pods -A -o json | jq -r '.items[] | select(
    any(.spec.volumes[]?; .projected.sources[]?.serviceAccountToken))
    | .metadata.namespace+"/"+.metadata.name+" sa="+
      .spec.serviceAccountName'
Audit log, service accounts reading secrets:
  jq -r 'select(.stage=="ResponseComplete" and
    .objectRef.resource=="secrets" and (.verb|IN("get","list","watch"))
    and (.user.username|startswith("system:serviceaccount:")))
    | [.user.username,.verb,.objectRef.namespace,.objectRef.name]
    | @tsv' audit.log | sort | uniq -c | sort -rn
Flags: a token mounted into a pod that never calls the API (set
automountServiceAccountToken: false on the pod or the service account);
a hand-made service-account-token Secret (it does not expire); an
unexpected bootstrap token in kube-system; a service account listing
secrets across namespaces. A token copied out of a pod stays valid until
it expires or its pod or service account is deleted, and the token the
kubelet mounts is extended to up to 1 year by default (kube-apiserver
--service-account-extend-token-expiration), so deleting the pod is what
revokes it.

6. ADMISSION AND PERSISTENCE (ATT&CK T1053.007, T1543.005, T1036.005)#

Inspect:
  kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations
  kubectl get mutatingwebhookconfigurations -o json | jq -r '.items[]
    | .metadata.name as $n | .webhooks[] | $n+" "+.name+" -> "+
      (.clientConfig.url // (.clientConfig.service.namespace+"/"+
      .clientConfig.service.name))+" failurePolicy="+.failurePolicy'
  # static pods, on the node (kubeadm path; check staticPodPath):
  grep staticPodPath /var/lib/kubelet/config.yaml
  ls -la --time-style=full-iso /etc/kubernetes/manifests/
  # their mirror pods (owned by a Node, named <pod>-<node>):
  kubectl get pods -A -o json | jq -r '.items[]
    | select(any(.metadata.ownerReferences[]?; .kind=="Node"))
    | .metadata.namespace+"/"+.metadata.name'
  kubectl get daemonsets,cronjobs,jobs -A
  kubectl get ns --sort-by=.metadata.creationTimestamp
Flags: a webhook that calls an external URL or a service in an odd
namespace; failurePolicy: Ignore on a security webhook (it fails open); a
manifest in staticPodPath that your installer did not write (kubeadm
writes etcd, kube-apiserver, kube-controller-manager, kube-scheduler); a
DaemonSet in kube-system you cannot explain; a CronJob in a new namespace;
names that copy system ones (kube-proxy-x, coredns-backup). Admission
webhooks and static pods have no ATT&CK technique of their own.

7. NODE AND CONTAINER RUNTIME (ATT&CK T1609, T1611)#

On the node, containerd (or CRI-O) through crictl:
  crictl pods --namespace <ns>
  crictl ps -a --label io.kubernetes.pod.namespace=<ns>
  crictl ps -a --label io.kubernetes.pod.name=<pod>
  crictl inspect <cid> | jq '{pid: .info.pid, image: .status.image.image,
    imageRef: .status.imageRef, mounts: [.status.mounts[].hostPath]}'
  crictl logs --timestamps --tail 200 <cid>
  crictl images --digests
  ctr namespaces ls                       # k8s.io = kubelet; others: why?
  ctr -n k8s.io containers ls
  ctr -n k8s.io tasks ls                  # running tasks with host PIDs
Docker hosts:
  docker ps -a --no-trunc
  docker inspect -f '{{.HostConfig.Privileged}} {{json .Mounts}}' <cid>
  docker inspect -f '{{.HostConfig.RestartPolicy.Name}}' <cid>
  docker diff <cid>                       # A added, C changed, D deleted
  docker logs --timestamps <cid>
  docker history --no-trunc <image>
Flags: a containerd namespace other than k8s.io (moby is Docker's); a
container on a Kubernetes node that the kubelet did not start; restart
policy "always" on a container nobody deployed; docker.sock or
containerd.sock mounted into a container; files added under /tmp,
/dev/shm or a web root in docker diff.

8. READ-ONLY COLLECTION#

  docker export <cid> -o <cid>.tar        # filesystem; volumes not incl.
  crictl inspect <cid> > <cid>.inspect.json
  PID=$(crictl inspect <cid> | jq .info.pid)       # containerd nodes
  ls -la /proc/$PID/root/tmp              # the container's root fs
  tr '\0' ' ' < /proc/$PID/cmdline; echo
  tar --one-file-system -C /proc/$PID/root -cf <cid>-root.tar .
  nsenter -t $PID -n -- ss -tunap         # its network namespace
  nsenter -t $PID -m -- ls -la /tmp       # its mount namespace
Caveats: nsenter runs commands inside the container's namespaces; it
stays read-only only while the command does (ls, cat, ss). kubectl debug
is NOT read-only: an ephemeral container added to a pod can never be
removed or changed. kubectl debug node/<node> creates a pod running in the
node's PID, network and IPC namespaces with the node's root filesystem
mounted at /host (writable) even with --profile=general; delete it with
kubectl delete pod <name> --now when done. Hash every file you collect.

9. IMAGE PROVENANCE (ATT&CK T1525, T1204.003)#

Inspect:
  # images referenced by tag instead of digest:
  kubectl get pods -A -o json \
    | jq -r '.items[].spec.containers[].image' | sort -u | grep -v '@sha'
  crictl images --digests                 # digests present on the node
  docker buildx imagetools inspect <image>   # what the registry serves
  # signatures (Sigstore cosign; identity and issuer from your policy):
  cosign verify <image> --certificate-identity <signer> \
    --certificate-oidc-issuer <issuer>
  # example, Kubernetes' own images (identity from the Kubernetes docs):
  ID=krel-trust@k8s-releng-prod.iam.gserviceaccount.com
  cosign verify registry.k8s.io/kube-apiserver-arm64:v1.34.0 \
    --certificate-identity "$ID" \
    --certificate-oidc-issuer https://accounts.google.com
  # SBOM and build provenance stored as registry attestations:
  docker buildx imagetools inspect <image> --format '{{ json .SBOM }}'
  docker buildx imagetools inspect <image> --format '{{json .Provenance}}'
Flags: "no signatures found" on an image your policy says is signed; a
running imageID digest that your registry never built; a tag that now
points to a different digest than the one that ran; history steps that
download and run scripts. Many public images are unsigned, so a missing
signature alone does not prove tampering. Kubernetes signs each
per-architecture image; verifying the multi-arch tag reported "no
signatures found" in testing.

10. LOGS (ATT&CK T1685, T1070)#

On the node (systemd):
  journalctl -u kubelet --since "24 hours ago" -o short-iso --no-pager
  journalctl -u containerd --since "24 hours ago" -o short-iso --no-pager
  ls /var/log/pods/            # <ns>_<pod>_<uid>/<container>/<n>.log
  ls -l /var/log/containers/   # symlinks named <pod>_<ns>_<container>-<id>
  grep -E 'audit-(log-path|policy-file)' \
    /etc/kubernetes/manifests/kube-apiserver.yaml
Audit event fields: stage, verb, user.username, user.groups,
impersonatedUser, sourceIPs, userAgent, objectRef (resource, namespace,
name, subresource), responseStatus.code, requestReceivedTimestamp and the
annotation authorization.k8s.io/decision. Stages: RequestReceived,
ResponseStarted (long-running only), ResponseComplete, Panic; keep
ResponseComplete to count each request once.
  # exec and attach into pods:
  jq -r 'select(.stage=="ResponseComplete" and
    (.objectRef.subresource|IN("exec","attach")))
    | [.requestReceivedTimestamp,.user.username,.objectRef.namespace,
       .objectRef.name] | @tsv' audit.log
  # impersonation:
  jq -r 'select(.stage=="ResponseComplete" and .impersonatedUser != null)
    | [.user.username,.impersonatedUser.username,.verb,
       .objectRef.resource] | @tsv' audit.log | sort | uniq -c
  # denied requests (probing, stolen low-privilege tokens):
  jq -r 'select(.stage=="ResponseComplete" and .responseStatus.code==403)
    | [.user.username,.verb,.objectRef.resource] | @tsv' audit.log \
    | sort | uniq -c | sort -rn | head
  # anonymous requests that succeeded:
  jq -r 'select(.stage=="ResponseComplete" and
    .user.username=="system:anonymous" and .responseStatus.code < 400)
    | .requestURI' audit.log | sort | uniq -c
Flags: audit flags removed from the kube-apiserver manifest or the policy
set to level None (no flag = no audit log); gaps in the audit log or the
kubelet journal; exec into pods by an account that never does it; a
service account with a kubectl userAgent; source IPs outside your ranges.

QUICK TRIAGE ORDER#

1. Contain without deleting: cordon, quarantine label and deny-all policy;
   save pod YAML, describe output and logs.
2. Timeline: events (1 hour only), rollout history, the newest
   namespaces, roles and bindings, audit-log writes.
3. Workloads: privileged and host-namespace pods, hostPath, image digests.
4. RBAC: cluster-admin holders, anonymous bindings, wildcards, escalate,
   bind and impersonate, then can-i --list for each suspect account.
5. Secrets: which pods mount which secrets and tokens; secret reads by
   service accounts in the audit log.
6. Persistence: webhooks, staticPodPath, DaemonSets, CronJobs, lookalike
   names.
7. Node: crictl, ctr and docker views; containerd namespaces.
8. Collect read-only (export, /proc/<pid>/root, nsenter), then check
   image signatures and SBOMs.
9. Only then rotate credentials and rebuild nodes from known-good images
   (see the Kubernetes Security and Container Security sheets).

Commands and flags checked against the Kubernetes documentation (kubectl
reference, auditing, RBAC good practices, service accounts, network
policies, static pods, ephemeral containers and node debugging), the
cri-tools crictl docs, containerd ctr help, the Docker CLI reference, the
Sigstore cosign docs and nsenter(1); technique IDs per MITRE ATT&CK
Enterprise v20. Commands tested on a kind cluster (Kubernetes v1.34.0,
containerd 2.1.3, kubectl v1.34.0), Docker 29.6.2 and cosign v2.4.1.

Defensive reference on CyberRamen. Offensive / red-team sheets live on OffensiveRamen.com.