Back to articles

How to read Kubernetes events without getting overwhelmed

A systematic method for reading and filtering Kubernetes events. Not an answer to one specific problem, but a repeatable process you can apply to any incident.

Kubernetes events are one of the most underused diagnostic tools in cluster operations. They tell you what the system is doing — pod scheduling decisions, container restarts, volume mounts, image pulls — but the default experience is a wall of text that scrolls by too fast to parse. Most operators glance at kubectl get events, see fifty lines of noise, and move on to logs instead.

This article gives you a systematic method for reading and filtering Kubernetes events. Not an answer to one specific problem, but a repeatable process you can apply to any incident. By the end, you'll have a mental model for event triage and a set of kubectl commands you can use without reaching for the documentation.


What Kubernetes events actually are

Events are first-class API objects that record state transitions observed by Kubernetes components. They live in events.k8s.io/v1 (GA since Kubernetes 1.19), though the legacy core/v1 API still exists for backward compatibility. You don't need to worry about this distinction — kubectl get events handles the mapping automatically.

When the scheduler assigns a pod to a node, it creates an event. When the kubelet fails to pull an image, it creates an event. When a ReplicaSet scales up, it creates an event.

Every event has a type field: either Normal or Warning.

The distinction matters for filtering. If you're troubleshooting an incident, you often want to start with Warning events only.

A caveat: Events are not guaranteed to be emitted. Under heavy load, the EventRateLimit admission plugin may throttle event creation. In large clusters experiencing cascading failures, some events may never appear. Don't assume that the absence of events means nothing happened.

The opposite problem also exists: in large clusters, the challenge is often not missing events but too many — hundreds per second — making filtering essential.


The anatomy of an event

Before you can filter events effectively, you need to understand their structure. Here are the fields that matter:

Key fields

Field What it tells you
type Normal or Warning
reason Short code describing what happened: Scheduled, Pulled, Failed, FailedScheduling, BackOff, etc.
message Human-readable description with context (node names, error messages, resource quantities)
involvedObject The object this event relates to — a Pod, Node, Deployment, PersistentVolumeClaim, etc. Includes kind, name, namespace, and uid
reportingController Which component reported the event: default-scheduler, kubelet, replicaset-controller, etc. (In older clusters, this field is called source.component)
count How many times this event has occurred (events with identical fields are deduplicated and this counter increments)
firstTimestamp When the event first occurred
lastTimestamp When the event most recently occurred
eventTime High-precision timestamp (events.k8s.io/v1 API, but not always populated by all components)

The count field deserves attention. If you see count: 847 on a FailedMount event, that's not 847 separate problems — it's one problem that has been happening repeatedly. The combination of count and timestamps (eventTime, or the legacy firstTimestamp/lastTimestamp) tells you whether something is ongoing or historical.

Events are deduplicated based on reason, message, and involvedObject. If any of these differ — even slightly — Kubernetes creates a new event object instead of incrementing the count. This explains why you sometimes see multiple similar-looking events with different counts: small variations in the message text prevent aggregation.

The involvedObject reference

Every event points to the object it describes via involvedObject. This is how you connect events to your actual resources:

involvedObject:
  apiVersion: v1
  kind: Pod
  name: api-server-7f8b9c4d5-xk2pl
  namespace: production
  uid: 3d7e8f9a-1234-5678-abcd-ef0123456789

When you're looking at events cluster-wide, this reference is how you filter down to relevant objects. The uid ensures you're looking at the right instance, even if a pod with the same name existed previously.


The default experience and why it fails

Running kubectl get events in any active namespace gives you something like this:

LAST SEEN   TYPE      REASON              OBJECT                          MESSAGE
2m          Normal    Scheduled           pod/api-7f8b9-xk2pl             Successfully assigned...
2m          Normal    Pulling             pod/api-7f8b9-xk2pl             Pulling image...
2m          Normal    Pulled              pod/api-7f8b9-xk2pl             Successfully pulled...
2m          Normal    Created             pod/api-7f8b9-xk2pl             Created container...
2m          Normal    Started             pod/api-7f8b9-xk2pl             Started container...
90s         Normal    Scheduled           pod/worker-8c9d0-yl3qm          Successfully assigned...
90s         Normal    Pulling             pod/worker-8c9d0-yl3qm          Pulling image...
87s         Warning   Failed              pod/worker-8c9d0-yl3qm          Failed to pull image...
87s         Warning   Failed              pod/worker-8c9d0-yl3qm          Error: ImagePullBackOff
45s         Normal    BackOff             pod/worker-8c9d0-yl3qm          Back-off pulling image...

The problems:

  1. Normal events dominate. Healthy workloads generate far more Normal events than Warning events. The signal is buried.
  2. Default sort is confusing. The default sort order is based on a "last seen" concept, but is not strictly chronological and varies across Kubernetes versions. The output doesn't make temporal relationships obvious.
  3. Truncation hides details. The MESSAGE column gets truncated to fit terminal width, cutting off the most useful part.
  4. Namespace scoping. By default, you see only the current namespace. Problems often span namespaces.
  5. Retention is short. Events are typically garbage-collected around an hour after creation by default (controlled by --event-ttl on the API server, though this varies by distribution). If you're investigating something that happened this morning, the events may already be gone.

A systematic method for event triage

Here's a method that works across different incident types. It starts broad and narrows down.

Step 1: Get Warning events only

Start by filtering to Warning events. This eliminates the majority of noise immediately.

kubectl get events --field-selector type=Warning -A

The -A flag shows all namespaces. The --field-selector filters server-side, which is faster than piping through grep.

If you're troubleshooting in a specific namespace:

kubectl get events --field-selector type=Warning -n production

Step 2: Sort by time

Sorting events by time is less straightforward than it should be. Kubernetes events have multiple timestamp fields, and the right choice depends on what you're looking for:

The tradeoff: metadata.creationTimestamp is always present but shows first-seen time. lastTimestamp shows last-occurred time but may be unset. Pick based on your question:

# When did this problem start? (first seen)
kubectl get events --field-selector type=Warning -A --sort-by='.metadata.creationTimestamp'

# What just happened? (last occurred — use if your cluster populates this)
kubectl get events --field-selector type=Warning -A --sort-by='.lastTimestamp'

To reverse the order (most recent first), pipe through tac on Linux or tail -r on macOS:

# Linux
kubectl get events --field-selector type=Warning -A --sort-by='.metadata.creationTimestamp' | tac

# macOS
kubectl get events --field-selector type=Warning -A --sort-by='.metadata.creationTimestamp' | tail -r

Step 3: Get full event details for specific objects

Once you've identified a problematic pod (or node, or PVC), get its events with full messages:

kubectl describe pod api-server-7f8b9c4d5-xk2pl -n production

The Events section at the bottom of kubectl describe output shows the same events but with complete messages and better formatting. This is often more readable than raw event queries.

For events only (no other pod details):

kubectl get events --field-selector involvedObject.name=api-server-7f8b9c4d5-xk2pl -n production

Step 4: Check event counts

High count values indicate repeated occurrences. Get events in YAML format to see counts:

kubectl get events --field-selector type=Warning -n production -o yaml

Or use JSON output with jq for specific extraction:

kubectl get events --field-selector type=Warning -n production -o json | \
  jq '.items[] | {reason: .reason, count: .count, message: .message, object: .involvedObject.name}'

This gives you a compact view of what's failing and how often.

Step 5: Look for patterns across objects

Some problems affect multiple objects. Group events by reason:

kubectl get events --field-selector type=Warning -A -o json | \
  jq -r '.items[] | .reason' | sort | uniq -c | sort -rn

Output might look like:

     47 FailedScheduling
     12 BackOff
      8 FailedMount
      3 Unhealthy

Forty-seven FailedScheduling events suggests a cluster-wide scheduling problem — maybe node pressure, resource exhaustion, or taint/toleration issues. Now you know where to dig.


Common event patterns and what they mean

Not all warnings are created equal. Here's a reference for patterns you'll see repeatedly.

A real debugging walkthrough

Before the reference, here's what event-driven debugging looks like in practice:

$ kubectl get events --field-selector type=Warning -n production
LAST SEEN   TYPE      REASON             OBJECT                    MESSAGE
2m          Warning   FailedScheduling   pod/api-7f8b9-xk2pl       0/3 nodes are available: 3 Insufficient cpu.

What this tells you:

  1. FailedScheduling → scheduler event → placement problem
  2. Insufficient cpu → no node has enough allocatable CPU
  3. 0/3 nodes → checked all nodes, none qualified

Next steps: Either reduce the pod's CPU request, or scale the node pool. Check current allocation with kubectl describe nodes to see how much headroom exists.

This is the pattern: event reason tells you the category, message tells you the specifics, and the source tells you which component to investigate further.

Scheduling failures

Reason: FailedScheduling
Source (reportingController): default-scheduler

The scheduler couldn't place the pod on any node. The message tells you why:

What to do: Check node resources with kubectl describe nodes and look for the "Allocated resources" section. Check taints with kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints.

Image pull failures

Reason: Failed (with message about image pull), ErrImagePull, ImagePullBackOff
Source (reportingController): kubelet

The kubelet couldn't pull the container image. Common causes:

What to do: Check the exact error in the message. For auth issues, verify the imagePullSecret exists and is referenced correctly. For Docker Hub rate limits, the message explicitly mentions rate limiting.

Mount failures

Reason: FailedMount, FailedAttachVolume
Source (reportingController): kubelet, attachdetach-controller

The pod's volume couldn't be mounted. This is especially common with persistent volumes.

What to do: Check PV/PVC status with kubectl get pv,pvc. For multi-attach errors, verify the volume's access mode. For cloud volumes, check cloud provider logs.

Probe failures

Reason: Unhealthy
Source (reportingController): kubelet

A liveness, readiness, or startup probe failed.

What to do: Check probe configuration in the pod spec. A single probe failure is often transient. Repeated failures (high count) indicate a real problem — the application isn't responding as expected.

Container restarts

Reason: BackOff
Source (reportingController): kubelet

The kubelet is backing off from restarting a container that keeps failing. The message shows the backoff duration.

What to do: This is a symptom, not a cause. Look for preceding events that explain why the container failed: OOMKilled, probe failures, application errors. Check container logs with kubectl logs <pod> --previous.

Eviction events

Reason: Evicted
Source (reportingController): kubelet

The kubelet evicted the pod, usually due to node resource pressure.

What to do: Check node conditions with kubectl describe node <node-name>. Look for MemoryPressure, DiskPressure, or PIDPressure conditions. This indicates node-level resource exhaustion.


Filtering techniques

By involved object kind

Get events for all pods, all nodes, or all PVCs:

kubectl get events --field-selector involvedObject.kind=Pod -A
kubectl get events --field-selector involvedObject.kind=Node -A
kubectl get events --field-selector involvedObject.kind=PersistentVolumeClaim -A

By source component

Filter by which component reported the event. The field name depends on your cluster's Kubernetes version.

For clusters using events.k8s.io/v1 (Kubernetes 1.25+):

kubectl get events -A -o json | \
  jq '.items[] | select(.reportingController == "default-scheduler")'
kubectl get events -A -o json | \
  jq '.items[] | select(.reportingController == "kubelet")'

For older clusters or mixed environments, check both fields:

kubectl get events -A -o json | \
  jq '.items[] | select(.source.component == "kubelet" or .reportingController == "kubelet")'

By time window

Events from the last 10 minutes only:

kubectl get events -A -o json | \
  jq --arg cutoff "$(date -u -d '10 minutes ago' +%Y-%m-%dT%H:%M:%SZ)" \
  '.items[] | select((.eventTime // .lastTimestamp // .metadata.creationTimestamp) > $cutoff)'

(On macOS, use date -u -v-10M +%Y-%m-%dT%H:%M:%SZ instead. The fallback pattern eventTime // lastTimestamp // metadata.creationTimestamp handles clusters with different event API versions.)

Combining filters

Field selectors can be combined:

kubectl get events --field-selector type=Warning,involvedObject.kind=Pod -n production

For more complex filtering, JSON output with jq is more flexible than field selectors.


Building event monitoring into your workflow

Events are ephemeral by default. For production clusters, consider these approaches:

Extend event TTL (self-managed clusters only)

The kube-apiserver flag --event-ttl controls how long events are retained. The default is 1 hour. If you manage your own control plane, you can increase it:

--event-ttl=12h
Important

This flag is not configurable on managed Kubernetes services (EKS, GKE, AKS). For those environments, external event storage is your only option for longer retention.

The tradeoff is etcd storage. Events are small individually, but high-churn clusters generate many of them.

Ship events to external storage

Tools like Eventrouter, Kubewatch, or the Kubernetes Event Exporter can forward events to logging systems (Elasticsearch, Loki) or alerting systems (Slack, PagerDuty). This gives you event history beyond the TTL and enables alerting on specific event patterns.

Watch events in real time

During active troubleshooting, watch events as they appear:

kubectl get events --watch -A

Or with filtering:

kubectl get events --watch --field-selector type=Warning -A

This is useful when you're deploying a change and want to see what happens.

Use the kubectl-events plugin

The kubectl-events plugin (available via krew) provides a cleaner interface than raw kubectl get events:

kubectl krew install events
kubectl events -A --watch

It handles timestamp sorting more reliably and produces better-formatted output. If you debug clusters regularly, it's worth installing.


Quick reference: commands for common situations

Something's wrong, what's happening cluster-wide?

# Sort by first-seen (when did problems start?)
kubectl get events --field-selector type=Warning -A --sort-by='.metadata.creationTimestamp'

# Sort by last-occurred (what just happened?) — if your cluster populates this
kubectl get events --field-selector type=Warning -A --sort-by='.lastTimestamp'

# Reverse order: most recent at top (Linux: tac, macOS: tail -r)
kubectl get events --field-selector type=Warning -A --sort-by='.metadata.creationTimestamp' | tac

Why won't this pod start?

kubectl describe pod <name> -n <namespace>

What's failing most often?

kubectl get events --field-selector type=Warning -A -o json | \
  jq -r '.items[] | "\(.reason) \(.involvedObject.kind)/\(.involvedObject.name)"' | \
  sort | uniq -c | sort -rn | head -20

Is this problem getting worse?

kubectl get events --field-selector involvedObject.name=<name> -n <namespace> -o yaml | \
  grep -E '(count|eventTime|firstTimestamp|lastTimestamp):'

What happened to pods in this namespace in the last hour?

kubectl get events --field-selector involvedObject.kind=Pod -n <namespace> \
  --sort-by='.metadata.creationTimestamp'

The mental model

Think of Kubernetes events as a transaction log for cluster operations. They record what controllers and kubelets are doing, not what applications are doing. Events tell you:

A useful shortcut when triaging — check reportingController to know where to look:

They don't tell you why your application is returning 500 errors or why queries are slow — that's what application logs and metrics are for.

The effective workflow is: events first for infrastructure problems, logs for application problems, metrics for trends over time. Events are your starting point when something that was working stops working.


Summary

Kubernetes events are noisy by default but valuable when filtered. The method:

  1. Start with Warning events only (--field-selector type=Warning)
  2. Sort by time to see what's current
  3. Use kubectl describe for full messages on specific objects
  4. Check counts to distinguish ongoing problems from historical ones
  5. Group by reason to find patterns across objects

Keep these commands accessible. When something breaks at 2 AM, you don't want to be searching the documentation.