You're upgrading your cluster. It's 2 AM because that's when traffic is lowest. You run kubectl drain node-3 and wait. And wait. The drain hangs. Nothing moves. Your maintenance window is burning.
Eventually you discover the problem: a PodDisruptionBudget is blocking the eviction. You either didn't know about the PDB, forgot it existed, or never considered how it would interact with your maintenance procedures.
This happens more often than you'd expect in real clusters. PodDisruptionBudgets are one of those Kubernetes features that seem optional until they become urgent.
What PodDisruptionBudgets actually do
A PodDisruptionBudget (PDB) tells Kubernetes how many pods from a workload must remain available during voluntary disruptions. The key word is voluntary. When something intentionally tries to evict a pod—a node drain, a cluster autoscaler removing a node, a spot instance termination handler—the PDB can block that eviction if it would violate the budget.
Involuntary disruptions don't respect PDBs. If a node catches fire (metaphorically or otherwise), those pods are gone regardless of what your PDB says. Hardware failures, kernel panics, network partitions—these are involuntary. The PDB only governs planned, API-driven pod evictions.
Here's a minimal PDB:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-pdb
namespace: production
spec:
minAvailable: 2
selector:
matchLabels:
app: webThis says: when evicting pods with label app: web, ensure at least 2 remain available. If you have 3 replicas and try to evict 2 simultaneously, the second eviction will be blocked until a replacement pod is running.
The mental model
When thinking about PDBs:
- They don't keep pods alive
- They don't prevent failures
- They only gate voluntary evictions
A PDB is a rate limiter on disruption, not a guarantee of availability. It tells Kubernetes "slow down" during planned maintenance—nothing more. And because PDBs are evaluated per eviction request, multiple concurrent disruptions (parallel drains, autoscaler actions) can interact in non-obvious ways.
How node drains interact with PDBs
When you run kubectl drain, Kubernetes doesn't delete pods directly. It evicts them through the Eviction API, which respects PDBs. The drain process:
- Cordons the node (marks it unschedulable)
- Attempts to evict each pod via the Eviction API
- For each eviction, checks if any PDB would be violated
- If a PDB blocks the eviction, retries with backoff until it succeeds or times out
The drain command keeps retrying blocked evictions. If you set --timeout, it will eventually give up. But kubectl drain has no default overall timeout and may wait indefinitely unless --timeout is set.
This is where things get stuck. Consider this scenario:
- You have a Deployment with
replicas: 3 - A PDB requires
minAvailable: 2 - All 3 pods are on the node you're draining
The drain tries to evict pod 1. The PDB checks: are there at least 2 Ready pods available not counting this one? Yes, pods 2 and 3. Eviction proceeds. Now pods 2 and 3 remain on the draining node, pod 1 is starting elsewhere.
The drain tries to evict pod 2. The PDB checks: are there at least 2 Ready pods available? We have pod 3 on this node, and pod 1 is... still starting. It's not Ready yet. So we have 1 Ready pod. Eviction blocked.
The drain waits. Pod 1 becomes Ready. Now we have 2 Ready pods (pod 1 on the new node, pod 3 on the draining node). Pod 2's eviction proceeds.
This cascading eviction is intentional. It's the whole point of PDBs—ensuring continuous availability during maintenance. But it means drains take time proportional to how long your pods take to start up.
When drains get permanently stuck
Sometimes drains don't take longer—they hang indefinitely. The common causes:
The replacement pod can't schedule. You evict pod 1, but the replacement can't start because all other nodes are full, or the pod has node affinity that no available node satisfies, or a required PersistentVolume is locked to the draining node. Pod 1 never becomes Ready, so subsequent evictions are blocked forever.
The replacement pod keeps crashing. Same outcome—the pod never reaches Ready state, so the PDB never allows further evictions.
You have a single-replica Deployment with a PDB. This is the classic mistake:
apiVersion: apps/v1
kind: Deployment
metadata:
name: critical-service
spec:
replicas: 1
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: critical-service-pdb
spec:
minAvailable: 1
selector:
matchLabels:
app: critical-serviceThis PDB says "always keep at least 1 pod available." With only 1 replica, evicting it would leave 0 available. The eviction is blocked. The drain hangs. Forever.
You can't drain a node with this pod on it unless you delete the PDB, scale up the deployment, or use --disable-eviction (which deletes pods directly, bypassing PDBs and graceful disruption guarantees, and can cause service interruption).
PDB covers pods without a controller. If you have bare pods (not managed by a Deployment, StatefulSet, etc.) and a PDB covers them, evicting them leaves no way to replace them. Without a controller to create a replacement, the PDB constraints never become satisfiable again, blocking evictions indefinitely.
minAvailable vs maxUnavailable
PDBs offer two ways to express your availability requirement:
spec:
minAvailable: 2 # At least 2 pods must remainspec:
maxUnavailable: 1 # At most 1 pod can be downYou can't use both. They seem equivalent but behave differently when replica counts change.
With minAvailable: 2 and 3 replicas, you can lose 1 pod. If you scale to 10 replicas, you can lose 8 pods. The absolute floor stays at 2.
With maxUnavailable: 1 and 3 replicas, you can lose 1 pod. If you scale to 10 replicas, you can still only lose 1 pod. The disruption tolerance doesn't scale with replicas.
Both support percentages:
spec:
minAvailable: 80% # At least 80% of selected podsspec:
maxUnavailable: 20% # At most 20% of selected podsPercentages round up for minAvailable and round down for maxUnavailable, always erring on the side of availability. Watch out for small replica counts: 1 replica with maxUnavailable: 30% rounds down to 0, blocking all evictions.
Which should you use?
maxUnavailable is usually more practical for stateless services. It says "I can handle losing N pods at once" regardless of scale. If you know your service can absorb 1 pod disappearing, maxUnavailable: 1 works whether you have 3 replicas or 30.
minAvailable makes sense when you have a hard minimum for correctness. Quorum-based systems (etcd, ZooKeeper, consensus protocols) need a specific number of nodes running. If your etcd cluster needs 2 of 3 nodes for quorum, minAvailable: 2 expresses that directly.
For percentage-based PDBs, maxUnavailable: 25% is common—it lets you drain one node at a time in a 4-node cluster without blocking.
unhealthyPodEvictionPolicy
Kubernetes 1.26 introduced the unhealthyPodEvictionPolicy field, which controls whether unhealthy pods (not Ready) can be evicted even when doing so would violate the PDB:
spec:
maxUnavailable: 1
unhealthyPodEvictionPolicy: AlwaysAllow
selector:
matchLabels:
app: webThe two options are:
IfHealthyBudget(default): Unhealthy pods count toward the budget. An unhealthy pod still blocks eviction if the budget is exhausted.AlwaysAllow: Unhealthy pods can always be evicted, regardless of the budget. This prevents stuck drains caused by crashing pods that will never become Ready.
Use AlwaysAllow for stateless workloads where a crashing pod provides no value and shouldn't block node maintenance. Stick with the default for workloads where even unhealthy pods might be serving traffic or holding state.
PDBs for stateful workloads
StatefulSets bring additional complexity. Pods have stable identities and often persistent volumes with node affinity. PDBs for StatefulSets need extra thought.
Storage constraints: If your StatefulSet uses local persistent volumes, the replacement pod can only schedule on the same node. You can't drain that node until the pod is gone, but the pod can't move because storage is locked there. PDBs make this worse by blocking the eviction. You need to either migrate the data first or accept downtime.
Ordered startup: StatefulSets start pods in order. If you're draining a node with pod-1 and the PDB blocks until a replacement is ready, that replacement won't start until pod-0 is healthy. Chain these dependencies across nodes and drains become very slow.
Headless services and client connections: Stateful workloads often have clients connected to specific pod IPs. A PDB ensures you don't evict too many pods at once, but clients still see disruption. The PDB doesn't make connections graceful—it limits concurrency.
For StatefulSets with 3 replicas that require quorum:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: etcd-pdb
spec:
maxUnavailable: 1
selector:
matchLabels:
app: etcdThis ensures you always have quorum (2 of 3) during voluntary disruptions. But remember: involuntary disruptions can still kill 2 nodes simultaneously. The PDB doesn't provide fault tolerance—it provides disruption governance.
Common mistakes
PDB on a single-replica workload. Already covered, but it's so common it bears repeating. If you set minAvailable: 1 on a 1-replica Deployment, you've created an undrainable pod. Either scale up to 2+ replicas, or remove the PDB entirely if the workload truly can't be replicated.
PDB selector matches no pods. Typos happen. Your PDB selects app: webapp but your pods have app: web-app. The PDB exists but protects nothing. Kubernetes doesn't warn you. Use kubectl get pdb -o wide to see how many pods match. Note that since Kubernetes 1.21 (policy/v1), empty selectors are rejected—you must specify at least one label. But a non-empty selector that matches nothing is still allowed.
Forgetting PDBs exist when troubleshooting stuck drains. You'll run kubectl get pods, see pods pending eviction, and wonder why they won't move. Check PDBs early:
kubectl get pdb -A
kubectl describe pdb <name> -n <namespace>The describe output shows current/desired healthy pods and any evictions being blocked.
PDB without corresponding readiness probe tuning. PDBs count Ready pods. If your readiness probe is slow (high initialDelaySeconds or frequent failures), replacement pods take longer to count as available, which slows drains. Tune your probes and your PDBs together.
Setting maxUnavailable: 0. This blocks all voluntary disruptions. Every drain hangs until you remove the PDB. Sometimes this is intentional (critical singleton that must never move), but usually it's a mistake.
Real scenarios
Cluster upgrades
GKE, EKS, and AKS all drain nodes during control plane or node pool upgrades. Your PDBs determine whether those upgrades succeed or stall.
Before upgrading:
# Check all PDBs in the cluster
kubectl get pdb -A
# Look for PDBs with ALLOWED DISRUPTIONS = 0
kubectl get pdb -A -o wideIf ALLOWED DISRUPTIONS is 0, that workload is at capacity—no pods can be evicted without violating the PDB. Either scale up the workload or remove the PDB before upgrading.
Node maintenance
When you need to patch a node's OS or replace hardware:
# Cordon first to stop new pods scheduling
kubectl cordon node-5
# Check which PDBs might block
kubectl get pdb -A -o json | jq '.items[] | select(.status.disruptionsAllowed == 0)'
# Drain with a timeout so you don't wait forever
kubectl drain node-5 --ignore-daemonsets --delete-emptydir-data --timeout=300sIf the drain times out, investigate which PDB blocked it:
kubectl get events -A --field-selector reason=FailedEvictionAutoscaler scale-down
The cluster autoscaler respects PDBs when removing underutilized nodes. If a PDB blocks eviction, the node won't scale down even if it's nearly empty. This can lead to cost waste—nodes sitting idle because one pod with a restrictive PDB is stuck there. The autoscaler logs will explicitly indicate when a PDB is blocking scale-down.
Review PDBs when investigating autoscaler behavior:
kubectl describe configmap cluster-autoscaler-status -n kube-systemkubectl commands for working with PDBs
# List all PDBs with status
kubectl get pdb -A -o wide
# Detailed status including blocked evictions
kubectl describe pdb <name> -n <namespace>
# Check which pods a PDB covers
kubectl get pods -n <namespace> -l <selector-from-pdb>
# See current disruption allowance
kubectl get pdb <name> -n <namespace> -o jsonpath='{.status.disruptionsAllowed}'
# Delete a PDB (allows drains to proceed)
kubectl delete pdb <name> -n <namespace>When a drain is stuck: the debugging flow
Follow this sequence:
- What pods are stuck on the node?
kubectl get pods -A -o wide --field-selector spec.nodeName=<node> - Which PDBs might be blocking?
Look at the ALLOWED DISRUPTIONS column. If it's 0, you've found your blocker.kubectl get pdb -A -o wide - Why is allowed disruptions zero?
Check if replacement pods are Pending, CrashLoopBackOff, or failing readiness probes.kubectl get pods -n <namespace> -l <pdb-selector> - Fix the root cause: Scale up the deployment, fix the crashing pod, free up node resources, or—as a last resort—delete the PDB.
When NOT to use PDBs
PDBs aren't always appropriate:
Single-replica workloads that can tolerate downtime. If your batch job or internal tool can be offline during maintenance, don't add a PDB. You'll block drains for no benefit.
Development and staging environments. Drains should be fast in non-production. PDBs add friction that's rarely worth it when availability doesn't matter.
Workloads that need to drain fast in emergencies. Sometimes you need a node evacuated immediately. PDBs can't distinguish between routine maintenance and urgent situations. If your incident response involves draining nodes rapidly, restrictive PDBs will slow you down.
DaemonSets. PDBs for DaemonSets are possible but usually wrong. DaemonSets have one pod per node by design—there's nothing to evict to elsewhere on the same node, and the pod can't move while the node exists. Drain operations use --ignore-daemonsets precisely because DaemonSet pods aren't migratable.
When you have no idea what availability you actually need. A PDB based on guesswork either blocks drains unnecessarily (too restrictive) or provides false confidence (too permissive). Measure your actual traffic patterns and failure modes first.
The preventive care mindset
PodDisruptionBudgets belong to a category of Kubernetes features you configure proactively and then forget about—until the moment they save you or the moment they block you. Like resource limits, pod anti-affinity, and network policies, they're infrastructure hygiene.
The right time to set up PDBs is when you deploy the workload, not when you're planning maintenance. Review them during your regular cluster hygiene sweeps. Test them in staging before you test them at 2 AM during a production upgrade.
Every stateless service with more than one replica should probably have a PDB with maxUnavailable: 1 or a reasonable percentage. Every stateful workload with quorum requirements should have a PDB that enforces that quorum. Everything else is a judgment call about how much you care about availability versus operational speed.
In practice, PDB issues often only surface during node drains or autoscaler activity, which is why they tend to go unnoticed until maintenance windows. The feature everyone forgets exists specifically so you don't have to remember to protect your workloads manually during every maintenance window. Configure it once, correctly, and let Kubernetes enforce it forever.