Kubernetes StatefulSet Gotchas in Production

A routine Kubernetes cluster upgrade was underway. The automation began cordoning and draining nodes sequentially. In the middle of the operation, a high availability, quorum based metadata service (such as ZooKeeper, etcd, or a Kafka controller quorum) lost consensus and went offline. The platform team was surprised. The service was deployed as a StatefulSet with three replicas, and a PodDisruptionBudget was active. Yet, the drain operation evicted two out of the three replicas simultaneously, violating the quorum requirements and stalling the entire database pipeline. The upgrade passed staging tests, but failed under production load.

This incident highlights a major trap in Kubernetes operations. StatefulSets look like Deployments with stable network names and persistent storage, making it tempting to treat them the same way. They are not. The guarantees of StatefulSets, such as ordered pod startup, sequential rollout, and persistent volume associations, are managed by the StatefulSet controller. However, cluster level operations like node drains and scheduling decisions are completely unaware of these ordinals. When these two layers interact, they create race conditions and failure modes that can take down your stateful workloads.

Understanding OrderedReady Semantics and Rollout Blocks

The default behavior of a StatefulSet is governed by the podManagementPolicy: OrderedReady configuration. This policy enforces strict sequential execution during scaling and updates. During a rolling update, the controller updates pods in reverse ordinal order, from the highest ordinal down to zero. For a StatefulSet with three replicas (pod-0, pod-1, pod-2), the rollout progresses as follows:

1. The controller terminates pod-2.
2. The controller schedules the replacement pod-2 with the new container image.
3. The controller waits for pod-2 to transition to the Running state and pass all active readiness probes.
4. Only after pod-2 is marked Ready does the controller proceed to terminate and update pod-1.

This ordering is designed to preserve application availability. By waiting for the highest ordinal to become healthy before touching its predecessor, the controller ensures that the cluster maintains replication quorum. However, this policy introduces a significant risk. The official Kubernetes documentation explicitly warns that using Rolling Updates with the default OrderedReady policy can leave a StatefulSet in a broken state that requires manual intervention to repair, as updated in the StatefulSet concepts guide in 2026.

If a bug in the new container image or a database migration failure causes the replacement pod-2 to fail its readiness checks, the rollout stalls. Because pod-2 never becomes Ready, the StatefulSet controller will block indefinitely and will never attempt to update pod-1 or pod-0. If the service experiences a crash on pod-1 while the rollout is blocked, the application can lose quorum. The entire rollout is held hostage by a single failing pod, and SREs must manually rollback the image configuration or delete the blocked pod to restore reconciliation.

The PVC Ownership Model and Garbage Collection

Another major difference between Deployments and StatefulSets is how they handle storage. Deployments typically share network attached storage or use ephemeral volumes. StatefulSets use the volumeClaimTemplates specification to dynamically provision a dedicated PersistentVolumeClaim (PVC) for each ordinal pod.

To prevent accidental data loss, Kubernetes separates the lifecycle of the pods from the lifecycle of their PVCs. When a StatefulSet pod is terminated or deleted, its associated PVC is not deleted. The PVC remains bound to the PersistentVolume (PV) in the cluster, ensuring that when the pod is rescheduled (whether due to a rollout or a node failure), it reconnects to the exact same storage volume, preserving the database state.

To manage this relationship, the StatefulSet controller writes owner references to the PVCs it creates. The garbage collector only deletes these PVCs after the owning pod has transitioned out of the active state. This sequencing is necessary to ensure the pod can cleanly unmount the volume before the storage backplane reclaims the underlying block device. If an SRE force deletes a stuck pod (using the --force --grace-period=0 flags), this lifecycle sequence breaks. The scheduler may attempt to bind the replacement pod before the volume is detached from the old host, leading to volume attachment locks that keep the new pod stuck in ContainerCreating for hours.

The Local PV Race Condition: Issue #128164

The risks of PVC garbage collection are magnified when using local PersistentVolumes. Unlike network attached storage (such as AWS EBS or Google Persistent Disk) which can be mounted on any node in a zone, a local PV is pinned to a specific physical host. This physical binding creates a scheduling race condition, which was documented in Kubernetes GitHub issue #128164, filed in October 2024.

The bug occurs when a pod and its associated PVC at a given ordinal are deleted at nearly the same time, a common occurrence during automated cluster scale downs or node decommissioning. Under the hood, the Kubernetes scheduler and the volume controller run as independent loops, relying on separate informers to track cluster state. Because the scheduler's pod informer often runs faster than its PVC informer, the scheduler can observe the replacement pod before it realizes that the associated PVC is in the process of being deleted. The sequence of events is detailed below:

This race condition specifically affects local PVs because they are bound to a single node's physical storage. The scheduler cannot choose an alternative node to resolve the conflict. The replacement pod remains stuck Pending until an administrator manually deletes the stale PV objects. As discussed in the GitHub issue, the primary mitigation is to prevent the scheduler from considering PVCs that are marked for deletion when scheduling new pods. A secondary fix is adding a PVC to pod owner reference so the scheduler can back off if the PVC is not owned by the pod being scheduled. However, as of late 2026, SRE teams operating local PVs must still watch for this race during node maintenance.

How Node Drains Bypass Ordinal Guarantees

The most common cause of StatefulSet outages in production is the node drain operation. A node drain is a cluster level administrative task. It cordons a node to prevent new pods from being scheduled on it, and evicts all active pods currently running on that host.

Crucially, the node drain process does not know about StatefulSet ordinals. It operates on a per node basis. The eviction API issues delete requests for pods based on their physical placement across the cluster, not their logical dependency hierarchy. If you have a three replica StatefulSet (pod-0, pod-1, pod-2), and pod-0 and pod-1 happen to be scheduled on separate hosts that are drained around the same time, both pods will be evicted simultaneously.

This parallel eviction violates the sequential shutdown assumption built into stateful systems. For example, in a Kafka cluster, broker-0 expects broker-1 to be online when it shuts down so it can replicate its remaining in flight messages. If a node drain evicts both brokers together, the cluster loses availability, causing client write failures. The StatefulSet's OrderedReady configuration is powerless here because the evictions are triggered by the node manager, not the StatefulSet controller.

The Local PV Node Lock

Using local PVs also complicates node drains. If a pod using network attached storage is evicted, the scheduler can run the replacement pod on any healthy node in the zone. Detaching and attaching the volume takes a few minutes, but the pod returns to service automatically.

If a pod using a local PV is evicted during a node drain, it is physically locked to that host. The local PV cannot be moved. Because the node is cordoned, the scheduler is forbidden from running the pod on that node, but it is also unable to run it anywhere else due to the PV node affinity. The pod gets stuck in a Pending state and remains offline until the physical node is uncordoned and returned to service. The application operates with degraded capacity throughout the maintenance window.

StatefulSet vs. Deployment

The table below summarizes the operational differences between Deployments and StatefulSets, compiled from the Kubernetes API reference guide in 2026:

The Production Checklist

To run stateful workloads safely on Kubernetes, implement these configuration patterns:

1. Configure a PodDisruptionBudget

Never rely on StatefulSet ordinals to protect availability during node drains. You must define a PodDisruptionBudget (PDB) that explicitly declares the minimum number of healthy replicas required for your quorum. During a drain, the eviction controller will block any eviction that violates this budget:

yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: zookeeper-pdb
  namespace: database-ns
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: zookeeper

2. Leverage Persistent Volume Retention Policies

Ensure that your StatefulSet defines a clear retention policy for PVCs. This ensures that PVCs are cleaned up automatically when the StatefulSet is scaled down, preventing orphaned volumes and reducing the risk of informer races:

yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: datastore
  namespace: database-ns
spec:
  serviceName: "datastore-service"
  replicas: 3
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Delete
  template:
    metadata:
      labels:
        app: datastore
    spec:
      containers:
      - name: datastore
        image: datastore:v2.1
        volumeMounts:
        - name: data
          mountPath: /var/lib/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Gi

3. Implement Application-Level Readiness Checks

A pod is not "Ready" simply because the application process has started. The readiness probe must check the internal sync state of the database. If a database replica has not caught up with the write log, it must remain unready. This ensures that the OrderedReady controller wait times are actually load bearing and represent real synchronization:

yaml
readinessProbe:
  exec:
    command:
    - /bin/sh
    - -c
    - "mysqladmin ping && [ $(mysql -e 'SHOW SLAVE STATUS\G' | grep 'Seconds_Behind_Master' | awk '{print $2}') -lt 10 ]"
  initialDelaySeconds: 30
  periodSeconds: 10

When to Avoid StatefulSets

StatefulSets are excellent for workloads that require stable names and ordinal properties, such as primary-replica database clusters or consensus engines. However, the ordering guarantees add operational complexity and increase the risk of blocked rollouts.

If your application can handle data replication internally and does not require fixed names, a Deployment configuration combined with a separate PVC strategy is often simpler. Modern cloud native databases like CockroachDB or Cassandra can run successfully under a Deployment model, allowing the scheduler to manage evictions and rollouts without the constraints of ordinal ordering.