Kaigents
Get Started
← All articles
kubernetesagent-sandboxwarm-poolsobservabilitycontrollers

Agent Sandbox v0.5.6 Fixes the Warm-Pool Race That Was Silently Duplicating Agent Pods

How strict OwnerReference UID mapping, semantic template hash checks, and cache lag resilience eliminate TOCTOU races under heavy concurrent agent load

Kaigents
Agent Sandbox v0.5.6 Fixes the Warm-Pool Race That Was Silently Duplicating Agent Pods

Your billing alert fires at 3:00 AM, showing a sudden spike in LLM API usage. You open your terminal and run kubectl get pods -n agent-sandboxes. Instead of the expected single worker pod running a code-execution tool, you see two pods associated with the same agent run. They both have similar names, both show running status, and both are actively processing the exact same user payload. One worker has duplicated the execution, written to the database twice, and triggered duplicate API billing events.

This is the silent duplicate-pod symptom. In high-throughput AI agent platforms, ephemeral sandboxes are created and destroyed in milliseconds. To eliminate cold-start latency, platform engineers use pre-warmed sandbox pools. However, when these pools meet Kubernetes controller cache lag, the result is a classic Time-Of-Check-To-Use (TOCTOU) race condition. The controller sees a SandboxClaim, checks its local informer cache, fails to see that a pod has already been claimed, and assigns a second pod to the same sandbox.

When you run LLM-driven agents that can execute arbitrary Python code or interact with external APIs, this duplication is more than a minor scheduling nuisance. It directly impacts your database consistency, external API rate limits, and LLM provider token costs. The agent assumes it is executing in an isolated, single-use sandbox, but under the hood, the Kubernetes control plane has cross-wired the execution environments. Understanding why these warm pools are where agent sandboxes break first requires analyzing the synchronization model of custom controllers.

How SandboxWarmPool Coordinates Ephemeral Environments

The Agent Sandbox project manages high-performance, isolated environments in Kubernetes using three primary Custom Resource Definitions (CRDs). The first is the SandboxTemplate, which acts as an administrator-defined blueprint specifying the container image, resource limits, environment variables, and security policies for the sandboxes. The second is the SandboxWarmPool, which maintains a set of pre-warmed, unassigned sandbox pods based on a SandboxTemplate. The third is the SandboxClaim, which is the user-facing request to adopt and provision an instance from the pool.

Pre-warming is essential because cold-starting a Kubernetes pod—pulling the container image, allocating IP addresses, mounting volumes, and initializing the container runtime—can take anywhere from two to ten seconds. For an interactive agent that needs to run a quick Python snippet or check a filesystem, a multi-second delay ruins the user experience. By maintaining N ready-to-use standby pods in a SandboxWarmPool, the Agent Sandbox controller reduces cold-start latency to less than 50 milliseconds.

When a SandboxClaim is created, the SandboxClaim controller reconciles the claim by selecting an available pre-warmed pod from the active warm pool. The controller adopts the pod by applying the necessary owner references, labels, and annotations. This transition reassigns the pod to the user's specific sandbox workspace. Simultaneously, the SandboxWarmPool controller detects that the number of standby pods has dropped below the desired replica count and immediately schedules a new pod to replenish the pool. This continuous cycle of claim, adoption, and replenishment keeps the pool ready for subsequent requests.

The Anatomy of a Stale Adoption Race

In theory, the transition of a pod from the warm pool to an active SandboxClaim is clean and instantaneous. In practice, the Kubernetes API server and controller runtime rely on an asynchronous, event-driven architecture that is highly susceptible to informer cache lag under heavy concurrent load. This lag is the root cause of the stale-adoption race condition.

When a controller reconciles a custom resource, it does not query the Kubernetes API server directly for every read operation. Doing so would overwhelm the API server under high-volume workloads. Instead, the controller runtime uses Informers, which maintain a local in-memory cache of the cluster state. The local cache is updated asynchronously via a watch stream from the API server. This means there is a non-zero time delay between an object being updated in the API server and that update being reflected in a controller's local cache.

Under high concurrency, this informer cache lag leads to a TOCTOU race. For example, during a sudden burst of agent activity, multiple SandboxClaims are created simultaneously. The claim controller reconciles the first claim, selects an available pre-warmed pod from its local cache, and updates the pod's metadata to adopt it. However, before the pod's updated metadata is synced back to the local caches of other concurrent reconciliation threads, a second claim reconciliation begins. The second thread reads the stale local cache, sees the same pod as still available and unassigned, and attempts to adopt it as well.

This problem becomes even more severe when templates are updated. Issue #764 in the agent-sandbox repository highlights a race condition during template rollouts. When a platform team modifies a SandboxTemplate spec, the SandboxWarmPool controller triggers its update strategy. If the update strategy is configured as Recreate, the controller must delete the existing standby pods and create new ones with the updated template hash. If a SandboxClaim is submitted during this rollover, the controller's stale local cache might return a pod that has already been marked for deletion or is still running the old template version. The controller mistakenly adopts the stale pod, leading to an environment running outdated code.

Similarly, Issue #418 describes a warm-pool pod adoption race where two separate warm pods were assigned to a single Sandbox. The controller checked if a pod was available, saw two pods in the cache, and assigned both to the claim because of a failure to handle concurrent write collisions. This resulted in duplicate service endpoints, meaning the ingress or internal routing layer sent requests to both pods simultaneously, duplicating every single execution.

Deep Dive into the v0.5.6 Controller Hardening

The release of Agent Sandbox v0.5.6 on August 20, 2026, introduces critical structural fixes to resolve these warm-pool race conditions. Rather than relying on simple metadata annotations or retry loops, the update hardens the controller's transaction boundaries and synchronization logic.

The most critical fix is Pull Request #1337, titled "fix: enforce strict Sandbox-to-Pod mapping," authored by @alanhuangch. Prior to this release, the controller relied primarily on mutable pod annotations to map a Sandbox to its backing pod. Under high concurrency, these annotations could fail to write or become stale, causing the controller to spin up duplicate pods. PR #1337 solves this by enforcing OwnerReference UIDs as the authoritative, immutable Sandbox-to-Pod mapping. During reconciliation, the controller checks the owner references of the pods directly. If it detects multiple pods owned by the same Sandbox UID, it refuses to proceed. The controller fails closed, updating the Sandbox status to Ready=False with the reason MultiplePods, and emits a Kubernetes warning event. This failure mode prevents the controller from executing commands in duplicate environments.

Additionally, v0.5.6 addresses the stale-adoption race during template updates under Pull Request #1078. The controller now enforces strict semantic blueprint and content hash checks. It compares the agents.x-k8s.io/pod-template-hash label on candidate warm pods against the computed hash of the active template before adopting them. If a pod's hash does not match, the controller rejects it, ensuring that stale template versions are never adopted.

To handle transient API server conflicts, Pull Request #1072 introduces cache lag resilience. When the createSandbox function encounters an AlreadyExists error due to informer cache lag, the controller does not fail immediately or trigger an exponential backoff that would stall the queue. Instead, it applies a bounded 200ms requeue and sets the SandboxCreatePending status condition. This brief, bounded pause allows the local informer cache to synchronize with the API server, ensuring that subsequent reconciliation attempts read the correct state without workqueue thrashing.

Observability and operational parameters also received significant upgrades. Pull Request #1290, authored by @yuzhiquan, introduces configurable flags to the controller: --sandbox-warm-pool-readiness-grace-period (defaulting to 5 minutes) and --sandbox-warm-pool-unschedulable-recheck-interval (defaulting to 1 minute). In elastic environments where node auto-provisioning or large image pulls introduce scheduling delays, these flags prevent the controller from marking a warm-pool pod as dead prematurely.

Furthermore, the release publishes 18 previously undocumented example architectures, demonstrating how to integrate these hardened warm pools with orchestrators like n8n and tools like the Pi coding agent.

Concrete YAML Blueprints and Alert Rules

To deploy a reliable warm pool in production, you must use the updated extensions.agents.x-k8s.io/v1beta1 API schema and configure the appropriate update strategies. The following YAML manifest defines a production-ready SandboxWarmPool that uses the Recreate update strategy to handle template rollouts cleanly.

apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxWarmPool
metadata:
  name: dynamic-python-pool
  namespace: agent-sandboxes
spec:
  replicas: 10
  updateStrategy: Recreate
  sandboxTemplateRef:
    name: python-execution-template

When an agent needs to execute code, the application submits a SandboxClaim specifying the target warm pool:

apiVersion: extensions.agents.x-k8s.io/v1beta1
kind: SandboxClaim
metadata:
  name: claim-user-exec-8921
  namespace: agent-sandboxes
spec:
  warmPoolRef:
    name: dynamic-python-pool

Monitoring these pools is critical to catching cache lag and scheduling bottlenecks before they impact users. The v0.5.6 release ships with an opt-in ServiceMonitor and starter alert rules under Pull Request #1355. You can enable these rules by setting metrics.prometheusRule.enabled=true in your Helm values. Below is a custom PrometheusRule manifest designed to alert your platform team if all controller targets go down or if sandbox suspension latencies spike, utilizing the new Prometheus metrics introduced in Pull Request #1143:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: agent-sandbox-alerts
  namespace: monitoring
spec:
  groups:
  - name: agent-sandbox.rules
    rules:
    - alert: AgentSandboxControllerMetricsTargetsDown
      expr: up{job="agent-sandbox-controller"} == 0
      for: 5m
      labels:
        severity: critical
    - alert: SandboxResumeLatencySpike
      expr: histogram_quantile(0.95, sum(rate(sandbox_client_resume_latency_ms_bucket[5m])) by (le)) > 200
      for: 2m
      labels:
        severity: warning

By tracking sandbox_client_resume_latency_ms at the 95th percentile, you can detect when warm-pool depletion or informer cache lag is forcing claims to block on pod initialization, allowing you to scale the pool replicas or adjust the controller's readiness grace period.

Upgrading and Adjusting Your Cluster Parameters Today

To eliminate the warm-pool adoption race in your cluster, you must upgrade your controller deployment to v0.5.6. This upgrade requires applying the consolidated installation manifest which includes the core controller, custom resource definitions, and extensions. Run the following command to apply the update:

kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/download/v0.5.6/sandbox-with-extensions.yaml

After applying the manifest, verify that your controller pods have restarted and are running the v0.5.6 image. You should then update your controller deployment configuration to tune the warm pool parameters. If you are operating in a multi-tenant cluster where node scaling can take several minutes, adjust the startup flags to prevent false-positive failures:

spec:
  template:
    spec:
      containers:
      - name: controller
        image: registry.k8s.io/agent-sandbox/controller:v0.5.6
        args:
        - --sandbox-warm-pool-readiness-grace-period=8m
        - --sandbox-warm-pool-unschedulable-recheck-interval=2m

This configuration increases the readiness grace period to 8 minutes and sets the recheck interval to 2 minutes, giving your cloud autoscaler sufficient time to provision nodes and pull heavy runtime images. Once applied, monitor your logs for any MultiplePods warning events to verify that the controller is successfully failing closed and preventing duplicate pod execution across your entire fleet.

Sources