A configuration service may be healthy while every request fails, and a request fleet may serve correctly while its configuration service is unavailable. Treating those components as one undifferentiated system hides this difference and encourages a damaging design: putting coordination work directly on the request path.

The control plane decides and distributes what the system should do. The data plane performs the high-volume work using an accepted local view of those decisions. Between them sits reconciliation: the machinery that turns desired state into observed, effective state despite delay, retries, partial rollout, and incompatible versions.

This separation is more than an architecture diagram. It defines availability, consistency, security, and operational contracts. A good data plane can continue useful work during a control-plane outage without accepting arbitrary stale state forever. A good control plane can evolve and repair configuration without becoming a synchronous dependency of every request.

Draw the Boundary Around the Hot Path

The data plane handles the workload the product exists to serve: route a packet, authorize an API request, answer a query, stream a record, or execute a scheduled task. Its path is frequent and latency-sensitive. The control plane handles lower-frequency decisions: declare routes, assign partitions, publish authorization policy, schedule capacity, rotate trust material, or initiate rollout.

Frequency alone does not determine the plane. A status query can be a control-plane API even when called often, and a rare customer request still belongs to the data plane. Ask whether an operation changes coordination state or executes against an already accepted state.

Concern Control plane Data plane
Primary operation Validate, plan, assign, and publish Execute requests or move data
Typical state Desired configuration and rollout intent Locally effective snapshot and runtime counters
Latency sensitivity Usually moderate Usually strict
Consistency need Ordered, auditable changes Fast reads from an internally consistent view
Failure response Pause change and reconcile later Continue, reject, or degrade by explicit policy
Scale dimension Number of objects and changes Request, byte, or task volume

Do not put a remote control-plane lookup in every data-plane request. If an API gateway asks a central policy service for the current route table before routing each request, the policy service is now part of the data path regardless of its name. Its latency, outage, and rate limits multiply across all traffic.

The reverse leak is also costly. Letting data-plane workers directly edit global desired state because they observe load gives every worker broad authority and creates unstable feedback. Workers should report observations; a bounded controller decides whether those observations justify a change.

Model Desired, Observed, and Effective State

Plane separation becomes concrete when state has distinct meanings:

  • Desired state is the control plane’s declared intent, such as route /reports to backend group B with policy revision 17.
  • Observed state is what an agent reports it has downloaded, validated, or applied.
  • Effective state is the immutable snapshot currently used for requests.

These values may differ legitimately during rollout. Pretending that a successful control-plane write means instant global effect creates false guarantees. The control plane should return an object identity and revision, then expose rollout status separately. Operators need to know whether revision 17 is merely stored, distributed to 60 percent of instances, or active everywhere required.

Desired state should be declarative where possible. “Backend group B has these endpoints and this policy” can be reconciled repeatedly. An imperative sequence such as “add endpoint, wait, remove old endpoint” is harder to recover after the third command is lost. Controllers can derive ordered actions from old and new declarations while recording progress.

Effective state must be internally consistent. A request must not see routes from revision 17 and credentials from revision 16 if those objects were validated together. Build or download a complete candidate snapshot, validate all references, then atomically replace the active pointer. In-place mutation exposes partially applied configuration to concurrent requests.

Every resource needs stable identity, ownership, and a monotonic revision or comparable version. Wall-clock timestamps are poor ordering tokens when writers or regions have uncertain clocks. A control-plane database transaction, consensus log position, or per-object generation gives reconciliation an unambiguous order.

Reconcile Instead of Assuming Delivery

Distribution can push changes, let agents pull them, or combine both. Push reduces notification delay but requires reachability and retry state. Pull works through restrictive networks and naturally repairs missed notifications, at the cost of polling delay. A common design sends a lightweight invalidation hint and also polls periodically for convergence.

Delivery is not application. The agent follows a state machine:

flowchart LR D[Discover desired revision] --> F[Fetch immutable bundle] F --> V{Validate schema, references, signature} V -->|invalid| Q[Reject and report condition] V -->|valid| P[Prepare local resources] P --> A[Atomically activate snapshot] A --> O[Report observed and effective revision] O --> D Q --> D

Every transition must be retryable. Fetching the same immutable bundle produces the same candidate. Preparing connections or indexes may need cleanup after interruption. Activation should be idempotent: applying the already active revision is a no-op. Reporting success can be repeated because the report identifies both agent and revision.

Reconciliation also repairs drift. An instance may restart from an old disk snapshot, miss a notification, or fail midway through preparation. Periodic comparison between desired and observed revision finds that divergence without an operator replaying a fragile command sequence.

Controllers must avoid hot loops. An invalid desired object should set a durable condition with a classified error and bounded retry schedule, not consume CPU indefinitely. Status writes should be rate-limited and aggregated so a fleet does not overwhelm the control plane while reporting the same outage.

Work Through an API Gateway Policy Rollout

Consider a multi-tenant API gateway. Administrators define hostnames, routes, backend pools, request limits, and authorization policy. Gateways execute requests at the edge. The goal is to publish a policy update without making the administrator API a dependency of edge traffic.

The control plane accepts a tenant-scoped change, verifies that routes have valid backends and policy references, assigns revision tenant-42/184, and compiles an immutable bundle. The bundle contains all data needed for request decisions plus a compatibility level and expiry policy. It is signed by a control-plane publishing identity.

An edge agent learns that revision 184 is desired, downloads it, verifies tenant identity and signature, parses it with size limits, and checks that the gateway binary supports its compatibility level. It constructs matchers and connection pools outside the active request path. Only after preparation succeeds does it atomically swap activeSnapshot from revision 183 to 184. In-flight requests retain revision 183; new requests use 184.

ts
type PolicyBundle = Readonly<{
  tenantId: string;
  revision: number;
  compatibility: number;
  validUntil: string;
  routes: ReadonlyArray<RoutePolicy>;
  signature: string;
}>;

type EffectiveSnapshot = Readonly<{
  tenantId: string;
  revision: number;
  validUntil: Date;
  router: CompiledRouter;
}>;

class TenantPolicyAgent {
  private active: EffectiveSnapshot | undefined;

  async reconcile(desiredRevision: number): Promise<void> {
    if (this.active?.revision === desiredRevision) return;

    const bundle = await this.repository.fetch(desiredRevision);
    this.verifier.verify(bundle);
    if (bundle.tenantId !== this.tenantId) {
      throw new Error('Policy bundle belongs to another tenant');
    }
    if (bundle.compatibility > this.supportedCompatibility) {
      throw new Error('Gateway cannot interpret policy bundle');
    }

    const candidate: EffectiveSnapshot = {
      tenantId: bundle.tenantId,
      revision: bundle.revision,
      validUntil: new Date(bundle.validUntil),
      router: await compileAndPrepare(bundle.routes),
    };

    this.active = candidate;
    await this.status.reportEffective(this.tenantId, candidate.revision);
  }

  route(request: GatewayRequest): GatewayResponse {
    const snapshot = this.active;
    if (!snapshot || snapshot.validUntil <= new Date()) {
      return unavailable('No acceptable policy snapshot');
    }
    return snapshot.router.dispatch(request);
  }
}

The sketch makes the contract visible: request routing performs no control-plane call, candidates cannot cross tenants, unsupported policy is rejected before activation, and stale operation has a bound. Production code would preserve a last-known-good snapshot on durable local storage and retire resources from an old snapshot only after in-flight readers release it.

If revision 184 is malformed for one gateway version, those instances continue on 183 and report a condition. The control plane can halt rollout or republish a corrected revision. It must not overwrite 184 in place; immutable artifacts make diagnosis and rollback deterministic.

Version Configuration and Rollouts Explicitly

Configuration is an API between planes. It needs compatibility rules just like a network protocol. A schema version says how bytes are decoded; a semantic capability says which behavior the data plane can execute. Keeping those separate avoids treating every optional field as a breaking format.

Prefer additive evolution. New control-plane writers should not publish a required feature until the target data-plane cohort advertises support. During rolling upgrades, the controller may compile different but semantically equivalent bundles for compatibility groups. Record which artifact each instance received so rollback does not rely on guesswork.

Rollout strategy limits blast radius. Validate statically, exercise a test fleet, canary one region or cohort, and advance only when acceptance and data-plane outcome signals remain healthy. Separate “bundle downloaded” from “bundle effective” and from “requests successful.” A syntactically valid policy can still route traffic incorrectly.

Rollback should publish or select a new desired revision that refers to known-good content. Moving desired state backward without a recorded transition can confuse monotonic agents and audit history. Immutable revisions plus an explicit rollback event preserve order: revision 185 can restore the semantics of 183 while remaining newer than 184.

Garbage collection is part of versioning. Retain enough old bundles for active instances, rollback, and investigation, but bound storage and credential exposure. An agent should report if it is pinned to a revision scheduled for retirement.

Define Degraded Operation Before an Outage

During control-plane unavailability, a data plane can use its last-known-good snapshot, fail closed, fail open, or offer reduced behavior. The correct choice depends on the invariant, not convenience.

Routing to an already approved backend can often continue for a bounded interval. Granting a new permission cannot. Certificate or revocation state may require a shorter validity window than cosmetic routing metadata. One global staleness duration is therefore usually too coarse; policies should classify resources by consequence.

Bootstrap is distinct from steady-state outage. An instance with no validated snapshot may be unable to serve safely. An instance that has served revision 183 for ten minutes may continue while attempting reconciliation. Persisted snapshots improve restart availability, but they must be authenticated, bound to the instance or tenant, and checked for expiry.

Avoid an arbitrary hard expiry that causes every instance to stop simultaneously during a prolonged control-plane incident. Add randomized refresh schedules and consider staged stale thresholds: warn, restrict sensitive operations, then stop when the safety bound is reached. The bound must remain enforceable; “serve stale indefinitely” silently transfers authority from the control plane to forgotten local state.

Data-plane feedback must also degrade safely. Buffer bounded status or metrics, aggregate repetitive conditions, and drop noncritical telemetry rather than blocking requests. Never let an unavailable reporting endpoint exhaust request threads or local disk.

Separate Security Authority

The control plane is high leverage: a valid configuration change can redirect or deny an entire fleet. Protect it with strong administrator authentication, narrow tenant and resource authorization, approval for high-impact changes, immutable audit history, and rate or scope limits. A valid login must not imply permission to publish every policy.

The data plane needs read-only access to exactly its assigned artifacts. Give agents distinct workload identities and bind bundles to tenant, environment, and audience. Signatures protect integrity and publisher authenticity through caches and distribution layers; encryption protects confidential policy where needed. Neither replaces authorization at publication and fetch time.

Keep administrative endpoints off public data-plane listeners where practical. Separate credentials, network policy, capacity pools, and monitoring. An attacker who can flood requests should not starve reconciliation, while a compromised data-plane process should not acquire the right to change desired state.

Treat configuration as untrusted input even when signed. A compromised administrator or control-plane bug can publish pathological regexes, oversized tables, cyclic references, or unsafe resource counts. Validate complexity and allocate under limits before activation. Defense in depth protects the fleet from legitimate authority producing invalid intent.

Verify Convergence and Failure Behavior

Test each plane independently, then test the protocol between them. Control-plane tests cover authorization, semantic validation, ordered revisions, compilation, audit, and rollback. Data-plane tests cover bundle parsing, reference validation, atomic activation, concurrent readers, expiry, and last-known-good recovery.

Failure-injection scenarios provide the discriminating evidence:

  • Drop notifications and verify periodic reconciliation converges.
  • Interrupt fetch and preparation at every boundary, then retry safely.
  • Deliver old, duplicate, corrupt, cross-tenant, and unsupported bundles.
  • Restart with a valid, expired, and tampered disk snapshot.
  • Partition the control plane while sustaining representative requests.
  • Roll out a valid but harmful route to a canary and verify automatic halt.
  • Restore old semantics through a newer rollback revision.

Operate with plane-specific indicators. For the control plane, track accepted and rejected changes, compilation failures, reconciliation backlog, and rollout age. For the data plane, track effective revision, staleness, activation failures, request outcomes by revision, and resources retained by old snapshots. Fleet-wide version skew and time to converge connect the two.

Plane separation adds machinery: snapshots, agents, versions, status, and eventual convergence. Small systems with static configuration may not need a distinct service, but they still benefit from keeping configuration loading outside request execution. The abstraction earns its cost when independent availability, safer rollout, or fleet coordination matters.

The durable rule is simple: the control plane declares and reconciles intent; the data plane executes from a validated local truth. Make the bridge versioned, observable, bounded under failure, and protected as a security boundary. Then coordination can fail without immediately stopping useful work, and request scale can grow without turning every request into a global control decision.