Service Discovery and Health Checking

A service name such as payments.internal looks stable, but the processes behind it are not. Instances start, drain, move, fail, lose dependencies, and reappear with new addresses. Service discovery turns a logical identity into a changing set of reachable endpoints. Health checking decides which of those endpoints should currently receive work. Neither decision is instantaneous or perfectly informed.

That uncertainty is the central design fact. A registry can contain a process that died seconds ago. A process can answer a shallow probe while being unable to serve real requests. A healthy instance can be removed because a congested network delayed probes. Clients can cache different membership versions and still all be behaving correctly according to their contracts.

The right objective is therefore not a magical always-current list. It is a bounded-convergence system whose ownership and failure semantics are explicit. Registration identifies candidates; health evidence changes their eligibility; caches trade freshness for availability; observability reveals disagreement. Request-placement policy comes afterward and is deliberately outside this article’s focus.

Model Discovery as Time-Varying Membership #

A discovery response is better modeled as a versioned snapshot than as a timeless lookup:

service: payments
revision: 18427
endpoints:
  - id: pay-a17, address: 10.8.4.21:8443, zone: east-a, state: ready
  - id: pay-b03, address: 10.8.7.09:8443, zone: east-b, state: draining
validUntil: 2026-05-24T12:00:15Z

The endpoint identity should be stable for one process incarnation even if an address is reused later. Metadata may include zone, protocol, port name, deployment version, capacity class, and tenancy boundary. Keep it limited to facts needed for routing or policy. Turning the registry into a general configuration database increases update volume and makes stale metadata harder to reason about.

Membership has a lifecycle. An instance is created, registers or becomes discoverable through an orchestrator, proves readiness, serves traffic, enters draining, and is finally removed. Every transition needs an owner. Self-registration gives the process direct control but requires credentials, renewal, and cleanup after crashes. Platform registration lets an orchestrator derive membership from desired and observed workload state, reducing application code but tying discovery to that platform’s correctness.

Leases bound stale self-registration. An instance renews before expiry, and the registry removes it when renewals stop. A short lease improves failure convergence but amplifies write load and makes transient pauses dangerous. A long lease lowers control-plane traffic but leaves dead endpoints visible longer. Deregistration on graceful shutdown helps normal deployments, yet leases remain necessary because crashes do not execute cleanup.

The registry itself must survive faults without inventing conflicting membership. A strongly consistent registry makes updates and reads easier to order but may reject writes or stale reads during a partition, depending on its policy. An eventually consistent registry can remain locally available while regions disagree. The correct choice depends on the cost of a stale endpoint versus the cost of discovering no endpoint. Whichever model is chosen, expose revision and age so downstream components can detect what they are using.

Choose Where Resolution Happens #

Discovery can happen through DNS, a client library, a local agent, or an intermediary proxy. The wire format matters less than where membership is cached and who owns refresh behavior.

DNS discovery maps a name to address records, sometimes with service records for ports and priorities. It works with standard clients and has broad operational support. Its limitation is distributed caching: authoritative time-to-live values are hints unless every resolver and application honors them, and many APIs hide record age. DNS is suitable when endpoint changes tolerate cache delay and the application does not need rich per-instance metadata.

Client-side discovery lets each client query or watch a registry and choose an endpoint. It can use zone and protocol metadata, react quickly to updates, and avoid a central data-path hop. The cost is a substantial client contract: caching, watch reconnection, health interpretation, placement, telemetry, and compatible behavior must exist in every language and version.

Proxy-side discovery places those responsibilities in a load balancer, sidecar, node agent, or gateway. Applications connect to a stable local or network address while the proxy watches membership. This centralizes policy and upgrades, but adds another component and can hide endpoint-level detail unless telemetry propagates it. A node-local agent is a hybrid: one watcher maintains a cache and clients use a local API.

Push watches reduce polling delay but do not remove the need for refresh. A watch can disconnect silently, miss a compaction window, or resume from an obsolete revision. A robust watcher obtains a full snapshot, applies ordered deltas, detects revision gaps, and periodically reconciles with another snapshot. Until the first valid snapshot arrives, its fail-open or fail-closed behavior must be deliberate.

Give Health States Precise Meanings #

“Healthy” is too broad for a useful interface. At minimum, distinguish startup, liveness, readiness, and draining.

• A startup probe says initialization has not yet exceeded its allowed window. It prevents a slow but progressing process from being killed by liveness checks.
• A liveness probe says the process can make progress and should not be restarted. It should detect deadlock or a permanently broken runtime, not every dependency outage.
• A readiness probe says the instance is presently eligible for new work under a stated service contract.
• A draining state says existing work may finish but no new work should begin.

Conflating liveness and readiness creates restart storms. If a database outage makes every service fail liveness, the platform restarts otherwise functional processes, destroys warm caches, and adds startup load while the dependency remains down. The service should usually become unready for operations it cannot fulfill while remaining live enough to recover when the database returns.

Readiness itself may be operation-specific. A replica can serve reads while rejecting writes. A worker may accept ordinary jobs but not a model requiring an unloaded artifact. One global Boolean forces either unsafe routing or unnecessary removal. Separate ports, capabilities, or endpoint subsets can express these differences, but avoid a combinatorial health matrix. Publish only distinctions the caller can actually use.

Probe endpoints must be cheap, bounded, and truthful. A liveness check can validate the event loop or worker heartbeat without synchronously querying every dependency. A readiness check may inspect local dependency state maintained by background monitors rather than creating fresh downstream traffic per probe. Include a strict deadline; a probe that hangs consumes the very resources needed for recovery.

Avoid returning sensitive internals in unauthenticated probe bodies. Status codes and coarse reason codes are enough for routing. Detailed dependency errors belong in protected telemetry, where they can be aggregated without exposing topology or credentials.

Work Through a Checkout Deployment #

Consider a checkout service with instances in two zones. Each process requires a database connection pool and a locally loaded tax-rule bundle before it can serve POST /checkout. Deployment starts version 42 alongside version 41.

The process lifecycle is:

1. Register the new incarnation as starting, or let the orchestrator create that membership record.
2. Load configuration and tax rules, establish a minimum usable database pool, and begin background dependency monitoring.
3. Mark the checkout capability ready only after a synthetic in-process validation succeeds.
4. On shutdown, switch to draining, wait for admitted requests up to a bound, then close listeners and deregister.

Suppose the registry publishes revisions 700 through 704 during the rollout. A proxy watching from revision 701 disconnects and reconnects after the registry has compacted deltas through 703. It must not apply revision 704 to its old snapshot and assume completeness. It requests a full snapshot at 704, replaces its cache atomically, and reports the temporary watch gap.

Now the database becomes unavailable in zone A. Those checkout instances remain live but mark the checkout capability unready after a bounded number of failed observations. Instances in zone B remain ready. This is a discovery and eligibility outcome, not yet a statement about how requests are distributed among eligible endpoints.

When the database recovers, readiness should not oscillate on one successful ping. Each instance requires a small success threshold and randomized re-entry delay, then validates a real lightweight operation before becoming eligible. The stagger avoids every instance opening connections and receiving traffic simultaneously.

During normal termination, the ordering is important. Removing readiness before closing the listener lets discovery updates begin propagating while the process still handles stragglers. The drain period must exceed expected membership propagation and connection reuse, or old clients can send work after the listener disappears. Long-lived connections need their own graceful signal because resolving a name again does not affect an established connection.

This example exposes the key boundaries: the registry records incarnation and state; probes supply imperfect evidence; watchers reconcile versions; the data plane allows old knowledge for a bounded period; and service shutdown accommodates that delay.

Reason About Staleness and Convergence #

An endpoint can remain selectable after failure for approximately the sum of several delays:

This is a design budget, not a universal hard guarantee unless every term is bounded. T_detect includes probe interval, timeout, and failure threshold. T_publish covers registry processing. T_propagate covers watches, DNS, or proxy updates. T_cache includes resolver and application behavior. Retries and persistent connections can extend user-visible impact beyond membership convergence.

For an illustrative policy, probes might run every five seconds, time out after one second, and require three consecutive failures. Depending on failure timing, detection alone can then take more than ten seconds. Those numbers are not recommendations; they demonstrate why a dashboard saying “registry updated in 50 ms” does not establish fast failure removal.

False positives and false negatives pull thresholds in opposite directions. Aggressive checks remove real failures quickly but can eject healthy instances during packet loss, CPU pauses, or control-plane congestion. Conservative checks preserve capacity through noise but send traffic to dead endpoints longer. Use hysteresis: require several failures to remove and several successes to restore, and keep the two thresholds independently configurable.

A stale-but-nonempty cache may be safer than returning no endpoints when the registry is unreachable. That fail-open choice preserves availability but can retain terminated addresses indefinitely, so cap stale age and surface it. Security-sensitive membership changes, such as revoking a compromised endpoint, may require fail-closed behavior or a much smaller maximum age.

DNS adds nested caches. The authoritative server, recursive resolver, operating system, runtime, connection pool, and application may each retain information. Test the actual client stack rather than assuming an advertised TTL controls all layers. Even perfect DNS refresh does not close existing HTTP/2 or database connections.

Avoid Correlated Health-Check Failures #

Health infrastructure can cause the outage it is meant to detect. Thousands of synchronized probes can create periodic load spikes. A readiness endpoint that queries the database on every check multiplies an upstream incident. Randomize schedules, cache dependency observations briefly, bound concurrency, and make probes cheaper than ordinary work.

An overloaded instance may fail probes because probe execution competes with requests. Removing it shifts traffic to peers, which become overloaded and fail in turn. This positive feedback is especially dangerous when every observer makes an independent decision. Reserve a small resource budget for health handling, use passive request evidence carefully, and rate-limit how quickly a fleet can be removed or restored.

Health should not encode every local preference. If an instance marks itself unready at modest load, the remaining pool may absorb its work and cross the same threshold. Admission control and request shedding often handle overload better than pretending the process is dead. Discovery answers eligibility; capacity management decides how much work can be admitted.

Network perspective matters. A central checker may reach an instance that clients in another zone cannot, or fail along a path clients do not use. Combine active checks from relevant network locations with passive observations from real requests. Passive failures need classification: an application 400 says nothing about endpoint health, while repeated connection refusal is stronger evidence.

Do not let a single client globally eject an endpoint based on one failure. Local outlier detection can temporarily avoid it, but authoritative membership changes should aggregate enough evidence or come from the owning platform. Otherwise one faulty client network can erase healthy capacity for everyone.

Secure and Observe the Control Plane #

A registry is a routing control plane. Unauthorized registration can redirect credentials and traffic; unauthorized reads can reveal internal topology. Authenticate workloads with short-lived identity, authorize which service names and metadata they may publish, encrypt registry traffic, and audit mutations. Do not trust a self-reported zone or version if placement policy depends on it; derive such fields from the scheduler when possible.

Protect the registry from cardinality and churn attacks. Bound metadata size, registration rate, watch count, and service namespace creation. Separate tenants where discovery information itself is sensitive. Administrative operations such as force-removal should require stronger authorization than ordinary lease renewal.

Useful registry metrics include registration and expiration rates, watch lag, snapshot size, update revision, lease-renewal latency, and failed authorization. At each resolver, report cache age, last applied revision, watch reconnects, snapshot repairs, endpoint count, and time spent with no eligible endpoints. At services, report readiness transitions by coarse reason and drain duration.

Correlate three viewpoints during an incident: what the registry believed, what a particular client or proxy had cached, and what happened on actual connections. A single global endpoint-count graph cannot explain why one zone used revision 910 while another had revision 907. Include endpoint incarnation and resolver revision in traces or structured logs, with cardinality controls.

Alert on symptoms and disagreement, not every transition. Deployments naturally create registration churn. More actionable signals include a service unexpectedly reaching zero ready endpoints, clients exceeding maximum cache age, zones disagreeing persistently on membership size, probe failures spreading faster than request failures, or drain completion exceeding its bound.

Test Failure Timelines and Run Recovery Drills #

Unit tests should cover lifecycle transitions, lease expiry, hysteresis, capability-specific readiness, and atomic snapshot replacement. Use a fake clock to test exact boundaries without sleeping. Feed a watcher duplicated deltas, revision gaps, reordered messages, compaction errors, and disconnects; it should either produce a valid ordered state or request a fresh snapshot, never silently merge an unknown sequence.

Integration tests should terminate an instance without deregistration, partition it from the registry, delay probes, make only one dependency fail, and hold an old connection open through draining. Observe the complete timeline from fault to probe evidence, registry revision, resolver update, last attempted request, and final removal. Repeat from different zones because network paths differ.

Load tests must include the control plane. Measure registration storms during a fleet restart, synchronized lease renewal, many watch clients reconnecting, and full-snapshot transfer for large services. Verify that the registry applies backpressure without losing established leases and that data-plane caches continue within their stated stale-age policy.

Runbooks should cover registry unavailability, widespread false-negative probes, accidental zero-endpoint publication, stuck draining, expired credentials, and a bad metadata rollout. Preserve a way to freeze the last known good snapshot, disable a faulty health criterion, or roll back resolver policy without redeploying every application. Manual endpoint insertion is risky and should be audited, bounded, and automatically expired.

The acceptance criteria should be framed as observable invariants:

• No endpoint receives new work before the required capability is ready.
• A normally draining endpoint remains available long enough for membership and connection convergence.
• A dead endpoint disappears within the documented failure budget under the assumed network conditions.
• A resolver never applies deltas across an unknown revision gap.
• Loss of the registry produces the documented stale-cache behavior rather than an accidental empty set.
• Health-check traffic remains bounded during dependency and fleet-wide failures.

Service discovery is successful when callers can use a logical service identity despite continuous endpoint churn, and when operators can explain every period of disagreement. That requires more than a registry lookup. It requires explicit lifecycle states, health meanings, versioned caches, convergence budgets, secure ownership, and failure-timeline tests. Once that candidate set is trustworthy enough, a separate placement policy can decide where each request should go.