Service Discovery & Health Checks

The Problem

In a horizontally-scaled system, instances come and go constantly — autoscaling, deploys, crashes. Clients and load balancers need to know, at any moment, which instances are actually alive.

Heartbeats

Each instance periodically pings a health-check service.

• Miss one heartbeat → marked "suspect."
• Miss two consecutive → marked "dead." Load balancer stops routing to it, orchestrator restarts it.

flowchart LR
    S1[Server 1] -->|heartbeat| H[Health Service]
    S2[Server 2] -->|heartbeat| H
    H -->|dead: stop routing| LB[Load Balancer]
    H -->|dead: restart| O[Orchestrator]

Two-Way Health Checks

Check both:

1. Is the server up — process/network reachable.
2. Is the service inside it actually working — an app-level /health endpoint that verifies real dependencies (e.g., DB connectivity), not just "process exists."

A server can be up (responds to ping) while its application has silently hung — only the second check catches this.

Design Bias

Favor availability over strict efficiency here — a slightly-stale "alive" list that occasionally routes to a dying instance (retried by the client) is usually better than an overly aggressive health check that flaps healthy instances in and out under transient load.

Interview Drill

Q: Your health check only pings the process (not the app). What failure mode does this miss? A: A "zombie" instance — process is running and responds to network pings, but the application logic inside has hung or lost its DB connection. Only an app-level /health endpoint that exercises real dependencies catches this.

Cross-links

[[Single Point of Failure]] · [[Load Balancing]]