Back to Notes

Design a Payment System

1. Problem & Real-World Context

Design a system that processes payments between users/merchants — charge a card, move money, record the result — reliably enough that money is never lost, never duplicated, and always accounted for, even under network failures and retries. This is the HLD problem where the usual "eventual consistency is fine" escape hatch (used everywhere in feeds, chat, notifications) explicitly does not apply — the entire design exists in service of a much stricter correctness bar than most other HLD problems in this KB.

2. Requirements

Functional: initiate a payment (charge a payment method, transfer between accounts); query payment status; support refunds; reconcile with external payment processors (Stripe/card networks).

Non-functional: exactly-once effect — a network retry must never result in a user being charged twice (idempotency, §6); durability — a confirmed payment must never be lost; auditability — every balance must be reconstructable from an immutable history of transactions, not just a mutable current-balance field; strong consistency for balance reads (this is the rare HLD problem where CP, not AP, is the right default — see [[CAP Theorem & Consistency Models]]).

3. Capacity Estimation

  • At, say, 10K payments/sec at peak for a large platform, the actual bottleneck is rarely raw throughput — it's the serialization required for correctness: any two operations affecting the same account's balance must be strictly ordered relative to each other, which caps how much a single account's write path can be parallelized regardless of overall cluster capacity.
  • Ledger storage: an append-only ledger of every transaction (not just current balances) at 10K/sec × ~500 bytes/entry × a year ≈ ~150TB/year — large, but the append-only, rarely-queried-by-range access pattern fits a write-optimized store well (see [[SQL vs NoSQL]]), and this data is often the single most important artifact in the entire system from a compliance standpoint.

4. API Design

POST /v1/payments
Idempotency-Key: <client-generated UUID>
{ "amount": 4999, "currency": "USD", "source": "card_tok_abc", "destination": "merchant_123" }
→ { "payment_id": "pay_xyz", "status": "pending" | "succeeded" | "failed" }

GET /v1/payments/{payment_id}   → current status (poll or webhook-driven)

The Idempotency-Key header (see [[API Design Principles]]) is not optional here — it's the single most load-bearing detail in the whole API, discussed in depth in §6.

5. High-Level Architecture

flowchart LR
    C[Client] --> API[Payment API]
    API --> IK[(Idempotency Key Store)]
    API --> LEDGER[(Append-only Ledger<br/>strongly consistent)]
    API --> PSP[Payment Service Provider<br/>Stripe / card network]
    PSP -->|webhook: async result| RECON[Reconciliation Service]
    RECON --> LEDGER
    LEDGER --> BAL[Balance Read Service<br/>materialized view]

Walking a payment: client submits a payment request with an idempotency key → the API checks the idempotency key store first (§6) → if new, it records a pending ledger entry before calling the external payment processor → calls the PSP (Stripe, a card network) to actually move money → the PSP's result (often async, via webhook, since card authorization can take time) updates the ledger entry to succeeded or failed → a balance read service maintains a materialized, fast-to-query view derived from the ledger, rather than the ledger itself being queried directly for every balance check.

6. Idempotency — the Load-Bearing Mechanism

A client's network request can fail after the server successfully processed the charge but before the client received the response — the client, seeing what looks like a failure, retries. Without idempotency, this retries into a second real charge.

def process_payment(idempotency_key: str, amount: int, source: str, destination: str) -> dict:
    existing = idempotency_store.get(idempotency_key)
    if existing:
        return existing["response"]          # exact same response as the original attempt — no reprocessing

    # Reserve the idempotency key BEFORE calling the external processor,
    # so a concurrent retry arriving mid-flight sees "in progress", not "not found"
    idempotency_store.set(idempotency_key, {"status": "in_progress"}, if_not_exists=True)

    ledger_entry_id = ledger.append_pending(amount, source, destination)
    try:
        psp_result = payment_processor.charge(amount, source)
        ledger.mark_succeeded(ledger_entry_id, psp_result.charge_id)
        response = {"payment_id": ledger_entry_id, "status": "succeeded"}
    except PaymentDeclinedError:
        ledger.mark_failed(ledger_entry_id)
        response = {"payment_id": ledger_entry_id, "status": "failed"}

    idempotency_store.set(idempotency_key, {"status": "done", "response": response})
    return response

Why the idempotency key is reserved with if_not_exists before calling the PSP, not after: if two requests with the same key arrive nearly simultaneously (a genuine retry racing the original, not a sequential one), the second must see "already in progress" and wait/return the eventual result — not independently proceed to charge the card a second time while the first is still in flight. This check-and-reserve step is itself an atomicity requirement (a compare-and-set / conditional write), the same class of problem as the rate limiter's atomic check-and-increment in [[Rate Limiter]].

7. The Ledger: Append-Only, Never Mutated

The core data model is an immutable log of transactions, not a mutable balance column that gets updated in place.

transaction_id | account_id | amount | type    | timestamp | related_transaction_id
tx_001         | acct_A     | -5000  | debit   | ...       | tx_002
tx_002         | acct_B     | +5000  | credit  | ...       | tx_001

A transfer is recorded as two linked entries (a debit and a credit, referencing each other) rather than one row updated twice — this is double-entry bookkeeping, and it's not accounting tradition for its own sake: it makes every transfer independently auditable (the debit and credit must always sum to zero across the pair) and makes "what is this account's balance" a derived, recomputable value (sum of all entries for this account) rather than a single mutable number that could silently drift from the true history if a bug ever wrote to it directly.

Why not just an UPDATE accounts SET balance = balance + ? WHERE id = ?: that pattern has no audit trail — if the balance is ever wrong, there's no way to reconstruct how it became wrong, since the history of individual changes was never preserved, only the running total. The append-only ledger trades a small amount of read-path complexity (balance = a query, not a column) for auditability that's often a hard legal/compliance requirement in payments specifically, not just good practice.

8. Balance Reads: Materialized View, Not Live Aggregation

Summing every historical transaction on every balance check would get slower as the ledger grows — instead, maintain a materialized balance view, updated incrementally as new ledger entries are appended (not recomputed from scratch), and treat it as a fast cache of a value that's always re-derivable from the ledger if it's ever suspected to be wrong.

def apply_to_balance_view(entry: dict) -> None:
    balance_view.increment(entry["account_id"], entry["amount"])   # incremental, not a full re-sum

def reconcile_balance(account_id: str) -> int:
    """Recompute from the source of truth — used to verify/repair the materialized view."""
    return ledger.sum_entries_for(account_id)

This mirrors the general CQRS pattern from [[Message Queues & Kafka]] — the ledger is the write-optimized source of truth, the balance view is a read-optimized derived projection, and the two are allowed to be reconciled/repaired against each other precisely because the ledger, not the view, is authoritative.

9. Consistency Requirements

Unlike almost every other problem in this KB, this is a CP system by default for anything touching a specific account's balance: a payment operation must see a consistent, up-to-date view of that account, and it's acceptable (necessary, even) to sacrifice some availability under a partition rather than risk a double-spend or an incorrect balance. See [[CAP Theorem & Consistency Models]] — this problem is the canonical real-world example interviewers use to test whether a candidate reflexively defaults to "eventual consistency is always fine" without recognizing when it genuinely isn't.

10. Deep Dives / Follow-ups

  • Distributed transactions across services (debit one service's ledger, credit another's, e.g., a marketplace paying out a seller): the Saga pattern — a sequence of local transactions, each with a defined compensating action if a later step fails, rather than a single distributed ACID transaction across service boundaries (which doesn't scale well and couples services tightly). Worth naming explicitly as the standard answer to "how do you keep this atomic across microservices."
  • Reconciliation with the external PSP: since the PSP's webhook can be delayed, lost, or arrive out of order, a periodic reconciliation job compares the ledger's pending entries against the PSP's own transaction records, resolving any that never received a terminal webhook — a real production necessity, not a hypothetical edge case.
  • Refunds: modeled as new ledger entries (a reversing credit/debit pair referencing the original transaction), never as a deletion or mutation of the original entry — preserving the "immutable log" property even for corrections.

11. Interview Drill Questions

  1. A client's payment request times out on the network, and it retries the exact same request. How do you guarantee the user isn't charged twice? — The Idempotency-Key header, checked and atomically reserved before the external charge call — a retry with the same key returns the original result rather than re-executing the charge, and a key reserved via conditional write prevents a concurrent (not just sequential) retry from racing past the check.
  2. Why store an append-only ledger instead of just updating a balance column? — The ledger provides a full, immutable audit trail (required for compliance in most real payment systems) and makes the balance a derived, always-recomputable value rather than a single mutable number with no history — if a balance is ever suspected wrong, it can be reconstructed and verified from the ledger; a mutated column offers no such recovery path.
  3. Why is this problem CP rather than the AP default seen in most other HLD problems in this KB? — Because the failure mode of choosing availability here (allowing a stale or conflicting balance to be read/acted on) is a double-spend or an incorrect charge — genuinely unacceptable, unlike a stale social-media like count. This is the clearest real-world instance of CAP's trade-off actually mattering, not a theoretical exercise.
  4. How do you keep a payment atomic when it involves calling both your own ledger and an external payment processor you don't control? — You can't get true atomicity across that boundary — instead, record a pending ledger entry before calling the external processor, and use the processor's (possibly async, webhook-driven) result to transition the entry to a terminal state, with a reconciliation job as a safety net for results that never arrive via webhook.
  5. How would you implement a refund without violating the immutable-ledger design? — Append a new pair of ledger entries reversing the original transaction's debit/credit, referencing the original transaction ID — the original entries are never modified or deleted; the refund is its own auditable event layered on top of the history.

Cross-links

[[CAP Theorem & Consistency Models]] · [[API Design Principles]] · [[Rate Limiter]] · [[Message Queues & Kafka]] · [[SQL vs NoSQL]]