The Refund Bench

• Section 1 · Main idea

Not because three people are always smarter than one. One judge can have a bad day. One judge can misread a document or miss a detail. You cannot see that from the outside. So you seat three judges, let each one decide alone, and count the votes.

A 2–1 split is information. It tells you the case was hard. A single confident ruling cannot show that. A 3–0 ruling is information too — it tells you the case was clear.

A large language model (LLM) is like a judge who has a different day each time you ask. You can give the same question and the same evidence and still get a slightly different answer. That happens because the model samples from a distribution, which means it chooses from likely next tokens instead of looking up one fixed answer. So use the court pattern: ask three times, independently, and count.

This is self-consistency (Wang et al., 2022). When the model has strong evidence, the reasoning paths usually agree. When it is guessing, the answers spread out. The verdict you keep is the majority; the spread becomes your confidence. The cost is simple: K samples cost K times as much.

flowchart LR
    E1["EVIDENCE IS HARD"] --> E2["“Delivered 85 minutes after ordering”"]
    E2 --> G1["grievance<br>g1"]
    G1 --> J1["judge 1"]
    G1 --> J2["judge 2"]
    G1 --> J3["judge 3"]
    J1 --> U["UPHELD<br>3 – 0<br>confidence 0.98"]
    J2 --> U
    J3 --> U
    U --> PAY["the bench agrees — pay it, no human needed"]
    E3["NO EVIDENCE EXISTS"] --> E4["“The raita leaked inside the bag”"]
    E4 --> G3["grievance<br>g3"]
    G3 --> J4["UPHELD"]
    G3 --> J5["ESCALATE"]
    G3 --> J6["ESCALATE"]
    J4 --> V["split 2–1 → a human decides this one"]
    J5 --> V
    J6 --> V

• Section 2 · The case

The rest of the page follows one dispute. Real disputes contain 1–8 grievances. This one has four.

upheld  rejected  escalate  not a grievance

“We had guests over, it was embarrassing” gives context and feeling. There is no evidence to check against it. “The worst experience I've had” is an opinion. “Refund everything” is a demand, not a claim. None of these can be marked true or false, so they do not enter the pipeline.

That leaves the main design question. You must answer it before you write code: what counts as a grievance?

Use this working definition in the prompt: a grievance is a statement about something that happened to this order, and evidence could in principle confirm or contradict it. “Arrived at 9:40pm” qualifies. “It was embarrassing” does not.

“Two of the four were missing” cannot be judged on its own. Four of what? The extraction prompt requires every grievance to be self-contained, with references matched to the order:

• Section 3 · Bench rulings

Each grievance goes to the bench three times, independently — with no shared history and no chaining. Each judge sees the grievance and the evidence snapshot. Twelve calls.

g4 shows why the system matters. The accusation sounds plausible and sympathetic. The evidence contradicts it: the same customer gave this partner five stars at the door, one minute after handover. A tired support agent at 10pm might approve it. The bench does not.

g3 shows why you sample three times. One judge was ready to approve ₹60 on a claim nothing can prove. Two judges said there was not enough evidence. Majority rules, confidence drops to 0.35, and it goes to a human. With one call, you would have a coin-flip chance of paying an unprovable claim — every time, at 200 disputes a minute.

• Section 4 · Architecture

The pipeline has four logical components. Three always use one LLM call each, no matter what input you give them. The fourth is N × K of them.

flowchart LR
  I[Intake<br>complaint + evidence] --> E[Extract<br>1 sequential call]
  E --> G1[Grievance g1]
  E --> G2[Grievance g2]
  E --> GN[Grievance gN]
  G1 --> J1[Judge 1]
  G1 --> J2[Judge 2]
  G1 --> J3[Judge 3]
  G2 --> K1[Judge 1]
  G2 --> K2[Judge 2]
  G2 --> K3[Judge 3]
  GN --> N1[K independent samples]
  J1 --> R[Majority rule]
    J2 --> R
    J3 --> R
    K1 --> R
    K2 --> R
    K3 --> R
    N1 --> R
    R --> S[Settle<br>1 low-temperature call]
    S --> L[Ledger]
  NOTE["independent · no shared history · same evidence · T > 0"]
  NOTE --> R
  DATA["disputes · grievances · evidence · rulings"]
  DATA --> L
  NOTE["independent · no shared history · same evidence · T > 0"]
  NOTE --> R
  DATA["disputes · grievances · evidence · rulings"]
  DATA --> L

Use the same dispute and look at one stage at a time. Watch the cost on the right. That is the main point.

Change the input and only one segment changes. Each bar is the same four stages, drawn to the same scale.

• Section 5 · Call math

The formula is small enough to remember:

calls = 1 (extract) + N × K (judge) + 1 (settle)

      = 1 + (4 × 3) + 1 = 14 calls per dispute

Notice what is not in the formula: the vote. Counting three rulings is a Counter(), not a model call. The refund amount is not one either. That is a price lookup against the order.

• Average load20/mindisputes across the day
• Dinner peak200/min8–10 pm, ten times average
• Peak call rate2,800LLM calls per minute

Your provider cares about the last number: 200 × 14 = 2,800 calls a minute, sustained for two hours every evening. Plan for the average and you will hit your rate limit during Friday dinner. That is the worst time to learn about it.

You have 12 calls to make. Each takes about 1–2 seconds. How long until you are done?

One at a time:

12 calls × 1.5 sec = 18 seconds

That is too slow. An angry customer is watching a spinner. So you run several calls at once. “4-way parallelism” means 4 at a time — four phone lines, each handling one call, all working at the same time:

Line A:  call 1   call 5   call 9    3 calls
Line B:  call 2   call 6   call 10   3 calls
Line C:  call 3   call 7   call 11   3 calls
Line D:  call 4   call 8   call 12   3 calls
                                   ────────
                                   12 calls ✓

12 calls ÷ 4 lines = 3 calls per line

3 × 1 sec = 3 seconds   ← if calls run fast
3 × 2 sec = 6 seconds   ← if calls run slow
                        → call it 4.5 seconds typical

Twelve lines per dispute looks free until you include concurrency. At peak there are roughly 25 disputes in flight at once. Four lines each is 100 simultaneous requests; twelve lines each is 300. Your total rate limit divided by your concurrency sets how much parallelism each dispute can use — not by what one dispute would prefer.

intake        0.2 sec
extraction    1.5 sec
judging       4.5 sec   ← the part above
settlement    1.0 sec
              ────────
              ~7.2 sec

Why does the requirement say p50 < 15 seconds? Because 7 seconds is an estimate of the work and 15 is a promise to the customer. The gap covers queue wait, 429 retries, complaints with 8 grievances instead of 4, and a slow evening at your provider. Set the SLA equal to the estimate and you will miss it about half the time.

• Work, typical~7 swhat the pipeline costs
• p50 promise< 15 shalf of disputes
• p90 promise< 30 snine in ten

• Section 6 · Capacity estimation

The question is narrower than it looks. Not “how big is our data”. But: when one dispute settles, which rows did we just write?

Four tables. Count the rows in each.

Id, complaint hash, order id, customer id, status, final amount, timestamps.

1 row × 1 KB = 1 KB

One per grievance. Text, category, character span, final ruling, vote split, confidence, amount, reasoning.

4 rows × 1.5 KB = 6 KB

The frozen copy: order lines, timestamps, GPS trail, weights, ratings, as they stood at 21:52.

1 row × 3 KB = 3 KB

People often forget this table. You made 12 judging calls and you keep every single ruling, not just the winners. The raw JSON each judge returned, around 2 KB:

{"ruling":"ESCALATE","confidence":0.31,
 "reasoning":"No field in the evidence records
  packaging condition on arrival…"}

12 rows × 2 KB = 24 KB

Why keep all 12? When a customer escalates, a regulator asks, or your ops lead asks why ₹640 left the company. the statement “the bench ruled 3–0” is credible only if you can show the three rulings. If you throw them away, you cannot explain the payout later.

   1 KB   disputes
+  6 KB   grievances
+  3 KB   evidence_snapshots
+ 24 KB   judge_rulings
───────
  34 KB ≈ 35 KB per dispute

Raw rulings make up 69% of the data. The reason is the same as the call budget: K-sampling dominates both.

35 KB × 30,000 disputes/day = 1,050,000 KB
1,050,000 KB ÷ 1,000        = 1,050 MB
1,050 MB ÷ 1,000            = ~1 GB per day

1 GB × 365 days             ≈ 380 GB per year

That decision is a retention policy, and the four tables need different rules:

Move the 69% to cold storage after 90 days and your hot footprint drops from 380 GB to about 150 GB a year. Which can sit on a cluster you already run.

• Section 7 · Production shape

Use a dry cleaner as the analogy. You hand over a shirt. They write ticket #47. You leave in thirty seconds. The cleaning takes two days. You come back with the ticket and ask if it is ready.

Nobody would accept the other design: standing at the counter for two days because you must still be there when the shirt is ready. That is a synchronous design. At 200 disputes a minute, it fails in four ways at once: open connections time out, every server and LLM credit must be available at once, a crash loses work, and bursts have nowhere to wait.

flowchart LR
  APP[App<br>spinner] -->|POST dispute| API[Thin REST API]
  API --> Q[Kafka / SQS FIFO<br>dispute ids]
  Q --> W[Bounded worker pool<br>one dispute at a time]
  W -->|14 calls| L[LLM providers<br>429 retry with jitter]
  API --> DB[(Ledger<br>freeze evidence)]
  W -->|checkpoint each stage| DB
  APP -->|GET status| API

Teams often skip the pile, but it does real work. Two hundred disputes land in a minute and you can process forty at a time. Without a pile, you would have to reject 160 of them. With a pile, everyone gets a ticket, nothing is dropped, and later tickets wait longer.

429 is the HTTP status for “Too Many Requests.” At 2,800 calls a minute, you will see it every evening. The response needs three ideas:

• Retry — do not give up. Send the request again shortly.
• Exponential backoff — wait longer each time: 1 s, then 2, then 4, then 8. Retrying immediately adds to the same flood that caused the refusal.
• Jitter — add a random offset to each wait. Without it, forty workers refused at the same instant all wait exactly 2 s, all return at the same instant, and all get refused again, locked in the same pattern. Backoff lowers pressure. Jitter spreads the retries out. You need both.

When a worker takes a message from SQS, the message is not deleted. It is hidden for N seconds. Finish and acknowledge within N and SQS removes it. Let N pass without a heartbeat and SQS assumes the worker died and puts it back on the pile for someone else.

Now set N to 10 seconds for a job that takes 30 seconds:

t=0s    Worker A takes dispute d7a41…c39
t=10s   A is at grievance 2 of 4 — but N expired
        SQS assumes A died, requeues the dispute
t=11s   Worker B picks up the SAME dispute, starts over
t=20s   SQS does it again → Worker C starts it too
t=30s   Three workers settled the same dispute
        The customer was refunded ₹689 three times.

Nobody crashed. The work was fine. The deadline was shorter than the job. In a fact-checking system, that bug costs API credits. Here it sends real money out of the company three times, silently.

Use three fixes: set the timeout above your p99; send heartbeats with ChangeMessageVisibility while you work; or use Kafka, which has no such timeout and lets a consumer keep its partition until the work is done.

• Section 8 · The agent loop

The pseudocode shows the happy path. The annotations are where the money lives.

dispute_id = hash(complaint_text + order_id)  # [1]

if ledger.get(dispute_id).status == SETTLED:  # [1]
    return ledger.get(dispute_id).settlement  # never pay twice

evidence = evidence_service.snapshot(order_id)  # [2]
ledger.save(dispute_id, evidence, status=INTAKE)

grievances = grievance_extractor.run(complaint_text, evidence.line_items)  # [3]
ledger.save(dispute_id, grievances, status=JUDGING)

judging_tasks = []  # [4]
for g in grievances:
    for judge_index in range(K):
        judging_tasks.append((g, evidence, judge_index))

rulings = parallel_execute(judge_grievance, judging_tasks,
                          batch_size=BATCH_SIZE)

verdicts = bench.tally(rulings)  # [5]
amount   = pricing.compute(verdicts, evidence)   # code, not LLM  # [5]
if amount > AUTO_APPROVE_CAP:  return escalate(dispute_id)  # [5]
ledger.save(dispute_id, verdicts, amount, status=TALLIED)

message = settlement_writer.run(complaint_text, verdicts, amount)
payments.refund(dispute_id, amount)   # idempotency key = dispute_id
ledger.save(dispute_id, message, status=SETTLED)

1. Idempotency is not just an optimization here. It is part of the product. The complaint text becomes the key. In the fact checker, a duplicate run wasted API credits. Here it wires money out of the company twice. The same key also goes to the payments API, so even two payment calls settle once.
2. Evidence is frozen once, before any judging. All K judges on all N grievances see the identical snapshot. Re-fetch per call and a retry can legitimately produce a different verdict on the same dispute, which is indefensible when someone asks why.
3. Grievances are persisted before judging begins. Save them for resumption and for the customer-facing “here's what we checked.” A crashed worker must not extract them again.
4. The K judges share nothing. No conversation history between them. If judge 2 could see judge 1's ruling, it could anchor on it. You would count one opinion three times instead of three opinions once.
5. The bench counts, the pricing table calculates, the cap decides. The model never touches these three things. It rules on facts; it does not do arithmetic and it does not send payments.

• Section 9 · Prompts

You are a precise dispute-intake assistant.

Given a customer complaint and the order's line items,
extract every atomic, checkable grievance.

Rules:
- A grievance is a statement about something that happened
  to THIS order, which evidence could confirm or contradict
- Exclude feelings, opinions, demands, and rhetorical questions
- Do NOT judge whether the grievance is true — only extract it
- Each grievance must be self-contained: resolve pronouns and
  vague references against the order's line items
- Assign one category from the enum below
- Return ONLY a JSON array, no preamble, no markdown fences

Categories: LATE_DELIVERY | MISSING_ITEM | WRONG_ITEM |
            QUALITY | PACKAGING | PARTNER_CONDUCT | BILLING

Schema:
[{ "grievance_id": "<uuid>",
   "text": "<the grievance as a standalone sentence>",
   "category": "<one of the above>",
   "span_start": <char offset>, "span_end": <char offset> }]

Complaint:  {{complaint_text}}
Order:      {{line_items_json}}

You are an impartial claims adjudicator.
Rule on the following grievance using ONLY the evidence given.

Grievance: {{grievance_text}}
Evidence:  {{evidence_snapshot_json}}

Rulings:
- UPHELD:   the evidence supports the grievance
- REJECTED: the evidence contradicts the grievance
- ESCALATE: the evidence is silent or ambiguous on this point

Rules:
- Do not hedge — pick the single best ruling
- Rule ONLY on the evidence provided. Do not reason about what
  is typical, likely, or fair
- If the evidence does not speak to this grievance, the ruling
  is ESCALATE — never REJECTED
- Return ONLY valid JSON, no preamble, no markdown fences

Schema:
{ "ruling": "UPHELD" | "REJECTED" | "ESCALATE",
  "confidence": <float 0.0 to 1.0>,
  "evidence_cited": "<which field(s) decided it>",
  "reasoning": "<one to three sentences>" }

You are a customer-support writer for a food delivery app.

Below is a customer's complaint, the final ruling on each
grievance, and the refund amount already calculated.

Write a 3-4 sentence reply that:
- States the refund amount and when it will arrive
- Names what was resolved and what a colleague will look into
- Acknowledges the experience without grovelling

Rules:
- Do NOT re-adjudicate any ruling — treat all rulings as final
- Do NOT state, recalculate, or negotiate any amount other
  than the one given
- Never blame the restaurant or the delivery partner
- Plain language. No apology more than once.

Complaint: {{complaint_text}}
Rulings:   {{verdicts_json}}
Amount:    {{refund_amount}}

Notice what the message does not do: it doesn't mention g4 at all. Leaving out a rejected grievance is a deliberate product choice. Telling a customer “we checked and you're wrong about the delivery partner” does not help. The bench ruling is for your ledger. The customer gets the outcome.

• Section 10 · Design choices

• Section 11 · Next upgrades

Three samples from one model share that model's blind spots. If a model often reads pickup-weight evidence badly, asking it three times can give three confident wrong rulings, and the vote makes them look like consensus. Use one GPT judge, one Claude judge, and one Kimi judge. The failures are less likely to match. This is the same idea as a random forest.

Then weight their votes. If one model is better at timestamp arithmetic and another is better at free-text quality complaints, weight votes by category. The weights come from your own evals, or tests, and remain somewhat subjective, but a weighted bench across different models estimates better than an unweighted bench of one model sampled three times.

Today, stage 1 fetches all evidence up front, so you fetch everything that might be relevant for every dispute, and a packaging grievance still carries the full GPS trail in its prompt. Give the judge a tool call instead. It fetches only what that grievance needs: get_delivery_timeline() for g1, get_pickup_weight() for g2.

The architecture stays the same. The fan-out, bench, checkpointing, cap, and 60% rule all still hold. Only the inside of judge_grievance changes. That shows the decomposition was right.

• Section 12 · Other uses

Dispute resolution is one example of a general pattern: when being wrong costs more than asking again, ask again and count.