Skip to content

The second AI function, Merchant Eligibility Matching, has its own eval: can the AI tell an independent business from a chain?

Redemption Fraud Detection

How often does the AI agree with a person?

We wrote 90 synthetic redemptions and a person labeled each one normal or suspicious by hand, before the AI saw them. Then the real fraud check (the same prompt and model as production) scored every case. An AI hold counts as suspicious and a clear counts as normal. The holdout 3 cases (50 of them) were labeled and locked after the last prompt change and before the AI saw them, so they are the cleanest test of the current prompt. Earlier groups are smaller and are kept below. Catch rate is how many of the suspicious cases the AI held; hold precision is how many of the AI's holds the person had labeled suspicious.

Headline result

Caught 17 of 26 suspicious cases; 17 of 19 holds were suspicious.

Measured on 50 unseen cases, 26 of them labeled suspicious by a person, with the current prompt. 95% confidence intervals: catch rate 46.2–80.6%, hold precision 68.6–97.1%; overall agreement 78% (39 of 50, 64.8–87.2%).

The AI misses more often than it raises false alarms. It cleared 9 of the 26 suspicious cases, mostly ones with a single strong signal (a big jump over the employee's usual spend, or nearly the whole balance) at normal hours and inside the category's typical range; 4 of the 9 were at the salon. A cleared redemption goes through without a person looking at it, so a miss is the costlier mistake. Holds are reliable: 17 of 19 were suspicious. The 2 false alarms are frequent visitors spending a round amount well above their usual, which the prompt treats as possible credit-to-cash collusion; a held redemption waits for a person, so a false alarm costs a delay, not a wrong payout.

From the run on Oct 14, 2026 (prompt version 8d5e1e70, holdout_3 cases).

Locked before the AI saw them: the 50 cases were labeled by hand in commit 5f594c1 and locked on Oct 14, 2026 with a SHA-256 of every row (combined e8799c6d6407, lock file). The eval refuses to run if any of them changes.

Misses and false alarms by pattern

On the 50 holdout_3 cases. Tags come from each case's facts only, never from the person's label or the AI's answer, and a case can have several. Missed: labeled suspicious, AI cleared. False alarm: labeled normal, AI held. The "why" comes from the AI's own explanations.

Misses and false alarms by pattern
Pattern Cases Labeled suspicious Missed False alarms Why
Round amount ($20+, whole $10s) 21 18 3 1 A round amount is one weak signal to the AI: it cleared round amounts inside the category's normal range (FE-58, FE-89) or on part of the balance mid-afternoon (FE-86), and held one from a very frequent visitor (FE-75).
3x or more the usual amount 20 18 3 2 The AI called a big jump over usual spend "worth watching" but not enough alone when the amount was not round, used part of the balance and came in business hours (FE-45, FE-86, FE-89); with very frequent visits it held (FE-62, FE-75).
Uses 90%+ of the balance 15 15 2 0 The AI let one recent visit to the merchant and a normal-for-the-category amount outweigh using nearly the whole balance (FE-58, FE-71).
New merchant (0 visits in 30 days) 16 15 2 0 The AI read "no repeat visits" as reassuring, because the collusion pattern it looks for is repeat visits, so a first visit with a big amount was cleared (FE-45, FE-86).
Outside business hours 11 8 0 1 The one false alarm (FE-62) was a 6:40 AM coffee purchase; the early hour added to the frequent-visitor signal.
5+ visits here in 7 days 3 0 0 2 Both false alarms (FE-62, FE-75): the AI reads many visits in a week plus a much larger round amount as possible collusion, where the person saw a loyal regular. Same pattern as the earlier holdout_2 false alarms.
Above the category's typical range 21 19 3 1 Slightly above the range but not round and during the day, so the AI saw one signal and cleared it (FE-45, FE-56, FE-86); with frequent early visits it held (FE-62).
Misses and false alarms by merchant category
Merchant category Cases Labeled suspicious Missed False alarms Why
Cafe 7 3 1 1 Missed FE-86 (a first visit at 4 times the usual spend, at 3 PM); false alarm FE-62 (a regular at 6:40 AM spending 4 times the usual).
Fitness 7 2 0 0
Restaurant 18 8 2 1 Missed FE-56 and FE-80 (about 2.5–2.8 times the usual at dinner time, read as a larger dinner); false alarm FE-75 (a very frequent visitor, round $40).
Retail 11 7 2 0 Missed FE-45 (6.5 times the usual but under half the balance, at 1 PM) and FE-71 (nearly the whole $50 balance, but a recent visit and a normal retail amount).
Salon 7 6 4 0 4 of 6 suspicious salon cases were missed (FE-50, FE-58, FE-74, FE-89): the salon's wide $25–$90 typical range made amounts 2–3.5 times the employee's usual look normal to the AI.

Latest run

Note on this run: Prompt unchanged (8d5e1e70). Adds 50 holdout_3 cases, labeled by hand and locked at commit 5f594c1 before the AI saw them; 26 are labeled suspicious.

Holdout 3 agreement
78%
39 of 50 cases (26 labeled suspicious)
Caught 17 of 26 (65.4%, 95% CI 46.2–80.6%)
Holds suspicious: 17 of 19 (89.5%, 95% CI 68.6–97.1%)
Holdout 2 agreement
80%
8 of 10 cases (4 labeled suspicious)
Caught 4 of 4 (100%, 95% CI 51–100%)
Holds suspicious: 4 of 6 (66.7%, 95% CI 30–90.3%)
Holdout agreement
90%
9 of 10 cases (4 labeled suspicious)
Caught 4 of 4 (100%, 95% CI 51–100%)
Holds suspicious: 4 of 5 (80%, 95% CI 37.6–96.4%)
Dev agreement
90%
18 of 20 cases (7 labeled suspicious)
Caught 7 of 7 (100%, 95% CI 64.6–100%)
Holds suspicious: 7 of 9 (77.8%, 95% CI 45.3–93.7%)
All cases
82.2%
74 of 90 cases (41 labeled suspicious)
Caught 32 of 41 (78%, 95% CI 63.3–88%)
Holds suspicious: 32 of 39 (82.1%, 95% CI 67.3–91%)

Run Oct 14, 2026 4:01 PM · model claude-opus-5-5 · prompt version 8d5e1e70

Where they agreed and disagreed

Holdout 3 cases: person's label (rows) vs the AI's call (columns)
AI: clear AI: hold
Person: normal 22 agree 2 false alarm
Person: suspicious 9 missed 17 agree
Holdout 2 cases: person's label (rows) vs the AI's call (columns)
AI: clear AI: hold
Person: normal 4 agree 2 false alarm
Person: suspicious 0 missed 4 agree
Holdout cases: person's label (rows) vs the AI's call (columns)
AI: clear AI: hold
Person: normal 5 agree 1 false alarm
Person: suspicious 0 missed 4 agree
Dev cases: person's label (rows) vs the AI's call (columns)
AI: clear AI: hold
Person: normal 11 agree 2 false alarm
Person: suspicious 0 missed 7 agree

Blind adjudication of the disagreements

Where the person's label and the AI disagree, invited BID staff and contest judges rate each case blind, with agreed cases mixed in as hidden controls. Voting is open until Nov 13. Results appear here once the first adjudicator submits.

Every disagreement (16)

  • FE-08 · $40.00 at Farragut Grind, 13:00 dev

    Person: normal

    regular customer

    AI: hold, risk 85/100

    This is a round $40.00 charge at a cafe, more than six times this employee's usual $6.40 spend and well above the normal $3–$15 cafe ticket. The employee has also made 9 visits to this same cafe in the past 7 days (22 in 30 days).

  • FE-15 · $100.00 at Connecticut Noodle Bar, 14:00 dev

    Person: normal

    treating oneself

    AI: hold, risk 88/100

    This $100.00 redemption is a round amount, about 7 times the employee's usual $13.87 spend, and well above the typical $10–$45 restaurant ticket. It also follows 6 visits to the same merchant in the past 7 days (14 in 30 days).

  • FE-21 · $60.00 at Press Room Deli, 12:30 holdout

    Person: normal

    previous customer making a bigger purchase

    AI: hold, risk 82/100

    This $60.00 restaurant redemption is a round amount, about 4.3 times the employee's usual $13.87 spend, and above the normal $10–$45 restaurant ticket. The employee has also made 5 prior redemptions at this same merchant in the last 7 days (11 in 30 days), so several signals point the same way.

  • FE-34 · $50.00 at Press Room Deli, 12:10 holdout_2

    Person: normal

    large amount but has visited here previously and the account is not drained

    AI: hold, risk 82/100

    This is a round $50.00 charge, about 4 times the employee's usual $12.00 spend and above the normal $10–$45 restaurant ticket. It also follows 7 prior visits to the same restaurant in the last 7 days (15 in 30 days), so several signals point to possible collusion.

  • FE-38 · $20.00 at Farragut Grind, 09:15 holdout_2

    Person: normal

    recurring customer at a reasonable time in the morning

    AI: hold, risk 72/100

    Several signals point the same way: an exactly $20.00 charge at a cafe is above the normal $3–$15 ticket and about 3.5 times this employee's usual $5.80 spend, and this is their 4th visit to the same cafe in 7 days (10th in 30). On its own this would be a small amount, but the round, inflated figure combined with frequent repeat visits warrants a review.

  • FE-45 · $95.00 at Page & Quill Books, 13:30 holdout_3

    Person: suspicious

    no visits and way past median amount

    AI: clear, risk 58/100

    The $95.00 retail purchase is about 6.5 times this employee's usual $14.50 spend and somewhat above the typical $10–$80 retail range. However, it is not a round amount, uses less than half of the $210.00 balance, and happened at 1 PM with no repeat visits, so it does not warrant a hold.

  • FE-50 · $65.00 at K Street Cuts, 10:30 holdout_3

    Person: suspicious

    almost 3x the median amount

    AI: clear, risk 25/100

    This $65.00 salon redemption is about three times the employee's usual $22.00 spend, but it falls within the normal $25–$90 range for a salon visit. It also happened at 10 AM, is not a round amount, and uses only part of the $190.00 balance, so there is no pattern of abuse.

  • FE-56 · $52.30 at Seventeenth Street Kitchen, 19:45 holdout_3

    Person: suspicious

    3x median amount with little visits

    AI: clear, risk 45/100

    This $52.30 restaurant bill is about 2.8 times the employee's usual $18.40 and somewhat above the typical $10–$45 restaurant range, but it is not a round amount, uses well under the $175.00 balance, and happened at 7 PM. It looks consistent with a larger dinner rather than collusion.

  • FE-58 · $40.00 at K Street Cuts, 08:30 holdout_3

    Person: suspicious

    drains balance and double the median amount

    AI: clear, risk 55/100

    This $40.00 salon redemption is a round amount that uses nearly all of the $41.00 balance and is twice the employee's usual $20.00 spend. However, the employee has visited this merchant in the past 30 days, the amount is normal for a salon, and it happened during business hours, so it does not rise to a hold.

  • FE-62 · $25.00 at Farragut Grind, 06:40 holdout_3

    Person: normal

    frequent visitor

    AI: hold, risk 78/100

    Several signals point the same way: this $25.00 cafe redemption is about 4 times the employee's usual $6.20, above the normal $3–$15 cafe ticket, and made at 6 AM, outside business hours. It also follows 6 prior visits to the same merchant in the last 7 days (14 in 30 days).

  • FE-71 · $48.00 at Page & Quill Books, 11:15 holdout_3

    Person: suspicious

    drains account

    AI: clear, risk 45/100

    This $48.00 retail purchase uses nearly all of the $50.00 balance and is about twice the employee's usual $25.00. However, the employee visited this merchant once in the past 30 days, the amount is not round, it falls within the normal range for retail, and it happened at 11 a.m., so it does not warrant a hold.

  • FE-74 · $55.00 at K Street Cuts, 12:00 holdout_3

    Person: suspicious

    2x the median with little past redemptions and a large chunk of the balance being taken out

    AI: clear, risk 20/100

    This $55.00 salon redemption is within the normal $25.00–$90.00 range for salons, uses about a third of the $160.00 balance, and happened at midday. It is about 2.3 times this employee's usual $24.00 spend, but that is the only signal and is expected for a salon visit.

  • FE-75 · $40.00 at Press Room Deli, 13:05 holdout_3

    Person: normal

    even though 4x the median, they have enough past redemptions and prior visits to be seen as normal

    AI: hold, risk 74/100

    This is a round $40.00 redemption, about 3.6 times the employee's usual $11.00 spend, at a restaurant they have already used 6 times in the past week and 13 times in the past month. The amount is within a normal restaurant bill and the time (1 PM) is ordinary, but the round amount, the jump above usual spending and the very frequent repeat visits together warrant a review.

  • FE-80 · $41.20 at Seventeenth Street Kitchen, 20:40 holdout_3

    Person: suspicious

    not enough past visits with 2.5x past the median

    AI: clear, risk 25/100

    This $41.20 restaurant redemption is about 2.45 times the employee's usual $16.80, but it falls within the normal range for a restaurant, is not a round amount, uses only part of the $72.00 balance, and happened at 8 PM. One elevated signal on its own is not enough to hold.

  • FE-86 · $30.00 at Farragut Grind, 15:30 holdout_3

    Person: suspicious

    no past visits with the amount being 4x the median

    AI: clear, risk 62/100

    This $30.00 cafe redemption is a round amount, four times the employee's usual $7.50, and above the typical $3.00–$15.00 cafe ticket, so it is worth watching. It uses only a quarter of the $120.00 balance and happened at 3 PM with no repeat visits, so it does not reach the hold threshold.

  • FE-89 · $90.00 at K Street Cuts, 17:00 holdout_3

    Person: suspicious

    high amount and significantly above the median with little past redemptions

    AI: clear, risk 35/100

    The $90.00 salon charge is a round number and about 3.5 times this employee's usual $26.00 spend, but it falls within the normal range for a salon visit, uses only part of the $250.00 balance, and happened during business hours. These mild signals together are not enough to hold the payout.

Earlier runs

Every run is kept. A new prompt version means the prompt was changed between runs. Each cell: agreement, and how many of the suspicious cases the AI caught.

Agreement for every eval run
Run Prompt version Dev Holdout Holdout 2 Holdout 3 Note
Oct 11, 2026 9:44 PM e5bcf207 85% (17/20), caught 6 of 7 80% (8/10), caught 3 of 4 n/a n/a First run, with the original prompt. Its holdout cases (FE-21 to FE-30 group) were unseen at the time.
Oct 11, 2026 9:49 PM 8d5e1e70 90% (18/20), caught 7 of 7 90% (9/10), caught 4 of 4 n/a n/a Prompt change 1 (hold full-balance redemptions at a merchant not visited in 30 days) was based on dev case FE-18 only, but it was made after the first run's holdout results had been seen, so this run's holdout score is not fully independent.
Oct 11, 2026 10:03 PM 8d5e1e70 90% (18/20), caught 7 of 7 90% (9/10), caught 4 of 4 80% (8/10), caught 4 of 4 n/a First run including holdout_2 (FE-31 to FE-40): hand-labeled after prompt 8d5e1e70 was written and never seen while changing it. Same prompt as run 2. The holdout (FE-21 to FE-30 group) score is still not fully independent.
Oct 14, 2026 4:01 PM 8d5e1e70 90% (18/20), caught 7 of 7 90% (9/10), caught 4 of 4 80% (8/10), caught 4 of 4 78% (39/50), caught 17 of 26 Prompt unchanged (8d5e1e70). Adds 50 holdout_3 cases, labeled by hand and locked at commit 5f594c1 before the AI saw them; 26 are labeled suspicious.