Merchant Eligibility Matching

Can the AI tell an independent business from a chain?

Finn hand-labeled 60 real businesses inside the Golden Triangle BID against the independence rule, with a public source for each, and locked the labels before the AI saw them. The AI saw only each business's name, address, category and license activity. Dev cases (40) may be used to improve the prompt; holdout cases (20) are scored once per prompt version and never used to change it. The two are always reported separately.

What the AI knows: Judged from the AI's own training knowledge and the public license facts shown. It did not search the web, so it may not know about recent sales, closures or new owners.

Candidate — not a Local Goods participant. None of these businesses has joined Local Goods.

How to read these numbers

The AI is a triage tool: it screens out obvious chains and franchises, routes anything it can't verify to a person, and never approves a merchant on its own.

An AI “eligible” is only a recommendation: the BID admin sees “AI recommends eligible — confirm before approving”, and no business is approved until a person presses Approve. Without web access, it recognized only 2 of 11 independent businesses on dev; most small independents come back undetermined and go to a person. Giving it a vetted web search is future work, and would need a new prompt version and a fresh unseen holdout.

Final prompt: 458e8c2d, frozen before the holdout was run once.

Locked before the AI saw them: the 60 labels were made in commit 30b610e and locked on Oct 15, 2026 with a SHA-256 of every row (combined 23bb40320097). The eval refuses to run if any label changes.

Totals only on this page

These are real businesses that have not joined Local Goods, so this public page shows totals only: no names, notes or per-business results. Signed-in reviewers with their own link see every disagreement with the hand label's note and the AI's reasoning.

Current prompt version 458e8c2d. Prompt versions that have seen the holdout: 458e8c2d .

Dev

40 cases · run Oct 15, 2026 · model claude-opus-5-5 · prompt 458e8c2d

Eligible yes/no agreement: 28 of 40; 70%, 95% CI 54.6–81.9%
Baseline: always answer “not eligible”: 29 of 40; 72.5%. Most labeled businesses are not independent, so agreement alone flatters.
Independent businesses the AI found: 2 of 11; Of Finn's independent ones, the AI called eligible. 95% CI 5.1–47.7%
AI “eligible” calls that were right: 2 of 5; Of the AI's eligible calls, Finn labeled independent. 95% CI 11.8–76.9%

Triage

Chains and franchises correctly called not eligible: 22 of 29; 75.9%, 95% CI 57.9–87.8%
Chains and franchises wrongly called eligible: 3 of 29; 10.3%, 95% CI 3.6–26.4%. The costliest mistake; a person still decides.
All cases routed to human review: 32 of 40; 80%, 95% CI 65.2–89.5%

Exact classification match: 11 of 40 (27.5%, 95% CI 16.1–42.8%)
AI said undetermined on a case Finn could decide: 13 of 40 (32.5%, 95% CI 20.1–48%)
Cases Finn labeled undetermined: none, so “did the AI also say undetermined instead of guessing?” can't be measured on this set.
Flagged for a person by the AI or the confidence rule: 32 of 40

Confusion matrix (Dev): rows are Finn's labels, columns the AI's answers
Finn	AI: Eligible	AI: Not eligible	AI: Undetermined
Independent	2	0	9
Not independent	3	22	4
Undetermined	0	0	0

Not independent, by the reason Finn gave (Dev)
Reason	Cases	AI: not eligible	AI: same reason	AI: undetermined	AI: eligible
Local chain	10	5 of 10	2 of 10	2	3
Franchise	12	12 of 12	3 of 12	0	0
National chain	7	5 of 7	4 of 7	2	0
Outside control	0	n/a	n/a	0	0

Category disagreements (13)

In these cases both answers mean "not eligible"; only the reason differs. Finn notes that some of Finn's own fail_reason labels may be inaccurate (for example, company-operated locations labeled franchise). The labels stay locked and unchanged, so these still count as classification disagreements.

Holdout

20 cases · run Oct 15, 2026 · model claude-opus-5-5 · prompt 458e8c2d

Eligible yes/no agreement: 17 of 20; 85%, 95% CI 64–94.8%
Baseline: always answer “not eligible”: 17 of 20; 85%. Most labeled businesses are not independent, so agreement alone flatters.
Independent businesses the AI found: 0 of 3; Of Finn's independent ones, the AI called eligible. 95% CI 0–56.1%
AI “eligible” calls that were right: n/a; Of the AI's eligible calls, Finn labeled independent. none to measure

Triage

Chains and franchises correctly called not eligible: 15 of 17; 88.2%, 95% CI 65.7–96.7%
Chains and franchises wrongly called eligible: 0 of 17; 0%, 95% CI 0–18.4%. The costliest mistake; a person still decides.
All cases routed to human review: 14 of 20; 70%, 95% CI 48.1–85.5%

Exact classification match: 8 of 20 (40%, 95% CI 21.9–61.3%)
AI said undetermined on a case Finn could decide: 4 of 20 (20%, 95% CI 8.1–41.6%)
Cases Finn labeled undetermined: none, so “did the AI also say undetermined instead of guessing?” can't be measured on this set.
Flagged for a person by the AI or the confidence rule: 14 of 20

Confusion matrix (Holdout): rows are Finn's labels, columns the AI's answers
Finn	AI: Not eligible	AI: Undetermined
Independent	1	2
Not independent	15	2
Undetermined	0	0

Not independent, by the reason Finn gave (Holdout)
Reason	Cases	AI: not eligible	AI: same reason	AI: undetermined
Local chain	3	3 of 3	1 of 3	0
Franchise	9	8 of 9	3 of 9	1
National chain	5	4 of 5	4 of 5	1
Outside control	0	n/a	n/a	0

Category disagreements (7)

Run history

Every merchant eval run, oldest first
Run	Split	Prompt	Agreement	Independent found	Eligible calls right	Note
Oct 15, 2026 6:13 PM	Dev	`1abdea31`	70% of 40	2 of 11	2 of 5	First dev run, prompt unchanged.
Oct 15, 2026 6:21 PM	Dev	`458e8c2d`	70% of 40	2 of 11	2 of 5	Prompt change 1 (dev only): independent only when sure of local ownership and 5 or fewer locations.
Oct 15, 2026 6:27 PM	Holdout	`458e8c2d`	85% of 20	0 of 3	n/a	The one holdout run, with frozen prompt 458e8c2d.