🔬 Healthcare Fraud Detection: Full Session Report

1. The Autoresearch Pattern

Inspired by Andrej Karpathy's autoresearch project, the core loop is simple:

The key insight from Karpathy: you need an automated, objective oracle. In his case it was validation loss on a language model. In ours it's the OIG LEIE — the federal database of providers excluded from Medicare for fraud, abuse, or license violations. Each eval.py run downloads the live LEIE, matches it to CMS billing data, and returns AUC-ROC. The agent iterates on detector.py freely.

2. Data Infrastructure

All stored in a single 6GB DuckDB database on a Hetzner server (32GB RAM, 8 vCPU). Python venv at /home/dataops/cms-data/.venv/. DuckDB opened read-only to protect data integrity.

3. Model Iterations — AUC Progression

4. Key Discoveries

5. Ground Truth Deep Dive: The 181 Label Question

Blake flagged an important concern: with only 181 matched labels out of 82,714 LEIE entries, how reliable is our AUC?

Why So Few Matches?

Is 181 Enough?

Honestly — it's tight. Key implications:

• The 95% confidence interval on our 0.81 AUC is roughly ±0.03–0.04
• V11 (0.8013) vs V12 (0.8098) are likely not statistically distinguishable — we chased noise in those final iterations
• The large jumps (baseline→V2, V2→V7) are real; the marginal improvements (<0.01) are uncertain
• The model works — but expansion of ground truth is the highest-ROI next step

6. Fuzzy Matching Experiment

We attempted to expand ground truth from 181 → 720+ labels by fuzzy-matching the 74,241 no-NPI LEIE entries against NPPES using last_name + state + specialty.

Results

Why Did AUC Drop?

Investigation revealed: the fuzzy-matched providers were excluded in 1995–2010 — 20-30 years ago. Their current billing patterns are completely normal (they may have been reinstated, or the NPI was reassigned). Adding them as fraud labels teaches the model the wrong thing. The old LEIE entries without NPIs are not worth adding.

What would actually help: DOJ press releases (OIG publishes ~100/month with specific provider names), CMS Program Integrity dataset, state medical board disciplinary actions.

7. Feature Importance Analysis

The ensemble effect is the whole game. No single signal is powerful, but they catch different fraudsters. That's exactly why max() works: it asks "is this provider extreme on anything?" rather than "is this provider above average on everything?"

8. Top Suspect List — Physicians Only

After removing PECOS-dominated results and filtering to individual physicians (removing ambulance services, labs, IDTFs), the top billing anomaly suspects:

9. LLM Web Validation Pipeline

Built validate_suspects.py — a batch pipeline that takes the physician suspect list and for each provider:

1. Builds 3–5 targeted search queries (OIG news, medical board, fraud indictment, settlement)
2. Calls Brave Search API (3 searches × 3 results = up to 9 unique sources per provider)
3. Feeds CMS data context + search results to Claude (claude-haiku) for structured analysis
4. Returns structured JSON: verdict | fraud_confidence | fraud_indicators | legitimate_explanations | recommended_action
5. Generates styled HTML report with color-coded verdicts

Verdict Categories

To Activate

# On Hetzner server:
export BRAVE_API_KEY=...       # free tier at https://brave.com/search/api/
export ANTHROPIC_API_KEY=...

# Run top 25 suspects:
cd /home/dataops/fraud-detector
python3 validate_suspects.py --limit 25

# Output:
# validation/validation_report_2026-03-09.html  (HTML report)
# validation/validation_results.json            (structured data)

10. Next Steps — Prioritized

Repository & Files

Session: March 8–9, 2026 (~6 hours)  |  GitHub: blakethom8/cms-fraud-detection  |  Server: Hetzner 5.78.148.70  |  Data: All CMS data publicly available at data.cms.gov