What we tested, and what we didn’t
Every number on this page is produced by benchmark/run_benchmark.py — a script anyone can run. We publish methodology, honest limitations, and negative results. The numbers are not cherry-picked from multiple runs.
93.3%
overall recall
5 seeds, n=9,333 train trials
99.8%
exclusion precision
when APIE flags a trial, it is genuinely corrupted
1.4s
validation latency ±0.1s
full APIE cascade, CPU-only, n=9,333 rows
Dataset — production scale
Corruption model
Architecture: APIE three-stage cascade
Stage 2 Orchestrator (0ms det. / 2–5s AI): Translates fingerprint into a parametric test plan. In deterministic mode, a rule-based meta-selector picks checks and calibrates thresholds from the fingerprint signals. In AI mode, the fingerprint is sent to an LLM which returns a JSON test plan. The LLM reasons about what to check, not individual rows — it cannot hallucinate row-level decisions.
Stage 3 Filter Bank (~200ms–2s): Executes only the requested checks with the AI-specified parameters. Eight parametric checks: ratio invariant, pairwise ratio drift, ensemble predictor, copy-paste block, distribution shift, local neighbor anomaly, power law, joint skew outlier. Each check fits its model exclusively on the running clean inlier set.
AI integration — what it actually does
In benchmark mode (no API key), a deterministic meta-selector produces the same test plan. The AI upgrades parameters when available; the deterministic plan is the safety net. Both are merged and tested honestly.
Baselines
Runs & variance
Latency — the honest number
Reproducibility
python -m benchmark.run_benchmark. No hidden data files. Output writes to benchmark/results.json, which this page reads directly. With Anthropic key: ANTHROPIC_API_KEY=sk-... python -m benchmark.run_benchmark.Limitations — read before citing
- Synthetic data. Generated from known physical relationships. Real datasets have correlated noise, multi-physics coupling, and instrument-specific failure modes not captured here. These numbers are a controlled proof of mechanism.
- Single domain. All runs use aerodynamics data. APIE is domain-agnostic but per-category recall depends on which physical invariants are present. Other domains will have different fingerprint signals and require different checks.
- Measurement noise floor. 39% of measurement noise rows are missed (61.0% caught). These are rows where the ±12% perturbation falls within natural data variance for that specific feature value combination. This is a physical detection limit, not a software bug.
- GBT improvement is modest. GBT improved 0.8% vs corrupted and 4.6% vs naive. Tree models are inherently robust to outliers — the value here is precision (knowing WHICH rows to debug) and downstream MLP/neural pipeline improvement, not MAPE on GBT itself.
- MLP improvement of 73% is upper bound. MLP is maximally sensitive to distribution shift. Neural networks in production pipelines will see improvement proportional to their sensitivity to the corruptions present in their specific dataset.
The honest numbers
n=9,333 training trials, 30.2% corrupted across 6 failure modes. Three-way comparison: no filtering vs naive IQR/z-score vs SimAPI APIE. Mean ± std across 5 seeds.
30%
of trials corrupted
6 categories including the hardest: measurement noise and sensor drift
93.3%
of corruptions caught
precision 99.8% — near-zero false positives
99.8%
exclusion precision
when flagged, it is genuinely corrupted
| Model | Corrupted | Naive (IQR+Z) | SimAPI APIE | Clean ceiling |
|---|---|---|---|---|
| Neural net (MLP) distribution-sensitive | 3.94% MAPE | 1.13% MAPE | 1.06% MAPE | 0.68% MAPE |
| Gradient boosting robust to outliers | 0.42% MAPE | 0.44% MAPE | 0.42% MAPE | 0.39% MAPE |
Why MLP improves 73%
APIE removes 100% of sensor drift rows — the velocity creep shifts the entire feature distribution. Neural networks are maximally sensitive to this: an MLP trained on drifted velocity learns the wrong v→Cd relationship for 15% of the dataset. Removing these rows lets the model learn the true distribution. MLP goes 3.94% → 1.06%, beating naive filtering by 4.0%.
What naive filtering can’t see
A naive IQR filter removes rows that are statistical outliers column-by-column. Cross-variable corruptions (Re inflated 1.7–2.2× while v is unchanged) produce values inside each column’s individual bounds — IQR catches zero of them. APIE catches 100% by checking Re/v as a physical invariant. Similarly, Pa→kPa unit errors produce plausible individual pressure values — only P/(ρT)≈0.287 instead of 287 reveals the error.
Reproduce: python -m benchmark.run_benchmark — 5 seeds · n≈9,333 train · 63.7s runtime · numbers vary ±1.8% (GBT) / ±3.8% (MLP) across seeds
Per-category detection
n=9,333 training trials · 5 seeds · Deterministic (no key)
Joint Mahalanobis outlier on correlated (Cd, Cl) pair — kurtosis~16 at 5.5σ threshold. Zero false positives at n=9,333.
RANSAC invariant on P/(ρT). A Pa→kPa swap gives 0.287 instead of 287 J/kg·K — a 1000× deviation, detectable at any scale.
Early-segment baseline tracker on Ma/v. Uses first 8% of data as clean anchor. Catches drift even when it starts at row 1000 in a 9K dataset.
Cosine similarity scan with window=8, threshold=0.999. Standardised feature vectors; perturbed duplicates still exceed threshold.
Ratio invariant on Re/v at σ=3.5. At n=9,333: 0 clean false positives, 100% recall. Corrupted Re is 1.7–2.2× larger than expected from velocity.
Local k-NN regression anomaly + multi-model ensemble. 39% missed are perturbations below local variance floor — physical detection limit.
What the AI actually sees
At n=9,333, the fingerprint sent to the AI is ~685 tokens. Here is a real example of the diagnosis Claude produces from the fingerprint alone, without seeing any rows:
The AI then specifies which checks to run and with which thresholds. It cannot modify the filter bank code — only parametrise it. This bounds the blast radius of any AI reasoning error.