Eval-driven AI engineering. Dated, reproducible benchmarks. Model-agnostic on principle.

License: MIT Dated benchmarks paiteq.com Model-agnostic

What we publish here

Datasets here mirror public benchmark runs so anyone can reproduce our scores on their own infrastructure. The harness is MIT. Corpora are MIT or permissively re-licensed.

Dataset What it covers Status
paiteq-ai/rag-bench-2026q2 1,840-document RAG retrieval benchmark. Recall@5, MRR, faithfulness, citation accuracy, latency p95, $/1k queries across Claude Opus 4.7, Sonnet 4.6, GPT-4o, Gemini 2.5 Pro, Llama 3.3 70B. 🟡 In flight · 2026-06
paiteq-ai/agent-reliability-2026q3 100-task agent reliability benchmark covering tool-calling, multi-step execution, error recovery. Pass@1, pass@5, mean steps, mean cost. 🔵 Planned · 2026-09

The harness behind the numbers

github.com/paiteq/ai-eval-harness — MIT

Wraps Ragas (retrieval + generation), promptfoo (regression), and a custom rubric layer for agent reliability. Same code that runs on every client engagement and every public benchmark.

git clone https://github.com/paiteq/ai-eval-harness
cd ai-eval-harness

ai-eval run benchmarks/rag-2026-q2.yaml \
  --provider claude --model claude-opus-4-7

Your scores should land inside the 95% confidence intervals published on each benchmark page. If they don't, the harness writes a diff log we'd like to see.

Methodology — composed, not invented

Built on published evaluation research and AI risk-management standards:

Reference Used for
Ragas (Es et al. 2023) Faithfulness, answer relevance, context precision scoring.
BEIR (Thakur et al. 2021) Retrieval ranking metrics (recall@k, MRR, NDCG@10).
AgentBench (Liu et al. 2023) Agent rubric shape — pass@k, recovery rate.
NIST AI RMF 1.0 (Jan 2023) MEASURE-2.3 anchors walk-away metric. MANAGE-2.4 anchors weekly cadence.
ISO/IEC 42001:2023 §6.1 risk treatment + §8 operation drive governance review structure.
EU AI Act, Regulation (EU) 2024/1689 Articles 12–14 (logging, transparency, human oversight) drive audit-log requirements.

Full methodology: getwidget.dev/methodology/eval-driven-delivery

Three rules every benchmark we publish meets

  1. Dated in the URL and H1. Undated benchmarks rot. Every slug carries the quarter so readers know what's current.
  2. Reproducible by anyone. Code MIT, corpora mirrored here, prompts versioned in the harness repo. Run it on your own infra and your scores should match.
  3. Cost on the same axis as quality. Recall@5 and pass@1 are meaningless without $/1k queries on the same dated run. Every benchmark reports both.

Who we are

Same engineering team, three sites, one entity:

Paiteq
AI engineering studio.
Claude · OpenAI · open-source.
Eval-first delivery. 		GetWidget
AI engineering studio.
Flutter heritage — 4,800+★ open-source kit.
Founded 2017, Dallas + Bengaluru. 		Hire Flutter Dev
Vetted senior Flutter engineers.
AI-augmented delivery.
Claude Code in our repos.

Citation

If you use these benchmarks or the harness, please cite:

@misc{paiteq2026harness,
  title  = {paiteq/ai-eval-harness — Open-source eval harness for RAG and agent systems},
  author = {Paiteq},
  year   = {2026},
  url    = {https://github.com/paiteq/ai-eval-harness},
  note   = {MIT. Wraps Ragas + promptfoo + custom agent rubrics.}
}

Per-benchmark BibTeX is published on each dataset card.

Maintained by Paiteq · Benchmarks published at getwidget.dev/benchmarks · Harness MIT-licensed