WBSP
ProvidersWikiSign in

claud

A variant of AI Evaluation & Benchmarking.

View the interactive variant page →

AI Evaluation & Benchmarking

Part of the worlds-biggest-software-project initiative.

An open, AI-native platform for LLM output quality testing, regression detection, and human feedback loops.

AI Evaluation & Benchmarking is a candidate project to build an open-source evaluation platform for teams shipping LLM and agent applications. It targets AI engineers, QA teams, and product leaders who need trustworthy quality signals across prompts, models, and multi-step agents — and combines the metric depth of frameworks like DeepEval with the production observability of Langfuse and Arize Phoenix.

Why AI Evaluation & Benchmarking?

  • Existing evaluation tooling is fragmented: DeepEval leads on metric breadth but lacks production monitoring; Langfuse and Arize Phoenix lead on tracing but have shallower evaluation depth; Braintrust is polished but proprietary and costly ($249/mo Pro, no self-hosting).
  • Promptfoo — the leading open-source red-teaming tool — was acquired by OpenAI in March 2026, leaving the open ecosystem exposed if its licence trajectory shifts.
  • Galileo's low-latency Luna-2 evaluators are proprietary; after Cisco's April 2026 acquisition, roadmap and pricing are uncertain.
  • Agent trajectory evaluation (planning, tool selection, error recovery) is poorly served across every incumbent surveyed.
  • Regulated industries need self-hosted, privacy-first evaluation with audit-ready exports for NIST AI RMF and the EU AI Act (August 2026 deadline) — only Langfuse and Arize Phoenix offer full-featured self-hosting today, and neither produces compliance-aligned reports.

Key Features

Core Metric Library

  • Faithfulness, Answer Relevance, Hallucination, Contextual Precision, Contextual Recall, and Toxicity out of the box
  • Configurable LLM-as-a-judge scoring (G-Eval-style custom criteria)
  • Transparent, explainable scores with reasoning strings for every judgment
  • Multi-provider judge support: OpenAI, Anthropic, Google, and local/Ollama models

Developer & CI Workflow

  • Pytest-native test runner with pass/fail thresholds
  • JSON, HTML, and XML output formats for CI/CD gating
  • Dataset management: create, version, and query test datasets from files or production traces
  • Trace capture for RAG and single-turn LLM calls with span-level timing and metadata

Agent & Production Evaluation

  • Agent trajectory scoring, tool-call accuracy, and planning-quality metrics
  • Real-time production monitoring dashboard: score trends, latency, cost, and quality
  • OTel-compatible trace ingestion (OTLP over HTTP, protobuf or JSON) for framework-agnostic instrumentation
  • Human annotation queues for last-mile review and judge calibration

Safety, Security & Compliance

  • Red teaming and adversarial test generation aligned to the OWASP LLM Top 10
  • Prompt management with environment-gated deployment (score thresholds required to promote)
  • Compliance-ready export aligned to NIST AI RMF and EU AI Act documentation requirements

AI-Native Advantage

AI is used not just as the subject of evaluation but as a core part of the platform. Synthetic adversarial test cases are generated to probe edge cases and failure modes, reducing the manual burden of dataset curation. Natural-language evaluator authoring lets product teams define custom judge criteria in plain English. Automated regression root-cause analysis compares trace deltas between prompt versions to surface causal changes, and adaptive sampling selects the most informative production traces to evaluate rather than relying on random sampling.

Tech Stack & Deployment

The project is designed self-hosted-first (Docker and Kubernetes) with an optional managed cloud, mirroring the deployment model that makes Langfuse and Arize Phoenix viable for regulated industries. Instrumentation is OpenTelemetry-native, building on the OpenInference span conventions so traces from any OTLP-speaking framework — LangChain, LlamaIndex, LiteLLM, OpenAI Agents SDK, Claude Agent SDK, LangGraph, CrewAI, DSPy, Vercel AI SDK — ingest without custom adapters. Python and JavaScript/TypeScript SDKs cover the common application surfaces, with a Pytest plugin for the developer workflow.

Sign-in and Access

  • People sign in with single sign-on. The dashboard has no passwords and no sign-in form of its own: people sign in through the platform's identity service (HAP, an OpenID Connect provider). A first sign-in creates the account with the organisation's default role; administrators are recognised from the identity provider's administrator claims, and roles can be raised, lowered or withdrawn per project. Someone without access sees a page naming who to contact, never a loop back to sign-in.
  • Programs use API keys. Tests, CI jobs, trace exporters, the Python and TypeScript clients and agents connecting over MCP authenticate with API keys created by an administrator. Each key carries scopes (trace, evaluate, annotate, manage, admin), can be limited to one project, can expire, is shown once and stored only as a hash, and can be revoked at once.
  • Every change made by a person or a key is written to the audit trail.

Running It on Your Machine

Under the platform's dev destination only PostgreSQL (port 54343) and Redis (port 63343) run in containers, started by wbsp deploy --destination dev. The API, the background worker and the dashboard run directly on your machine: in VS Code, start the launch configuration Full stack (API + worker + dashboard + browser); from terminals, from the repository root:

set -a; source .env; source .env.dev; set +a; cd target && uv run aeb api --port 63959   # API
set -a; source .env; source .env.dev; set +a; cd target && uv run aeb worker             # worker
set -a; source .env.dev; set +a; cd target/web && pnpm dev                               # dashboard

Open http://localhost:30343. Under dev the local sign-in bypass is on, so you arrive signed in as Local developer, an administrator. The ports come from the project number (343); nothing listens on 3000, 8000 or 8080. Full steps, including the one-time uv sync and pnpm install, are in user-docs/getting-started.md; every other command is in user-docs/useful-commands.md.

Market Context

LLM evaluation sits inside the broader MLOps platform market, valued at approximately USD 2.0 billion in 2024 and projected to exceed USD 10 billion by 2029, and is one of its fastest-growing sub-segments. Pricing tiers are well-defined: open-source self-hosting is free; developer SaaS (Langfuse Pro) starts at $29–$49/mo; professional platforms (Braintrust Pro) run $249/mo; enterprise contracts (Maxim, Arize AI) reach the six-figure range. Primary buyers are AI engineers and ML teams, QA engineers building prompt regression suites, product managers tying LLM quality to business metrics, and compliance officers in regulated industries.

Project Status

The first release is implemented: scoring, regression suites and the pytest gate, run review, trace capture, agent evaluation, red teaming, monitoring and alerts, prompt promotion, human review, compliance evidence, AI-assisted authoring, the TypeScript client and the MCP server. Start with user-docs/getting-started.md. Contributions, feedback, and domain expertise are welcome.

Contributing

We welcome contributions from developers, domain experts, and potential users. See CONTRIBUTING.md for guidelines.

Important: All contributions must be your own original work or clearly attributed open-source material with a compatible licence. Copyright infringement and licence violations will not be tolerated and will result in immediate removal of the offending contribution. If you are unsure whether a piece of code, text, or other material is safe to contribute, open an issue and ask before submitting.

Licence

Licence to be determined. See discussion for context.