WBSP
ProvidersDocsSign in

base

A variant of AI Prompt Management System.

View the interactive variant page →

AI Prompt Management System

Part of the worlds-biggest-software-project initiative.

An open-source, AI-native platform for versioning, evaluating, A/B testing, and cost-tracking LLM prompts across the full development lifecycle.

The AI Prompt Management System is a unified tool for ML engineers, AI/LLM engineers, and product teams shipping production LLM applications. It treats prompts as first-class versioned artefacts — with immutable history, environment labels, evaluation pipelines, and cost analytics — while using AI itself to automate optimisation, regression detection, and cross-provider adaptation.

Why AI Prompt Management System?

  • Existing open-source tools cover only parts of the lifecycle. Langfuse leads on observability but treats prompt management as secondary; Agenta is fully MIT-licensed but has a smaller community; Promptfoo is CLI-only with no production management. No single open-source tool spans versioning, testing, deployment, and monitoring.
  • Proprietary SaaS gates core capabilities behind paywalls. PromptLayer ($99/month+), LangSmith ($39/month+), Vellum (enterprise pricing), Braintrust (usage-based), and Maxim AI (custom enterprise) all keep production-grade features behind subscriptions, with no self-hosted open-source option.
  • Enterprise compliance is a moat for incumbents. Only Maxim AI offers SOC 2, HIPAA, GDPR, and in-VPC deployment among dedicated tools — leaving regulated teams without an open alternative.
  • Consolidation pressure is reshaping the category. Humanloop was acquired by Anthropic (Aug 2025); Promptfoo is now OpenAI-owned. An independent, vendor-neutral open-source platform reduces lock-in risk for teams managing multi-provider deployments.
  • Underserved capabilities have no production-ready solution. No tool today offers automated semantic regression detection, pre-deployment cost forecasting, cross-provider prompt adaptation, or prompt dependency graphs.

Key Features

Versioning and Deployment

  • Prompt versioning with immutable history and auto-incrementing version numbers
  • Named deployment labels (production, staging, dev) as mutable pointers to specific versions
  • One-click rollback by reassigning labels to any prior version
  • Branch-based versioning for parallel independent development (inspired by Agenta)
  • GitHub sync via webhooks for Git-native prompt workflows

Evaluation and Testing

  • Dataset-based evaluation with LLM-as-judge, code-based, and human evaluators
  • Pass/fail thresholds and quality gates blocking promotion when evaluations fail
  • Pairwise comparison of two prompt versions against shared test cases
  • Multi-provider playground for interactive testing across OpenAI, Anthropic, Gemini, and more
  • Adversarial and red-teaming test case generation with OWASP LLM Top 10 mapping

Production Operations

  • Production A/B traffic splitting by percentage or user segment
  • Token cost tracking per request and per prompt version
  • Distributed tracing with token usage, latency, and cost metrics
  • Human annotation queue for collecting feedback on production traces
  • REST API for runtime prompt retrieval with SDK-side caching and configurable TTL

AI-Assisted Workflows

  • Automated prompt optimisation using evaluation feedback
  • Semantic change classification to flag high-risk vs low-risk edits before deployment
  • Adversarial test case synthesis from a small set of human-written examples
  • Cross-provider prompt adaptation rewriting prompts to perform equivalently across models
  • Pre-deployment cost forecasting across model tiers

AI-Native Advantage

Where incumbents treat AI primarily as the workload being managed, this project uses AI inside the management layer itself. Automated semantic regression detection flags when a new version changes output meaning even when surface metrics look stable. Cross-provider adaptation rewrites prompts optimised for one model to run equivalently on another, accounting for provider-specific formatting. Cost forecasting models projected token spend across model tiers before changes reach production. Adversarial test generation expands human-written examples into edge cases and injection attempts automatically.

Tech Stack & Deployment

  • Deployment modes: self-hosted (Docker, Kubernetes) and managed cloud, with full feature parity for the open-source tier
  • SDKs: Python and JavaScript/TypeScript at minimum, following the patterns used by Langfuse, Agenta, and Braintrust
  • APIs: REST API with OpenAPI specification for runtime prompt retrieval and management
  • Observability: OpenTelemetry-compatible tracing aligned with GenAI semantic conventions
  • Integrations: OpenAI SDK, Anthropic SDK, LangChain, LangGraph, LiteLLM, Vercel AI SDK
  • CI/CD: GitHub Actions integration for prompt validation pipelines

Demo / sample data

target/scripts/seed-demo.mjs loads realistic, credible sample data into a destination's database so every dashboard screen looks populated for demos, screen-recordings, and screenshots. It is separate from migrations — it writes demo content only (provider pricing is seeded by a Prisma migration and is left untouched).

Run it (from target/):

# Local "dev" destination (start data services + apply migrations first):
docker compose up -d              # Postgres + Redis  (repo root)
cd target
npm run db:migrate                # apply schema
npm run db:seed:demo -- dev       # load demo data

# Other local destinations:
npm run db:seed:demo -- compose
npm run db:seed:demo -- standalone

# Cloud (AWS) destinations — DATABASE_* are injected by the WBSP platform;
# --force is required so you never wipe real data by accident:
node scripts/seed-demo.mjs stage --force
node scripts/seed-demo.mjs prod  --force

The destination argument selects which database to load. Connection details come only from the platform-injected discrete DATABASE_HOST / DATABASE_PORT / DATABASE_NAME / DATABASE_USER / DATABASE_PASSWORD / DATABASE_SSL variables (the same vars src/lib/config.ts and scripts/migrate.mjs use — WBSP never provides a single DATABASE_URL). For local destinations, any of those not already in the environment are read from the repo-root .env.<destination> (or its committed .env.<destination>.example); values already set in the environment always win.

See your own account fully populated after login: the platform authenticates via HAP OIDC, so a real login creates a fresh user that isn't a member of the seeded projects. Pass your HAP subject so the primary demo user is you:

SEED_OWNER_SUB="$MY_HAP_SUB" SEED_OWNER_EMAIL=me@example.com \
  npm run db:seed:demo -- dev

Idempotency / flags: the script is safe to re-run. By default it does a truncate-then-insert (TRUNCATE … RESTART IDENTITY CASCADE on every demo-owned table, which bypasses the append-only UPDATE/DELETE triggers, then re-inserts fresh rows). Pass --no-wipe (or SEED_WIPE=false) to insert without wiping, and SEED_DRY_RUN=1 to print the row counts it would insert without touching any database. Full usage lives in the script's header comment.

It seeds ~2,000+ rows across 4 sample projects: users & memberships, API keys, provider credentials, prompts with branches/versions/deployment labels & promotions, datasets/versions/cases (incl. an OWASP red-team suite), evaluators, quality gates, evaluation/pairwise/security runs with results & findings, A/B experiments, traces & spans, cost roll-ups, the annotation queue, AI suggestions, GitHub-sync config/events, and audit history.

Market Context

The prompt engineering and management market was valued at approximately $850M globally in 2024 and is projected to reach $1.52B by 2026 as part of the broader LLMOps and AI observability category. Incumbent SaaS pricing ranges from $39/month (LangSmith) and $49/month (Langfuse Cloud) for individual teams, through $99/month (PromptLayer) for production usage, up to custom enterprise contracts (Vellum, Maxim AI). Primary buyers are ML and AI/LLM engineers building production applications, data science leads needing governance over prompt changes, AI product managers monitoring quality regressions, and enterprise IT teams requiring audit trails for compliance.

Project Status

This project is in the research and specification phase.
Contributions, feedback, and domain expertise are welcome.

Contributing

We welcome contributions from developers, domain experts, and potential users. See CONTRIBUTING.md for guidelines.

Important: All contributions must be your own original work or clearly attributed open-source material with a compatible licence. Copyright infringement and licence violations will not be tolerated and will result in immediate removal of the offending contribution. If you are unsure whether a piece of code, text, or other material is safe to contribute, open an issue and ask before submitting.

Licence

Licence to be determined. See discussion for context.