A/B Testing SDK & Platform

Part of the worlds-biggest-software-project initiative.

Open-source, AI-native experimentation infrastructure with client-side and server-side SDKs and a centralised platform for managing experiments, analytics, and rollouts.

The A/B Testing SDK & Platform gives product and growth teams a rigorous experimentation framework spanning browser, server, mobile, and edge runtimes. It combines local-evaluation SDKs with a management layer for defining experiments, targeting audiences, computing statistical significance, and graduating winners — addressing the gap between client-side tools that flicker and server-side tools that are hard to set up.

Why A/B Testing SDK & Platform?

Client-side experimentation tools suffer from rendering flicker because assignment is evaluated after page render; server-side or edge evaluation is needed but often missing from lower-cost tools.
Closed-source SaaS leaders (Statsig, LaunchDarkly, Optimizely) lock experiment history, statistical engines, and event schemas behind proprietary platforms; pricing escalates sharply at large event volumes.
Open-source incumbents are split between warehouse-native depth (GrowthBook) and unified analytics breadth (PostHog), but no project combines warehouse-native rigour with polished managed UX at a mid-market price.
LaunchDarkly's experimentation module is a paid add-on on top of base flag pricing, and its statistical engine is less advanced than dedicated experimentation platforms.
Statistical pitfalls — peeking, sample ratio mismatch, mutually exclusive experiments, and guardrail metric regressions — are not consistently enforced across existing tools.

Key Features

SDKs and Local Evaluation

Multi-language SDKs for browser JS/TypeScript, Node.js, Python, iOS (Swift), and Android (Kotlin) as a minimum baseline
Local flag evaluation after a one-time payload fetch — zero added latency per request
Edge runtime support targeting Cloudflare Workers, Fastly Compute, and Lambda@Edge
Server-side evaluation to eliminate client-side flicker and cover backend logic

Feature Flag and Experiment Management

Percentage rollouts, cohort targeting, and environment separation (development, staging, production)
Experiment configuration with variant definition and traffic allocation
Audit log of every flag and experiment change with actor, timestamp, and diff
Approval workflow for production flag changes

Statistical Analysis

Bayesian or frequentist analysis with significance thresholds and confidence intervals
Sequential testing to allow safe early stopping without inflating false-positive rates
CUPED variance reduction for faster experiment convergence
Sample ratio mismatch (SRM) detection and data quality checks

Warehouse-Native Analytics

Direct metric sourcing from BigQuery, Snowflake, and Postgres without re-instrumenting events
Reusable metric definitions shared across experiments for consistency
Guardrail metric tracking that alerts on regressions in latency, error rate, or revenue
Cross-experiment interaction detection for overlapping flags on the same user surface

AI-Native Advantage

AI augmentation targets the parts of experimentation that are tedious or easy to get wrong. Candidates include automated experiment design (suggesting sample size, run duration, and metric mix from a hypothesis), natural-language result summaries for non-technical stakeholders, automated guardrail monitoring before harmful variants ship, and predictive holdout analysis estimating long-term impact from short-run data. These directly address weaknesses in incumbents whose statistical engines are powerful but operationally opaque.

Tech Stack & Deployment

The platform is designed to be self-hostable for full data sovereignty, with a managed cloud option for teams that prefer not to operate the stack. SDKs perform local evaluation after a single payload fetch, with optional edge deployment via Cloudflare Workers, Fastly Compute, and Lambda@Edge. Warehouse-native integration with BigQuery, Snowflake, and Postgres lets experiment analysis run against existing data infrastructure rather than a duplicated pipeline.

Market Context

The experimentation platform market is crowded across price points, from open-source warehouse-native tools (GrowthBook, PostHog — both MIT) to enterprise SaaS (Optimizely, LaunchDarkly, Statsig, Amplitude Experiment) and free managed offerings (Firebase A/B Testing). Buyers are product, growth, and platform engineering teams; primary pain points are SDK breadth, statistical rigour, warehouse integration, and pricing predictability. Candidate scoring (per candidate-projects.md row #382): complexity 6, domain availability High, demand High.

Project Status

This project is in the research and specification phase.
Contributions, feedback, and domain expertise are welcome.

Contributing

We welcome contributions from developers, domain experts, and potential users. See CONTRIBUTING.md for guidelines.

Important: All contributions must be your own original work or clearly attributed open-source material with a compatible licence. Copyright infringement and licence violations will not be tolerated and will result in immediate removal of the offending contribution. If you are unsure whether a piece of code, text, or other material is safe to contribute, open an issue and ask before submitting.

Licence

Licence to be determined. See discussion for context.