#031

A/B Testing Platform

ServiceWebsite

Experiment management, statistical significance engine, rollout integration

Project Variant:
Dark factory developmentPublic
Candidate
2
Dark factory
3
Guided development
4
Raw
5
Custom development
6
Alpha
7
Beta
8
Production

Collaborators

philcal (project owner)

How do I use this software?

This software runs wherever suits you — you just need somewhere to host it. Pick the option that fits your team:

OptionWhat it means
Self-hostingSet up the environment and run it yourself, on your own infrastructure.
Tooltwist hostingTooltwist can host and run it for you.
Other providersFind a host in the provider directory — or, if you already have a support company, we're happy to give them the tools to deploy the application for you.
Licensing

This variant is open source — you're free to use it and modify it at no cost. Hosting and support arrangements are provided separately and are not covered by this licence.

Who can help me?

Plenty of people can help you get the most from this software — browse the provider directory. Some providers can host it for you, others can customise it to your needs, and others again offer technical support and a helpdesk.

Tooltwist can host and customise the software for you, and Twist Teams provides technical support services.

Already have a support provider? We're happy to give them the tools to fully support the platform.

Not sure who to ask? Feel free to .

How can I help?

If you would like to help develop or test this project, go to the Collaborators tab (after you log in) and request to join. Your help will be appreciated!

Try me

Read This First

This page is a plain-language introduction to Stratum. It assumes no technical background and uses no jargon — every specialist word is explained the moment it appears. Read it once, top to bottom, and you will know what this product is, whether it is for you, and what living with it looks like.

This page is deliberately not two things. It is not an installation manual — if your job today is to get the software running on a machine, go to Getting started. And it is not a reference — the fuller guides are linked where they help and listed at the end.

Who is this for?

Stratum is for teams that run a website or an app and want to improve it without guessing.

  • Engineers use it to switch new features on gradually and safely — and off again instantly if something goes wrong — without redeploying the site.
  • Product managers use it to test ideas on real visitors and get a trustworthy answer to "did this change actually help?"
  • Analysts use it to check the numbers behind every result, because the product shows its working rather than asking to be believed.
  • Administrators use it to control who can change what, with a permanent record of every change.

It suits any organization whose website or app matters to its business — an online shop, a software product, a media site, a booking service. Because you run it on your own machines and it computes results inside your own database, it especially suits organizations that want customer data kept in-house.

What does it do?

Stratum does two closely related jobs.

Job one: it lets your website show different things to different visitors, under your control. Your site asks Stratum, for each visitor, "which version should this person see?" — version A or version B of a page, a green or a blue button, the old checkout or the new one. You change the answer from a control panel (the dashboard), and the change takes effect on your live site within about a minute, with no code release. These remote-controlled switches are called feature flags.

Job two: it measures which version actually works better. Show the new checkout to half your visitors and the old one to the other half, and Stratum will tell you — from your own business data — which half bought more, signed up more, or stayed longer, and whether the difference is real or just luck. This is called an experiment.

flowchart LR
    visitor([Visitor]) --> app["Your website<br/>renders A or B"]
    app -- "user attributes" --> sdk["Stratum SDK<br/>in your app"]
    sdk -- "variant A or B" --> app
    server["Stratum server<br/>+ dashboard"] -. "rules, polled ~30 s" .-> sdk
    sdk -- "exposures" --> server
    server --> results["Results, computed<br/>in your warehouse"]

In pictures: your website asks a small built-in helper (the SDK — a piece of Stratum that lives inside your site) which version each visitor gets; the helper follows rules you set in the dashboard; and reports of "who saw what" flow back so results can be measured.

What field is this?

This product belongs to the practice of A/B testing — also called online controlled experimentation — which is how modern websites and apps are improved: instead of arguing about whether a change is better, you show it to some visitors and not others, and let the outcome decide. The occupation it serves is digital product development — the everyday work of product teams, growth teams, and marketing teams.

Typical questions it settles: Does the green "Buy now" button sell more than the blue one? Does the shorter sign-up form finish more often? Does the redesigned checkout flow increase orders — or quietly lose the customers who used coupons?

What surrounds it?

Stratum never works alone. Here is the cast of characters around it:

  • Your website or app, and its visitors. This is where all the action happens. Stratum is a companion system standing beside your site — the site consults it, shows visitors whatever it decides to show, and reports back what each visitor saw.
  • Your team. People sign in to the dashboard to set up switches and experiments and read results. Four permission levels exist, from administrator down to read-only viewer, so the intern can look without touching.
  • The SDK inside your site. A small helper your developers add to the website once. It is the messenger between your site and Stratum.
  • Your data warehouse. The database where your business already records what customers do — orders, sign-ups, page visits. Stratum measures experiment results inside that database, so your raw customer data never leaves your control. (A "warehouse" can be as ordinary as a PostgreSQL database.)
  • Your sign-in system. Team members sign in with your organization's existing single sign-on; Stratum does not keep its own passwords.
  • Your notification channels. Stratum can send messages to other systems when something noteworthy happens — an experiment finishing, a result looking untrustworthy — so news reaches the tools your team already watches.

What it does NOT do

Three common assumptions about tools like this are not true of Stratum, and knowing that early will save you confusion:

  • It does not sit between your visitors and your website. It is not a gateway, proxy, or layer your traffic passes through. If Stratum were switched off entirely, your website would keep serving visitors — they would simply all see the standard version.
  • It does not watch, detect, or identify your visitors. Stratum learns only what your website chooses to tell it — "this visitor is id 123, on the pro plan, in Australia." It has no ability to discover a visitor's address, location, or identity on its own.
  • It does not host or serve your website. Your site stays exactly where it is, built with whatever it is built with. Nothing moves.

The first thing to do

Sign in, open the demonstration project, and wander around. A standard installation comes preloaded with a realistic pretend company — switches already configured, experiments already run, results already computed. Nothing in it is real, so you cannot break anything. Click into a feature flag and see its rules; open a finished experiment and look at its results page; find the button that shows the exact calculation behind a number. Fifteen minutes of this teaches more than any manual.

Two situations change that advice slightly:

  • If your instance is brand new and empty (a production installation is deliberately set up clean, with no demonstration data), the first action is instead to create your organization's first project — the workspace that will hold your flags and experiments. Whoever installed the system has this covered in Production setup.
  • If you don't have a running instance at all yet, that's the installation job — Getting started — and it ends with the demo data described above.

Setting up your complete environment

When you are ready to move from exploring to really using Stratum, there are five connections to make, in this order. If your team only wants the on/off switches for now — no measuring — you can do just steps 4 and 5 and come back for the rest later.

  1. Connect your data warehouse. Tell Stratum where your business data lives so it can measure results there. You'll paste in connection details and press "Test connection". (Guide: User guide → Metrics and data sources.)
  2. Point it at your existing records. Show Stratum which table holds the events you care about — orders, sign-ups — and which columns identify the customer and the time. (Same guide as step 1.)
  3. Define your metrics. Turn "success" into named, reusable definitions — conversion rate, revenue per visitor — so every experiment measures things the same way. (Same guide as step 1.)
  4. Put the SDK into your website. A developer adds Stratum's small helper to your site and passes it the visitor details you want to target on. This is the one step that needs a code change, and it is done once. (Guide: SDK guide.)
  5. Invite your team. Add colleagues and give each a permission level — administrator, experimenter, analyst, or viewer. (Guide: User guide → Governance.)

Operators tuning the installation itself (addresses, secrets, environments) have their own reference: Configuration.

A normal day

Day to day, Stratum is used in two rhythms.

Rolling out a feature. A developer ships a new capability switched off, hidden behind a flag. From the dashboard, someone turns it on for a small slice of visitors — say 10% — and watches. All quiet? Widen to 25%, then half, then everyone. Something wrong? One click turns it off for all visitors, instantly, with no emergency release. The same visitor always stays in the same group, so nobody sees a feature flicker on and off between page views.

Running an experiment. Someone writes down what they expect ("the shorter form will finish more often"), sets up the versions to compare, picks the metrics that define success, and presses Start. Then — mostly — everyone waits, because experiments need enough visitors to be trustworthy. Results arrive as regularly refreshed report cards. One rule matters more than the rest: if a result carries the data-quality warning (called SRM), don't act on it. That warning means the visitors didn't split into groups the way they were supposed to — a sign of a plumbing problem, not a real winner — and the fix is to investigate, not to celebrate. A clean, convincing result — one where the difference is too large to be luck — ends with a decision: keep the winner for everyone, or discard the change and move on, either way knowing rather than believing.

Ongoing care

Stratum needs little tending, but a healthy installation has a rhythm to it:

  • Look at running experiments on a schedule, not obsessively. Results refresh themselves at a cadence you choose; a periodic review is more honest than hitting refresh hourly.
  • Retire flags whose job is done. Once a feature is at 100% for everyone and here to stay, remove its check from the website's code and archive the flag. A pile of stale flags is clutter that slowly makes the site harder to reason about.
  • Archive finished experiments and unused metrics. Archiving tucks things away without destroying them — history is kept, lists stay clean.
  • Back up the database. Stratum keeps its configuration and history in an ordinary database; back it up the way your organization backs up any production database. This is standard operator practice, not a built-in Stratum feature.
  • Rotate warehouse credentials periodically. The stored connection secrets can be replaced but never read back out — rotating them on your security team's schedule is painless.
  • Skim the audit log now and then. Every change ever made — who, what, when, before and after — is permanently recorded. Administrators reviewing it occasionally keeps the record useful rather than forensic.
  • Stay reasonably current. Updates are applied by your operator; the system's database upgrades are designed to be applied safely and stepped back if needed.

The demonstration data

The pretend company you meet on first sign-in is part of the product. An evaluation copy comes preloaded with two invented organizations — an online shop and a media business — complete with projects, feature flags, experiments at every stage of life (draft, running, paused, and finished), metrics, a connected warehouse, and results already computed. Every name, customer, and number in it is fictional. Explore freely; nothing you do there touches anything real.

When you want the pristine version back, start a fresh evaluation copy — the all-in-one evaluation container rebuilds its demonstration data every time a new one is started (unless you deliberately asked it to keep data between runs), and a developer installation reloads it with one command (Getting started). A production installation starts completely clean: no demonstration data is ever present there (Production setup).

Roles at a glance

Four permission levels cover who can do what. In brief:

RoleIn a sentence
AdminCan do everything, including managing members and settings and reading the audit log.
ExperimenterCan create and change flags, experiments, and metrics, and start or stop experiments — the hands-on editor role.
AnalystCan read everything, including the audit log, but can change nothing.
ViewerCan look at screens and results only — strictly read-only.

Roles are set for the whole organization and can be adjusted per project. The full detail lives in User guide → Governance.

Quick questions

Do we need a data warehouse before we can start? No. The on/off switches (feature flags) work with no warehouse at all. You connect a warehouse when you want to measure — it is the first step of "Setting up", not a requirement for day one.

We turned a flag off — why did a visitor just see the feature? Your website checks in for fresh rules roughly every half a minute, so a change can take up to about a minute to reach every visitor. That is the design working, not a fault.

A result is showing the traffic-split warning (SRM). Can we still use the numbers? No. That warning means visitors did not divide into groups the way the experiment intended, so the comparison is not fair. Investigate the cause; do not act on the result.

The new version's number is higher — why does the verdict say "not significant"? Because the difference is still small enough that luck could explain it. The verdict changes when enough visitors have been through to rule luck out. The honest answer to impatience is to wait.

Could Stratum slow down or take down our website? It is not in the path of your visitors' traffic. If it were unreachable, your site would carry on serving everyone the standard version.

A small glossary

Every one of these is explained in context somewhere above; here they are again in one place.

  • Feature flag — a remote-controlled switch inside your website, changed from the dashboard without a code release.
  • Experiment — a fair comparison of two or more versions, measured on real visitors.
  • Variation — one of the versions being compared; the unchanged original is the control.
  • Metric — a named definition of success, such as checkout conversion or revenue per visitor, defined once and reused by any experiment.
  • Dashboard — the control panel where your team manages all of the above.
  • SDK — the small helper your developers embed in your website once; it asks Stratum which version each visitor should get.
  • Exposure — a recorded fact that one visitor saw one variation; the raw material results are computed from.
  • Data warehouse — the database where your business already keeps its records; Stratum computes results inside it. It can be as ordinary as a PostgreSQL database.
  • Fact table — a table in that warehouse holding the events you measure, such as orders or sign-ups.
  • SRM (sample ratio mismatch) — the data-quality warning that visitors did not split as designed; a result carrying it should not be trusted.
  • Environment — a separate copy of a flag's rules, so a change can be tried somewhere safe before it reaches the live site.
  • Project — the workspace holding one product's flags, experiments, and metrics; an organization can hold several projects.
  • Audit log — the permanent, unchangeable record of who changed what, and when.

Where to go next

You now have the whole picture. Here is the rest of the bookshelf, and when to take each volume down:

Read……when you want to
How it worksSee the machinery behind this page: exactly how your website and Stratum talk to each other, step by step.
Getting startedInstall Stratum on your own machine and sign in for the first time.
User guideLearn the product properly: flags, experiments, reading results, metrics — with the statistics explained in plain language.
SDK guideAdd Stratum's helper to your website (for your developers).
API referenceDrive everything from your own scripts and systems (for your developers).
ConfigurationTune the installation: settings, secrets, environments (for your operators).
Production setupPrepare a real, clean production system for first use (for your operators).

Where to get help

Stratum is part of the wbsp.ai community. That is the place to ask questions, report something confusing, suggest a feature, or find people who customise builds like this one — restyled to a company's brand, extended with new capabilities, or adapted to a different domain. While you are there, have a look around the catalogue: the community has already built many applications, including other versions of this one, and a close match to what you need may already exist.