Back to index
OtherSelf-paced reference, a weekend to work through the statistics and SDK material·Free and open source under MIT, with an enterprise licence covering some directories

GrowthBook Documentation and Statistics Guide

4.2

The rare experimentation tool where you can read the statistics implementation rather than trusting a vendor's description of it. The docs are practical, the warehouse-native design avoids duplicating your data, and the honesty about method limitations is unusual.

What We Liked

  • Mostly MIT licensed, so the stats engine is inspectable rather than a black box you are asked to trust
  • Warehouse-native: it queries your data where it already lives instead of ingesting a second copy
  • Covers CUPED, sequential testing, Bayesian analysis, bandits and sample ratio checks in one place
  • Twenty-four SDKs means the flagging half is genuinely usable across a polyglot stack
  • Self-hosting is a real supported path, not a technicality

What Could Be Better

  • Open core, and the boundary between MIT directories and the enterprise licence is easy to misjudge until you need a feature
  • Documentation assumes you already have a warehouse with clean event data, which is the hard prerequisite
  • No structured course or certification, so it is reference material rather than teaching
  • Bayesian and frequentist modes are both offered with limited guidance on choosing between them
  • Self-hosting the stats engine at scale is more operational work than the docs imply

Detailed review

There is a specific problem with commercial experimentation platforms that nobody in the industry likes discussing. You feed them your data, they run some statistics, and they hand back a result that says a change is significant and worth a two percent lift. You cannot check the arithmetic. You cannot see whether the variance reduction was applied correctly, whether the sequential boundaries were computed the way the marketing page claims, or whether the confidence interval accounts for the fact that your randomisation unit and your analysis unit are different things.

You are asked to trust the vendor on the single question that determines whether the entire programme is producing knowledge or expensive superstition. GrowthBook's answer to that is to make the code readable. The bulk of it is MIT licensed, the stats engine is in the repository, and if you want to know exactly what CUPED adjustment is being applied you can go and read it. For a certain kind of data scientist this is the whole argument, and I think they are right.

The docs themselves are practical rather than academic. The feature flagging material is competent and covers what you expect: targeting rules, percentage rollouts, environments, SDK integration across twenty-four platforms including React, Python, Android and iOS. That breadth matters more than it sounds, because feature flags are one of the few pieces of infrastructure that genuinely has to work identically in your web app, your mobile app and your backend, and a flagging tool with good JavaScript support and a neglected Go SDK causes exactly the kind of drift you do not want in release control. The experimentation half is where it gets interesting.

The stats engine documentation covers CUPED, sequential testing, Bayesian analysis, post-stratification, bandits and sample ratio checks, which is a serious list. More to the point, it explains what each method is for rather than presenting them as a feature checklist. The warehouse-native architecture deserves a note of its own because it is a real design decision rather than positioning. Most experimentation platforms want you to send them events, which means a second copy of your user data lives in a vendor's system, subject to its own retention policy, its own access controls and its own deletion workflow.

GrowthBook queries your warehouse. Your data stays where your governance already applies to it, your event definitions stay in one place, and when someone asks whether the experiment's definition of an active user matches the dashboard's definition, the answer is that they are the same SQL. Anyone who has spent a week reconciling two systems that disagreed about what a signup was will appreciate this more than the feature list suggests. The honest problems start with the open core boundary.

Most of the code is MIT, some directories are governed by a commercial enterprise licence, and the line between them is the sort of thing that seems academic during evaluation and becomes urgent when your compliance team asks what exactly you are running. The documentation could be considerably clearer about which capabilities sit on which side. This is not a criticism unique to GrowthBook, it applies to nearly every open core project in this catalogue, but it applies here too. The prerequisite problem is larger and less discussed.

Warehouse-native is only an advantage if you have a warehouse with trustworthy event data in it. If your events are inconsistently named, if a tracking change six months ago silently altered what a conversion means, or if your identity resolution is approximate, GrowthBook will compute beautifully rigorous statistics on top of rubbish. The docs do not really warn you about this, and it is by some distance the most common reason experimentation programmes fail. The methodological fork is the other omission.

GrowthBook offers both Bayesian and frequentist analysis, and gives you relatively little help deciding which to use. In practice this matters, because the two frameworks answer different questions and a team that switches between them depending on which gives the answer they wanted is doing something considerably worse than using either consistently. A page of blunt guidance saying pick one, here is how, and do not change it mid-programme would improve the product. What you do not get is teaching in any structured sense.

There is no course, no learning path, no certification. This is reference documentation for people who already know roughly what they are doing, and a product manager who has never run an experiment will find it hard going. The right use of it is alongside something more pedagogical, and then the code when you want to know what is really happening. Four point two.

Inspectable statistics, a sensible architecture and practical documentation, marked down for open core ambiguity, no teaching layer, and silence on the data quality prerequisite that decides whether any of it works.

[ final ]

The verdict.

The best option if you want to understand experimentation rather than outsource it. Read the statistics section alongside Statsig's for two independent explanations of the same methods, then pick whichever tool your data situation suits.