HeyJaredAtticus Li - Better Decisions
Product, growth, and experimentation leaders responsible for revenue use this newsletter to improve decision quality and scale profitable growth.
- Indexed issues, last 90 days
- 28
- Latest publication
- Oct 1, 2026
- Audience
- Checking…
- Earliest in this view
- Sep 9, 2026
Latest issues
Minimum Detectable Effect (MDE): The Most Important Number You're Not Setting Correctly (opens the original)
Read excerpt
Here’s a conversation I’ve had more times than I can count.“Our test has been running for 8 weeks and hasn’t reached significance.”“What MDE did you set?”“What’s MDE?”That’s the problem. Minimum Detectable Effect is the single most important number in experiment design, and it’s the number teams are most likely to either skip entirely or set arbitrarily. The result is either tests that run forever chasing an effect too small to matter, or tests that declare winners on effects too small to detect
Bayesian vs Frequentist Testing in Optimizely: Which Should You Choose? (opens the original)
Read excerpt
The question I get more than any other from teams setting up Optimizely for the first time: “Should we use Bayesian or Frequentist?”It’s the right question to ask. The answer changes your UI, your interpretation, your stopping rules, and how you communicate results to stakeholders. Get it wrong and you’ll either run tests for too long, call winners too early, or confuse your entire leadership team.Here’s what actually changes between the approaches — and a decision framework for picking the righ
How Optimizely Calculates Statistical Significance (opens the original)
Read excerpt
You ran a test. The dashboard shows 95% statistical significance. Your variant is up 4%. You’re ready to ship.Stop.That number does not mean what most people think it means. And if you’re making ship/no-ship decisions based on a misreading of statistical significance, you are guaranteed to ship losing tests eventually — probably already have.After 100+ experiments, I’ve watched smart teams freeze at p-values and confident teams barrel through with fundamentally broken interpretations. This artic
Why Your Optimizely Results Keep Changing (And When to Worry) (opens the original)
Read excerpt
Tuesday morning: your test shows 94% confidence and a 12% lift. You tell your team it’s almost ready to call.Friday afternoon: 71% confidence. 3% lift. Same experiment.You didn’t change anything. The test is just... different now. Is something wrong with your experiment? With the platform? With your data? Should you be worried?Usually, no. But sometimes yes. The ability to tell the difference is one of the most underrated skills in experimentation.The Three Types of Result ChangesNot all result
False Discovery Rate in Optimizely: Why Running Many Tests Simultaneously Is Riskier Than You Think (opens the original)
Read excerpt
You’re running 15 active experiments. You’ve set each one to 95% statistical confidence. You’re shipping winners and killing losers with discipline. Your testing program looks mature.Here’s the problem: you’re almost certainly making more wrong decisions than you think.The math of multiple testing means that running many experiments simultaneously — even with rigorous individual confidence thresholds — produces a program-level false positive rate that’s far higher than 5%. Understanding this is
Publishing over time
Last 90 days. Choose a month to open its work.
Recurring subjects
Named in the text we hold. One piece can cover several.
Audience
No verified audience measurement yet.
About this data
Counts cover the work we have indexed. Tone needs enough text and a confident classification. Excerpts and episode notes are not full articles or transcripts.
Identity or attribution wrong? Suggest a correction.