post

Footer About

Footer Get Involved

Footer Links

Mind the Product

General

Product Unplugged Episode 3 with Pierce Healy

Product Unplugged Episode 2 with Lorilyn McCue

Product Strategy

Building Product Teams

Diversity and Inclusion

Digital Transformation

Artificial Intelligence (AI) & Machine Learning (ML)

Product Leadership

Product Management Career

Product Ethics

Minimum Viable Product

Enterprise Product Management

OKRs

Customer Research

Hypothesis and Testing

Business Strategy

User Research

👩‍🚀🚀   Live now for MTP members: <a href="https://mtp2017stg.wpengine.com/membership/#events" style="color:#FFFFFF;text-decoration:underline;">Driving Innovation AMA with Tendayi Viki</a>

Roadmaps: Break the rules without breaking the principles

Got questions about your product career?

Product roadmaps vs data – Jeremy Levy on The Product Experience

From engineering to product management at Google

Mind the Product joins Pendo

Louron is an editor at Mind the Product. He has had a background in writing and editing while previously working as a journalist for several business publications. He spent the first few years of his career working in content marketing and audio production. You can find him on Twitter as @LouronPratt22.

Louron Pratt

<p>In this <span style="text-decoration: underline;"><a href="https://mtp2017stg.wpengine.com/mtpcon/digital/americas/">#mtpcon Digital Americas</a></span> session, <span style="text-decoration: underline;"><a href="https://www.linkedin.com/in/arindone">Anthony Rindone</a></span>, product manager for experimentation at Split discusses how to make better product decisions through experimenting.</p>
<h2>Breaking metrics down</h2>
<p>Anthony says that experimenting with products is a humbling experience, if you succeed in 30% of the experiments you make, then you are in a world-class position, meaning that the number of times experiments you invest in turn out to fail. At Split, he explains that every change is a feature, and every feature is an experiment. He goes on to discuss three case studies from the experimentation trenches.</p>
<p>In his first example, he explains that he has encountered using the wrong overall evaluation criterion (OEC) to initiate experiments. A better approach to experimentation is to leverage a broader field of possibilities. By opening your range, you’re able to break things down and create sub-metrics and track what is specifically changing through that.</p>
<h2>Integrating multiple variations over time</h2>
<p>Throughout his second case study he talks about when introducing a new feature, it experiences a novelty effect. This can be misleading to how well received a new feature will be. If it’s a brand new addition it’s bound to receive more traction, but this reception may not necessarily last for long. By running multiple variations over four weeks, you can see what actual effect these experiments have. While running several variations, it’s important to sweat out every piece of detail.</p>
<p>“When you think about your whole application from end to end, there’s going to be some overlapping through new campaigns or feature roll-outs,” Anthony says, “improving your segmentation and being more precise with testing is key to tracking new variations effectively over time.”</p>
<h2>Have a specific hypothesis</h2>
<p>Finally, Anthony talks about his third example of having a vague hypothesis. The wider you cast your net, the more difficult it is to have a clear decision process or answer, he explains.</p>
<p>“If you don’t have a clear decision-making process, it creates loads of answers of a wide variety,” he says. Consequently, you end up setting yourself up for failure where you have two opposite conclusions. Both can’t be right. “The conclusion is normally decided by the highest-paid person in the room, which may not always be the right one.”</p>
<p>A hypothesis is more than a guess, it’s a foundational element on how to make decisions, he says. Build a falsifiable hypothesis upfront, identify the target segment at the start of your experiment. Doing so will provide an action plan for each outcome which everyone agrees on.</p>
<h2>To wrap up</h2>
<p>Anthony closes by providing some best practices with experimentation:</p>
<ol>
<li>Focus on the bigger picture</li>
<li>&#8230;But still, sweat the details</li>
<li>Having precise, falsifiable hypothesis’ are foundational needs to effective decisions-making</li>
</ol>
<h2>Explore more content like this</h2>
<p>Ordinarily, premium content such as this can only be accessed by Mind the Product members. Today, we’re offering up this talk to everyone in the community. If you like the content and want more like it, you can check out our <a href="https://mtp2017stg.wpengine.com/membership/compare/" target="_blank" rel="noopener">Mind the Product membership plans</a>.</p>
<p>Still in need of inspiration? Browse even more <a href="https://mtp2017stg.wpengine.com/category/user-testing/" target="_blank" rel="noopener">user testing</a> content or use our <a href="https://mtp2017stg.wpengine.com/content-a-z/" target="_blank" rel="noopener">Content A-Z</a> to find specific topics of interest.</p>
<p>If you’re a Mind the Product member, you can also check out more <a href="https://mtp2017stg.wpengine.com/tag/mtpcon/" target="_blank" rel="noopener">#mtpcon video content</a>.</p>
<div class="g g-21"><div class="g-single a-77"><a class="gofollow" data-track="NzcsMjEsMSw2MA==" href="https://mtp2017stg.wpengine.com/product-management-training/public-workshops/"><img decoding="async" src="https://mtp2017stg.wpengine.com/wp-content/uploads/2021/10/General-Training-Next-Level-FOOTER-scaled.jpg" /></a></div></div>

core/freeform

User Testing

Stay Humble or Be Humbled — Case Studies from the Experimentation Trenches by Anthony Rindone

Looking at the bigger picture by Rita Cervetto

Uncovering Your Most Pivotal Users by Marieke McCloskey and Doug Puett

Quantifying Qualitative Research by Leisa Reichelt

Naimish Gohil – Adapting Your Product for a Changing Market

The Typology of Design Sprints

ProductTank Columbus

ProductTank Tauranga

ProductTank Edinburgh

Prioritised Member AMA - Perfecting product roadmaps

ProductTank San Diego

Navigating difficult relationships panel

ProductTank Philadelphia

ProductTank Bucharest

ProductTank NoVa

Remote Training Workshops

ProductTank London

ProductTank Birmingham

ProductTank Oslo

ProductTank Astana

ProductTank NoVA

ProductTank Tokyo

ProductTank Belgrade

ProductTank Madrid

ProductTank Leeds

ProductTank Zürich

ProductTank Perth

ProductTank Chicago

ProductTank Auckland

ProductTank Richmond

ProductTank Charlotte

ProductTank Reston

ProductTank Washington

ProductTank Sofia

ProductTank Raleigh

ProductTank Ljubljana

ProductTank Kraków

ProductTank Athens

ProductTank Barcelona

ProductTank Dublin

ProductTank Montréal

ProductTank Bogotá

ProductTank Seattle

ProductTank Madison

ProductTank Singapore

ProductTank Portland

ProductTank Budapest

ProductTank Valencia

ProductTank Porto

ProductTank Los Angeles

ProductTank Austin

ProductTank Norwich

ProductTank Des Moines

ProductTank San Francisco

ProductTank Helsinki

ProductTank Reading

ProductTank Brighton

ProductTank Bielefeld

ProductTank Malmö

ProductTank Canberra

ProductTank Paris

ProductTank Manchester

#mtpcon SF+Americas 2022 Conference

#mtpEngage Hamburg 2022 Leadership Forum

#mtpEngage Hamburg 2022 Workshops

#mtpEngage Hamburg 2022 Conference

#mtpcon London+EMEA 2022 Conference