Mapping an Experiment: Identifying and Testing Your Riskiest Assumption in Social Impact Work

This how-to explores rapid experimentation as a core principle of product and design. By identifying and testing the riskiest assumptions early, teams can validate ideas, reduce uncertainty, and build more sustainable solutions.

Preparation

• Duration: 1 hour 
• Artifact: Experiment Map 
• Participants: Works best with 2-3 collaborators, but can be done individually.
• Supplies: Recommend using a physical or digital whiteboard like Mural or Figjam.

Purpose

• Goal
  • This method ensures that teams test their ideas effectively before significant investment, helping avoid wasted resources and false confidence.
• Practicing this in Teams & Organizations: 
  • Builds a culture of evidence-based decision-making.
  • Tests assumptions at various stages of product development.
  • Aligns design decisions with measurable business outcomes.

Learning Outcomes:

• Validate the Core Problem & Solution Fit – Ensure you’re solving the right problem and addressing real needs with evidence, not assumptions.
• Make Smarter, Faster Decisions – Reduce risk, adapt quickly, and refine your idea based on real-world feedback.
• Increase Long-Term Impact & Support – Build stakeholder trust, attract funding, and create sustainable and effective solutions.

Steps to Experimentation

Step 1: Identify Your Riskiest Assumption

Every social innovation is built on assumptions—beliefs about the problem, the people affected, and how they’ll engage with your solution. The riskiest assumption is the one that, if proven false, would cause your entire idea to fail. Some common risky assumptions in social impact work include:

• Are people actually trying to solve this problem? A solution without demand is doomed.
• Does this address the actual root cause? You might be solving a symptom of a problem rather than its underlying cause.
• Do people want help solving it? Just because a problem exists doesn’t mean people seek external solutions.
• Will people pay for it? Viability often depends on willingness to pay.
• Can this scale or sustain itself over time? Social impact efforts might be funded by investors, grants, donations, or government, or even yourself. Regardless of who is funding the effort, stakeholders need confidence in long-term viability.

Common Pitfalls of Not Experimenting

• False Confidence: Believing in assumptions rather than facts.
• Wasted Resources: Time, money, and effort spent on unused features.
• Missed Opportunities: Ignoring ways to improve the product.

Step 2: Declare a Hypothesis

A hypothesis is an educated guess about how things work. It must be something you can actually test.

What Makes a Good Hypothesis?

• Statement of Expectation: Predicts what will happen.
• Testable Belief: Must be possible to prove or disprove.
• Grounded in Assumptions: Based on existing knowledge.
• Specific & Measurable Outcome: Includes clear criteria for success.
• Targeted Aspect of the Solution: Focuses on one key factor.

Step 3: Design a Simple Experiment

After defining a hypothesis, the next step is to create an experiment to validate or invalidate it.

Components of a Good Experiment

• Clear Hypothesis: What are you testing?
• A Way to Measure: Quantitative vs. qualitative metrics.
• Right-Sized: Statistical significance vs. practical feasibility.
• Limited Bias: Designed to avoid skewed results.

Examples of Experiments

• Co-Design Sessions: Work directly with the people impacted to see if your idea fits their needs and realities.
• Prototype Testing: Build a minimal version of your solution (a flyer, a chatbot, a pop-up service) and see how people respond.
• Behavior-Based Validation: Instead of asking if people like an idea, test if they engage with it—for example, do community members attend an event or sign up for a service? Measure engagement rather than opinions (e.g., track sign-ups instead of surveys).
• MVPs: Launch a Minimum Viable Product with just enough features to be usable by early customers.

Example MVP Experiments

• Amazon: Started without inventory, buying books on demand.
• Uber: Launched with three cars to test demand.
• Bumble: Gave women control over initiating conversations.
• Netflix: Shipped DVDs by mail before streaming.

Planning Your Experiment

• Is the experiment cost-effective? Can it be done for less than $100?
• Is it quick to execute? Can it be tested in an hour?
• Is it generative? Does it provide more than a simple yes/no answer?

Step 4: Pick a Metric & Success Criteria

Not all metrics are created equal. The right metrics help teams make better decisions, while vanity metrics can be misleading.

Types of Metrics

• Vanity Metrics: Numbers that may look good but don’t indicate real progress (e.g., page views, social media likes).
• Actionable Metrics: Tied to user behavior and decision-making (e.g., conversion rates, retention, purchase intent).

How to Measure Experiment Success

• Define a Target Outcome: Example: “If at least 30% of users sign up within 7 days, we consider this validated.”
• Set a Threshold for Change: If the metric is below a certain level, reconsider the solution. 
• Illustrate Trends vs. Absolutes: Track how engagement changes over time.

Step 5: Run the Experiment & Measure Results

After launching the experiment, analyze the data to determine if your hypothesis was correct. If validated, proceed with confidence. If not, iterate and test again.

Good Test Design

• Clear Hypothesis: What do you expect to happen?
• Measure Outcomes: Use both qualitative and quantitative data.
• Limit Bias: Ensure results are valid and reliable.
• Right-Sized Experiments: Balance statistical significance with practical feasibility.

Why This Matters in Social Impact Work

To succeed, validate ideas through experimentation. Learn to align design decisions with business goals, master product roadmaps, and position yourself as a strategic leader.

A well-intentioned but untested idea can waste resources or—even worse—cause unintended harm. Testing your riskiest assumption ensures your solution is truly effective, community-driven, and sustainable before rolling it out at scale.

Experimentation is the backbone of innovation. Before you invest heavily in your social innovation, ask yourself: What’s the one thing that could make or break this idea? Then, experiment to find out.