Data and Testing Approach

Principles

• All of us apply bias to the numbers. Making data-driven decisions means fighting the urge to make exceptions to the data in front of you.

• Start with documentation and a 6 week period for new tests. Only apply one test per page.

• Always close and complete the test after the period ends. Analyze the data to determine whether to Roll out, Roll back, or re-test (inconclusive).

• Have a threshold for ranking or traffic that the page must achieve through regular improvements before becoming eligible for a test.

• Observe and collect data outside of official experiments so you can validate larger batches in testing.

In other words:

You don’t need to test much unless you’re on the first or second page

Optimizing means you already have traction, enough data, clicks, impressions, and real context to experiment.

Until you’re there, your priority is something else: improving content, authority, intent, structure, or whatever your page needs to reach the point where a test actually makes sense. Any experiment outside that range is like trying to fine tune an instrument that doesn’t have strings yet.

All experiments must be time-boxed, documented, and have clear success criteria

SEO, CRO, UX, and content experiments can live in one documentation area. But the key is that they all get documentation, including:

• A defined time period (I use 6 weeks)

• What is being tested and why

• Criteria for success, what defines failure

• What is considered inconclusive

• Metrics to monitor

• Previous context

• Hypothesis

These elements can turn a gut feeling into something that can be replicated or scaled.

When the time box ends: roll out or roll back

Make sure that you check in to see where you are in terms of having enough data to make a confident choice; aka statistical significance.

Mark the experiment:

Roll out: it worked, it gets implemented officially.

Roll back: it didn’t work, the changes are reverted.

Inconclusive: If is considered not reliable enough to make a business decision on. It gets documented and tested again later with a better-defined experiment.

Our decisions are only as good as our data and our ability to see our own bias

Without structure, testing lacks the rigor to be reliable. And without clear data, we tend to make decisions based on how we’re feeling at any given moment. These principles prevent chaos and maintain something fundamental; your pursuit of making decisions with integrity.