Testing can be used in all aspects of your business to improve performance and optimize your operations.  Almost any CEO or founder that I speak to will describe their organization as 'data driven', but most are not.  In fact, most organizations do not make full use of testing because they:

• Are convinced that they know the 'right' answer before they test
• Don't want to risk failure by suggesting a test that might not work out
• Set up the test without thinking about the end state first
• Have not developed the testing 'muscle' within their organization

A Fractional CMO can help many organizations embrace disciplined testing as a lever of growth by working through each area described in this post.

Embrace Failure

When it comes to testing, the only way to fail is to run a test that fails to provide any useful information.  An A/B split test in which the variation loses to the control is not a failed test.  It is simply an indication that the variation is no better than the existing state.  If the test is structured properly, this provides insight into directions you can take to improve performance in the future.

When I ran a marketing organization years ago, I would regularly tell my team that we had the best job in the world - we only had to be right 1 in 10, or even 1 in 20 times.  In any other profession we would be fired for being wrong 90%-95% of the time, but in growth marketing, if 1 in 10 or 1 in 20 tests is successful, you are a rock star.

Start With the End in Mind

When you plan out a test you should start at the end.  Ask yourself the following question:

"Once this test is complete how will I know if the item I am testing introduced a beneficial change?"

By thinking about the end first you can structure the test to give you the best chance of seeing a reaction or change, assuming that the item being tested does actually impact the result you are seeking.  

For example, let's say you are running a simple website A/B test on two home pages.  You want to see which home page leads to more demo/trial signups.  Your baseline/control data is:

       Average Home Page Visitors per week - 10,000

       Average Trial/Demo Signups per week - 500

The first thing you need to think about is how much of a difference you want to be able to detect.  It will take many more samples to detect a 1% difference than it will to detect a 20% difference because it will be much harder to tell if the new variation is actually different from the existing.  The table below shows the traffic you would need to detect the difference shown.  We used the calculator located on the Optimzely web site to generate this table.  

A key finding from this data is that most websites need to focus on large potential changes because they do not have the traffic to detect smaller ones.  This is extremely important.  You will not be able to tell if a small change had a positive effect on your site, so don't spend your time testing font colors or images - think about the factors that really drive behavior and focus on them instead.  

Test Concepts - Don't Debate Ideas

Everyone has opinions on what will drive beneficial change to your site.  That's great!  Test as many of them as you can and don't spend time debating which one may work and which one may not.  Just make the tests quick and aim to fail (or succeed) quickly.  Think about how you can test ideas without building everything out first - what is the smallest effort that will yield insight into the behavior you are trying to learn about.

Make Testing a Common Occurrence

When I ran a marketing organization, we each had to come up with 3-4 tests every quarter.  This was not optional.  We saw our site traffic as a scarce commodity and we needed to make the most of it to learn all we could about our customers.  Almost every test failed to improve performance but over 3 or 4 years, three tests produced significant gains and these changes allowed us to lower our CPA by more than 50% while simultaneously increasing our marketing spend.   This led to phenomenal, profitable and consistent growth.

Summary

Testing is one of the most important tools you have when it comes to growing your business.  Don't let the fear of a test failing keep you from trying new things. If you knew what to do from the beginning, testing wouldn't be necessary.  The testing muscle takes time to develop, but over time your company will certainly benefit from the effort.

Mike Morris is the founder of Kettle Hole Partners. When he is not trying to figure out mathematical models for marketing and customer acquisition he is probably riding his bike.