In Market Research, Marketing

I was at a networking event recently talking to a self-professed online marketing expert, who told me a story about a recent “success” with A/B testing on a client’s website.  I’m not going to reveal any of the parties involved, so I’ve changed the product, but not by enough to change the meaning.

The basis of his claim was “We A/B tested copy to see what worked best – and when we tried two different banners on the homepage, one got nearly twice as many clicks than the other, and both got more than the original copy.”

He then told me that the two banners were something like:

  • “Protect upholstery from children’s spills”
  • “Protect your upholstery from pet’s little accidents”

His results are doubtless interesting, but his interpretation of them seems to miss out an important question about audience and segmentation.  A/B testing is intended to look at subtle differences in copy and design, but here the difference is huge.  In fact, the two pieces of copy target completely different segments – one is pet owners, the other is parents.  It is good that the number of clicks went up with either version, but I suspect that the difference is more down to the ratio of parents to pet owners in the audience than anything else.

What the client really needs is something that appeals to parents AND pet owners, or the ability to target different messages to those different groups.

I challenged the self-professed expert on this point, and he replied, “You are entitled to your opinion, but I’ve done the test the numbers don’t lie”.

As A/B testing gets easier and gains popularity, I see more and more articles and examples of A/B tested copy where the different versions target different markets, segments, preferences or buyers motivations.  In these cases, the results of A/B testing may tell you which is the largest group in your audience, but tends to yield a result that targets that one group at the expense of all others.

There are lots of ways around this – a few of the more obvious are

  • Inclusive copy with messages that appeal across segments
  • Copy that directs different segments to a page specifically for them
  • Different pages for different segments (with different advertising, keywords etc to attract different audiences)

A/B testing is a hugely powerful technique when used properly, but as with most research techniques it is open to hideous misunderstanding and misinterpretation.

Although the context for A/B testing is online, I found this a useful reminder that when thinking about any marketing material I need to consider how to address different segments and buying motivations.  Do different segments need different copy, images and information? Will I create different material for different segments or make it generic?  It’s just as important in the offline world as online!

Leave a Comment