A/B Testing Is Alive and Well

Share with:

A/B testing is the bedrock of a good PPC campaign. It’s so important that I’ve written about it on this blog 46 times. Just last week, I wrote a review of AdAlysis, an A/B testing and multivariate testing tool. And just 2 years ago, I asked, can too many ads ruin PPC ad copy testing?

Spoiler: The answer is yes. Testing too many ads at once creates a myriad of issues, including taking forever to reach statistical significance in all but the highest-volume PPC accounts.

And yet, in their infinite wisdom, Google is now recommending that advertisers forgo A/B testing, and instead run at least 3 ads per ad group. In fact, Google representative Matt Lawson, in an article for Search Engine Land, this week went so far as to claim that using more than 2 ads per ad group is a “foolproof step to excellent Adwords ads.” In the article, he says, “I think the A/B approach to message testing is becoming outdated.”


I think what he means is: “At Google, we’d really rather decide what ads are performing best for you. We want you to use the ‘optimize’ ad rotation settings and let us choose which ad to serve.”

That’s right. Google is telling us to forget ad copy testing and just let Google pick the winners.

To a novice PPC advertiser, I’m sure this is music to the ears. Small business owners and in-house marketers who are dipping their toes into Adwords management are probably thrilled to hear that they don’t have to worry about A/B testing ad copy. They can just throw a few random ads into their account, and let Google pick the winner.


How many successful business owners do you know who let their vendors tell them what products to stock in their stores? When pharmaceutical companies started paying big bucks to get doctors to prescribe their medication over others, the public lost its collective mind. “How dare they buy off the doctors?” If you walked into a clothing retailer who claims to carry multiple brands, and only found Calvin Klein, wouldn’t you wonder about the store owner’s sanity?

It’s called putting all your eggs in one basket. It’s not smart business. And it’s not smart advertising.

I get it. PPC is complicated. And hiring a professional PPC manager is expensive. That’s why many novice business owners and in-house marketers try to tackle PPC on their own. But it’s too complex. PPC is not something you can DIY. You wouldn’t try to fill a cavity yourself. Or replace the roof on your house. Or do your own business taxes. Or elect a president who stands to personally line his pockets using the office. (Wait, did I say that out loud?). The point is, you shouldn’t try to do PPC alone. Years ago, you could. Today, it’s just too complicated. And Google is out there trying to get you to turn the whole thing over to them.

Don’t fall for it. Hire a professional. Use A/B testing liberally. Make your own business decisions.

Julie Friedman Bacchini wrote a great post breaking down the fallacy of the Google article on SEL – go give it a read. And let me know what you think about the Google article. Do you see any merit in it? Are we really heading toward a world where we just let Google make all the decisions? Share in the comments!

Related Posts:


  1. Google charges money only when someone clicks the ad. So the more accurate retail industry analogy would be where vendors tell stores which products to stock against the commitment that they will take money only when the product leaves the shelf i.e. it’s sold by the store. It so happens that this is a fairly common practice in the retail industry. It’s called “consignment basis”. Many retailers are used to this practice and will find merit in Google’s current offer. TBH, it’s too late to worry about putting all eggs in one Google’s basket. That ship sailed long ago. With or without this offer, what great choice did I have other than Google anyway for digital advertising in general and PPC in particular?

    • Melissa Mackey says

      But Google wants us to bet on them making the right decision, rather than let us make our own decision. So this would be like clothing manufacturers forcing stores to take clothing that may or may not sell, on consignment.

      I wholeheartedly disagree that Google is the only choice, at least in the US. We have clients for whom we don’t use Google at all, because it’s cost-prohibitive or doesn’t meet the client’s objective. Assuming that there are no other choices out there is common, but untrue. Bing, Facebook, and other social engines work great, as do several programmatic engines. No one should feel that Google is their only choice.

  2. Great article.
    I agree that A/B/C+ testing can slow down our ability to learn from ad performance. This brings up a question for me. To my knowledge AdWords is not a true testing platform: the setting “rotate evenly” does not imply “rotate randomly,” and AdWords retains the right to choose which ad it serves based on perceived profitability based on early data and other factors. Is a winning variation “statistically significant” if the segmentation / grouping was based on choice rather than random chance?

  3. There is an issue with data quantity too. High volume ads generate data quickly, obviously, so A/B/C might work for them (for the record I think it is a redundant idea) But most ads in most campaigns for most advertisers do not generate sufficient traffic to get any usable data in any practical timeframe. So A/B/C dilutes that data even more, stretching the usability unrealistically.

  4. I lobbied our Google reps for years to give us the option to optimize ads based on “Conversions” and not just “Clicks”. I was very happy when the day came that the rolled that out, but discovered via testing that it didn’t work very well. Clicks play a factor in the way it works as far as I can tell, and that muddies the water. We still make use of unlimited equal ad rotation and manually decide what ads stay and which go. Sometimes good converting ads get low CTR. Sometime high CTR ads don’t convert well. And sometimes ads with good CTR & Conv. result in a lot of trash leads.

    I think it would be great if Google could free us from much of the testing. Moving from A/B to multi variate testing would be awesome. Especially if you can add in things like device type, GEO location, TOD, etc. that are all very difficult or time-consuming to manage. But trusting Google to do things in our best interests would be foolish. Until they can clearly forgo profits in favor of providing more client value I will keep them at arm’s length which is about how far I can throw them.

  5. I see that I had already commented on this subject, but I just tried the feature to “optimize for conversions” again and it confirms what I had seen before.

    It may work well if you have an adgroup that enjoys a large number of conversions, but if not I think you will see that “optimize for conversions” really does not work as expected. In my case the adgroup was not getting many conversions and the ads getting the most impressions were ones with higher CTR. Part of this effect could be that the keyword activated with the ads selected by Google to show to users were considered to be more “relevant”. But it seems little consideration is made for ads with conversions over those with higher CTR. To me, it appears that Google has provided this option more to appease all of us that have asked for the option, rather than something we can rely on to perform as we expect. We will continue to rotate ads indefinitely and optimize manually.


  1. […] A/B Testing Is Alive and Well, Beyond the Paid […]

Speak Your Mind


This site uses Akismet to reduce spam. Learn how your comment data is processed.