NEW YORK – A few weeks back, I asked whether online ads do any good at all. I cited a paper that had done experiments on the effectiveness of Google ads. Jim Manzi, who runs a whole lot of experiments for companies, writes to offer his perspective:
“Megan McArdle laments that online journalism may be going the way of print journalism. In doing so, she cites a research paper for a series of randomized field experiments at eBay showing that a set of paid search buys had causal effects on sales that were a small fraction of those estimated by nonexperimental methods, and that in certain cases had no measurable effect on sales at all.
“I am always heartened to see controlled experiments take some of the air out of claims derived from nonexperimental methods. The eBay experiments seem to have been well-designed and analyzed, but I don’t think the implications of this are as broad as it might seem.
“Imagine measuring the return on investment in television advertising in 1990 by comparing total revenue achieved through the TV. … Because no material commerce was transacted via your TV, you would always have zero incremental revenue, and therefore terrible ROI. Of course, you would never do this calculation, as it would be clear that TV acted as a pure advertising medium. The transaction media where the causal benefits of the ad showed up were stores, call centers, direct sales forces and so forth.
“Online commerce is both an advertising medium and a transaction medium. For some kinds of businesses (like eBay), it is reasonable to assume that all of the incremental revenue caused by an online ad (such as a search heavy-up, a high concentration of ads in a given time frame) will be captured in the online channel. But for many large advertisers, the online ad will also cause sales effects offline.
“What is your guess for e-commerce as a proportion of U.S. retail sales? According to the Census Bureau, it’s about 6 percent. That is, 94 percent of retail commerce occurs offline. These multichannel effects are a huge deal, because very few of the largest advertisers in America transact purely, or even predominantly, online.
“In partnership with Google, my company Applied Predictive Technologies designed and analyzed 15 randomized field experiments between 2008 and 2011 to test the causal impact of search ad buys for multichannel marketers. These were very similar to the eBay experiments: A randomized set of geographic markets was exposed to higher or lower levels of online search ads, then compared with a matched set of control markets to evaluate impact. These 15 experiments evaluated sales impacts on all sales across both online and offline channels.
“The data were released to an academic who conducted a meta-analysis of them. The results are eye-opening. Twelve of these 15 experiments demonstrated statistically significant causal impacts on offline sales. Unless you think that markets somehow have failed and the enormous growth in online advertising spending over more than a decade has been a product of some kind of mass delusion, this makes sense.
“Online advertising is not the always-and-everywhere sure thing that publishers can sometimes claim, but is often very economically effective — when you carefully measure effects across all channels.”
Megan McArdle (email@example.com) is a Bloomberg View columnist.