September 08, 2018

By Steve Latham

The promise of digital marketing is very exciting: delivering the right message to the right person in the right place and at the right time. In marketers' quest for this holy grail, much attention and budget has been focused on the media side of the equation (i.e., the person, time, and place).

What's the true measure of an ad?

While marketers universally agree that advertising creative matters, the measurement of an ad's effectiveness hasn't received the same level of attention or resources as programmatic buying and audience targeting. Almost every advertiser still relies on arbitrary metrics for measuring and optimizing an ad's performance.

It's important for marketers to remember that the purpose of measurement is to guide optimization for maximum ROI. It's also important to remember that the investment required (i.e., the time, energy, and expense) should be relative to the size of the prize.

In this context, a few key questions need to be answered. Namely, what is the impact of a good ad versus a bad one, and what metrics can marketers use to measure an ad's performance?

Flawed Performance Metrics

Renowned business and management wonk Peter Drucker, paraphrasing the 19th century physicist Lord Kelvin, popularized the maxim "what gets measured gets managed." When looking inward at the advertising industry, there is still much work to do. This is especially true in how the industry measures ads.
A New Policy from Google Has Major Implications for Marketers
Specifically, the venerable click-through rate (CTR) metric, which was introduced the same year Forrest Gump won the box office in 1994, might have made sense when display CTRs were 8 to 10 percent, but those days are long gone. The objective of display and video advertising is to create brand awareness and consideration. With few exceptions, display advertising is not a direct response vehicle and, therefore, should not be measured by clicks. But due to its ubiquitous nature and the lack of alternatives, the CTR has been the default creative performance metric for 24 years, despite marketers' reservations about its accuracy in measuring effectiveness.

CTR has long been suspected of being a flawed metric — there's now data that proves it is.

Analyzing more than 10 billion impressions for 24 consumer brands across several industries, Flashtalking used machine-learning attribution models to measure the contribution of each ad creative based on its propensity to influence online conversions. The models scored each concept or version using a creative performance index (CPI) to quantify the attributed impact of each creative. It then ranked and segmented the creative pieces into quartiles based on CPIs for each brand. Several key insights emerged from the study, which will be published in a forthcoming whitepaper.

1. The gap between best and worst is huge.

Creative performance varies — some concepts, versions, and offers resonate with customers more than others. While this is generally accepted, few realize the breadth of the gap between best and worst. That gap is really more of a chasm.

Across all advertisers, the average CPI for the top quartile of ad creatives was 2.27, which was 7.9 times higher than the 0.29 score for the bottom quartile creative pieces. In other words, the top 25 percent of creative content in each campaign outperformed the bottom 25 percent by 700 percent. (See figure 1.)

The Gap Between Best and Worst Creative Pieces Is Typically Large
The average CPI for top and bottom quartiles by client

Such high variances in creative performance beg the obvious question: If creative piece A outperforms creative piece B by 700 percent, why would marketers continue to serve B? The answer is that they wouldn't — if they were measuring each creative unit properly.

2. Current metrics do not accurately identify good versus bad creative.

Data shows that CTRs are negatively correlated with performance, as measured by CPI. The worst creative pieces (i.e., those in the bottom quartile) have an average CTR of 0.017 percent. That is 2.4 times higher than the best creative pieces (i.e., those in the top quartile), which have an average CTR of just 0.007 percent. (See figure 2.) There are occasions in which CTR and CPI are correlated, but it's not the norm. In those cases, the difference in CTR is much less pronounced.

Click-Through Rates Are Not a Good Indication of an Ad's Effectiveness
Click-through rates for top and bottom quartiles
source: 2018 Flashtalking internal study

Hopefully this comes as no surprise to most marketers. It's long been argued that display ads can be very effective in creating awareness and/or consideration, even with a zero percent CTR. Now there is data that proves it.

Evolving How We Measure Ad Creative

To realize the true promise of digital marketing, brands must rethink their approach to how they measure media and advertising creative. While media measurement (i.e., multitouch attribution, marketing mix modeling) has consumed most of the spotlight to date, it's time for advertisers to get serious about measuring and optimizing the effectiveness of ad creative as well.

Changing directions is difficult, but the tide is turning as a growing number of forward-leaning brands are taking advantage of machine-learning models to measure ad creative performance in a more appropriate and responsible way.

However, it's important to remember that algorithms themselves are not the silver bullet. The quality of the insights is based first and foremost on the quality of the data and how well it represents the scenario marketers are seeking to understand. In the journey to become data-driven in their approach to creative measurement, here are some key issues for marketers to keep in mind:

  •     Metrics. Marketers must look beyond CTRs or funnel position to properly measure the performance of each creative piece in achieving its goal (e.g., creating awareness or consideration).
  •     Audience bias. If a user is already a fan and therefore predisposed to convert, they may skew the results. Conversely, if a user is loyal to a competitor, they may skew results the other way. If marketers are trying to determine which creative executions are most effective in attracting new customers, they have to account for user bias.
  •     Media bias. Bad placement can negate the affect of a great creative piece, just as good placement can make mediocre creative content look like a top performer. To isolate the impact of the creative component, marketers have to account for media bias.

How marketers measure their ad creative is important to their broader marketing strategies. Lacking the proper metrics for measuring creative, most brands are leaving money on the table (e.g., serving underperforming ads by optimizing on CTRs). The good news is that data-driven solutions now exist, and every brand has the ability to properly align its metrics with its strategy to become more proficient.

Steve Latham is the global head of analytics at Flashtalking. You can email him at


Leave a reply

Enter the characters shown in the image.