The Great CTR Debate

I often have conversations with clients about what their key performance indicators should be for their campaigns.  Quite a few times, clients want to use the CTR as the main proxy for performance and I squirm.  While I can talk for hours about why this is not a good idea 99% of the time, I wanted to hear the voice of the industry talk about it.

I asked a few friends to write a a few sentences about the famous “click thru rate” and if it should die or not.

While this is August 2010, the debate has been around for the past ten years.  I uncovered an article on CNET from 2001 entitled “Is the click-through history?

The Click Thru Rate Debate

I’m sure you have opinions and would love to hear them.  Please leave a comment on the blog or use the hashtag #deathofctr on twitter and tweet me at @dherman76

  • http://www.optimizeandprophesize.com/ jonathanmendez

    I was starting to comment here and it became a full blown post of my own. Great debate (and one I always love participating in – just ask Gian Fulgoni. :)

    My “comment” http://bit.ly/bdDk9b

    • http://www.darrenherman.com dherman76

      Jonathan – thanks so much for stopping bye. Love how you extended the post and just left you a juicy comment.

    • http://twitter.com/eranshir eran shir

      I’m doing the other way around :) Here’s a post I wrote after last year’s Monaco Media Forum talking about the death of the CTR: http://shir.posterous.com/insights-from-the-monaco-media-forum

      One interesting metric that we’re successfully promoting is the “effective click” – a visit to the advertiser’s site within 5-10min of viewing the ad. Maintains causality while dealing with the fact that people don’t click on ads.

      In fact, for savvy DR guys, even conversions and CPA is not a good enough metric. Rather you want to measure and optimize towards ROI and better yet, LTV of the customer..

      • http://www.optimizeandprophesize.com/ jonathanmendez

        Eran – I really like the effective click metric. Curious what the average latency is for those? And you’re right – LTV is the single most important metric (and justly the toughest to optimize for).

        The problem is that CTR was at any point *the* metric to optimize for. It never should have been (or be). All that showed was the immaturity of digital marketers. I think it’s great that we’re rising above that simple thinking to more advanced metrics and more advanced delivery platforms. The rise of A/B testing tools and the continued mindshare around web analytics are clear sign of the maturation. But killing CTR?

        Digital marketing will always be balance of volume and cost per. CTR is the best single metric we have to optimize volume so please, let’s not kill it – especially in display where there are more than enough click volume issues to keep everyone optimizing for some time to come.

  • http://www.optimizeandprophesize.com/ jonathanmendez

    I was starting to comment here and it became a full blown post of my own. Great debate (and one I always love participating in – just ask Gian Fulgoni. :)

    My “comment” http://bit.ly/bdDk9b

  • Martin Kelly

    Hmmm, my initial, gut reaction is that I think CTR’s are ridiculous as a measure of ROI for an advertiser and when I’m asked to report on this metric or it’s commented on I roll my eyes, discretely of course. But then when I think about it, I also know that I find the CTR’s of the campaigns that I run of interest to me, the differences between advertisers within a vertical and between different verticals. I also think that the CPC remarketer will lose sleep over a campaigns CTR’s as their margin directly relates to the clicks they generate. So CTR as a measure that I would report to a client or feel is relevant to them, never. But CTR as a metric that I find interesting as a component of a campaign, absolutely.

    • http://www.darrenherman.com dherman76

      Martin – thanks for commenting. I do like using CTR for internal purposes as you mentioned but that’s about it

  • Martin Kelly

    Hmmm, my initial, gut reaction is that I think CTR's are ridiculous as a measure of ROI for an advertiser and when I'm asked to report on this metric or it's commented on I roll my eyes, discretely of course. But then when I think about it, I also know that I find the CTR's of the campaigns that I run of interest to me, the differences between advertisers within a vertical and between different verticals. I also think that the CPC remarketer will lose sleep over a campaigns CTR's as their margin directly relates to the clicks they generate. So CTR as a measure that I would report to a client or feel is relevant to them, never. But CTR as a metric that I find interesting as a component of a campaign, absolutely.

  • http://www.darrenherman.com dherman76

    Jonathan – thanks so much for stopping bye. Love how you extended the post and just left you a juicy comment.

  • http://www.darrenherman.com dherman76

    Martin – thanks for commenting. I do like using CTR for internal purposes as you mentioned but that's about it

  • Eric Brown

    A similar article came out on Mobile Marketer a few weeks ago…

    http://www.mobilemarketer.com/cms/opinion/columns/7044.html

  • Eric Brown

    A similar article came out on Mobile Marketer a few weeks ago…

    http://www.mobilemarketer.com/cms/opinion/colum

  • http://www.adbuyer.com Tim Ogilvie

    Interesting debate. I’d add a few thoughts in the pro-CTR camp:

    1) CTR is the “cheapest signal” for measuring performance. You need significantly more data to measure brand impact, conversion, etc. So unless the client has a HUGE budget, you can usually get better results by optimizing first to CTR, then to the secondary metric, and finally circling back to back-test some of the early losers.

    2) CTR is highly correlated with online performance metrics. Specifically, we’ve seen ROAS and CPA metrics (both view-through and click-through based) are better from sites with high CTRs. The big issue here is that it leaves you vulnerable to click-fraud, so protection and a secondary metric are mandatory, as is a universal ban for anyone gaming the system.

    3) I haven’t seen anyone show that CTR is negatively correlated with any other performance metric. Put a different way, while CTR may not be a good metric to use, nobody I’ve seen has _shown_ that it is actually a bad one. Natural-born-clickers often turn out to be natural-born-buyers.

    • Guest

      My thoughts on 2 of your points below. I agree that CTR is a useful metric for advertisers whose sales chains end online, no doubt. But for advertisers who rely on their revenue streams to come from brick and mortar retail store sales, CTR is an irrelevant, and potentially harmful, metric to base your campaign decisions on.

      1) Certainly contingent on how “performance” is defined.
      3) Nielsen has proven that higher CTR has a slightly negative (-0.07) correlation with offline retail sales ROI.

      • http://www.adbuyer.com Tim Ogilvie

        Do you have a link to the Neilsen study? I’d argue that -0.07 is uncorrelated unless the data set is really really large, but still would be very interested.

        On the contingency to performance, I don’t think so. Key issue is the signal / impressions ratio. If there’s a “good” performance metric that can be measured and has a higher frequency than ctr, I’m probably just not thinking hard enough. This is really just a math question.

        • http://www.adbuyer.com Tim Ogilvie

          Answering my own question. If it is this study:

          http://blog.nielsen.com/nielsenwire/consumer/is-the-ad-biz-through-with-click-through/

          It’s a strawman argument. What they are saying is that looking at the Gap campaign versus an Apple campaign, there’s no correlation between the CTR and whether that campaign was successful. No kidding.

          The key issue that hasn’t been addressed is whether, for an individual advertiser, are the high-CTR rates of the placements within that campaign associated with higher offline sales? I have a strong suspicion that people have run these studies and found that CTR is, in fact, correlated with higher offline sales.

          • Guest

            The data is for more than 300 advertisers. The results are for CPG products. If you’re CPG brand that needs to move undreds of thousands of cases in store, why hang your hat on a metric that represents less than a percent of the audience (and off your demo target?)

            As I said before, CTR has utility for advertisers whose sales chains end (or can end) online. You pointed out Gap and Apple … you can buy their products online. No one buys toothpaste, macaroni and cheese, or toilet paper on the internet.

  • Niraj Nagpal

    I 100% agree that the click is an archaic metric. If marketers and brands can shift the conversation to ROI, time spent on site and brand lift, the entire industry will be winners.

  • Togilvie

    Interesting debate. I'd add a few thoughts in the pro-CTR camp:

    1) CTR is the “cheapest signal” for measuring performance. You need significantly more data to measure brand impact, conversion, etc. So unless the client has a HUGE budget, you can usually get better results by optimizing first to CTR, then to the secondary metric, and finally circling back to back-test some of the early losers.

    2) CTR is highly correlated with online performance metrics. Specifically, we've seen ROAS and CPA metrics (both view-through and click-through based) are better from sites with high CTRs. The big issue here is that it leaves you vulnerable to click-fraud, so protection and a secondary metric are mandatory, as is a universal ban for anyone gaming the system.

    3) I haven't seen anyone show that CTR is negatively correlated with any other performance metric. Put a different way, while CTR may not be a good metric to use, nobody I've seen has _shown_ that it is actually a bad one. Natural-born-clickers often turn out to be natural-born-buyers.

  • Niraj Nagpal

    I 100% agree that the click is an archaic metric. If marketers and brands can shift the conversation to ROI, time spent on site and brand lift, the entire industry will be winners.

  • Guest

    My thoughts on 2 of your points below. I agree that CTR is a useful metric for advertisers whose sales chains end online, no doubt. But for advertisers who rely on their revenue streams to come from brick and mortar retail store sales, CTR is an irrelevant, and potentially harmful, metric to base your campaign decisions on.

    1) Certainly contingent on how “performance” is defined.
    3) Nielsen has proven that higher CTR has a slightly negative (-0.07) correlation with offline retail sales ROI.

  • Togilvie

    Do you have a link to the Neilsen study? I'd argue that -0.07 is uncorrelated unless the data set is really really large, but still would be very interested.

    On the contingency to performance, I don't think so. Key issue is the signal / impressions ratio. If there's a “good” performance metric that can be measured and has a higher frequency than ctr, I'm probably just not thinking hard enough. This is really just a math question.

  • Togilvie

    Answering my own question. If it is this study:

    http://blog.nielsen.com/nielsenwire/consumer/is

    It's a strawman argument. What they are saying is that looking at the Gap campaign versus an Apple campaign, there's no correlation between the CTR and whether that campaign was successful. No kidding.

    The key issue that hasn't been addressed is whether, for an individual advertiser, are the high-CTR rates of the placements within that campaign associated with higher offline sales? I have a strong suspicion that people have run these studies and found that CTR is, in fact, correlated with higher offline sales.

  • Russell Glass

    Darren, great topic. Bizo just finished a study of every one of our FH 2010 campaigns,and optimizing to the click was effectively wrong over 75% of the time, which basically means that you’re better off optimizing with a coin-flip than the click. We’ll be publishing the study in the next few weeks with the details…

    To the point that “CTR is correlated with success”, maybe this was true at one point, but of our top performing 25 sites from a CTR perspective in 2010, only SIX have been in the top 25 from a conversion perspective.

    Simply, optimizing to click is just dead wrong.

    • http://www.darrenherman.com dherman76

      Russell, would love to get ahold of that study ahead of the market…

      • Russell Glass

        sure hit me with an email and i’ll send you the article that we’re going to publish.

  • Russell Glass

    Darren, great topic. Bizo just finished a study of every one of our FH 2010 campaigns,and optimizing to the click was effectively wrong over 75% of the time, which basically means that you're better off optimizing with a coin-flip than the click. We'll be publishing the study in the next few weeks with the details…

    To the point that “CTR is correlated with success”, maybe this was true at one point, but of our top performing 25 sites from a CTR perspective in 2010, only SIX have been in the top 25 from a conversion perspective.

    Simply, optimizing to click is just dead wrong.

  • http://www.darrenherman.com dherman76

    Russell, would love to get ahold of that study ahead of the market…

  • http://www.darrenherman.com dherman76

    Russell, would love to get ahold of that study ahead of the market…

  • Russell Glass

    sure hit me with an email and i'll send you the article that we're going to publish.

  • Guest

    The data is for more than 300 advertisers. The results are for CPG products. If you're CPG brand that needs to move undreds of thousands of cases in store, why hang your hat on a metric that represents less than a percent of the audience (and off your demo target?)

    As I said before, CTR has utility for advertisers whose sales chains end (or can end) online. You pointed out Gap and Apple … you can buy their products online. No one buys toothpaste, macaroni and cheese, or toilet paper on the internet.

  • http://twitter.com/kirbywinfield Kirby Winfield

    With display CTR’s running far below 1% for the past decade, we do seem to spend a lot of time measuring, optimizing for, and planning around clicks. What about the 10%+ impressions that create an engagement action? This type of inefficiency creates a huge opportunity for buyers and sellers willing to dig in and value media on more relevant metrics. To wit: If I am buying RON display for $1CPM, and garnering 5% engagement with my ad, and I decide an engagement is worth $0.20 to me, I am buying inventory worth $10CPM at a cost of $1. I will do that all day long. There’s a ton of unvalued (not just Undervalued – Unvalued) inventory out there hidden in display buys – will publishers or buyers get smart first and start using data to take advantage?

  • http://twitter.com/kirbywinfield Kirby Winfield

    With display CTR's running far below 1% for the past decade, we do seem to spend a lot of time measuring, optimizing for, and planning around clicks. What about the 10%+ impressions that create an engagement action? This type of inefficiency creates a huge opportunity for buyers and sellers willing to dig in and value media on more relevant metrics. To wit: If I am buying RON display for $1CPM, and garnering 5% engagement with my ad, and I decide an engagement is worth $0.20 to me, I am buying inventory worth $10CPM at a cost of $1. I will do that all day long. There's a ton of unvalued (not just Undervalued – Unvalued) inventory out there hidden in display buys – will publishers or buyers get smart first and start using data to take advantage?

  • http://twitter.com/eranshir eran shir

    I'm doing the other way around :) Here's a post I wrote after last year's Monaco Media Forum talking about the death of the CTR: http://shir.posterous.com/insights-from-the-mon

    One interesting metric that we're successfully promoting is the “effective click” – a visit to the advertiser's site within 5-10min of viewing the ad. Maintains causality while dealing with the fact that people don't click on ads.

    In fact, for savvy DR guys, even conversions and CPA is not a good enough metric. Rather you want to measure and optimize towards ROI and better yet, LTV of the customer..

  • http://www.optimizeandprophesize.com/ jonathanmendez

    Eran – I really like the effective click metric. Curious what the average latency is for those? And you're right – LTV is the single most important metric (and justly the toughest to optimize for).

    The problem is that CTR was at any point *the* metric to optimize for. It never should have been (or be). All that showed was the immaturity of digital marketers. I think it's great that we're rising above that simple thinking to more advanced metrics and more advanced delivery platforms. The rise of A/B testing tools and the continued mindshare around web analytics are clear sign of the maturation. But killing CTR?

    Digital marketing will always be balance of volume and cost per. CTR is the best single metric we have to optimize volume so please, let's not kill it – especially in display where there are more than enough click volume issues to keep everyone optimizing for some time to come.

  • Justin Terry

    This is a great discussion and it’s one I seem to have quite often with clients and account teams. One aspect of CTR that I didn’t see as part of the discussion here was the utility of using CTR solely as a creative assessment metric.

    From a search perspective, if we first optimize against keywords that have been identified as high-converting keywords (based on CPA, ROI, LTV, etc), we can then evaluate creative for those keywords based on CTR. If the conversion rate remains constant, a higher CTR would mean more conversions. On the display side, the same can be said for optimizing against placements that generate the highest number of conversions and testing creative.

    CTR isn’t the bad guy here, it’s the use of CTR as a “pure” success metric by some marketers. CTR can accurately measure whether a message resonates with a particular audience. CTR has its uses. For search, it’s for optimization of creative for conversion-driving keywords (not to mention rank-based bid strategies). For Display, it’s for optimization of creative on conversion-driving placements.

  • Justin Terry

    This is a great discussion and it's one I seem to have quite often with clients and account teams. One aspect of CTR that I didn’t see as part of the discussion here was the utility of using CTR solely as a creative assessment metric.

    From a search perspective, if we first optimize against keywords that have been identified as high-converting keywords (based on CPA, ROI, LTV, etc), we can then evaluate creative for those keywords based on CTR. If the conversion rate remains constant, a higher CTR would mean more conversions. On the display side, the same can be said for optimizing against placements that generate the highest number of conversions and testing creative.

    CTR isn’t the bad guy here, it’s the use of CTR as a “pure” success metric by some marketers. CTR can accurately measure whether a message resonates with a particular audience. CTR has its uses. For search, it’s for optimization of creative for conversion-driving keywords (not to mention rank-based bid strategies). For Display, it’s for optimization of creative on conversion-driving placements.

  • Pingback: Marketing Wednesdays: Display Impressions | Darren Herman