Facebook’s new relevancy metrics

After years of even Facebook themselves saying that it’s not a good metric, they have finally made a move to replace Relevance Score in Ads Manager, and it’s not a moment too soon.

The reason I say this is that – to most of the digital specialists I know with a lot of experience in managing large Facebook budgets – the Relevance Score metric seems to correlate poorly with other more meaningful metrics, and is even harder to take action on. Even Facebook has acknowledged this, and their recommendation to advertisers has been to focus more on business-related metrics, such as ROI, instead of the increasingly ambiguous Relevance Score.

Throughout this post, we will break down and analyse Facebook’s own explanation of these new metrics – so just keep the above paragraph in mind for now.

These are the three new metrics to replace Relevance Score, quoted directly from Facebook’s Help Center:

• Quality ranking: How your ad’s perceived quality compared with ads competing for the same audience.
• Engagement rate ranking: How your ad’s expected engagement rate compared with ads competing for the same audience.
• Conversion rate ranking: How your ad’s expected conversion rate compared with ads that had the same optimisation goal and competed for the same audience.

In order to (at the very least attempt to) simplify this further, the metrics can only assume one of 3 different values: above average, average or below average. While it might make sense, it also makes it harder to analyze the metrics in a data-driven way and figure out how the Facebook system actually uses the metrics – but more on that later.

In order to more thoroughly understand how to interpret these values, we’ve been provided with a table (above). Going into detail on each of the suggested actions would require more time and space than this blog post allows, so instead, I’ll offer a short summary:

Basically, if your rating is above average or average – do nothing, it’s fine. If you are below average on one point, Facebook suggests some actions to take, and if two or more metrics are “below average” they essentially suggest that you should change things drastically – creatives, targeting… everything.

However, a complication arises after the detailed explanation of these new metrics:

When an ad is meeting your advertising objectives

If an ad is already meeting your advertising objectives, reviewing ad relevance diagnostics may not be necessary. Sometimes high-performing ads have below-average ad relevance diagnostics rankings and that’s OK. Optimise for your advertising objectives, not for quality ranking, engagement rate ranking or conversion rate ranking.

In a rather straightforward manner, Facebook is effectively telling us not to bother with these obscure relevance “metrics” at all, but instead to focus on business outcomes.

Note that after this entire exercise in mental gymnastics, we have gone full circle and returned to square one: don’t bother with any of these new metrics; look at the ones that are important to your business objectives instead.

What’s with the obscurity by design?

It is interesting that Facebook is struggling so much with coming up with a decent metric for how “relevant” your ad is given that the entirety of the Facebook ad platform is built on its unique ability to target an audience with relevant content, so much so that it looks as if Facebook is trying to obscure their own system rather than making it more transparent. In order to explore this line of thinking, we will have to understand the basics of how the ad platform operates.

Without going into too much detail about how Facebook’s ad delivery and auction operates, we need to cover the very basics: every time a user logs on to Facebook, an auction takes place between all of the advertisers bidding to show that user an ad. The auction is a complicated process in itself, but like any other auction, in order to declare a winner, you have to know what the highest bid is. There are two groups of variables Facebook considers when calculating the winning bid: advertiser value (value generated for the advertiser) and user value (value generated for the user seeing the ad). It’s the latter that is of interest here.

It’s relatively straightforward to quantify advertiser value, but quantifying user value is difficult in comparison – and Facebook manages to do it extremely well.

Here is where things get muddy: how do we actually know that Facebook really is good at quantifying user value? Well, because their systems work. They even work so well that some people are certain they must be listening in on our conversations (in order to save you a click, I can tell you right now that they aren’t).

So if we know that Facebook is great at estimating user value, then calculating a solid relevancy metric should be quite easy. After all, they are already doing it for every auction – so why are we being shown these unnecessarily obscure ordinal-level metrics?

Here is what Facebook themselves have to say:

Improving how we report relevance

Relevance score measures whether the ads that you ran were relevant to the audience that you reached. Rather than measure relevance in one metric, over the next few months, we will replace relevance score with three new, more granular ad relevance diagnostics metrics. Similar to relevance score, these ad relevance diagnostics are not factored into an ad’s performance in the auction. We think that this level of granularity will offer reporting that’s more actionable for businesses.

From the above paragraph, we can deduce the purposes for why this change is taking place: adding granularity and improving actionability of reporting. While that sounds great, there are a few problems with this statement:

• Actionability
  We already concluded that even Facebook is telling us to ignore these metrics unless your business metrics are also poor in which case you should focus on those instead.
• Granularity
  It is true that we now have 3 metrics instead of one, but since they only exist on a 3-level scale, there isn’t actually much – if any – granularity being added.

This is not to say that these new metrics will be worse than Relevance Score, but it’s hard to see why exactly Facebook chose to make what was a much-needed change in this particular manner.

Treading a delicate line

An alternative explanation as to why these metrics look this way might be found when considering the challenges Facebook is facing as a company right now. On the one hand, they must keep developing their ad platform, improving it over time and providing valuable data to advertisers in order to keep advertisers profitable (and happy). On the other hand, after the recent scandals, we know what kind of chaos and mayhem someone could unleash on the Facebook platform if given a specific advantage or some unique insights into how the systems work. Furthermore, everyone – from small companies to large corporations and even state-actors – is actively trying to break, exploit or abuse different Facebook systems in some manner in order to get an advantage over competitors, increase profitability or alter the outcome of an election.

In light of this, Facebook is treading a very delicate line between granting access to data that advertisers rely on in order to be able to maintain profitability and not giving away too many details around how their systems work. The more information & data available to outsiders, the more vulnerable the systems become.

This will likely continue to be a trend when it comes to different Facebook products – a trend in which data is being simplified, aggregated and dumbed-down in order to keep up appearances of “actionability” while actually guarding the walled garden of data that Facebook has become.