Vanity Check: “Keyword Intelligence Tools”

ducks-rubber

It’s not about the tools you have, it’s about how you interpret the data.

Most importantly, it’s how you evaluate what your data is really telling you.

Look at this chart:

sm

Now, ask yourself what you see. Say it out loud, or in your head, whatever.

I bet the first thing you said was:

“The rankings on this site dropped”.

That’s not correct.

If you said,

“The keyword rankings for this site have dropped according to Searchmetrics.”

You’d be perfectly right.

The problem with keyword intelligence

The very name implies someone else is doing the thinking for you. I think that’s the problem. Our industry loves a new tool – and I’m no exception. SEOgadget is a happy customer of Searchmetrics (and SEMrush) and we have been for a long time. I use the tools as useful starting points for keyword research and competitive analysis and sales calls!

But, a few weeks back, one of our clients got in touch, extremely concerned, because their Searchmetrics visibility had dropped.

As it turns out, nothing was actually wrong, in fact, they’d had a 30% increase in their organic traffic, rankings were up all over the place and therefore there’d been a positive net revenue uplift on their SEO channel. Everyone had become so dependent on referring to visibility metrics via this data, and no one had actually checked what the data meant, or how it correlated with the real world. In this case, there were enough keywords outside of the SM database referring traffic to the site to make the aggregated visibility score irrelevant.

We had to point out to our team that everyone should be careful how we interpret single points of data. It happens.

What’s keyword intelligence good for?

Well, it’s definitely good for getting a flavor of how websites generate traffic from search. You can see where a site ranks for a keyword, provided that keyword is in the intelligence database, you can see what paid ads your competitors might be running (provided that keyword’s in the database). You can see how the rankings for those keywords have changed changed over time.

It’s fair to hope that the bigger the keyword database is, the more likely it is we’ll see data resembling actual analytics data. The reality is that the data is gappy, and therefore you need to be very careful how you interpret it.

Never forget the classic advice: don’t rely on a single data point to draw a conclusion. With that most basic of principles in mind, you can’t go wrong.

What I found

Well, first a caveat – we know this will be specific to individual domains. So you have to try this yourself.

Now I’ve said that, here’s what we did for that particular client:

We looked at Searchmetrics and SEMrush. We found that these tools have a spread of coverage – defined in our case by the number of keywords found in either tool that match actual traffic driving keywords coming from Google.

I think the right way to describe both tools for this domain is “generally gappy, more accurate for higher volume keyword coverage than long tail”.

To calculate spread, I exported 5,000 keywords from both SEMrush and Searchmetrics databases and compared those to the top 5,000 keywords we retrieved from the client’s analytics tool.

What we looked for is the count of keywords found in the site’s top 5,000 referring keywords from both tools.

Search metrics contained 1891 – 37.82% coverage, SEMrush 613 keywords – 12%. Essentially, there were lots more higher volume terms appearing in the analytics data than the intelligence tools databases.

You could be more wrong than you are right

Think about that for a second. If you make a judgement on your “SEO visibility” from an incomplete list of keywords, what’s going to happen? If you’ve got less than a 40% spread, won’t you have more chance of being wrong about the outcome as you could have to be correct? That’s precisely what happened – the keywords that were included in the intelligence database *did* drop, while may volume terms that were not in the database shot up.

I asked Pete to take a look at the data, and he gave me this response (we looked at SEMrush at the same time)

Interestingly, he pointed out, the higher the search volume, the more likely the keyword is to be in the data. When we ringfenced “head” terms that had “very high traffic” levels, SearchMetric’s coverage was much better: 78%

Here’s what we saw:

- SM knows about 80% of VHT keywords (very high traffic)
- SM knows about 64% of HT keywords (high traffic)
- SM knows about 33% of LT keywords (lower traffic)

- SEMr knows about 69% of VHT keywords
- SEMr knows about 21% of HT keywords
- SEMr knows about 8% of LT keywords

So…

The odds that reports are accurate for this particular site are a reflection of the distribution of their high traffic phrases (15% high traffic / 85% low traffic), and how often Searchmetrics has the data on the site (38% of the time SearchMetrics had the data, 12% of the time SEMrush did).

Following on from this, we examined the distributions for specific volumes of phrases as well, and compared the estimated volume accuracy against the actual traffic. The estimated average volume for SearchMetrics for very high traffic phrases was 3802, out by 16%, with a standard deviation of 962. Across the bottom 15% of phrases, these numbers became 10.1, 51% and 32.

The takeaway

The takeaway from this is clear: whilst SM and SEMr are undoubtedly useful tools for alerting you to things that are going on with the rankings they track, what they track isn’t necessarily what you actually rank for and what drives your most important traffic, nor may the scoring values actually equate to traffic. That’s what we found, for this domain, and now we’re taking a look at more, to see what the differences are. Go do the same and let me know in the comments. I wish I had more time, and I certainly wish I had the chance to take a look at Sistrix.

Pete and I discussed this for some time – the only way you’d get a really accurate impression of a domain’s referring keyword portfolio is if you had many, many more keywords available. There’s probably a better package on all the tools for that, but that is not the point.

A more complete view (in the UK, at least) is most likely Hitwise territory – and if you’re arriving at that point, you’ve got to ask yourself what the business case should be for increasing spend, massively on this type of tool.

For me, I see SM and SEMr as good tools for sites that target high traffic, generic “head”  keywords. For an enterprise class site, where often the most keywords that form the organic search traffic portfolio are “long tail”, or highly seasonal, emerging terms, Searchmetrics aggregate rank tracking (the visibility charts) could be missing out.

If you really want to track your search visibility, and your competitors, best to devise a large, potentially seasonal keyword tracking list and use a tool like AWR to monitor daily.

Indeed – SEOgadget.com is quite a good example. Searchmetrics reports on about 8% of the keywords that actually drive traffic to the site. You could easily derive a partial head term keyword strategy from the data, but you’d struggle to gain any clear idea of our real rankings.

Anyway, that’s what we found, for this particular domain, and now we’re taking a look at more sites, to see what the differences are. Go do the same and let me know in the comments.

Image: Tambako The Jaguar
Vanity Check: "Keyword Intelligence Tools", 5.0 out of 5 based on 3 ratings

Comments

  1. Birthe Stuijts

    Hi,
    We do this research always with more databases – mostly we work with sistrix AND searchmetrics. They both have (very) different keyword-sets. It differs per customer and project, which database reflects the most and best keywords. Often I also find rankings for keywordcombinations, that don’t interest me (ranking for Faxnumber as an example).
    Also keep in mind, that sistrix shows keywordrankings in the Top 100, searchmetrics doesn’t, and the actuality of both differs (it can differ upto 2 months, if you take the “worstcase” in the longtail-directory of Searchmetrics. Unfortunately you can’t influence, which keywords are checked weekly – except for you own keywordset of course)
    For local search / local keywords in Germany, you should take a look at Manhattan tools.
    I always use tools as an indicator – and make my own keywordsets – probably what you do with awd – to monitor the changes that really interest me and give me a proper benchmark.
    Greetings from Germany
    Birthe

  2. Sören Bendig

    The tool vendor “fixed” databases for SEO Visibility, Visibility Rank or VisibilityIndex are only good for a quick ad-hoc analysis, if you’re not a very large website like a price comparison and rank for “everything” anyway. The issues with these reference panels:

    # You can’t adjust the database as a user
    # In some tools you can’t even judge which keywords are used to calculate the KPI, which makes it hard to judge the correlation from keywords included compared to your business model. Is this KPI 80% or only 20% “spot on” for your business?
    # Often it’s just a very small sample like eg. 250k Keywords – as an average through all industries.
    # There might be “collateral keywords” you rank for in the tool database, like “terms and conditions”, that distort your website data
    # Most users still fail to really understand how these algorithms calculate (usually a mix from rankings, CTR weighting (first page results more important), search volume and sometimes CPC) the metric.

    This counts for all SEO tools that offer this kind of reference, even our own. Within our Starter version we let users access these KPIs on millions of keywords for free. We feel a Visibility Rank can be very helpful to understand the changes in SEO much better and easier. In addition you can analyze any domain you want without setup necessary. But it’s not the best Indicator you can have, due to the issues above.

    Since nearly two years now we tell our users to not rely on those KPIs for strategic purposes. It’s easy to take tool data and not reflect on it too much, due to work load that’s the case in many marketing fields, and people often use data just “as it is” without challenging the data.

    What is the solution?
    To understand the algorithm and CONTROL the database your KPIs are calculated on. Within SEOlytics for example you can create your individual “Daily Visibility Rank” in the SEO Monitoring. The database is your keywords, you define, and it’s calculated on a daily basis. The algorithm works similar, but you have full control of the database and if something is affecting your Visibility Rank, you can look at every single URL/paths the DVR is calculated on. Larger clients even utilize “Industry Visibility Ranks”, where they can also adjust the algorithm with eg. conversion weighting, etc. Despite the SEO Tools becoming more and more powerful and being able to handle loads of data. The best Business Intelligence still comes from exact data.

  3. Christopher Skyi

    I thought that searchmetrics and semrush were (imperfect) competitive analysis tools.

    If you want to know your SEO visibilty, that’s easy — just look at your Google Webmaster (or Bing Webmaster) Tools data. This is (or should be) 100% accurate, even better than Google analytic data because the latter depends on java script running correctly and cookies (both can glitch up depending the browser, it’s settings, the op. sys.) while the former does not.

    Why in the world was the client looking at searchmetrics for information about how visible they were? Plus, GWT & BWT are free — searchmetrics is not . . .

  4. Elroy van Ouwerkerk

    I partly agree with Christopher. Webmastertools gives you probably the most accurate and reliable data. Especially now when organic traffic keywords become 100% not provided you will have to trust on Webmastertools even more.

    But on the other hand if you would like to know the visibility on high volume keywords, Searchmetrics still gives you an idea and you’re also able to compare your data with your competition. So I do understand why it’s a paid tool.

  5. David Adress

    This counts for all SEO tools that offer this kind of reference, even our own. Within our Starter version we let users access these KPIs on millions of keywords for free.