How to Use TF-IDF To Get More SEO Traffic in 2019

 Psssst.

Wanna know how to make your content more SEO friendly…for free?

How to make it rank higher for its target keyword in just a few days.

And also rank for a whole new set of keywords at the same time?

That’s exactly what something called TF-IDF can help you achieve.

What is it?

Patience Padawan, patience.
The really cool bits of what I’m about to share with you are:

  1. Anyone can do it
  2. It takes almost no effort
  3. There’s a free tool to help

The net result here is you can potentially get more organic traffic for spending 10 – 15 minutes “upgrading” your pages.

This is what we have done on several pages of this site, and here are the traffic bumps we got in most cases:

Google Analytics Organic Traffic Bump

And this:

Google Analytics Organic Traffic Bump

Or maybe even this:

Google Analytics Organic Traffic Bump

And while you are at it, you can apply our date update freshness trick (which we did apply to several pages shown above) to get an even bigger bump.

Note however that some of the date ranges above include the Christmas period which overinflates some numbers.

We also had some pages that did NOT see an increase in organic traffic despite being run through the same process. So don’t expect a magic pill tactic. For us, we saw traffic increases about 75% of the times we applied TF*IDF (not scientifically tracked). 

So if you are in a competitive space, it’s probably worth a look.

Even though the traffic is up here, given how low it is, I wouldn't consider it a win.
This one even went down a bit.

So yes, it’s not a one hit wonder, but overall, the experience has been positive enough for us that we will keep using it in the foreseeable future.

Here’s a quick promise: By the time you finish reading this blog post you’ll know enough to go use this SEO tactic to give your rankings and traffic a boost.

Excited?

You should be!

What Is TF-IDF?

It stands for Term Frequency-Inverse Document Frequency.

Wait, come back, this isn’t a math lesson!

It’s not as complicated as it sounds.

But it’s also why mathematicians shouldn’t be allowed to name things.

TF- IDF has been around since 1957 when computer scientist (yes, there were computers at that time) Hans Peter Luhn first came up with it.

Hans Peter Luhn
Image: Hans Peter Luhn generating the first ever keyword density report.

It’s a really simple idea, but a brilliant one.

How TF-IDF Works

Let’s look at this from the perspective of a search engine since they are the ones using this kind of algorithm to establish relevancy.

Let’s say you have an index with just 10,000 pages, but you want to quickly find pages about a certain topic.

Pages Index

You decide the best way to narrow your search is to look for pages that only contain certain words.

So you create a piece of software that scans every single word in every page and collates the most frequently used strings of 1, 2 and 3 words.

Term Frequency (tf)

That is Term Frequency (TF).

Unfortunately, the results include lots of very common words like “the”, “and”, “but”, “where”, etc which pollutes the results.

Irrelevant Keywords

This is where Inverse Document Frequency (IDF) comes in.

IDF tries to estimate the value of a word or word string by looking at how unique it is across the entire index of pages.

Inverse Term Frequency

Thus weeding out common language from establishing relevancy.

Low Relevance Terms

By combining these 2 algorithms, search engines manage to extract keyword relevance from web pages.

It also gives them a good idea of which ones should show up at the top of the SERPs.

Once they have done that, search engines can then use the few pages deemed as relevant for a term and look for correlations with new pages entering the index.

That means search engines will not only look at the one keyword you target but at the entire TF*IDF spectrum of the relevant pages, and then compare it to yours.

So, for example, for this article about TF*IDF, adding complex notions like the fact that IDF is “logarithmically scaled” and therefore harder to abuse by sneaky SEO’s will increase our relevancy for the term “TF*IDF” because the Wikipedia page for TF*IDF also talks about the same thing.

And this kids is why cheap generic writers are not a good fit for SEO.

They lack the specific niche vocabulary that will make your content be seen as relevant by these algorithms.

And as a result, your cheap content will plain not rank.

Let’s talk about how we can fix this for you.

Wait…Is This Keyword Density 2.0?

Nope…not really.

It’s not keyword density as you might remember it.

Like when all the “gurus” were advising site owners to include their primary keyword once in the first paragraph, twice in the body, and once at the end of the blog post?

This isn’t the same thing.

That approach actually worked once upon a time.

Until Google released the Florida update in 2003, which was the first warning sign that keyword stuffing is a bad idea.

The 2011 Google Panda update wiped tens of thousands of low quality (thin content) affiliate sites off the face of the Internet for much the same reason.

People stopped talking about keyword density, except in hushed voices at SEO conferences.

Just in case Matt Cutts had the place bugged.

TF-IDF is more about overall page relevancy and far less about “keyword density”.

And this relevancy comes from focusing on all the other words related to a specific search query, and not just your core keyword.

You’re not looking to shoehorn keywords into certain parts of a page.

Instead, you’re just feeding Google what it wants- thematically relevant content.

Why TF-IDF Is Important for SEO

It’s important because you have two choices when it comes to ranking in Google:

  • Use your best guess at what works
  • Reverse engineer Google’s results

As much as Google would like you to believe that there’s no “formula” applied in how they rank content (ummm…algorithm), there absolutely is.

And we’re not speculating here – we’ve tested it and we can and do back that up with results.

When To Use TF-IDF

The best time to use it is when you’ve invested a lot of resources in creating a great piece of content that’s not doing so well.

Maybe it’s lingering on page two of the SERPs.

Or maybe it’s been around for a while and the traffic and rankings are starting to drop off.

Google Analytics Organic Traffic Before Tf Idf

Established content is a prime candidate for a TF-IDF makeover.

But it has just as much value when you’re creating new blog posts.

You can use it to optimize your content in a way the majority of your competitors simply are not.

Then you get results like this:

Now, let’s look at how you actually use this SEO tactic, but using a tool to make your life easier.

What TF*IDF Tools Are Available?

TF-IDF has been around for a long time, but it’s only recently that mainstream SEOs have started to pay attention to it.

So we’ve yet to be swamped by dozens of software developers all shilling for your monthly subscription monies.

But there are already some very interesting on-page tools that can help with your TF-IDF analysis.

Some of them are more generalized on-page SEO tools, while others are dedicated TF-IDF solutions.

Before we dive into them, I think it’s important to note that real TF*IDF is not really what any tool does.

If you remember the previous sections, IDF is supposed to check your page against every single page on the internet.

Low Relevance Terms

And well, that’s not exactly practical in terms of resources.

So each tool kind of comes up with an alternative solution that’s close enough.

In most cases, they either have a reference index they maintain internally to compare any given page to, or they focus more on correlation across the set of pages that rank for a given keyword.

We will be reaching out to them to highlight the process they use in each mini review below. 

Dedicated TF-IDF tools

Text Tools

Text Tools Homepage

General On-page Analysis Tools

The software and services listed here are not 100% dedicated TF-IDF tools, so it’s not fair to refer to them as such.

It also has come to our attention that several of these tools do not use TF-IDF per se but variations of it such as LSI keywords or “most prominent keywords” for various computing and SEO orientation reasons.

But each of them does provide you with the ability to at least figure out what words are missing from your page and produce roughly similar results to what we outlined previously in this post so we deemed them relevant.

MarketMuse

Marketmuse Homepage
Cora Seo Software Homepage
Pageoptimizer Pro Homepage

And yes, we have tested each of these tools.

In terms of overall value for money, it’s hard to beat Website Auditor, because it doesn’t cost anything, or at least the free version doesn’t.

For a grand total of zero dollars/euros/pounds per month, you get pretty comprehensive TF-IDF research functionality.

Is there a downside?

Yup, you can’t save projects with the no-cost version, so you need to start every project from scratch each time you’re doing any analysis.

That might not seem like a big deal, but it can grate on you after a while.

Now, the obvious question you’ll ask is what on-page tool we use for Authority Hacker content, and our other sites?

Although we have a lot of love for Website Auditor, we recently swapped over to Surfer SEO.

Website Auditor is an excellent tool, but in terms of user-friendliness, range of features, and overall results, we prefer to use Surfer

The fact it’s cloud-based is another perk – we’re not tied to a local software installation.

One thing to note is that Surfer SEO actually uses a “prominent phrases” algorithm that cleans up the most common words of the english language.

Basically it’s not pure TF*IDF but the idea is the same and results have been similar to what we experienced with Website Auditor.

But for now, let’s show you the free way of adding TF*IDF keywords to your pages with Website Auditor.

How To Run A TF-IDF Analysis – Free

First things first, you’ll need to download, install and launch Website Auditor – it’s free, so you have no excuses.

Now choose a URL you want to analyze for a specific keyword.

In our case, we’re going to analyze one of our own pages for the keyword “make money blogging”.

Step 1 – Enter your URL

Website Auditor Enter Url

Click ‘Finish’

P.S. We’re not creating projects because we’re using the free version of the tool – it doesn’t allow you to save any data.

You might notice a little spinning logo in the top left-hand corner of the page saying ‘Rebuild Project’:

Website Auditor Rebuild Project Icon

This is normal, so give it a few minutes to finish doing its thing.

Step 2 – The Audit

The next step is to click on the ‘Content Analysis’ menu and ‘Page Audit’ should load automatically.

Website Auditor Content Analysis

We then choose what page we want to audit and click ‘Next’.

Website Auditor Page Audit

Now it’s time to choose the keyword we want to analyze.

We want to optimize the page for “make money blogging”, so that’s the keyword we enter.

Website Auditor Page Audit Keywords

Click on ‘Finish’ to get…started.

Watch out for the progress bar in the bottom left-hand corner of the page.

Website Auditor Page Audit Progress

It takes a few moments for the software to fully analyze your page, so don’t interrupt it.

Take a few seconds to check out the ‘Page Audit’ report once it loads:

Website Auditor Page Audit Report

We can see here that our keyword isn’t in our Meta Description, so that’s something we need to take care of later.

Spending a minute or two in the ‘Page Audit’ section is always a good idea – it’s easy to miss those little things that can make a big difference.

Let’s keep moving.

Step 3 – TF-IDF Analysis

Now let’s get to the TF-IDF analysis by clicking on ‘TF-IDF’ in the left-hand menu.

Website Auditor Tf Idf

Website Auditor automatically scans the top 10 results in Google and compares your page against them.

It’s always worth checking that the sites you’re compared against are blogs on the same topic.

Why?

Because there’s no point comparing your post to a Wikipedia page, Amazon product page, or something like that.

You do that by clicking on the little gear icon in the top right-hand corner.

Website Auditor Competitors Analysis

Now for the really cool bit – the keyword results.

The ‘Keyword’ column shows you all the keywords that are relevant to the pages analyzed for your core keyword.

Basically, these are words that Google expects to find in a blog post on “make money blogging”.

We only use the “Multi-word Keywords” for our TF-IDF process, but not everyone uses this approach.

Website Auditor Multi Word Keywords1

This is usually a pretty long list, but don’t worry about that for right now.

Website Auditor Multi Word Keywords2

There are 4 possible recommendations for each keyword:

  1. Ok – no need to do anything
  2. Use more – add additional instances of this keyword
  3. Use less – remove some instances of this keyword
  4. Add – add this keyword at least once

Do you need to include every single one of the keywords on the list?

In an ideal world, you would, but you’ll notice that you’ve already included some of them without even trying.

This is one of the other benefits of using writers who specialize in a niche – they naturally include many of these key terms without being asked to.

For the rest, focus on the keywords used by the top 40% – 50% of your competitors.

These are likely to hold more “weight” in Google than keywords featured towards the end of the list.

Upgrading Your Content

Website Auditor can only check live pages, which means you need to add the missing keywords to your content, update it and then run the analysis again.

You’ll have to do this anyway, but it’s a shame that Website Auditor doesn’t have a preview function/text editor like MarketMuse.

But hey, it’s free, so I’ll stop nitpicking.

How can you make sure your keywords are included on your page?

The first thing is to use notepad, Sheets or Numbers to create a list of the keywords you want to include.

Then copy and paste your blog post into Google Docs.

Work through the post, adding the missing TF-IDF keywords into your headings, subheadings, and page content.

Take your time – this isn’t a race.

When you’re done just press CTRL + F (CMD +F on a Mac) and type in the keyword you’re trying to optimize for:

Content Keyword Count Analysis

You’ll see how many times that keyword now occurs in your document.

In our earlier analysis we saw that “make money blogging” was only included 8 times in our post:

Website Auditor Multi Word Keywords Count

So we’ve bumped that up to 11 within the post, and we can squeeze another instance into the Meta Description tag, for a grand total of 12.

Why are we not including exactly 16 instances of our phrase/keyword?

Because we like to test making incremental changes and see what results we get.

All part of the “minimum viable product” approach.

Remember to run this same check for all the other keywords you add to your page.

The purpose of TF-IDF is to improve the relevancy of your blog post using several multi-word phrases – not just focus on a single phrase or keyword.

The Final Step

Once you’re happy you’ve added enough keywords into your document, upload the new version of your page, and run the TF-IDF analysis again.

How quickly can you expect to see a bump in traffic?

That all depends on how much domain authority you have and how often your site gets spidered.

Experiences vary, but you should see a change within a few weeks.

What Do The Results Look Like?

So, yeah this all sounds great.

But the real question is – does it actually work?

Here’s some proof.

Google Analytics Organic Traffic Tf Ids Results

This is the net result of only using TF-IDF to optimize a single page of content.

And these results took less than a month to show up.
‘Page Sessions’ increased by 98.01%.

Or in real numbers, this one page went from 902 sessions in December 2018 to almost 1,800 sessions in January 2019.

No vanity metrics here – just lots of extra traffic.

And from doing nothing more than adding some relevant keywords to our content mix.

There’s no such thing as a “ranking formula”…right?

Summing It Up

Is TF-IDF some kind of secret sauce for on-page SEO?

No.

But it is a scientific way of upgrading your content to match Google’s “requirements”.
It’s also a perfect addition to regular keyword research.

But which of the above tools should you choose?

If you’re working on a budget then Website Auditor makes the most sense.

Surfer SEO, on the other hand, comes highly recommended as a “premium solution” – you do get a whole lot of value for $29 per month.

Your biggest competitors are using TF-IDF, so shouldn’t you be doing the same?

Share on facebook
Share on twitter
Share on linkedin
Share on pocket
Share on email
Share on print

Do you want to learn The SEO Tactics We Use To Build 6 Figure Sites?

Subscribe to join our FREE training and…

  • Learn how to build white hat links to your site without headaches
  • Finally have a proven method to finding profitable niches
  • Get access to our foolproof keyword research methods
  • Learn how to outsource high quality content

13 Comments

  1. Thanks, good post!
    I’d like to ask some questions, see if you guys had any feedback:

    1. When I do the changes on my pages based on the suggestions of Web Site Auditor and then refresh, it doesn’t seem to be registering the changes because the count on some keywords stays the same. I have tried refreshing many times to see if it was that, but it stays the same. Is anyone else experiencing this.

    2. When choosing the keywords, is it better to pick the closest keywords (for example, the exact ones in our page title) or go for similar keywords that have more traffic? It would seem to me that as long as we are trying to optimize a page, we might as well do it for a keyword that has the most traffic?

    3. To choose the competitors for the TF-IDF analysis, I’m doubting between choosing based on Google rank or optimization (smallest to largest and largest to smallest respectively of course). Any ideas?

    Cheers,

    Dirk

  2. So, I saw this post yesterday and it motivated me to reoptimize a few pages with TF*IDF.

    I’ve been using Website Auditor for a while now and think it’s just fine for TF*IDF analysis.

    I did one thing different this time around, though…

    I first looked at the target SERPs historical rankings for the past 90 days on Serpwoo, and used that to identify the competitor pages I’d use in my TF*IDF data set.

    To be clear, these weren’t all on the first page of the target serp at the time of analysis. They were, however, on the first page of the target serp for the majority of the past 90 days.

    Approaching TF*IDF by only comparing your site to the pages that have ranked the best, for longest seemed to be the missing piece.

    The page I optimized had been stuck in spots 12-15 for 6 months. Within 2 hours of doing the TF*IDF analysis and optimization, it moved up to spot #6.

    #winning

  3. Weren’t we all supposed to remove Java from our machines a few years ago because it was so dangerous? If so, why are folks (WebSite Auditor) still writing in Java and thus requiring us to have it (or reinstall it) on our machines? The hoops you go through to do so are a bit much.

  4. SEO Powersuite sucks. In their blog they mentioned “The TF-IDF tool is fully available in WebSite Auditor’s free version”, although I have their enterprise license, the feature is not working and asking me to update my search engine algo subscription.

    Such a shady company with unethical business practices.

    When selling their enterprise license, they sold it as if it’s a lifetime plan and I was unaware of their search algo updates that costs a lot.

    Thanks.

    1. Hah, I got the app sumo deal which includes search updates. Yeah, I think their business model is weird in terms of pricing but you should be able to use tf-idf with the free version.

  5. Thanks for this comprehensive post. We will dig that for sure.
    Just an extra tip for checking the keywords Google associates with your target keyword, (1) go to Google Images, (2) type your keyword and (3) look at the suggested related keywords at the top.
    “Make money blogging” = “wordpress”, “writing”, “ways”, “per month”, “affiliate marketing”, “writers”, “tips”, “income”, “problogger”, “nigeria” !!!, “monetize”

    1. That’s a good tip, not too sure about Nigeria though :D. I’d say apply critical sense to it but still, a great tip! Thanks, Erwan!

  6. This is a great post! Would you wait to attempt this tactic until after you have seen organic results or would you apply it as soon as the post was live? I.e. is this tactic vest applied to posts 6,9, 12 months old or would it be beneficial for new posts? Obviously with new posts it is impossible to track increases or decreases

    1. It does work on new posts, obviously though it’s hard to determine how much it helps if you are optimized from the get-go :).

Leave a Comment

Your email address will not be published. Required fields are marked *