Back in December 2019, search intent was mentioned as the most impactful factor by top SEO specialists in the world, like Viola Eva, Sławek Czajkowski, and Josh Hardwick.
If you want to rank and keep ranking in 2023, nothing changes in that matter – you have to get serious about user intent.
I believe that search intent is becoming the foundation of content optimization. It’s now equal to the backlinks, technical SEO, and content in the page evaluation process. And that’s something that everyone I asked to contribute to this article agrees on.
Especially that Google we know Google does tweak its search intent recognition. Major changes to the intents have started after the BERT update in October 2019. Many SEO folks reported ranking drops for keywords they were ranking for without giving those keywords special attention. And so, with no attention to the overall SERP (search engine results pages) intent.
Seeing the tweaks and the changes, we decided to investigate them.
This article will open your eyes on:
1. How to determine user intent, thanks to my awesome guests.
I want you to know the smart way to determine search intent. You can do all the work manually, the old-school way, or you can learn from the best.
I asked for opinions from top brains from the SEO industry like Viola Eva, Robbie Richards, James Dooley, Steve Toth, Kevin Indig, Lukasz Zelezny, Miles Beckler, Gael Breton, Matthew Woodward, Nathan Gotch, Adam Chronister, and Matt Diggity.
2. How user intent is changing over time—based on data study of 37k keywords.
I asked the freshly-baked Surfer (our new team member, get it?) and machine learning wizard, Maciej Gruszczyński to help me with this one. Maciej and I used BERT for determining user intent based on a search query, meta, and URL from SERP.
These two sections make up for a comprehensive overview of why search intent needs special attention in your SEO process—from the perspectives of top marketers AND the unyielding hard data.
Key takeaways
Here’s the TL;DR version:
- Search intent is a critical ranking factor recognized by top SEOs all over the world. They actively base their strategies on search intent.
- The SEOs’ methods of investigating search intent are not all the same, but most are based on an analysis of actual search results.
- In a year, our 37k-keyword sample’s search intent changed around 12% (this is the average drawn from results gathered between four time stamps, including three major algorithm updates).
- Nearly half of the post-BERT changes were reverted to the original within less than a year.
- After the December 2020 Core Update, 24% of the keywords that were changed during either the May or the BERT update got reverted to the intent from before BERT.
- 23% of shopping intent keywords turned into informational or commercial.
Is the 12%-change big enough to care?
As Matt Diggity himself said,
12% of search queries sounds like a significant enough change for me.
It’s hard to disagree. 4,5k keywords out of 37k have changed their user intent in a year. It’s hard to walk by those numbers
I asked top experts for their opinions on the subject, how we obtained the results, and our actionable conclusions.
Question 1: Why is the user intent important to analyze?
If you don't check and align with the user intent, you won't rank.
Gael Breton, Overlord at Authority Hacker
This simple sentence says it all. If you don’t meet your visitors' needs, your content efforts will fail. Or, at the very least, reaching and maintaining a high position (with the power of domain authority, backlinks, etc.) will be much more expensive.
You can’t ignore intent when creating your content strategy.
When you create content in alignment with what Google already sees as relevant, you have a higher likelihood of ranking. If you ignore intent and just start writing about a keyword, the odds of you missing the correct intent (and thus not ranking) are drastically higher. Understanding intent means you understand what is required to rank.
Miles Beckler, The Most Helpful Marketer In The World
Google's job is to return the most relevant answer to a search query. If you understand what Google deems relevant, you can create a more targeted content briefing and SEO strategy and increase your chances of success dramatically.
Viola Eva, SEO Consultant and Founder at Flow SEO
User Intent is a ranking enabler. Without meeting it, no other ranking factor applies or makes sense.
Kevin Indig, VP SEO & Content at www.kevin-indig.com
You simply won’t rank if you don't deliver the right type of article that Google expects to see for a query.
Matt Diggity, Founder at Diggity Marketing
Keyword intent will dictate how you structure the page and the angle of the content you'll use. For example, informational content must be written and structured differently than transactional content.
Nathan Gotch, Founder of Gotch SEO
Search intent is a buzz word in the SEO communities and rightly so because no matter how powerful your backlinks if you don’t meet the intent of the query no amount of link building will rank your site.
James Dooley, Founder at FatRank
Gone are the days of plain and simple link building and stuffing your page with keywords. If your content doesn’t match with user intent, then there’s not much you can do—except for adjusting the content to be in accordance with what the searcher wants. While matching search intent is not the only thing you need to do to rank, this is the backbone of any page.
Stefan Batory, CEO at Booksy
What does it all mean to you?
If you try to rank a product landing page (shopping intent) when there are informational articles ranking in a SERP exclusively, your chances to appear in the top ten are non-existent.
You may have the strongest backlinks, the highest Content Score, and still get no organic traffic from your target keywords. That’s just money flushed down the drain.
I analyze user intent to improve desired metrics agreed with a client who I am working for.
Lukasz Zelezny, Founder at SEO.London
Intent is the most important factor when it comes to prioritizing, and accurately qualifying your keyword opportunities. Get this step wrong and you'll waste a lot of resources going after keywords that you can't realistically rank for, or ones that will not support your site monetization model.
Robbie Richards, Founder of robbierichards.com
That’s very interesting. According to Robbie, search intent can have a big impact on your business model as well.
This is how I see it: if you have a multi-product e-commerce site, and you manage to rank products from one category, you should expand your inventory by products from this very category. Those products rank well because the dominating search intent of their keywords is shopping.
Chances are, other items you sell rank low, because they target phrases with informational or commercial investigation search intent.
Therefore, your final offer will depend greatly on search intent.
User intent helps you make a decision on how and even whether you should attempt to compete in that SERP.
Steve Toth, Creator at SEO Notebook
Search intent is important because it indicates that the searcher is further along in the buying process. It also can inform the quality of your traffic.
Adam Chronister, Director of Operations at Enleaf
Search intent matters as it cuts right to the heart of what the user is actually after. Every time somebody conducts a search, they have a goal in mind, whether it’s to make a purchase or to simply collect information. Your job as a marketer is to figure out what that goal is in order to deliver. Matching search intent leads to happier customers which leads to better ranking for your site.
Milosz Krasinski, Managing Director at web consulting company at ChilliFruit
Search intent is about figuring out why somebody is conducting a search. Is there intention to buy something or do they want to do a price comparison or simply find out more about a certain product? These are the questions that you tackle with search intent and, this allows you to deliver more targeted results which, subsequently, will rank you higher with Google as well as improving your conversion rate.
Neal Taparia, Founder of Solitaired.com
Here’s the final thought from Matthew Woodward that sums it up best:
Intent can literally make or break a business in both organic search and paid.
If you spent tens of thousands of dollars developing your site, content, and link profile to rank for high search volume keywords with "free" in them—it's unlikely to bring a net positive.
But if you spent the same money purely focused on keywords that are aligned with the intent of your business, you will grow.
One of the biggest mistakes I see is people sacrificing intent for search volume, which in turn—sacrifices their budget.
Matthew Woodward, SEO at MatthewWoodward.co.uk
Matthew argues that the conversion rate from keywords with small search volume, can be much higher than for many queries with high search volume if the user intent is not there.
If someone uses the word “free” in their search query, it means they have no intent in leaving you their money. If your goal is to make money out of this page, your chances are low. There are other reasons why you may want to rank for those keywords though. Just be realistic about your goal.
Question 2: How to determine user intent?
There are many ways of analyzing search intent: from eyeballing, through analyzing SERP features, to reviewing each competitor page by page.
I’ve tried them all. And I was asking myself for a while, how the top SEOs are going through the process. This is what they said:
Gael Breton analyzes his keywords one-by-one.
Pay attention to title tags and URLs.
Miles Beckler suggests just googling away.
Take time to understand what Google believes is relevant content for that phrase before writing.
Viola Eva looks into SERP features to pick the right content form:
I look at the whole of page 1 and start by reviewing media elements. Does Google show images, videos, classic, organic results, map pack? This helps me understand what Google thinks the user intent is—and it also helps me understand if writing a page is the right move to win real estate on page 1.
Lukasz Zelezny bases his process on Google Analytics and other data providers:
I am reversibly looking at organic keywords in Google Analytics after these are pulled from Search Console via Keyword Hero. Google Analytics gives a better description of behavioral aspects than Google Search Console. However, GA initially doesn't have the ability to show organic keywords behind organic traffic. With Keyword Hero, I am able to see not only the number of visits (which normally correlate with the number of clicks) but also time on page, bounce rate, conversion, conversion rate, and for e-commerce websites bunch of e-commerce related metrics.
Robbie Richards has a whole process based on URLs and title tags.
Generally speaking, the SERPs will tell you the user intent behind a query. i.e., they rank the content types and formats that best address the intent of the user.
For example: if you see mostly e-commerce product or category pages ranking for a given term, you can infer that the user’s intent for that query is more transactional in nature.
Similarly, suppose you see mostly blog post content types in a listicle format ranking for a given "best X for Y" query. In that case, you can infer that the intent of the user is more investigational, so a post comparing different options would rank best.
You can quickly spot these trends by looking at either URL or title tag patterns in the SERPs.
In addition to analyzing the SERP content types and formats, you also need to consider the query itself. This comes down to understanding the audience and your common sense. Look for modifiers in the query that would indicate different stages of intent.
For example: if someone includes the "best, top, alternative, competitors" modifier in the query, you know they are actively researching their different options before making a purchase. This could indicate more mid-funnel investigational intent. Great for affiliate site models.
I wrote a comprehensive SERP analysis guide if you want to dive deeper.
Then I look at organic results and mainly content length and content type: Is page 1 full of long-form guides? Comparison articles with tables? Product category pages? This helps me to decide what type of page I should create or optimize.
Kevin Indig built a whole formula based on his process.
I look at three things:
1. What types of sites are ranking in top positions
2. SERP Features Google displays
3. Ranking changes over time for the query
This gives me the best idea of how Google interprets the user intent for a given query. The system I developed and wrote about in my article solving fragmented user intent shows how to use rank trackers to export SERP Features and identify user intent at scale. At the cherry on top, the concept allows you to track user intent changes over time.
Steve Toth bases his research on competitor analysis.
Mainly I look into types of articles on the SERP, e.g., lists, comparison sites, guides, marketplaces, aggregators, and less often, companies actually offering the product or service. I look at what keywords are ranking for my competitors. If I know the original keyword’s intent, it's fair to assume that other keywords are highly-ranking to share similar intent.
Adam Chronister shares examples, looking at a query itself.
When looking for signs of intent in a query, I’m usually looking at the incoming queries themselves, identifying particular modifiers. Some of the more obvious are how, what, who, where, etc.
Example:
- How to buy a life insurance plan
- Where to get a fishing license
Other modifiers might include terms like buy, top, order, best, etc.
- Top Pizza Restaurants in Seattle
- Best Spa in Chicago
- Where to buy snow tires
Google Search Console performance report measures the user intent of the traffic coming into a website. Lower in the report, I can see queries bringing traffic into the website, and from there, it's easy to prioritize them by impression and clicks and scan those terms for user intent verbiage.
Matt Diggity says that the truth is hidden in the page titles.
The titles usually tell the whole story. It takes time to sharpen your search intent detective skills, though. But eventually, you'll be able to look at a SERP result and simply see "when I Google 'best wireless router', I get a bunch of listicles. This must be review intent."
In rare cases, you may need to open the page and see the type of page (transactional, informational, directory, etc), but 99% of the time, you'll find the answer in the title.
Nathan Gotch relies on the types of ranking pages.
The query itself can often reveal the intent. However, nothing beats looking at the actual SERPs and then doing some qualitative analysis of the rankings pages. Ultimately, the question is, what TYPES of pages are ranking (informational, transactional, etc.)
James Dooley is a fan of keyword clustering.
If you do not have a keyword clustering specialist as part of your team, I would quickly recommend you fill this void. In my opinion, this is the most critical position within your team.
There are plenty of tools out there to assist your keyword grouping to meet the intent. A few are listed here if interested.
Milosz Krasinski makes sure that the search volume is right.
I analyse intent by first mapping it to create a more strategic list of keywords. I then create categories for my keywords by search volume and intent. I can then tailor my content accordingly.
Neal Taparia uses tools to determine user intent.
There are lots of clever and complex ways of figuring out search intent but, to be honest, I tend to use Cognitive SEO which is an online tool that helps with this. Identifying and matching search intent is really important and so its a good idea to get all the help that you can get.
Matthew Woodward sticks to search results.
There is a big difference between what you think the "user intent" is and what Google thinks it is.
Ultimately, we want to make sure our efforts align with Google’s definition of "user intent." And to figure it out, there’s no better tool than the power of observation.
You should search for your target keyword and use the power of observation to look at the complete picture you have in front of you—which is the SERP.
Search results are the output of an algorithm. That’s Google’s way of telling you: “Here’s what the users want.”
We pay close attention to intent during our manual keyword research process.
Unless the intent of a keyword matches our ideal customer profile, it's not worth spending any time or money going after it.
For example, you could probably get thousands of visitors per month in your niche ranking for keywords with the word "free" in them—but that intent does not usually align with the bottom line of the business.
Can you spot the patterns among the experts’ quotes? Titles, SERP features, search query itself... This is all you need to determine user intent with a high accuracy.
While most of our experts claimed to look into search intent themselves, here at Surfer, we decided to base our study on big data and automation. The model from the research is a part of Keyword Research Tool as well to recognize search intent for a single-page topic cluster.
The main goal of utilizing machine learning in this process is to analyze SERPs and point out the changing intents continually. Analyzing search intent by hand will tell you what works for your one particular keyword now.
In fact, we’re going to explore the automation of search intent analysis further so you can spot trends and intent changes.
User intent Big Data case study: How has search intent changed over time?
So here’s what happened.
Maciej, our big data specialist, took the distilled version of BERT, added his secret sauce, and trained the model. He used SERP features, geodata, semantics, and human input to the training data set so that BERT could determine the user intent based on the search query, title, URL, and description.
To give you a sneak peek into our process, here’s a simplified example of what kind of info we fed to our BERT.
Is there a map pack in the SERP? Most likely, the search intent is local. Just like in the example below:
The knowledge panel shows up, and there are no signs of commercial investigation (like words “review” or “best”)? The intent is informational.
All of these above mixed with query modifiers and human input turned out to be great. In short, this is how we trained the BERT. And hey—it works!
But before I show you the results, let’s brush up on the essential search intent aspect: its types.
Our classification of search intent types
There’s no one, simple classification of user intent.
The most popular classification divides search intent into the following categories: informational, navigational, transactional, and commercial.
But after analyzing the SERPs, we came up with a bit different yet equally simple classification. In our opinion, it showcases the division between SERPs better:
- Informational
The user wants to find information on a subject, learn, and get more knowledge.
- Customer investigation
The user knows they need to make a purchase but isn’t yet sure about its exact nature. For example, the searcher knows they need a new pair of headphones, but don’t know which brand to go for, whether they need them wireless or not, etc.
- Shopping
The user now knows exactly what they want to buy and is currently searching for the best place to get it.
- Local
The user needs a local service—a plumber, a hairdresser, a 24/7 pizza place. They want to see maps, prices, and contact info.
Each category got indicators assigned. For example, the local intent got the map pack, the commercial investigation—a query modificator, or shopping—the ad carousel.
Thanks to the set of rules and input from our SEO experts, we managed to prepare a dataset, and BERT was able to learn how to determine user intent.
Okay, but how the heck are you using BERT? It’s Google’s algorithm!
I asked Maciej to explain it as this question may be popping up in your head. According to him, BERT is just one of many Natural Language Processing models, but a real game-changer in machine learning.
He claims that its transformer-based architecture is both innovative and massive. Therefore, it can be used for a wide range of NLP tasks, like translation, question answering, text classification, sentiment analysis, proper name recognition, dialogue systems, and so on.
Google shared BERT with the public and started a new era in the field of AI. Many researchers used it to publish a lot of papers where they test, apply, or upgrade it. We decided to put it to the task that was probably never solved this way—user intent detection based on SERP content.
Many upgrades of the original BERT model have been proposed since the first release, and we have chosen the most fitting one. (I’m proud that we go hand in hand with the state-of-the-art.)
It works very simply. It returns the percent of each intent based on analyzed ranking pages’ snippets. Then, we determine which intent was dominant in the SERP.
In our research, the intent can be considered dominant when the majority of top 10 results are recognized to serve this particular intent.
Once you know why and how we used BERT to determine user intent, it’s time to move to the actual study!
The study: methodology and results
It all started with 37k keywords analyzed by Surfer users in January 2020. We recrawled the same 37k SERPs twice:
- after the May Core Update
- and in December 2020.
The cherry on top was the database from September 2019 for the exact same sample to cover the situation before BERT.
We wanted to answer one simple question:
The changes are significant after all three Google updates.
BERT update changed 10,5% of the keywords’ dominating user intent. What does it mean?
Almost 4k keywords turned from one intent type to another. For example, a SERP that was dominated by product pages (shopping intent) are now on the commercial investigation side.
If your keywords were among them, it means that your content relevance is lower. And this may be reflected in your organic traffic.
May Core update saw the biggest impact on user intent changes, resulting in a 13,3% change of the keywords’ dominating user intent.
For December 2020 Broad Core Update, this number was slightly smaller, changing 12,4% of the keywords.
Now, those changes happened from one period to the next. The big question is, what’s the cumulative impact of those changes?
51% of the user intents changed in the BERT update were reverted to the original intent by the May update! It appears that the changes were not the best from the users’ perspective and they adjusted the algorithm.
Did the December 2020 Core Update revert the May Core update then? Hell yeah! 39% of the keywords came back to the intent introduced by BERT yet again.
That’s quite a lot of changes! But if we wanted to sum up how many keywords got reverted to the pre-BERT state after the December 2020 Core Update...
It turns out the number is around 24%.
We considered only unique reverts (those keywords that were only reverted by the December 2020 Core Update) – for example, when the intent was informational before BERT, changed into shopping, remained shopping after May, and went back to informational after the December 2020 Core Update.
As you can see, the changes have been a real rollercoaster and the cumulative change of the search intent between the first and the last data check is 15.7%.
Who suffered the most from the intent changes?
The massive chunk of all shopping intent keywords turned into informational or customer investigation. Either way, stores were replaced by blogs.
From 5,5k keywords that used to be shopping, 1,6k changed the dominating intent, starting to display blogs after the May update. That’s 29,3%!
The December 2020 Core Update was just a bit smaller; 21,4% of shopping keywords turned into informational. The BERT turned out a bit weaker in this area, changing only 11,9% of shopping queries into Info.
These changes were reverted pretty often; however, if we take the first sample from September 2019 and compare it with December 2020…
Conclusion and action points: what do the results mean to us, SEOs?
The huge chunk of data we analyzed can shed light on a few SEO aspects of our work with search intent. The most important one is:
The SERPs we analyzed showed consistent user intents, and all of our expert guests confirmed that meeting the search intent brings results.
But there are a few other takeaways that you can take and apply to your strategy.
1. You can analyze user intent by looking at SERPs with decent accuracy.
It turned out that titles, URLs, descriptions, and on-SERP features were enough to determine the intent. The only problem here is the scale—if you have a lot of keywords to juggle, you will spend a lot of time monitoring SERPs for each of them.
2. Put a blog on your store's website.
Over 20% of shopping intent keywords turned into an informational or commercial investigation after BERT and/or the December 2020 Core Update.
E-commerce should invest in content beyond their product pages, like blog posts and articles.
3. Reactions to updates should be quick or not done at all.
There is a high chance that they’ll get reverted by the following update. So, either ride the high tide and keep changing, or stay calm and let things be. “Better late than never” doesn’t work for search intent.
4. Writing an article today based on the current user intent is a relatively safe practice.
If you’re worried future algorithm updates might change your evergreen content into a dud, don’t be. Data showed that most intents remained unchanged. If you’re not e-commerce, chances are the updates will leave your content as is.
You should go back to your content once in a while anyway. Make search intent check a point on your content optimization to-do list.
5. Do you have content that lost traffic after the Core Update? Check if you are aligned with the intent; there is an over 10% chance it changed.
Again, you might be one of the unlucky “few.” Do your research.
6. If you can’t satisfy the new intent for your target keyword, find a new one.
You may think, “alright, I’ll take over this keyword with my domain strength ALONE,” and try to do that… But even if you succeed, high ranking may be just temporary. If people won’t find what they want Google will drag you down.
Summary: How should we change the way we think about user intent?
I think the 10-13% changes every few months are pretty big. About 6k keywords out of the 37k sample changed user intent in a year.
Not to mention the massive shopping intent queries drop. This huge change could put a lot of e-commerce stores at a disadvantage.
I hope more and more businesses will consider search intent in their keyword research process and stop wasting time trying to rank their content that is just not right for the context of the query.
Maciej and I will keep on exploring the subject of search intent identification to streamline the process. In the meantime, keep creating the most relevant pages in line with the user's needs. Simple but powerful, that’s THE “content strategy” for searc
See also:
- Search Intent In SEO: How to Get It Right [A Quick Guide]
- How to Use Topic Clusters for Content Marketing
- 5 Ways To Search For Keywords
