Google Blocks Web Scrapers, Causing SEO Tool Outages

According to a story in Search Engine Journal, it appears that Google had taken drastic action against web scrapers that harvest search results data, leading to data outages among some of the most popular SEO rank tracking tools. This move from Google caused some upset in the SEO community and, of course, if it continues, there could be some pretty seismic long term consequences.

The Impact on SEO Tools

Many of the well-known rank tracking services, including SEMrush (my personal favourite) have been affected by this Google activity. These SEO tools rely on data from search results pages (SERPs) to provide valuable insights to users. However, it seems that the blocking of scrapers has not been entirely consistent, with some providers like Sistrix and MonitorRank reportedly still operational.

Now I don’t know how much data SEMrush scrapes, but there was always the caveat that their ranking figures were 85% accurate, so there was a degree of

Google’s Stance on Scraping

Google’s guidelines have long prohibited automated rank checking and scraping of search results. The company states that such activities consume resources, interfere with user experience, and violate their spam policies and Terms of Service (TOS). I had this very same conversation with a PPC colleague some weeks ago when they were planning to use a dedicated scraping tool to harvest some Google SERP data for a project. My first instinct was to remind them that Google took a dim view of scraping and it could indeed be a violation of their TOS. Of course, the whole time I’m using the SEMrush Position Tracking tool on almost 50 projects, so I couldn’t talk. 

Potential Consequences of Google Scraping Ban

There’s been a fair old bit of chatter in the SEO community speculating about the potential fallout from this action and there are some genuine concerns about the future of tools that use Google SERPs…

  1. Higher costs: The data extraction may become more challenging and expensive, potentially leading to increased subscription fees for SEO tools.
  2. Calls for an official Google API: Some industry professionals are urging Google to provide a paid API for search results as a solution
  3. Alternative data sources: SEO tools might need to rely more heavily on extrapolated data from various sources to compensate for the lack of direct SERP access

As of January 23, 2025, Google has not made an official announcement regarding this crackdown. SEOs are anticipating further developments and potential statements from Google on this matter. If we hear anything, we’ll post it here.

Leave a comment