Google Introduces New Crawler To Optimize Googlebot’s Performance


Google has lately launched a brand new net crawler known as “GoogleOther,” designed to alleviate pressure on Googlebot, its main search index crawler.

The addition of this new crawler will in the end assist Google optimize and streamline its crawling operations.

Internet crawlers, also referred to as robots or spiders, mechanically uncover and scan web sites.

Googlebot is answerable for constructing the index for Google Search.

GoogleOther is a generic net crawler that will probably be utilized by numerous product groups inside Google to fetch publicly accessible content material from web sites.

In a LinkedIn publish, Google Search Analyst Gary Illyes shares extra particulars.

Dividing Tasks Between Googlebot & GoogleOther

The principle function of the brand new GoogleOther crawler is to take over the non-essential duties at the moment carried out by Googlebot.

By doing so, Googlebot can now focus solely on constructing the search index utilized by Google Search.

In the meantime, GoogleOther will deal with different jobs, resembling analysis and improvement (R&D) crawls, which aren’t instantly associated to look indexing.

Illyes states on LinkedIn:

“We added a brand new crawler, GoogleOther to our record of crawlers that in the end will take some pressure off of Googlebot. It is a no-op change for you, nevertheless it’s fascinating nonetheless I reckon.

As we optimize how and what Googlebot crawls, one factor we needed to make sure is that Googlebot’s crawl jobs are solely used internally for constructing the index that’s utilized by Search. For this we added a brand new crawler, GoogleOther, that may exchange a few of Googlebot’s different jobs like R&D crawls to unlock some crawl capability for Googlebot.”

GoogleOther Inherits Googlebot’s Infrastructure

GoogleOther shares the identical infrastructure as Googlebot, which means it possesses the identical limitations and options, together with host load limitations, robots.txt (albeit with a distinct user-agent token), HTTP protocol model, and fetch measurement.

Basically, GoogleOther is Googlebot working beneath a distinct title.

Implications For SEOs & Website House owners

The introduction of GoogleOther mustn’t considerably affect web sites, because it operates utilizing the identical infrastructure and limitations as Googlebot.

Nonetheless, it’s a noteworthy improvement in Google’s ongoing efforts to optimize and streamline its net crawling processes.

In case you’re involved about GoogleOther, you may monitor it within the following methods:

  • Analyze server logs: Frequently evaluate server logs to establish requests made by GoogleOther. This may aid you perceive how usually it crawls your web site and which pages it visits.
  • Replace robots.txt: Guarantee your robots.txt file is up to date to incorporate particular guidelines for GoogleOther if obligatory. This may aid you management its entry and crawling habits in your web site.
  • Monitor crawl stats in Google Search Console: Control crawl stats inside Google Search Console to watch any adjustments in crawl frequency, crawl funds, or the variety of listed pages for the reason that introduction of GoogleOther.
  • Observe web site efficiency: Frequently monitor your web site’s efficiency metrics, resembling load occasions, bounce charges, and person engagement, to establish any potential correlations with GoogleOther’s crawling actions. This may aid you detect if the brand new crawler is inflicting any unexpected points in your web site.

Supply: Google

Featured Picture: BestForBest/Shutterstock


Scroll to Top