Googlebot?
Googlebot is the general name for the two types of web crawlers used by Google Search:
- Googlebot Smartphone: A mobile crawler that simulates a user browsing from a mobile device.
- Googlebot Desktop: A desktop crawler that simulates a user browsing from a desktop computer.
You can identify the type of Googlebot via the HTTP header. Both crawlers follow the same robots.txt rules, which means it is not possible to selectively block only the Googlebot Smartphone or only the Googlebot Desktop using robots.txt.
In most cases, Google Search primarily indexes the mobile version of your content. As a result, the majority of requests from Googlebot will come from the mobile crawler, while fewer requests will originate from the desktop crawler.
How Googlebot Accesses Your Website
Googlebot typically accesses websites once every few seconds. However, in short bursts, the crawl rate may increase. If your website has trouble handling the increased load from Google, it is possible to reduce the crawl rate.
Googlebot is capable of crawling up to 15MB of an HTML file or any other supported text-based file. Each resource referenced in the HTML (such as CSS or JavaScript) is loaded separately, and the same size limit applies to each of them. After reading the first 15MB, Googlebot stops parsing the page and only sends the first 15MB to the index. This size limit applies to uncompressed data. Other Google crawlers (e.g., Googlebot Video, Googlebot Image) may have different size limits.
When crawling occurs from IP addresses located in the United States, Googlebot uses the Pacific Time Zone (PT).
Additional technical characteristics of Googlebot are described in Google’s crawler overview.
Blocking Googlebot
Googlebot discovers new URLs by following links embedded in already known pages. Keeping a site “secret” is nearly impossible, as a single clicked link can expose a URL in the referrer tag of another site and allow it to spread from there.
If you want to limit crawling by Googlebot, use:
- robots.txt – to block crawling of a page
- noindex – to prevent indexing of a page
- Other methods (e.g., password protection) – to completely block access
Blocking Googlebot affects all of Google’s search services, including Google Discover, Google Images, Google Video, and Google News.
Authenticating Googlebot
Before deciding to block Googlebot, it’s important to know that the Googlebot user-agent can be spoofed by other crawlers. Make sure that the problematic request actually originates from Google.
The best ways to verify Googlebot are:
- Checking the source IP address using reverse DNS lookup, or
- Comparing the IP address against the official list of Googlebot IP ranges.
Authenticating Googlebot
Before deciding to block Googlebot, it’s important to know that the Googlebot user-agent can be spoofed by other crawlers. Make sure that the problematic request actually originates from Google.
The best ways to verify Googlebot are:
- Checking the source IP address using reverse DNS lookup, or
- Comparing the IP address against the official list of Googlebot IP ranges.
Web Development
Website Maitenance
Web Design
SEO
Pricing