Googlebot
Googlebot is the general name for two types of web crawler used by Google’s search engine:
- Googlebot Smartphone: a mobile crawler that simulates a user on a mobile device.
- Googlebot Desktop: a desktop crawler that simulates a user on a desktop computer.
You can identify which type of Googlebot made a request via the HTTP header. Both crawlers follow the same robots.txt rules, which means you can’t selectively block only Googlebot Smartphone or only Googlebot Desktop using robots.txt.
In most cases, Google’s search engine primarily indexes the mobile version of your content. As a result, most Googlebot requests come from the mobile crawler, with a smaller share coming from the desktop crawler.
How Googlebot Accesses Your Site
On average, Googlebot accesses a given site roughly once every few seconds. That said, crawl frequency can temporarily increase. If your site struggles to handle a spike in Google’s requests, it’s possible to reduce the crawl rate.
Important update: in February 2026, Google significantly reduced the maximum amount of data Googlebot will fetch from an HTML page or other supported text file — from 15MB down to 2MB (an 86.7% cut). Each resource referenced in the HTML (CSS, JavaScript) is fetched separately and is subject to that same 2MB limit. Past that limit, Googlebot stops reading the page and only submits what it already fetched for indexing. PDFs get a much higher allowance — 64MB. This size limit applies to uncompressed data. Other Google crawlers (Googlebot Video or Googlebot Image, for instance) may have different size limits.
The practical takeaway for site owners: if your page is loaded down with heavy inline scripts, unprocessed data, or large embedded code, Googlebot may only read part of it — one more reason clean, optimized code matters.
In July 2026, Google also updated its official crawl budget documentation: every site now starts with the same, conservative crawl limit, and that limit is shared across all of Google’s crawlers — meaning Googlebot, the image crawler, and other bots all draw from the same “budget.”
When crawling from IP addresses located in the US, Googlebot uses Pacific Time.
Other technical details about Googlebot are covered in Google’s crawler overview.
Blocking Googlebot
Googlebot discovers new URLs through links embedded in pages it’s already seen. Keeping a site truly “secret” is nearly impossible — even a single click on a link can land that URL in another site’s referrer tag, from which it can spread further.
If you want to limit how Googlebot crawls your site, use:
- robots.txt — to stop a page from being crawled
- noindex — to stop a page from being indexed
- Other methods (e.g., password protection) — to block access entirely
Blocking Googlebot affects Google Search (including Discover and other features), as well as Google Images, Google Video, and Google News.
It’s also worth remembering that Googlebot is no longer the only crawler that matters. AI bots like OpenAI’s GPTBot or Anthropic’s ClaudeBot are managed separately through robots.txt, and deciding whether to allow or block them is an independent decision that has nothing to do with Googlebot’s own rules.
Verifying Googlebot
Before deciding to block Googlebot, it’s important to know that Googlebot’s user-agent string is frequently spoofed by other crawlers too. Make sure a problematic request actually belongs to Google. The best way to verify Googlebot is to check the source IP address via reverse DNS, or compare the IP address against Googlebot’s official IP ranges.
“The details are not the details. They make the design.”
– Charles Eames, designer
