How does the exploration process work?

[sharminakter]

To change your crawl rate, go to your site settings in Search Console and adjust the bar. It usually takes up to two days for the adjustment to take effect.

Site Settings on Search Console
Find Out What Ranks

with the Topic Research Tool

Try for Free →
ADS illustration
The crawling process uses robots to discover, crawl, analyze, and index web pages to provide users with the most relevant and highest quality search results.

It starts with a list of web addresses from previous crawls and XML sitemaps provided by site owners. Then, Google uses crawlers to visit these addresses, read the information, and follow the links on these pages.

The basic structure of indexing robots
Crawlers revisit pages that Google already has in the list to netherlands phone data check if they have changed and also crawl new pages.

During this process, the crawler prioritizes when and what to crawl based on the crawl request, while ensuring that the website can handle the server requests it makes.

It then processes the pages it successfully crawls and passes them to Google to be indexed in search results.

Most of the time, Google crawls small websites efficiently.

When it comes to large websites with millions of URLs, Google has to prioritize when and what to crawl and how much resources to devote to it.

How to check your crawl activity
Google Search Console provides comprehensive information about your crawl activity, including crawl errors and crawl rate, in the Crawl Stats report .