Crawling budget – the volume of pages crawled by a search robot per unit of time and affecting site indexing

Crowling Budget – SEO WIKI

A crawling budget is the number of pages that a crawler crawls over a given period of time. The term was introduced by Google, which evaluates relevance based on the popularity of the request, the relevance of the content and the speed of crawling the site.

Crowling budget is formed by user demand and server availability. If there are a large number of URLs on one IP, indexing can slow down, so large web projects need to optimize content based on this indicator.

Saving the crawling budget is facilitated by removing duplicates and eliminating unnecessary ones from the index using robots.txt, competent linking with XML and HTML mapping, server response 304 when accessing scanned pages with unchanged content.