Crawl budget

The number of pages a crawler is willing to visit on a domain within a given period before moving on. For AI bots, a limited crawl budget makes it even more important to have an llms.txt and a clear structure that points the crawler to the most important pages instead of wasting visits on secondary content.