Don’t use 403s or 404s for rate limiting | Google Search Central Blog | Google for Developers
Friday, February 17, 2023 Over the last few months we noticed an uptick in website owners and some content delivery networks (CDNs) attempting to use 404 and other 4xx client errors (but not 429) to attempt to reduce Googlebot’s crawl rate. The short version of this blog post is: please don’t do that; we have … Read more