Rate limiting is when a server caps how many requests a single client can make within a set time window. When the cap is exceeded, the server stops serving that client for a while, usually replying with the HTTP status code 429 Too Many Requests. Sites use rate limiting to protect their infrastructure from overload and to block abusive automated traffic. For anyone scraping the web, rate limiting is the most common reason a run starts failing, and handling it gracefully is the difference between a scraper that finishes and one that gets blocked.
How rate limiting works
A server tracks how many requests come from a given source, usually identified by IP address, over a rolling window such as a minute. When the count crosses the limit, further requests are rejected with a 429 response until the window resets. The response often includes a Retry-After header telling the client how many seconds to wait before trying again. A scraper that ignores that signal and keeps hammering the server typically gets blocked harder, sometimes for hours.
Why rate limiting matters in web scraping
A scrape that runs too fast trips rate limits and starts returning 429 errors instead of data. Worse, in a spreadsheet tool like Google Sheets' native IMPORTXML, those errors often look like the data simply does not exist, returning blank cells with no explanation. Beyond the technical problem, scraping fast enough to degrade a site can have legal exposure as trespass to chattels, which is why a self-imposed limit, commonly around one request per second per domain, is both courteous and protective.
How a scraper handles rate limiting
A well-built scraper does three things. It paces requests so it stays under typical limits rather than bursting. It backs off when it does hit a 429, waiting longer between retries, often using exponential backoff that doubles the delay after each failure and respects any Retry-After header. And it retries failed requests automatically rather than dropping the row. The native import functions do none of this: there is no retry, no backoff, and no way to add one, which is why scale and dynamic pages need a server-side web scraper instead.
Rate limiting in ReplyLabs
ReplyLabs runs requests on its own servers, paces them, and backs off when a host pushes back, rather than flooding a site. Because each URL is its own request, a page that gets rate-limited fails on its own row without stopping the rest of the run, and failed rows are never charged. This is part of what makes scraping at scale reliable from inside Google Sheets, covered in full in web scraping in Google Sheets.