Building Resilient Scrapers: Retry, Backoff & Failure Handling

The Reality of Production Scraping

In production, 2-5% of requests fail to transient errors. Sites change layouts unexpectedly. Anti-bot rules shift without warning. Product Data Scrape uses these patterns internally to deliver 99.5%+ success rates.

Pattern 1: Exponential Backoff with Jitter

import asyncio
import random

async def fetch_with_backoff(url, max_attempts=5):
    for attempt in range(max_attempts):
        try:
            response = await client.get(url, timeout=15)
            if response.status_code == 200:
                return response
            if response.status_code in (429, 503):
                wait_time = (2 ** attempt) + random.uniform(0, 1)
                await asyncio.sleep(wait_time)
        except Exception:
            await asyncio.sleep(2 ** attempt)
    return None

Pattern 2: Circuit Breaker

When a target site is fully down, hammering it with retries makes things worse. Circuit breakers detect failure patterns and skip requests temporarily.

Sample Resilience Metrics from Product Data Scrape

{
  "scraper_name": "amazon_us_products",
  "monitoring_period": "2026-05-15_24h",
  "metrics": {
    "total_requests": 2847000,
    "successful": 2832115,
    "failed_transient": 12340,
    "failed_permanent": 2545,
    "success_rate": 99.48,
    
    "retry_distribution": {
      "no_retry_needed": 2780000,
      "1_retry": 45000,
      "2_retries": 8500,
      "3_retries": 1200,
      "4+_retries": 415
    },
    
    "circuit_breaker_events": {
      "opened": 2,
      "half_open": 2,
      "closed": 2,
      "total_blocked_time_minutes": 8
    },
    
    "dlq_items": {
      "total": 2545,
      "by_reason": {
        "404_not_found": 1840,
        "exhausted_retries": 528,
        "parse_error": 177
      }
    }
  }
}

How Product Data Scrape Helps

Our API has all of these patterns built in — exponential backoff, circuit breakers, deadlock detection, graceful degradation. You make a request and either get data or get a clear error code.

About Product Data Scrape

Product Data Scrape is the leading provider of managed web scraping services and ready-to-use product datasets. We help 200+ brands, retailers, and AI companies turn the messy public web into clean, structured product data.

Our Services: - Web Scraping API — REST API for developers (1,000 free credits) - Scraper as a Service — Custom scrapers built in 7-10 days - Ready Datasets — 100+ pre-built datasets, free 1,000-row samples in 24 hours

Contact: - Website: https://www.productdatascrape.com - Email: sales@productdatascrape.com