Building Async Scrapers That Actually Scale

You've got a list of 50,000 URLs to scrape. You fire up a for loop with requests.get(), and an hour later, you've processed 3,000. The bottleneck isn't your machine - it's waiting for each HTTP response before sending the next request.

Async scraping with asyncio and HTTPX changes the equation completely. Here's how to build scrapers that handle thousands of concurrent requests without melting your machine or getting banned.

Why Async Matters for Scraping

Scraping is I/O-bound work. For every request, your program spends 200-2000ms waiting for a response. In a synchronous loop, that's dead time - your CPU sits idle while the network does its thing.

Async lets you send hundreds of requests simultaneously. While you're waiting for response #1, you've already fired off requests #2 through #500. The CPU utilization is the same, but throughput goes through the roof.

# Sync: ~3 requests/second
import requests

for url in urls:
    response = requests.get(url)
    process(response)

# Async: ~200+ requests/second
import httpx
import asyncio

async def scrape(urls):
    async with httpx.AsyncClient() as client:
        tasks = [client.get(url) for url in urls]
        responses = await asyncio.gather(*tasks)
        for r in responses:
            process(r)

But the naive async approach above will blow up at scale. Here's how to do it properly.

Connection Pooling

HTTPX's AsyncClient maintains a connection pool. By default, it keeps 100 connections - which is fine for most cases, but if you're hitting one domain with thousands of requests, you want to tune it:

import httpx

limits = httpx.Limits(
    max_connections=100,        # Total connection pool size
    max_keepalive_connections=20,  # Persistent connections to reuse
    keepalive_expiry=30,        # Seconds before closing idle connections
)

async with httpx.AsyncClient(limits=limits, timeout=30.0) as client:
    # All requests share this pool
    pass

The key insight: reuse connections. TLS handshakes are expensive. A persistent connection pool means you pay the TLS cost once per host, then reuse that connection for hundreds of requests.

Semaphore-Based Rate Limiting

Firing 50,000 requests simultaneously will get you IP-banned in seconds. You need a concurrency limiter:

import asyncio
import httpx

CONCURRENCY = 50  # Max simultaneous requests

async def fetch(client, url, semaphore):
    async with semaphore:
        try:
            response = await client.get(url, timeout=15.0)
            return response
        except httpx.RequestError:
            return None

async def scrape(urls):
    semaphore = asyncio.Semaphore(CONCURRENCY)
    async with httpx.AsyncClient() as client:
        tasks = [fetch(client, url, semaphore) for url in urls]
        results = await asyncio.gather(*tasks)
        return [r for r in results if r is not None]

The Semaphore(50) ensures at most 50 requests are in-flight at any time. The rest queue up and fire as slots open. This gives you consistent throughput without overwhelming the target or your system.

Smart Retry with Exponential Backoff

Network requests fail. Connections time out, servers return 503s, rate limiters kick in. A good scraper handles this gracefully:

import asyncio
import random

async def fetch_with_retry(client, url, semaphore, max_retries=3):
    async with semaphore:
        for attempt in range(max_retries):
            try:
                response = await client.get(url, timeout=15.0)

                if response.status_code == 429:
                    # Rate limited - back off
                    wait = (2 ** attempt) + random.uniform(0, 1)
                    await asyncio.sleep(wait)
                    continue

                if response.status_code >= 500:
                    # Server error - retry
                    await asyncio.sleep(1)
                    continue

                return response

            except httpx.RequestError:
                if attempt < max_retries - 1:
                    await asyncio.sleep(1)
                    continue
                return None
    return None

The exponential backoff with jitter (2 ** attempt + random) prevents thundering herd problems - when all your retries fire at the same time and hit the rate limiter again.

Per-Domain Rate Limiting

If you're scraping multiple domains, you don't want a single slow domain blocking everything. Use per-domain semaphores:

from collections import defaultdict
from urllib.parse import urlparse

class DomainRateLimiter:
    def __init__(self, per_domain=10, delay=0.1):
        self.semaphores = defaultdict(lambda: asyncio.Semaphore(per_domain))
        self.delay = delay

    async def acquire(self, url):
        domain = urlparse(url).netloc
        semaphore = self.semaphores[domain]
        await semaphore.acquire()
        await asyncio.sleep(self.delay)  # Minimum delay between requests

    def release(self, url):
        domain = urlparse(url).netloc
        self.semaphores[domain].release()

limiter = DomainRateLimiter(per_domain=5, delay=0.2)

This gives each domain its own concurrency limit. You can hammer api-a.com at 5 concurrent requests while simultaneously hitting api-b.com at 5 more - without either affecting the other.

Processing Results in Batches

Don't try to hold 50,000 responses in memory. Process them as they complete:

import asyncio
import httpx

async def process_batch(client, urls, semaphore, callback):
    async def fetch_and_process(url):
        async with semaphore:
            try:
                response = await client.get(url, timeout=15.0)
                await callback(url, response)
            except httpx.RequestError as e:
                await callback(url, None)

    # Process in chunks to avoid creating 50k coroutines at once
    chunk_size = 1000
    for i in range(0, len(urls), chunk_size):
        chunk = urls[i:i + chunk_size]
        tasks = [fetch_and_process(url) for url in chunk]
        await asyncio.gather(*tasks)

async def save_result(url, response):
    if response and response.status_code == 200:
        # Write to file, database, etc.
        data = response.json()
        # ... process data ...

async def main():
    semaphore = asyncio.Semaphore(50)
    async with httpx.AsyncClient() as client:
        await process_batch(client, all_urls, semaphore, save_result)

Chunking at 1,000 means you never have more than 1,000 pending coroutines, keeping memory usage predictable.

Putting It All Together

Here's a production-ready async scraper structure:

import asyncio
import httpx
import random
import time
from dataclasses import dataclass

@dataclass
class ScrapeResult:
    url: str
    status: int
    data: dict | None
    error: str | None

class AsyncScraper:
    def __init__(self, concurrency=50, timeout=15, max_retries=3):
        self.concurrency = concurrency
        self.timeout = timeout
        self.max_retries = max_retries
        self.results = []

    async def fetch(self, client, url, semaphore):
        async with semaphore:
            for attempt in range(self.max_retries):
                try:
                    response = await client.get(url, timeout=self.timeout)

                    if response.status_code == 429:
                        wait = (2 ** attempt) + random.uniform(0, 1)
                        await asyncio.sleep(wait)
                        continue

                    return ScrapeResult(
                        url=url,
                        status=response.status_code,
                        data=response.json() if response.status_code == 200 else None,
                        error=None
                    )
                except httpx.RequestError as e:
                    if attempt == self.max_retries - 1:
                        return ScrapeResult(url=url, status=0, data=None, error=str(e))
                    await asyncio.sleep(1)

    async def run(self, urls):
        semaphore = asyncio.Semaphore(self.concurrency)
        limits = httpx.Limits(max_connections=self.concurrency)

        async with httpx.AsyncClient(limits=limits) as client:
            tasks = [self.fetch(client, url, semaphore) for url in urls]
            self.results = await asyncio.gather(*tasks)

        return self.results

# Usage
scraper = AsyncScraper(concurrency=50)
results = asyncio.run(scraper.run(urls))

success = [r for r in results if r.status == 200]
failed = [r for r in results if r.error]
print(f"Success: {len(success)}, Failed: {len(failed)}")

Performance Numbers

Real numbers from production scrapers I've built:

The sweet spot is usually 50-100 concurrent connections. Beyond that, you hit diminishing returns - the target's rate limiter kicks in, or your own network becomes the bottleneck.

Key Takeaways

• Async scraping is 20-60x faster than synchronous for I/O-bound workloads
• Always use a semaphore to cap concurrency - unbounded async will get you banned
• Reuse connections via AsyncClient - don't create a new client per request
• Implement exponential backoff with jitter for retries
• Process results in chunks to keep memory predictable
• 50-100 concurrent connections is the sweet spot for most targets
• Per-domain rate limiting is essential when scraping multiple hosts

The goal isn't maximum speed - it's maximum reliable throughput. A scraper that runs at 50 req/s consistently is better than one that bursts to 500 and gets blocked after 10 seconds.

Always respect robots.txt and terms of service. Scrape responsibly.