OpenClaw: Scraping JS Sites Without Getting Blocked

Here's the deal: web scraping javascript-rendered pages is not the simple Puppeteer-and-done problem that most tutorials make it out to be. When we built OpenClaw, we went in thinking the hard part was getting the browser to render the page. We were wrong. The hard part is everything that happens after the page loads, at scale, when the target site starts adapting to you.

This is the breakdown I wish existed before we started.

Why JavaScript Rendering Is Just the Entry Fee

The Playwright docs will get you a browser instance in ten minutes. That part is genuinely easy. What the docs don't cover is what happens when you're running hundreds of concurrent browser sessions against a site that has a dedicated anti-bot team.

In OpenClaw, we use Playwright over Puppeteer for one specific reason: Playwright's browser contexts are cheaper to spin up and tear down than full browser instances, and they give you proper isolation between sessions. Each scrape job gets its own context with its own cookie jar, localStorage state, and fingerprint profile. Here's what that initialization looks like in practice:

import { chromium, BrowserContext, Page } from 'playwright';

interface ScraperContext {
  context: BrowserContext;
  page: Page;
  sessionId: string;
}

async function createScraperContext(
  fingerprintProfile: FingerprintProfile
): Promise<ScraperContext> {
  const browser = await chromium.launch({
    headless: true,
    args: [
      '--disable-blink-features=AutomationControlled',
      '--disable-dev-shm-usage',
      '--no-sandbox',
    ],
  });

  const context = await browser.newContext({
    userAgent: fingerprintProfile.userAgent,
    viewport: fingerprintProfile.viewport,
    locale: fingerprintProfile.locale,
    timezoneId: fingerprintProfile.timezone,
    extraHTTPHeaders: {
      'Accept-Language': fingerprintProfile.acceptLanguage,
    },
  });

  // Patch navigator properties before any page load
  await context.addInitScript((profile) => {
    Object.defineProperty(navigator, 'webdriver', {
      get: () => undefined,
    });
    Object.defineProperty(navigator, 'hardwareConcurrency', {
      get: () => profile.hardwareConcurrency,
    });
    Object.defineProperty(navigator, 'deviceMemory', {
      get: () => profile.deviceMemory,
    });
  }, fingerprintProfile);

  const page = await context.newPage();
  const sessionId = crypto.randomUUID();

  return { context, page, sessionId };
}

Notice --disable-blink-features=AutomationControlled. That single flag patches the navigator.webdriver property at the Chromium level. Without it, any site running basic bot detection will flag you immediately.

The Fingerprinting Problem Nobody Talks About

Here's what most guides miss: browser fingerprinting checks are not just looking at your user agent string. They're running canvas fingerprinting, WebGL fingerprinting, font enumeration, audio context checks, and timing analysis on your JavaScript execution patterns. If you spin up 50 browser contexts that all report identical canvas hashes, you're going to get blocked regardless of how good your proxy rotation is.

In OpenClaw, we maintain a pool of pre-generated fingerprint profiles. These aren't random values we made up. We generated them by running actual browser instances across different hardware configurations and capturing the real output. Each profile stores the canvas hash, WebGL renderer string, audio fingerprint, and a set of navigator properties that are internally consistent with each other.

The consistency part matters more than the randomness. A fingerprint claiming to be a MacBook Pro but reporting a Windows timezone and a screen resolution that no Mac ships with is going to fail fingerprint coherence checks. Sites running Fingerprint Pro or similar services are specifically looking for incoherent profiles.

interface FingerprintProfile {
  userAgent: string;
  viewport: { width: number; height: number };
  locale: string;
  timezone: string;
  acceptLanguage: string;
  hardwareConcurrency: number;
  deviceMemory: number;
  canvasHash: string;
  webglRenderer: string;
  webglVendor: string;
  platform: string;
}

// Profiles are loaded from a pre-validated pool, not generated on the fly
async function getFingerprintProfile(
  pool: FingerprintProfile[]
): Promise<FingerprintProfile> {
  const index = Math.floor(Math.random() * pool.length);
  return pool[index];
}

JavaScript Rendering Delays and Why waitForSelector Lies to You

page.waitForSelector() tells you that a DOM element exists. It does not tell you that the data you want has finished loading. This distinction has burned us more than once.

A lot of modern SPAs render a skeleton UI almost immediately, then populate it with data from a subsequent API call. Your selector fires the moment the skeleton appears. You screenshot or extract at that point and get empty divs.

The pattern we settled on in OpenClaw is waiting for network idle combined with a content-presence check:

async function waitForContent(
  page: Page,
  contentSelector: string,
  options: { timeout?: number; minContentLength?: number } = {}
): Promise<void> {
  const { timeout = 30000, minContentLength = 10 } = options;

  await page.waitForLoadState('networkidle', { timeout });

  await page.waitForFunction(
    ({ selector, minLength }) => {
      const el = document.querySelector(selector);
      if (!el) return false;
      const text = el.textContent?.trim() ?? '';
      return text.length >= minLength;
    },
    { selector: contentSelector, minLength: minContentLength },
    { timeout }
  );
}

networkidle waits until there are no network requests for 500ms. Combined with a content length check, this catches the lazy-load case. It's slower than a naive selector wait, which is fine. Speed matters less than accuracy when you're building a production scraper. Bad data that arrives fast is worse than good data that takes an extra second.

Rate Limiting That Adapts to You

This is the part that actually kept me up at night during OpenClaw development.

Simple rate limiting is easy to work around. You add delays, you rotate proxies, done. But the sites worth scraping are running adaptive rate limiting. They track your request patterns over time. If you're hitting their API every 3.2 seconds like clockwork, that regularity is itself a signal. Real users have variance. Real users get distracted, click around, read things.

OpenClaw uses a jitter-based delay system that samples from a distribution rather than adding a fixed offset:

function humanDelay(
  baseMs: number,
  options: { jitterFactor?: number; burstProbability?: number } = {}
): Promise<void> {
  const { jitterFactor = 0.4, burstProbability = 0.05 } = options;

  // Occasionally simulate a user who got distracted
  if (Math.random() < burstProbability) {
    const longPause = baseMs * (3 + Math.random() * 5);
    return new Promise((resolve) => setTimeout(resolve, longPause));
  }

  // Normal case: gaussian-ish jitter via Box-Muller
  const u1 = Math.random();
  const u2 = Math.random();
  const gaussian = Math.sqrt(-2 * Math.log(u1)) * Math.cos(2 * Math.PI * u2);
  const jitter = gaussian * jitterFactor * baseMs;
  const delay = Math.max(baseMs * 0.5, baseMs + jitter);

  return new Promise((resolve) => setTimeout(resolve, delay));
}

The burst probability parameter simulates those moments where a real user walks away from their browser. That long tail of occasional multi-second pauses is actually important for evading pattern detection.

Beyond delays, we track per-domain request budgets. Each scrape job carries a budget object that tracks how many requests have been made to a given domain in the current session. When the budget runs low, the job slows down automatically rather than hitting a hard cutoff. Hard cutoffs look like bots. Gradual slowdowns look like users losing interest.

Infrastructure: Why You Need a Queue, Not a Loop

Running web scraping javascript-rendered pages at scale from a single process is how you end up with a cascade failure that takes down your entire scraping operation at 2am. OpenClaw runs jobs through a proper queue (we use BullMQ on Redis) with per-domain concurrency limits enforced at the queue level.

The architecture looks like this:

• A coordinator service receives scrape requests and enqueues jobs with domain metadata attached
• Worker processes pull from the queue, respecting per-domain concurrency limits
• Each worker manages its own Playwright browser instance with a pool of contexts
• Results go into a separate processing queue before hitting the database

The reason this matters: if a target site starts returning 429s or CAPTCHAs, you want to back off that specific domain without affecting scraping jobs for other domains. A monolithic loop can't do that cleanly. BullMQ's rate limiter handles this at the queue level, which means you're not writing backoff logic in every worker.

import { Queue, Worker, QueueScheduler } from 'bullmq';

const scrapeQueue = new Queue('scrape-jobs', {
  connection: redisConnection,
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: 'exponential',
      delay: 5000,
    },
    removeOnComplete: { count: 1000 },
    removeOnFail: { count: 500 },
  },
});

// Per-domain rate limiting enforced at enqueue time
async function enqueueScrapeJob(
  url: string,
  options: ScrapeOptions
): Promise<void> {
  const domain = new URL(url).hostname;

  await scrapeQueue.add(
    'scrape',
    { url, options, domain },
    {
      jobId: `${domain}-${Date.now()}-${Math.random()}`,
      rateLimiter: {
        max: 5,
        duration: 10000, // 5 requests per 10 seconds per domain
      },
    }
  );
}

Exponential backoff on retry means a job that gets blocked doesn't immediately hammer the target again. Three attempts with exponential backoff gives the site time to forget about you before you try again.

Resilience Over Speed

The broader point I keep coming back to with OpenClaw is that the instinct to optimize for speed is almost always wrong when you're building a scraper that needs to run for months. Speed gets you blocked. Resilience keeps you running.

This connects to something I've been thinking about as AI tooling gets more embedded in development workflows. Tools like Deltix are pushing AI-driven testing into production pipelines, and the same philosophy applies: you want systems that fail gracefully and recover predictably, not systems that are fast until they aren't. A scraper that runs at 60% speed indefinitely is more valuable than one that runs at 100% speed for two weeks before getting permanently banned.

The specific things that matter most for resilience in web scraping javascript-rendered pages:

Session rotation. Don't reuse browser contexts across domains. Don't reuse them for too many requests within a single domain either. Contexts are cheap. Bans are expensive.

Proxy health tracking. Proxies go bad. Track response codes per proxy per domain and retire proxies that start seeing elevated error rates before they get fully banned.

Circuit breakers per domain. If a domain starts returning consistent errors, stop hitting it immediately and alert. Don't let a bad domain drain your proxy pool.

Structured logging with enough context to debug. When a job fails, you need to know the fingerprint profile used, the proxy used, the exact URL, the response status, and the page content at time of failure. Logging job IDs without this context is useless.

The Concrete Recommendation

If you're starting a new scraper for javascript-rendered pages today: use Playwright with browser contexts (not full browser instances), build fingerprint profiles from real browser captures rather than generating random values, route everything through BullMQ or a similar queue with per-domain concurrency limits, and design for resilience from day one.

Don't start with raw speed as a goal. Start with "how do I keep this running for six months without manual intervention" and work backwards from there.

OpenClaw took several iterations to get to a place where it runs reliably without babysitting. Most of those iterations were removing clever optimizations that made things faster but fragile, and replacing them with slower, more predictable patterns. That's the actual lesson.

The Playwright documentation on browser contexts is worth reading in full if you're building anything serious. The section on context isolation specifically will save you from a class of fingerprinting mistakes that are hard to debug after the fact.