Crawlee and Playwright are often mentioned together because Crawlee can use Playwright under the hood. The decision is really about how much crawling infrastructure you want from a framework versus how much direct browser control you need.
Minimal Playwright route check
Raw Playwright is useful when the output model is specific, such as classifying runtime errors separately from HTTP status.
import { chromium } from "playwright";
const browser = await chromium.launch();
const page = await browser.newPage();
const jsErrors = [];
page.on("pageerror", (error) => jsErrors.push(error.message));
const response = await page.goto("https://example.com", { waitUntil: "networkidle" });
console.log({
status: response?.status(),
title: await page.title(),
jsErrors
});
await browser.close();Crawler-style queue shape
If your crawler becomes mostly queue/retry/storage logic, a crawler framework may be the better foundation.
const queue = ["https://example.com"];
const seen = new Set();
while (queue.length) {
const url = queue.shift();
if (seen.has(url)) continue;
seen.add(url);
// Fetch or render the page, extract internal links,
// classify the result, then push new URLs onto queue.
}Builder decision table
| Need | Crawlee | Raw Playwright |
|---|---|---|
| Crawler queue and retries | Built in | You build or adopt your own |
| Browser rendering | Available through Playwright/Puppeteer crawlers | Native |
| Custom page classification | Possible | Direct and explicit |
| Memory lifecycle control | Framework-assisted | You own every detail |
| Fast prototype | Often faster | More setup |
| Product-specific crawler behavior | Good if framework fits | Best when behavior is unusual |
Why The Product Uses Direct Browser Control
Choose Crawlee when crawling infrastructure is the product
If the hard part is request queues, retries, storage, proxy/session handling, and crawler orchestration, Crawlee gives you a lot of useful structure.
That structure helps most with data extraction, broad crawling, and workflows where common crawler abstractions fit the job.
Choose Playwright when browser behavior is the product
Raw Playwright is attractive when every detail of page lifecycle, event capture, authentication, and classification needs to map directly into your product model.
A route-integrity crawler may choose direct Playwright control even when a crawler framework would speed up some plumbing.