Crawl a site with Scrapy

Run a Scrapy spider in a disposable microVM and export only its JSON result. This example targets Books to Scrape, a public practice site. Use Playwright instead when content appears only after JavaScript runs.

Crawl a site

Create the spider

books_spider.py

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/"]
    allowed_domains = ["books.toscrape.com"]
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 2,
        "DOWNLOAD_DELAY": 0.25,
        "CLOSESPIDER_PAGECOUNT": 10,
    }

    def parse(self, response):
        for book in response.css("article.product_pod"):
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css(".price_color::text").get(),
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Prepare Scrapy

msb run --name scrapy-base --replace \
  --memory 1G --root-disk 3G --max-duration 5m \
  python:3.13.14-alpine3.23 -- sh -lc \
    'mkdir -p /work && pip install --no-cache-dir scrapy==2.17.0'

msb run --name scrapy-base --replace `
  --memory 1G --root-disk 3G --max-duration 5m `
  python:3.13.14-alpine3.23 -- sh -lc `
    'mkdir -p /work && pip install --no-cache-dir scrapy==2.17.0'

Capture the prepared environment:

msb snap create scrapy-runtime \
  --sandbox scrapy-base --integrity

msb snap create scrapy-runtime `
  --sandbox scrapy-base --integrity

Verify the snapshot before using it:

msb snap verify scrapy-base:scrapy-runtime

The snapshot avoids reinstalling Scrapy for every crawl.

Crawl the site

The destination name must be unused. Restore installs the deny-by-default allowlist, connection cap, restricted guest profile, and lifetime bound before boot. Copy the spider into the root-owned /work directory and make it read-only before running it as the unprivileged user; no host directory is exposed.

msb snap restore scrapy-base:scrapy-runtime --name scrapy-books \
  --user 65534:65534 --memory 1G --max-duration 2m \
  --net-default deny \
  --net-rule 'allow@books.toscrape.com:tcp:443' \
  --max-connections 8 --security restricted
msb cp ./books_spider.py scrapy-books:/work/books_spider.py
msb exec --user root scrapy-books -- chmod 0444 /work/books_spider.py
msb exec --workdir /work --user 65534:65534 --env HOME=/tmp \
  --timeout 2m --rlimit fsize=8388608 \
  scrapy-books -- scrapy runspider books_spider.py \
    --loglevel WARNING -O /var/tmp/books.json
msb stop scrapy-books

msb snap restore scrapy-base:scrapy-runtime --name scrapy-books `
  --user 65534:65534 --memory 1G --max-duration 2m `
  --net-default deny `
  --net-rule 'allow@books.toscrape.com:tcp:443' `
  --max-connections 8 --security restricted
msb cp ./books_spider.py scrapy-books:/work/books_spider.py
msb exec --user root scrapy-books -- chmod 0444 /work/books_spider.py
msb exec --workdir /work --user 65534:65534 --env HOME=/tmp `
  --timeout 2m --rlimit fsize=8388608 `
  scrapy-books -- scrapy runspider books_spider.py `
    --loglevel WARNING -O /var/tmp/books.json
msb stop scrapy-books

The crawler and the microVM policy both constrain navigation to the target host. Change the spider, start URL, and network rule together when adapting the example, and respect the site’s terms and robots policy.

Copy out the result

mkdir -p .artifacts
msb cp scrapy-books:/var/tmp/books.json .artifacts/books.json

New-Item -ItemType Directory -Force .artifacts | Out-Null
msb cp scrapy-books:/var/tmp/books.json .artifacts/books.json

Inspect the number of collected records:

jq 'length' .artifacts/books.json

Treat scraped values as untrusted data when rendering HTML, building shell commands, or exporting spreadsheets.

Clean up

msb rm -f scrapy-base scrapy-books

Remove the reusable snapshot:

msb snap rm scrapy-base:scrapy-runtime

Overview

AI Agents

Dev Environments

CI/CD & Jobs

Browser & Web Automation

File Processing

Docker & System Tools

Data & Storage

Observability

Crawl a site with Scrapy

Crawl a site

Reference

​Crawl a site

​Reference

Crawl a site

Reference