Crawl a site with Scrapy

Run a Scrapy spider in a disposable microVM and export only its JSON result. This example targets Books to Scrape, a public practice site. Use Playwright instead when content appears only after JavaScript runs.

Crawl a site

Create the spider

books_spider.py

import scrapy


class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/"]
    allowed_domains = ["books.toscrape.com"]
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 2,
        "DOWNLOAD_DELAY": 0.25,
        "CLOSESPIDER_PAGECOUNT": 10,
    }

    def parse(self, response):
        for book in response.css("article.product_pod"):
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css(".price_color::text").get(),
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Prepare Scrapy

msb run --name scrapy-base --replace \
  --memory 1G --root-disk 3G --max-duration 5m \
  python:3.13.14-alpine3.23 -- sh -lc \
    'mkdir -p /work && pip install --no-cache-dir scrapy==2.17.0'

msb run --name scrapy-base --replace `
  --memory 1G --root-disk 3G --max-duration 5m `
  python:3.13.14-alpine3.23 -- sh -lc `
    'mkdir -p /work && pip install --no-cache-dir scrapy==2.17.0'

Capture the prepared environment:

msb snapshot create scrapy-runtime \
  --from scrapy-base --integrity --force

msb snapshot create scrapy-runtime `
  --from scrapy-base --integrity --force

Verify the snapshot before using it:

msb snapshot verify scrapy-runtime

The snapshot avoids reinstalling Scrapy for every crawl.

Crawl the site

msb run --name scrapy-books --replace \
  --from-snapshot scrapy-runtime \
  --mount-file ./books_spider.py:/work/books_spider.py:ro \
  --workdir /work --user 65534:65534 --env HOME=/tmp \
  --memory 1G --max-duration 2m \
  --net-default deny \
  --net-rule 'allow@books.toscrape.com:tcp:443' \
  --max-connections 8 --security restricted \
  --rlimit fsize=8388608 \
  -- scrapy runspider books_spider.py \
    --loglevel WARNING -O /var/tmp/books.json

msb run --name scrapy-books --replace `
  --from-snapshot scrapy-runtime `
  --mount-file ./books_spider.py:/work/books_spider.py:ro `
  --workdir /work --user 65534:65534 --env HOME=/tmp `
  --memory 1G --max-duration 2m `
  --net-default deny `
  --net-rule 'allow@books.toscrape.com:tcp:443' `
  --max-connections 8 --security restricted `
  --rlimit fsize=8388608 `
  -- scrapy runspider books_spider.py `
    --loglevel WARNING -O /var/tmp/books.json

The crawler and the microVM policy both constrain navigation to the target host. Change the spider, start URL, and network rule together when adapting the example, and respect the site’s terms and robots policy.

Copy out the result

mkdir -p .artifacts
msb cp scrapy-books:/var/tmp/books.json .artifacts/books.json

New-Item -ItemType Directory -Force .artifacts | Out-Null
msb cp scrapy-books:/var/tmp/books.json .artifacts/books.json

Inspect the number of collected records:

jq 'length' .artifacts/books.json

Treat scraped values as untrusted data when rendering HTML, building shell commands, or exporting spreadsheets.

Clean up

msb rm -f scrapy-base scrapy-books

Remove the reusable snapshot:

msb snapshot remove scrapy-runtime

Agents

CI/CD

Docker

Sandbox Workflows

Automation & Tooling

Data & Observability

Crawl a site with Scrapy

Crawl a site

Reference

​Crawl a site

​Reference

Crawl a site

Reference