whatisgithub

What is scrapy-pyppeteer?

elacuesta/scrapy-pyppeteer — explained in plain English

Analysis updated 2026-07-18 · repo last pushed 2021-02-26

★ 58PythonAudience · developerComplexity · 3/5DormantSetup · moderate

In one sentence

Scrapy-Pyppeteer lets the Scrapy web scraping framework use a headless browser to render JavaScript-heavy sites, so you can scrape pages that build their content dynamically in the browser.

Mindmap

mindmap
  root((repo))
    What it does
      Renders JS with browser
      Feeds HTML to Scrapy
      Clicks and scrolls pages
    Tech stack
      Python
      Scrapy
      Pyppeteer
    Use cases
      Scrape single-page apps
      Scrape social feeds
      Automate page actions
    Audience
      Web scraper developers
    Status
      No longer maintained
      Use scrapy-playwright instead

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

What do people build with it?

USE CASE 1

Scrape a JavaScript-heavy single-page application by letting a real browser render it first.

USE CASE 2

Extract content from social media feeds or infinite-scroll pages that load data dynamically.

USE CASE 3

Automate page actions like clicking, scrolling, or taking screenshots before scraping the final HTML.

What is it built with?

PythonScrapyPyppeteer

How does it compare?

elacuesta/scrapy-pyppeteeramazinghorseli/rednote-insightlittlepeachs/naturepanelforge
Stars585858
LanguagePythonPythonPython
Last pushed2021-02-26——
MaintenanceDormant——
Setup difficultymoderatemoderatemoderate
Complexity3/53/54/5
Audiencedeveloperpm founderresearcher

Figures from each repo's GitHub metadata at analysis time.

How do you get it running?

Difficulty · moderate Time to first run · 30min

Project is no longer actively maintained, maintainers recommend scrapy-playwright for new projects.

Copy-paste prompts

Prompt 1
Help me configure a Scrapy spider to use scrapy-pyppeteer for a JavaScript-rendered page.
Prompt 2
Show me how to make scrapy-pyppeteer wait for a specific element before returning the page HTML.
Prompt 3
Explain how to add meta={"pyppeteer": True} to a Scrapy request to trigger browser rendering.
Prompt 4
Since this project isn't maintained, help me migrate my scrapy-pyppeteer spider to scrapy-playwright.

Frequently asked questions

What is scrapy-pyppeteer?

Scrapy-Pyppeteer lets the Scrapy web scraping framework use a headless browser to render JavaScript-heavy sites, so you can scrape pages that build their content dynamically in the browser.

What language is scrapy-pyppeteer written in?

Mainly Python. The stack also includes Python, Scrapy, Pyppeteer.

Is scrapy-pyppeteer actively maintained?

Dormant — no commits in 2+ years (last push 2021-02-26).

How hard is scrapy-pyppeteer to set up?

Setup difficulty is rated moderate, with roughly 30min to a first successful run.

Who is scrapy-pyppeteer for?

Mainly developer.

Open on GitHub → Ask about another repo

This repo across BitVibe Labs

Verify against the repo before relying on details.

What are similar repos?