What is anansi?

Anansi is a Python web scraping toolkit built for sites that actively try to block scrapers. The core idea is that the web changes constantly and fights back, so the scraper should adapt on its own instead of breaking every time a page layout shifts. When a site redesigns a page, Anansi does not just fail. It keeps a confidence score for each CSS selector it uses, and if one stops working it tries several backup strategies in turn: matching by the surrounding text, finding similarly named classes, following the structure of nearby elements, or falling back to XPath. Whichever approach works gets remembered for next time. The tool also pulls structured data straight from JSON-LD, Open Graph tags, and Microdata when a page offers it, skipping selector guesswork entirely in those cases. For sites that block bots, Anansi can mimic Chrome's network fingerprint so its traffic looks like a real browser, and it will automatically switch to a real headless browser (Playwright) when a page needs JavaScript to render, including handling Cloudflare style challenges. It also adjusts how fast it crawls based on how many errors it is getting, slowing down after repeated failures and speeding back up once things are clean. Repeat crawls are efficient: Anansi remembers what it already fetched using ETags and content hashes, so unchanged pages are skipped without even loading them again. Extracted data can be checked against a Pydantic schema before it gets saved, catching bad results early. The crawler itself runs on asyncio, stores its queue in SQLite so a crawl can be paused and resumed later, and supports rotating through a pool of proxies. A notable extra is a built in MCP server, which exposes the scraper's fetch, extract, and crawl functions as tools so an AI assistant can run a crawl through conversation instead of writing code. Anansi installs from its GitHub repository with pip, with optional extras for the browser and fingerprinting features. It is still an early project, with 61 stars at time of writing.