crawler

firecrawl/firecrawl

Turning the web into clean, LLM-ready context at scale — Firecrawl is an open-source API for crawling, scraping and searching the web, converting pages into markdown or structured data for AI agents. It covers a high share of sites reliably and is offered both self-hosted and as a managed cloud service, making it a building block for LLM and agent pipelines.

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

D4Vinci/Scrapling

A Python scraping framework built for stealth and adaptive selectors — Scrapling combines request handling, Playwright-based crawling, and anti-bot stealth in one framework, with AI-assisted scraping and an MCP server for agent integration. It targets developers building scrapers that need to survive site changes and detection, from single requests to full-scale crawls.

🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

ScrapeGraphAI/Scrapegraph-ai

LLM-driven scraping pipelines that extract data by description, not selectors — ScrapeGraphAI builds web scraping pipelines using LLMs and graph logic instead of hand-written selectors. You describe what data you want from a site or local document (XML, HTML, JSON, Markdown), and it extracts it automatically. Useful for developers who need resilient extraction without maintaining fragile scraping rules, with integrations across Python/Node.js frameworks.

Python scraper based on AI

feder-cr/invisible_playwright_mcp

A Playwright MCP agent built to slip past anti-bot defenses — This MCP server drives a stealth Firefox browser that AI agents like Claude Code or Gemini CLI can use to browse without triggering anti-bot systems or captchas. It plugs into assistants via MCP for tasks like job hunting, scraping, and general computer use.

Playwright MCP server undetected by anti-bots and captchas: AI agent browses the web on anti-detect stealth Firefox, Python, undetected browser automation, scraping, computer use.

apify/crawlee

A Node.js crawling toolkit that blends HTTP and browser scraping — Crawlee unifies Puppeteer, Playwright, Cheerio, JSDOM and raw HTTP under one API, letting developers switch between headless browsers and lightweight HTTP parsing as needed. Built-in proxy rotation and human-like request patterns help crawlers avoid bot detection, useful for feeding data to AI, LLM or RAG pipelines.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

projectdiscovery/katana

A fast Go crawler that adds headless JS parsing and ML page classification — Katana is a CLI web crawler built for security recon, supporting both standard and headless modes with JavaScript parsing, automatic form filling, and scope control via regex. It adds a knowledge-base feature using an auto-downloaded ML model to classify pages and forms, aimed at pentesters and bug bounty hunters needing flexible, scriptable crawling output.

A next-generation crawling and spidering framework.

s0md3v/Photon

Incredibly fast crawler designed for OSINT.

Tuhinshubhra/RED_HAWK

All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers

Autumn-27/ScopeSentry

Distributed recon platform that reacts fast when new vulns drop — Scope Sentry combines asset mapping, subdomain enumeration, directory scanning, crawling, and page monitoring across multiple distributed nodes. Users can pick nodes to run scans and quickly check whether existing assets are affected when new vulnerabilities emerge. Built for bug bounty hunters and security teams, with a plugin marketplace for extending scans.

ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes

iudicium/pryingdeep

Prying Deep - An OSINT tool to collect intelligence on the dark web.

RayenMlayeh/jevcrawl

A crawler that lets a small decision model find the answer, not just pages — jevcrawl uses a lightweight decision model instead of an LLM to pick links and passages, aiming to surface the exact answer page for AI agents. On WebWalkerQA benchmarks it beats Crawl4AI's goal-directed crawl and an LLM-driven crawl on accuracy, speed, and cost per task.

MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.