unclecode/crawl4ai

✨ Features

☁️ Same in the cloud: POST /scrape returns this Markdown, with no browser to run. Docs →

☁️ Same in the cloud: POST /extract, with no LLM key of your own. Docs →

☁️ Same in the cloud: up to 50 URLs in one streamed call, or 10,000 in a background job. Docs →

☁️ Rather not run a server? The cloud is the same idea, hosted. Get a key →

• 🧹 Clean Markdown: headings, lists, tables and code blocks, in a structure an LLM reads well.
• 🎯 Fit Markdown: filters remove menus, footers and boilerplate: PruningContentFilterLXML, BM25ContentFilter (for a query) and LLMContentFilter.
• 🔗 Citations: page links become a numbered reference list.
• 🛠️ Your own strategy: plug in a custom Markdown generator.
• 🔎 CSS and XPath schemas: fast extraction with no LLM (JsonCssExtractionStrategy, JsonXPathExtractionStrategy, RegexExtractionStrategy).
• 🪄 Schema generator: describe what you want once; generateschema writes a reusable schema.
• 🤖 LLM extraction: any LLM provider, open-source or hosted, into a typed JSON schema (LLMExtractionStrategy).
• 🧱 Chunking: topic, regex and sentence chunking for long pages.
• 🌌 Cosine similarity: find the chunks that match a query (CosineStrategy).
• 🖥️ Your own browser: persistent profiles with saved logins, cookies and settings.
• 🔄 Remote browsers: connect over the Chrome DevTools Protocol (CDP).
• 🔒 Sessions: keep a browser state across multi-step crawls.
• 🧩 Proxies: with authentication and rotation.
• 🕶️ Stealth mode: enablestealth, and an undetected-browser adapter for sites that detect automation.

🛠️ Installation

The server needs a token. Without one it answers only inside its container.

The dashboard is at http://localhost:11235/dashboard, the playground at http://localhost:11235/playground. LLM keys, MCP and every setting: Self-hosting guide.

pip install -U crawl4ai
crawl4ai-setup      # installs and sets up the browser
crawl4ai-doctor     # checks the installation

🔬 Advanced usage examples

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
from crawl4ai.content_filter_strategy import PruningContentFilterLXML
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator

async def main():
    run_config = CrawlerRunConfig(
        cache_mode=CacheMode.BYPASS,
        markdown_generator=DefaultMarkdownGenerator(
            content_filter=PruningContentFilterLXML(threshold=0.48, threshold_type="fixed", min_word_threshold=0)
 

Attribution Requirements

When using Crawl4AI, you must include one of the following attribution methods:

📈 1. Badge Attribution (Recommended) Add one of these badges to your README, documentation, or website:

• Disco Theme (Animated)
• Night Theme (Dark with Neon)
• Dark Theme (Classic)
• Light Theme (Classic)

☁️ Or use the cloud: no browsers, no proxies

The same key works for /extract, /search, /serp, /answer and many URLs at once (/scrape/batch, /scrape/jobs). Pay as you go: live prices. 3. Give it to your AI agent. Claude Code shown; Codex, Cursor and OpenCode →

• Verify your email and 1,000 free credits are yours every month. No card.
• Get any page as Markdown:

Which one?

I grew up on an Amstrad, thanks to my dad, and never stopped building. In grad school I specialized in NLP and built crawlers for research. That’s where I learned how much extraction matters.

In 2023, I needed web-to-Markdown. The “open source” option wanted an account, API token, and $16, and still under-delivered. I went turbo anger mode, built Crawl4AI in days, and it went viral. Now it’s the most-starred crawler on GitHub.

I made it open source for availability, anyone can use it without a gate. Now I’m building the platform for affordability, anyone can run serious crawls without breaking the bank. If that resonates, join in, send feedback, or just crawl something amazing.

• Runs the browsers — you, in your Python process — you, in Docker on your machine — we do
• JS-heavy pages and bot walls — your settings, your proxies — your settings, your proxies — handled for you, automatically
• Web search — – — – — /search, /serp and /answer
• Price — free, forever — free (your hosting) — 1,000 free credits a month; plans from $10
• LLM-ready output: smart Markdown with headings, tables, code and citation hints
• Fast in practice: async browser pool, caching, minimal hops
• Full control: sessions, proxies, cookies, user scripts, hooks
• Adaptive intelligence: learns site patterns, explores only what matters
• Deploy anywhere: no keys needed, CLI and Docker, or the hosted cloud

📖 Documentation

• Library docs, guides and API reference: docs.crawl4ai.com
• Cloud docs: crawl4ai.com/docs
• Release notes: releases · Roadmap: ROADMAP.md