CrawlForge MCP Server

MCPAI代理

提供方 mysleekdesigns · 包名 crawlforge-mcp-server

AI代理 stdio · 本地进程 OAuth 授权 社区 / 第三方 Claude DesktopClaudeCursorWindsurf
查看接入方式

28-tool MCP server for web scraping, crawling, deep research & autonomous extraction — clean Markdown & structured JSON for Claude, Cursor & any MCP client. 1,000 free one-time credits, local-Ollama LLM support.

一句话选型结论

按资料字段整理

CrawlForge MCP Server 属于AI代理类 MCP 服务器,stdio · 本地进程,OAuth 授权,可直接接入 Claude Desktop、Claude、Cursor 等,有可直接执行的安装命令。接入前先确认权限范围;涉及写入、删除或付款的操作,建议先在测试环境验证。

分类AI代理 传输方式stdio · 本地进程 鉴权OAuth 授权 部署方式本地部署
GitHub 星标 未标注 仓库热度,无公开仓库时显示未标注
工具数 未标注 暴露给模型的工具数量
资源 / 提示词 0 / 0 资源与提示词模板数量
维护方 社区 / 第三方 核验等级 仅资料核对

工具与能力

AI代理
数量来自来源目录的标注,具体工具清单以官方文档为准
工具数量未标注暴露给模型可调用的工具个数
资源数量0可读取的资源条目
提示词数量0内置提示词模板数量

能力标签

web scraping web crawler mcp server model context protocol data extraction content extraction deep research search api

支持客户端

适用场景

web scrapingweb crawlermcp server

安装与配置

stdio · 本地进程 OAuth 授权
客户端在本机拉起子进程,数据不出本机
npx -y crawlforge-mcp-server

这条命令只是入口,具体启动参数请按官方文档补齐,不要直接照搬到别的服务器。

  1. 1
    准备运行环境

    stdio 服务在本地起进程,需要先装好 node,再按安装命令拉取依赖。

  2. 2
    安装服务

    复制上面的安装命令执行;命令里带占位符的位置,换成自己的密钥或路径。

  3. 3
    写入客户端配置

    在客户端的 MCP 配置里填好启动命令和参数,保存后完全退出客户端再打开,让新配置生效。

  4. 4
    验证是否可用

    连接成功后随便调一个只读工具,确认能返回结果,再接入正式流程。

典型使用场景

3 类
按分类与标签整理的常见用法,具体以官方文档为准
给代理接外部能力

让代理在任务中途调用搜索、数据库等真实工具。

多步任务编排

把一串操作拆成步骤,由代理按顺序执行。

自动化重复流程

把固定流程交给代理跑,减少手工操作。

选择建议

选之前先看这几条
按条目字段给出的判断,不替你决定用不用

值得选的地方

  • 有现成的安装命令,不用自己找入口,照着跑就能起服务。
  • 已适配 Claude Desktop、Claude、Cursor,换客户端不用重新研究配置。

先想清楚的地方

  • stdio 方式要求本地装好运行时(如 Node.js),换机器要重新装一遍。
  • 使用前要准备好OAuth 授权对应的凭据,并按最小权限配置。
  • 资料没标注工具数量,实际能力要装好后确认。
  • 版本更新与行为细节以官方仓库为准。

更适合

  • 需要在AI代理场景里让模型直接调用外部能力的人。
  • 已经在用 Claude Desktop、Claude 的用户,接入路径最短。
  • 想先快速试一下、能接受在本地或自有环境跑服务的人。

不太适合

  • 完全不想在本地装环境的人。
  • 不方便管理密钥或第三方授权的人。
  • 需要厂商级 SLA 和技术支持的生产环境。

同类对比

6 个同类
同分类里挑最接近的几条,字段以各自详情页为准
MCP 服务器工具数传输方式部署方式安装命令最后核验
CrawlForge MCP Server当前 未标注 stdio · 本地进程 本地部署 有 未标注
Filesystem 560c4363 13 未标注 本地部署 无 2026-05-17
Server Everything (everything) 11 stdio · 本地进程 本地部署 有 2026-05-17
Time 7365cb6d 2 SSE · 服务端推送 远程部署 有 2026-05-18
Everything Claude Code 未标注 stdio · 本地进程 本地部署 有 2026-05-17
Openmemory 557e1585 未标注 stdio · 本地进程 本地部署 有 2026-05-17

服务器档案

AI代理
字段来自项目仓库与公开索引,中文说明由本站标注
提供方mysleekdesigns上游标注的维护方 mysleekdesigns
分类AI代理本站按上游分类展示
连接方式stdio · 本地进程客户端在本机拉起子进程,数据不出本机
鉴权方式OAuth 授权远程服务建议用最小权限凭据
维护方社区 / 第三方核验等级 仅资料核对
包名 / 运行时crawlforge-mcp-server · node本地安装时用到的包标识与运行时
部署类型本地部署上游给出的部署形态
最近核验未标注上游最后核验时间

详细介绍

26.5 KB
来自项目仓库与上游目录的原文,未做删改

31 web scraping, crawling, deep-research & autonomous-extraction tools for Claude, Cursor & any MCP client.

Clean Markdown & structured JSON from any site. Get started with 1,000 free credits — no credit card required.

⭐ Star us on GitHub to follow along — it genuinely helps others discover the project.

Table of Contents

  • Why CrawlForge?
  • CrawlForge vs. alternatives
  • Quick Start (2 Minutes)
  • Available Tools
  • Pricing
  • Advanced Configuration
  • Usage Examples
  • Security & Privacy
  • Support
  • Contributing

🎯 Why CrawlForge?

  • 31 MCP-native tools — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous agent, a unified multi-format scrape, document processing, stealth browsing, stateful browser sessions, and more, callable directly from your AI assistant.
  • Generous free tier — 1,000 credits to start instantly, no credit card. The grant is one-time rather than monthly, and the credits never expire.
  • Local-LLM by default — extract_with_llm runs against a local Ollama model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in.
  • LLM-ready output — clean Markdown, structured JSON (schema-driven), screenshots, links, and metadata from a single fetch.
  • Autonomous agent — describe what you need in natural language; it plans, gathers, and shapes an answer under orchestrator-enforced hard stops (max steps/URLs/wall-clock) — no URLs required.
  • Security-hardened — SSRF protection on every request, a fail-closed backend allow-list, a vetted action allowlist for browser automation, and per-tool credit gating.
  • Works everywhere MCP does — Claude Desktop, Claude Code, Cursor, and any other MCP-enabled client, configured in one command.

📊 CrawlForge vs. alternatives

CrawlForge MCP Firecrawl Raw scraping API
Native MCP server ✅ 31 tools ✅ ❌
Free tier ✅ 1,000 credits, one-time, never expire Limited Varies
Self-hosted / local LLM extraction (Ollama) ✅ default, $0/token ❌ ❌
Autonomous agent (no URLs needed) ✅ agent ✅ ❌
Deep research with source verification ✅ deep_research Partial ❌
Browser automation / actions ✅ scrape_with_actions ✅ Varies
Stealth / anti-detection engines ✅ Chromium + Camoufox ✅ Add-on
Pre-built site templates ✅ 10 sites ❌ ❌
License MIT AGPL-3.0 Proprietary

> Comparison reflects publicly documented capabilities at time of writing. CrawlForge is MIT-licensed and MCP-first — built to plug straight into AI coding assistants.

🚀 Quick Start (2 Minutes)

1. Install from NPM```bash

npm install -g crawlforge-mcp-server


### 2. Setup Your API Key (required)

Every tool requires a CrawlForge API key — new accounts get 1,000 free trial credits to start. The recommended path signs you in through the browser, so the key is never pasted into a terminal (a coding agent can run this for you and relay the URL):```bash
crawlforge login

It prints an approval URL; open it, approve, and the key is stored in ~/.crawlforge/config.json. Then run crawlforge init to register the MCP server with your client. Or use the interactive wizard, which also configures your clients:```bash
npx crawlforge-setup


This will:
- Guide you through getting your free API key
- Configure your credentials securely
- **Auto-configure Claude Code and Cursor** (if installed)
- Verify your setup is working

**Don't have an API key?** Get one free at https://www.crawlforge.dev/signup

> **One-step setup (v4.6.0+):** `crawlforge init` detects your API key, installs the agent skill, and idempotently merges the MCP config stanza into Claude Code, Claude Desktop, and Cursor. Use `crawlforge init --all --yes` to configure every detected client non-interactively.

### 3. Configure Your IDE (if not auto-configured)

🤖 For Claude Desktop

Add to `claude_desktop_config.json`:```json
{
  "mcpServers": {
    "crawlforge": {
      "command": "npx",
      "args": ["-y", "crawlforge-mcp-server"]
    }
  }
}

Location:
- macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
- Windows: %APPDATA%/Claude/claude_desktop_config.json
- Linux: ~/.config/Claude/claude_desktop_config.json

Restart Claude Desktop to activate.

🖥️ For Claude Code CLI (Auto-configured)

The setup wizard automatically configures Claude Code by adding to ~/.claude.json:```json
{
"mcpServers": {
"crawlforge": {
"type": "stdio",
"command": "crawlforge-mcp"
}
}
}


After setup, restart Claude Code to activate.

💻 For Cursor IDE (Auto-configured)

The setup wizard automatically configures Cursor by adding to `~/.cursor/mcp.json`:```json
{
  "mcpServers": {
    "crawlforge": {
      "type": "stdio",
      "command": "crawlforge-mcp"
    }
  }
}

Restart Cursor to activate.

🔁 For n8n (workflow automation)

n8n's built-in MCP Client Tool node connects over Streamable HTTP (works on n8n Cloud and self-hosted). Run the server in HTTP mode:```bash
export CRAWLFORGE_API_KEY=your_api_key
npm run start:http # Streamable HTTP endpoint at http://localhost:10000/mcp


Then point the MCP Client Tool node at `http://:10000/mcp` with transport **HTTP Streamable** and a **Bearer** credential set to the same API key. On self-hosted n8n you can instead use the community `n8n-nodes-mcp` node over STDIO (`npx -y crawlforge-mcp-server`).

Full guide: docs/n8n-integration.md

> **Which launch command?** `npx -y crawlforge-mcp-server` needs no global install and always runs the published version (recommended for Claude Desktop). For a global install (`npm i -g crawlforge-mcp-server`), use the dedicated `crawlforge-mcp` bin — it resolves on your `PATH`, so it survives Node/nvm version switches. The bare `crawlforge` command still launches the server when an MCP client spawns it over stdio (backward compatibility for configs created before v4.2.5); interactively it's the CLI — run `crawlforge mcp` to start the server by hand.

## 📊 Available Tools

CrawlForge requires a CrawlForge API key — **every tool is metered and consumes credits**. New accounts get **1,000 free trial credits** to start. Get a key at crawlforge.dev/signup.

**All Tools** (API key required)

Tool · Credits · What it does
`fetch_url` · 1 · Fetch content from any URL
`extract_text` · 1 (6 projected with `escalate: true`, charged 1 when the plain fetch works) · Extract clean text from web pages
`extract_links` · 1 (6 projected with `escalate: true`, charged 1 when the plain fetch works) · Get all links from a page
`extract_metadata` · 1 · Extract page metadata (title, OG tags, schema.org)
`scrape_template` · 1 · Structured data from well-known sites (Amazon, GitHub, LinkedIn, YouTube, Reddit, Hacker News, npm, and more) without writing selectors
`list_ollama_models` · 1 · List the Ollama models installed locally (helps you pick a `model` for `extract_with_llm`)
`get_batch_results` · 1 · Retrieve paginated results for a `batch_scrape` job by `batchId`
`read_result` · 1 · Search, slice, read lines or a JSON path from a result a tool returned with `truncated: true` and a `result_handle` (kept 1 hour on your own machine) — never fetch the page again
`scrape` · 2 · **Unified single-fetch, multi-format extraction.** Pass a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema, plus `{type:"highlights",query}` and `{type:"question",question}` for only the matching sentences, table rows and code blocks, verbatim with offsets into the markdown: +1 credit once per call, `mode:"model"` +3) plus `onlyMainContent`; one fetch serves every requested format with per-format partial-success warnings. `escalate:true` retries a blocked plain fetch once in the stealth browser and returns the page instead of the block (+5 as the projected ceiling; the charge drops back to 2 when the plain fetch succeeded and no escalation ran)
`scrape_structured` · 2 · Extract structured data with CSS selectors
`extract_embedded_state` · 2 (7 projected with `escalate: true`, charged 2 when the plain fetch works) · Read a page's embedded JavaScript state — `__NEXT_DATA__`, React Server Component payloads (references between rows resolved, plus a `data_rows` index), Nuxt (Nuxt 3's `__NUXT_DATA__` decoded), SvelteKit, Apollo, Redux, `ytInitialData`, Inertia, Shopify, JSON `data-*` attributes, `` — with a `path` to scope the result, `keys_only: true` to see its first two levels of keys, or `find: ""` to get every path where a field of that name lives. `escalate: true` re-reads a walled page in the stealth browser, runs the same parser and also reads the framework globals off `window` (`window_state`). No LLM in the extraction path
`extract_content` · 2 · Enhanced content extraction
`map_site` · 2 · Discover and map website structure (optional `search=` ranks the discovered URLs)
`process_document` · 2 · Multi-format document processing
`localization` · 2 · Look up a country's locale settings (Accept-Language, timezone, currency) to pass to other tools; returns values, applies none
`track_changes` · 3 · Monitor content changes over time
`analyze_content` · 3 · Comprehensive content analysis
`extract_structured` · 3 · LLM-powered schema-driven extraction (your own LLM key or local Ollama)
`extract_with_llm` · 3 · Natural-language extraction. Defaults to a local Ollama model; pass `provider: "openai" \ · "anthropic"` with the matching key for cloud models (external LLM billed by your provider)
`browser_session` · 3 · **A browser page that stays open across calls**, keeping its cookies and its login in between. `open` a session on a URL, `snapshot` it to list the interactive elements as stable refs (`@e1`, `@e2`), `act` on a ref, `read` the content, `close`. Priced per operation: open 3, read 2, snapshot/act/screenshot/close/list 1 each. Reach for it when you must see the page before choosing what to click, when the flow spans several calls, or when a login must hold across later reads; one fixed action chain on one page is `scrape_with_actions`
`summarize_content` · 4 · Generate intelligent summaries
`crawl_deep` · 4 · Deep crawl entire websites
`search_web` · 5 · Search the live web (Brave Search API, through the CrawlForge API)
`reddit_search` · 5 · Search Reddit posts/comments or read a full thread — reddit.com blocks direct scraping, so this reads the Arctic Shift community archive (free, no Reddit credentials). A Reddit-wide search spends a web search to discover posts, so it is priced with `search_web`. Optional: with your own Reddit app credentials (`REDDIT_CLIENT_ID`/`REDDIT_CLIENT_SECRET`), posts searches and thread reads come from Reddit's official API first (live scores, complete comment trees), with the archives as fallback; same price. Reddit no longer issues new credentials without an approval ticket, see docs/reddit-access-and-oauth.md
`serp_rank` · 5 · Check where a domain ranks in Google's **real organic SERP** for a keyword (the position `search_web` can't give). Powered by DataForSEO (`DATAFORSEO_LOGIN`/`DATAFORSEO_PASSWORD`, billed to your own DataForSEO account). Returns `{ configured:false }` and charges **0** credits until configured
`batch_scrape` · 5 · Process multiple URLs simultaneously
`scrape_with_actions` · 5 · Browser automation chains
`generate_llms_txt` · 5 · Generate AI interaction guidelines
`stealth_mode` · 5 · Anti-detection browser management
`agent` · 8 (+5 per stealth retry that gets the page, max 2) · **Autonomous research/extraction from a natural-language prompt — no URLs required.** Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM). A page that walls the plain fetch is retried in the stealth browser automatically — URLs you name first — and that evidence is marked `via: "stealth"`; each retry that runs adds 5
`deep_research` · 10 · Multi-stage research with source verification

Ten tools (`scrape`, `fetch_url`, `extract_content`, `crawl_deep`, `batch_scrape`, `stealth_mode`, `scrape_with_actions`, `process_document`, `deep_research`, `extract_embedded_state`) accept `max_inline_chars` (default 40,000; env `CRAWLFORGE_MAX_INLINE_CHARS`): a result over it comes back as a `preview` plus a `result_handle` for `read_result`, with the full result kept for 1 hour under `~/.crawlforge/results/` on your own machine — nothing is uploaded.

For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see SKILL.md.

↑ Back to top

## 💳 Pricing

**Every tool is metered and requires an API key.** New accounts get 1,000 free trial credits — no credit card required to start.

| Plan | Credits | Best For |
|------|---------|----------|
| **Free** | 1,000 one-time | Testing & personal projects |
| **Hobby** ($19) | 5,000 / month | Small projects & development |
| **Professional** ($99) | 50,000 / month | Professional use & production |
| **Business** ($399) | 250,000 / month | Large scale operations |

**All plans include:**
- Access to all 31 tools
- Credits never expire; paid-plan credits roll over month to month
- API access and webhook notifications

View full pricing

## 🔧 Advanced Configuration

### Environment Variables```bash
# Optional: Set API key via environment
export CRAWLFORGE_API_KEY="cf_live_your_api_key_here"

# Optional: Custom API endpoint (for enterprise)
export CRAWLFORGE_API_URL="https://api.crawlforge.dev"
# As of v3.0.18, this variable is validated against an allow-list of CrawlForge backend hosts.

# Optional: Local LLM (Ollama) overrides — extract_with_llm, extract_structured
# and deep_research all use Ollama when no cloud key is set
export OLLAMA_BASE_URL="http://localhost:11434"   # default; set https://ollama.com for Ollama Cloud
export OLLAMA_DEFAULT_MODEL="gemma3:4b"            # optional; unset = pick the best installed model automatically
                                                   # deep_research judges claims with gemma3:12b when it is installed (ollama pull gemma3:12b);
                                                   # conflict detection is on only with that model, or a cloud provider
export OLLAMA_EMBEDDING_MODEL="nomic-embed-text"   # default: OLLAMA_DEFAULT_MODEL; used for semantic ranking in deep_research
export OLLAMA_API_KEY="..."                        # only for authenticated endpoints (required by Ollama Cloud; a local instance needs none)
export DISABLE_OLLAMA="true"                       # skip Ollama entirely and use CSS/keyword fallbacks

# Optional: Cloud LLM keys — only needed when you pass provider: "openai" or "anthropic"
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."

# Optional: limit which tools this client sees — by name, by group, or both (comma-separated)
export CRAWLFORGE_TOOLS="scrape,search_web,extract_content"
export CRAWLFORGE_TOOL_GROUPS="basic,search,scrape"   # unset = all tools; unknown names/groups are ignored with a warning

# Optional: deep_research stealth extraction fallback (v4.6.6) — see below
export RESEARCH_STEALTH_ENGINE="auto"      # auto (default) | camoufox | chromium
export RESEARCH_STEALTH_FALLBACK="true"    # set to "false" to disable entirely
export RESEARCH_MAX_STEALTH_RETRIES="8"    # cap on stealth retries per research run

# Optional: your own proxies for the stealth paths that have no caller to ask —
# the scrape escalation stage, the deep_research retry, the agent, browser_session.
# Comma-separated; a proxy passed on a call always wins. CrawlForge supplies none.
export CRAWLFORGE_STEALTH_PROXIES="http://user:pass@gw.provider.net:8080"
# Unrelated to the PROXY_ROTATION_* variables, which belong to the localization tool.

# Optional: turn off the clearance jar (on by default) — see "Stealth engines and proxies"
export CRAWLFORGE_CLEARANCE_JAR="off"

# Optional: also refuse private networks (RFC1918, ULA, CGNAT, ...) — see "Security & Approvals"
export SSRF_STRICT="true"

# Optional: the only directories process_document may read local files from (":"-separated, ";" on Windows)
export CRAWLFORGE_ALLOWED_DIRS="$HOME/Documents:$HOME/Downloads"

MCP Spec Features

CrawlForge tracks the current MCP spec (2025-06-18) plus select experimental extensions:

  • Structured output — scrape, map_site, serp_rank, reddit_search, search_web, extract_structured, and crawl_deep return machine-parseable structuredContent alongside the usual text, validated against a published outputSchema; legacy clients keep working off the text.
  • Self-correctable errors — invalid tool input now comes back as an isError: true result the calling model can read and retry from, instead of a raw JSON-RPC protocol error.
  • JSON Schema 2020-12 tool schemas, deterministic tools/list ordering (client prompt-cache friendly), and cacheable-result hints on read-only tools.
  • Icons on the server, its tools, and its prompts.
  • Async tasks (experimental) on the four long-running tools — crawl_deep, batch_scrape, deep_research, agent — for clients that support polling; synchronous results are still returned for clients that don't.

See docs/mcp-spec-adoption.md for wire-level examples and client-compatibility notes.

Local-LLM quickstart (extract_with_llm with Ollama)

extract_with_llm defaults to a local Ollama model — no LLM-provider key, no per-token LLM costs, and no data leaving your machine (the CrawlForge credit cost still applies).```bash

1. Install Ollama: https://ollama.com

2. Pull any model from https://ollama.com/library

ollama pull llama3.2

3. Discover what's installed (from your MCP client)

list_ollama_models()

4. Extract — defaults to Ollama with the model from step 2

extract_with_llm({ url: "https://example.com", prompt: "…", model: "llama3.2" })


### Stealth engines and proxies

Every stealth entry point — `scrape` with `escalate: true`, `stealth_mode`, `browser_session`, `scrape_with_actions` and the `deep_research` retry — now defaults to **`auto`**: use Camoufox (Firefox anti-detect) when its binary is installed, fall back to Chromium when it is not, with the reason in the result's `warnings[]`. Every result names the engine that actually ran. Naming an engine explicitly behaves as it always has — `camoufox` fails rather than falling back, and `chromium` and `playwright` are the same engine (on `scrape`'s `escalate_engine`, spell it `playwright`).

On `browser_session` and `scrape_with_actions` the engine applies only with `stealth: true` — their ordinary path is the standard Chromium pool.

`auto` is not free. Measured once each on an Apple Silicon Mac on 2026-09-21 (launch plus a context and a blank page): Chromium 148 ms and 253 MB, Camoufox 946 ms and 667 MB — roughly **+0.8 s and +400 MB per stealth call**. Pin `engine: "playwright"` for high-volume work on sites that do not block.

**CrawlForge supplies no proxies**, and a datacenter proxy does not fix a block: Cloudflare scores the IP's ASN and the TLS/HTTP2 handshake before any JavaScript runs, so no browser-side patch compensates for a datacenter address. Bring your own residential exit with `CRAWLFORGE_STEALTH_PROXIES` (comma-separated URLs), which the escalation stage, `stealth_mode`, `browser_session`, `scrape_with_actions`, the `deep_research` retry and the `agent` tool's automatic stealth retry use when the caller passes none; a proxy passed on the call always wins. With a proxy, Camoufox derives its timezone, locale and geolocation from the exit IP.

**A challenge solved once is not solved again.** When a stealth render gets past a Cloudflare or DataDome wall, the vendor's clearance cookies (`cf_clearance`, `__cf_bm`, `datadome` — nothing else) are kept in `~/.crawlforge/stealth-clearance.json` (mode 0600) and replayed to the next stealth context with the same engine, User-Agent and proxy, for at most the cookie's own lifetime and never more than 24 hours. A render that still meets the wall drops them. Set `CRAWLFORGE_CLEARANCE_JAR=off` to disable.

Full detail, measurements and sources: docs/stealth-engines.md.

### Stealth extraction for `deep_research` (Camoufox)

`deep_research` automatically retries sources that block the normal fetch path (Reddit, Quora, forums, and Cloudflare/DataDome-protected pages return HTTP 403) through a **real fingerprinted browser**, then re-extracts from the rendered HTML. It's bounded (`RESEARCH_MAX_STEALTH_RETRIES`, default 8, plus a per-page timeout) and lazy — the browser stack only loads when a source is actually blocked.

Engine selection (`RESEARCH_STEALTH_ENGINE`):

- **`auto`** (default) — prefer **Camoufox** (Firefox anti-detect), fall back to Chromium stealth, then plain fetch.
- **`camoufox`** — force Camoufox.
- **`chromium`** — force the Chromium stealth engine.

Camoufox gets through walls that stop headless Chromium: in a benchmark run on 2026-09-21 from a residential IP it was the only engine to clear Cloudflare Turnstile (indeed.com) and Akamai (harrods.com), both of which blocked Chromium stealth. Neither engine cleared DataDome or an interactive Turnstile, so this is a better fetch path, not a bypass. To enable it, install the optional dependency and run its one-time binary fetch:```bash
# Camoufox is declared as an optional dependency, so a normal install already pulls it.
# If you installed with --no-optional, add it explicitly:
npm install camoufox

# One-time download of the Camoufox Firefox binary (~130 MB):
npx camoufox fetch

Without the Camoufox binary, deep_research silently falls back to Chromium stealth and then to plain fetch — no errors, just lower recovery on heavily-protected sites. Disable the whole fallback with RESEARCH_STEALTH_FALLBACK=false.

> Note: Hard IP-reputation blocks (e.g. Reddit's edge 403) resist headless stealth from any IP and require residential/mobile proxies, which CrawlForge does not provide — point the server at your own with CRAWLFORGE_STEALTH_PROXIES. See docs/stealth-engines.md for details.

Manual Configuration

Your configuration is stored at ~/.crawlforge/config.json:```json
{
"apiKey": "cf_live_...",
"userId": "user_...",
"email": "you@example.com"
}


## 📖 Usage Examples

Once configured, use these tools in your AI assistant:```
"Search for the latest AI news"
"Extract all links from example.com"
"Crawl the documentation site and summarize it"
"Monitor this page for changes"
"Extract product prices from this e-commerce site"

🔒 Security & Privacy

  • Secure Authentication: API keys required for all metered tools
  • Local Storage: API keys stored securely at ~/.crawlforge/config.json
  • HTTPS Only: All connections use encrypted HTTPS
  • No Data Retention: We don't store scraped data, only usage logs
  • Rate Limiting: Built-in protection against abuse
  • Compliance: Respects robots.txt and GDPR requirements

Security & Approvals

  • SSRF enforcement: Every scraped URL is validated before the request is sent, and every connection (each redirect hop included, capped at 5) is checked against the address it actually connects to — http/https only, cloud metadata hostnames (GCP, Azure) refused. Two levels:
  • Default (Stage 1) blocks loopback (127.0.0.0/8, ::1), the unspecified addresses (0.0.0.0/8, ::), link-local and cloud metadata (169.254.0.0/16 incl. 169.254.169.254, fe80::/10). It allows private networks (RFC1918 10/8, 172.16/12, 192.168/16, IPv6 ULA): run on your own machine, the server acts for you, and reading a router page, a NAS or an intranet docs site on your LAN is a legitimate use that the stricter level would break.
  • SSRF_STRICT=true adds every private and reserved range: RFC1918, ULA (fc00::/7), CGNAT (100.64/10), multicast, reserved, IPv4-compatible (::/96) and NAT64 (64:ff9b::/96) IPv6. Set it if an agent that reads untrusted pages (and could be prompt-injected into fetching a LAN address) should not reach your local network, or whenever the server is shared.
  • Local files (process_document with sourceType file or pdf_file; refused outright when the server is served to a network): a path is judged by its realpath, so a symlink counts as its target. Only .pdf, .docx, .txt, .md, .html and .htm files are read, and nothing inside a dot-directory (~/.ssh, ~/.aws, ~/.config, ~/.crawlforge, ...). CRAWLFORGE_ALLOWED_DIRS (directories separated by :, ; on Windows) narrows reads to those directories instead; inside them the dot-directory rule is lifted, the extension rule is not.
  • Backend endpoint guard (v3.0.18): The server's own calls to CrawlForge.dev use a separate fail-closed allow-list ({crawlforge.dev, www.crawlforge.dev, api.crawlforge.dev}, HTTPS required). Setting CRAWLFORGE_API_URL to an arbitrary host is blocked at parse time.
  • Action allowlist: scrape_with_actions accepts only 11 action types (snapshot, wait, click, type, press, scroll, screenshot, executeJavaScript, select, hover, navigate). No download, file-write, or arbitrary cross-page navigation primitives exist — navigate goes through the same SSRF and robots.txt gate as the initial URL.
  • JavaScript gate: The executeJavaScript action throws by default. Set ALLOW_JAVASCRIPT_EXECUTION=true at deploy time to enable (not recommended in production).
  • MCP Elicitation (v3.6.0): Four tools request user confirmation before executing expensive operations — deep_research (>50 URLs), batch_scrape (sync mode, >25 URLs), crawl_deep (projected >500 pages), extract_structured (schema has >3 required fields with no LLM configured). Credit-low situations also elicit. Confirmation is best-effort: if the MCP client does not support elicitation the tool proceeds (fail-open).
  • Per-tool credit gating: Every tool is wrapped with withAuth() and is metered — credits are checked and deducted before execution, and a valid API key is required for every tool (fail-closed since v3.0.18).

See docs/sandboxing-and-approvals.md for the full reference.

Security Updates

v3.0.3 (2025-10-01): Removed authentication bypass vulnerability. All users must authenticate with valid API keys.

For the full security policy and how to report a vulnerability, see SECURITY.md.

↑ Back to top

🆘 Support

  • Documentation: https://www.crawlforge.dev/docs
  • Issues: GitHub Issues
  • Email: support@crawlforge.dev
  • Discord: Join our community

📄 License

MIT License - see LICENSE file for details.

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide first.


Built with ❤️ by the CrawlForge team

Website · Documentation · API Reference

常见问题

8 问
答案由本站依据条目字段整理,不引用上游原文
CrawlForge MCP Server 怎么安装?

在客户端按下面的安装命令接入:npx -y crawlforge-mcp-server。对应的包名是 crawlforge-mcp-server。装完在客户端配置里引用同一个包或端点。

CrawlForge MCP Server 怎么在 Claude Desktop、Claude、Cursor 里配置?

它的传输方式是stdio · 本地进程,在客户端的 MCP 配置里填好启动命令和参数,保存后完全退出客户端再打开,配置才会生效。

CrawlForge MCP Server 收费吗?

本站不代收费用,也没有标注具体价格。是否收费取决于服务本身和背后的接口额度,正式使用前按官方说明确认。

CrawlForge MCP Server 需要 API Key 吗?

资料标注需要OAuth 授权认证,使用前准备好对应凭据,并按最小权限配置。

CrawlForge MCP Server 支持哪些客户端?

资料标注支持 Claude Desktop、Claude、Cursor、Windsurf、Codex、Zed;其他客户端只要能配置 MCP Server,也能用同样的命令或地址接入。

CrawlForge MCP Server 能提供哪些能力?

资料显示能力标签包括 web scraping、web crawler、mcp server、model context protocol、data extraction、content extraction。具体工具清单以官方文档为准。

CrawlForge MCP Server 需要什么运行环境?

运行时为 node;属于本地进程,需要在本机装好运行环境。

CrawlForge MCP Server 是本地还是远程?

它是本地进程(stdio),由客户端在本机拉起。传输方式为stdio · 本地进程。

热门专题

按标签挑服务器
全部专题 →

服务器信息以作者仓库与官方文档为准;分类、传输方式与鉴权方式的中文说明为中文整理,最近核验 未标注。