pagesift 1.6.0
Local-first web & document ingestion engine in Rust — HTTP/browser capture, layout-aware PDF/Office/OCR parsing, persistent sync, verifiable data graphs, and evidence-grounded AI context.
One self-hosted Rust binary with a CLI, Firecrawl v1/v2 & Jina Reader HTTP APIs, an MCP server, a C ABI, and native Python / Node.js packages.
English · 简体中文 · 日本語 · Español
Overview
pagesift turns live web pages, uploaded binary documents, archives, and long-lived monitored sources into clean Markdown, schema-validated JSON, provenance-tracked DataGraph relations, and budget-compiled ContextPack evidence inside a single memory-bounded Rust binary. HTTP fetching, native document readers, HTML extraction, adaptive sync state, incremental context indexing, and evidence storage run in-process without requiring Python, JVM, or Node workers; a headless browser pool, NVIDIA GPU OCR, and user-configured LLM endpoints attach only when requested.
Drop-in or companion for: Firecrawl (self-hosted v1/v2 API), Jina Reader (/r/ and /s/), Crawl4AI, trafilatura, Readability, pymupdf4llm, Docling, Kreuzberg, and xberg.
At a glance
- Layered runtime: Entry surfaces (
CLI/HTTP/MCP/ FFI) → policy & SSRF-guarded net → adaptive strategy (HTTP/ embedded JSON / browser) → native format readers →DataGraph,Context& append-only store → cgroup-sized NUMA worker pool (pagesift-flow) - Inputs: URLs, HTML, PDF (text + optional PP-OCRv6 / formulas), uploaded images (OCR when enabled), Office & OpenDocument, RTF/WordPerfect/HWP, e-books, emails, archives (
zip,tar,7z), SQLite/CSV, 373 code languages, and WARC (format matrix) - Outputs: Markdown, HTML, raw HTML, links, images/attachments, RAG chunks, schema JSON,
DataGraph(graph.jsonl, citations, claim lifecycles, conflicts,fact_clusters,?asOf=),ContextPack(KnowledgeUnit,ContextDelta,provenanceManifest), plus structured warnings on partial captures - Interfaces: CLI · Firecrawl v1/v2 HTTP API · Jina Reader (
/r/,/s/) · MCP (stdio / HTTP) · Python · Node.js / TypeScript · Rust crate · C ABI (Go, Java, Kotlin, C#, Ruby, PHP, Lua, Perl, Dart, Swift, Zig) · LangChain & LlamaIndex - Platforms & ops: Single binary (
serve --role gateway,worker,standby), Linux x86_64/aarch64 (glibc ≥ 2.28 & musl), macOS arm64, Windows x86_64, Dockerghcr.io/yolow92/pagesift:1.6.0 - Index & release: Machine-readable index llms.txt · Apache-2.0 or commercial support · 1.6.0 (release notes)
Install and quick start
| Surface | Install | Minimal call |
|---|---|---|
| CLI | cargo install pagesift --locked |
pagesift scrape https://example.com |
| Python | pip install pagesift |
pagesift.scrape("https://example.com").markdown |
| Node.js | npm install @pagesift/node |
(await new Scraper().scrape("https://example.com")).markdown |
| Server | docker run -p 3002:3002 ghcr.io/yolow92/pagesift:1.6.0 |
curl localhost:3002/r/https://example.com |
| MCP | — | claude mcp add pagesift -- pagesift mcp |
Quick start snippets: CLI, Python, Node.js, HTTP API, MCP
CLI```bash
pagesift scrape https://example.com # Clean Markdown to stdout
pagesift scrape --graph https://example.com # Markdown + DataGraph provenance & claims
pagesift scrape report.pdf # Local PDF, Office, e-book, email, archive, code or WARC
pagesift --ocr scrape scan.pdf # OCR scanned PDF pages or uploaded images
pagesift crawl https://docs.rs --limit 500 --out ./data/ # Crawl to disk (add --graph for graph.jsonl)
pagesift graph inspect ./data/graph.jsonl # Inspect citations, lineage, claims and conflicts
pagesift sync init --id docs --seed https://example.com # Initialize and run a persistent Loop sync source
pagesift context pack --file report.pdf "rate limits" # Compile a token-budgeted ContextPack with citations
pagesift extract --schema schema.json https://example.com # Schema JSON with CSS selector relocation
pagesift doctor # Inspect local browsers, GPU and models
**Python & Node.js**```python
import pagesift
page = pagesift.scrape("https://example.com", formats=["markdown", "links"])
print(page.markdown, page.metadata["title"], page.links)
with pagesift.Scraper(store_path="./data/store") as s:
pack = s.context(action="pack", source_ids=["docs"], query="rate limits", budget_tokens=1024)
print(pack["pack"]["compiledContext"])
``````typescript
import { Scraper } from '@pagesift/node'
const scraper = new Scraper({ storePath: './data/store' })
const page = await scraper.scrape('https://example.com', { formats: ['markdown', 'links'] })
const pack = await scraper.context({ action: 'pack', sourceIds: ['docs'], query: 'rate limits', budgetTokens: 1024 })
console.log(page.markdown, pack.pack)
HTTP server (Firecrawl v1/v2, Jina Reader, /v2/sources, /v2/graph, /v2/context) & MCP```bash
docker run -d -p 3002:3002 -v pagesift-data:/data -e PAGESIFT_API_KEYS=sk-test ghcr.io/yolow92/pagesift:1.6.0
curl -s localhost:3002/v2/scrape -H 'Authorization: Bearer sk-test' -H 'Content-Type: application/json' \
-d '{"url":"https://example.com","formats":["markdown"]}'
curl -s localhost:3002/r/https://example.com -H 'Authorization: Bearer sk-test'
curl -s localhost:3002/v2/parse -H 'Authorization: Bearer sk-test' -F file=@report.pdf -F 'options={"formats":["markdown"]}'
curl -s localhost:3002/v2/context/pack -H 'Authorization: Bearer sk-test' -H 'Content-Type: application/json' \
-d '{"sourceIds":["docs"],"query":"rate limits","budgetTokens":1024}'
pagesift mcp # stdio for a single agent
pagesift mcp http --port 3003 # shared local MCP daemon
MCP exposes `pagesift_scrape`, `pagesift_batch_scrape`, `pagesift_crawl`, `pagesift_poll_job`, `pagesift_cancel_job`, `pagesift_map`, `pagesift_interact`, `pagesift_interact_stop`, `pagesift_sync`, `pagesift_graph_query`, `pagesift_context`, and (with a configured model) `pagesift_agent` / `pagesift_agent_status`. See Shared MCP service and Context reference.
## Capabilities
Organized by architectural plane—showing implemented surfaces, data paths, and reference docs:
| Plane | Implemented capabilities | Reference |
|---|---|---|
| **Interfaces & compatibility** | Self-hosted CLI, Firecrawl v1/v2 HTTP API (`/v2/scrape`, `/v2/crawl`, `/v2/parse`, `/v2/monitor`, `/v2/sources`, `/v2/graph`, `/v2/context`, `/v2/agent`, `/interact`), Jina Reader (`/r/`, `/s/`), MCP (stdio/HTTP), Python, Node.js/TS, Rust library, C ABI bindings | parse-api.md · context.md · mcp-service.md |
| **Acquisition & safety** | Conditional HTTP (`ETag`/`Last-Modified`), SSRF & DNS-rebinding guards, `robots.txt` & per-host rate limits, identity leases, embedded JSON synthesis (`__NEXT_DATA__`/`__NUXT_DATA__`/JSON-LD), optional CDP browser pool (`Lightpanda`/`moli`/Chrome) + interactive `/interact` sessions, opt-in `[itna]` access-barrier classification | interact.md · http-sessions.md · bot.md |
| **Native parsing & OCR** | Pure-Rust readers down to container/file level (`yolowhtml`, `yolowpdf`, `yolowdoc`, `yolowsyntax`, `yolowzstd`, `yolow64`) covering web, Office/ODF, e-books, email, nested archives, SQLite/CSV, 373 code languages (AST function/class splits for 32+ languages), WARC I/O, and PP-OCRv6 + PP-FormulaNet on CPU/GPU | parse-api.md |
| **Extraction & agents** | Main-content Markdown with math/code/tables (`` & `hidden="until-found"`), RAG `queryFilter` & chunking, selector-memory schema extraction with automatic DOM relocation, and model-planned `/v2/agent` research verified against archived page evidence | structured-extract.md · agent-api.md |
| **Continuous Loop & monitors** | Persistent `pagesift sync` / `/v2/sources` sources with anti-aliased periodic detection, hierarchical Gamma-Poisson shrinkage & Lagrange water-filling under tick budgets, hub co-change propagation, modality cost & tab shadow pricing, bounded `DirectBrowser` recovery probes, surprise forgetting, tenant-isolated cold-start transfer, capture reliability censoring, and `/v2/monitor` webhooks & substantive block diffs | monitor.md · incremental-collection.md |
| **DataGraph & provenance** | Cross-page and cross-source evidence graph (`graph.jsonl`, `graph_summary.json`, `/v2/graph/claims`, `/history`, `/conflicts`, `?asOf=`): persisted document citations, redirect/canonical/attachment provenance, contextual unit/currency disambiguation, `OriginRelation` independent roots, `fact_clusters`, and claim lifecycles (`Active`, `Superseded`, `Withdrawn`, `Unverified`, `EvidenceDegraded`) | architecture.md |
| **Context & compilation** | Cross-version `KnowledgeUnit` identity & ambiguity tracking (`boundedUot` / `greedyDice` / `exactAnchorOnly`), atomic dependency-governed `ContextDelta` & idempotent consumer checkpoints, multi-strategy lexical/symbol/temporal/lineage `Search`, budget- & `MustKeep`-constrained `ContextPack` (`submodularClosure` / `boundedDpp` / `edgeSalience`) with `provenanceManifest`, and bounded VOI + shadow-price `Action` with drift-guarded online `Learn` | context.md |
## Performance at a glance
Benchmarks are tracked separately so this page stays focused on system structure; all recorded baselines, hardware details, caveats, and competitor comparisons live in docs/benchmarks.md:
- **Latency (recorded benchmarks, single-worker, lower is better):** PDF → Markdown `2.5–2.58 ms/page` (1,269 pages); HTML → Markdown `0.23–0.24 ms/page` (600 docs); main-content extraction `1.0 ms/page` (597 pages); Context `BoundedUot` reconcile `63.9 µs` and `ContextPack` compile `67–225 µs` (`192`/`384`-token budgets).
- **Throughput, quality & memory (recorded benchmarks):** Office in-process `81.9–210.0 files/s` (higher is better); HTML → Markdown `1.000` mean F1; WCXB `0.866` F1; RAG query filter `0.929` recall (higher is better); Context `Search + Pack` `2,708.8–24,434.4 QPS` across `1–16` threads (`25.9–28.8 MiB` RSS, higher QPS / lower RSS is better); `17–395 MB` peak memory/worker without a browser across domain runs.
- **Full scorecards & evaluations:** benchmarks.md · Full benchmark report · Latest Context & Loop release evaluation
## Architecture and core flows
The core runs as one compiled Rust binary with zero managed interpreter runtimes behind parsing, storage, or scheduling. Memory, CPU workers, and browser tabs pass through shared cgroup-aware admission, and internal runtime limits adapt only when online statistics clear significance thresholds (`δ = 0.05`).
### Request and ingestion pipeline```mermaid
flowchart TD
CLI["CLI / Rust caller"] --> Engine["pagesift-engine: scrape, crawl, Loop sync, Context"]
HTTP["HTTP API: Firecrawl v1/v2, Jina, /v2/sources, /v2/graph, /v2/context"] --> Engine
Bind["Python / Node / MCP / C ABI"] --> Engine
Engine --> Policy["pagesift-hooks & pagesift-policy<br/>robots, host rate limits, backoff, identity leases"]
Policy --> Net["pagesift-net + pagesift-compress (yolowzstd)<br/>HTTP/conditional fetch, SSRF guards, TLS/proxy"]
Net -.-> ITNA["pagesift-itna (opt-in)<br/>barrier class + shared budget"]
ITNA -.-> CDP
Net --> Plan{"pagesift-plan<br/>HTTP, JSON or browser?"}
Plan -->|HTTP or embedded JSON| DOM["pagesift-html (yolowhtml-dom): single-pass arena DOM"]
Plan -->|Browser escalation| CDP["pagesift-cdp: Lightpanda / moli / Chrome"]
CDP --> DOM
Net -->|PDF| PDF["pagesift-pdf (yolowpdf + yolow64)"]
Net -->|Office / e-book / email / archive / text| Office["pagesift-office (yolowdoc)"]
Net -->|Image / scanned PDF| OCR["pagesift-ocr, pagesift-infer, pagesift-gpu"]
Office -->|member / attachment| PDF
Office -->|source code| Syntax["pagesift-syntax (yolowsyntax)"]
PDF -->|scanned page| OCR
DOM --> Frontier["pagesift-frontier: URL scoring"]
Frontier -.->|next crawl / sync candidates| Engine
DOM --> Content["pagesift-content (yolowhtml) & pagesift-extract<br/>main content, selectors, evidence, chunks"]
Content --> Graph["pagesift-graph (DataGraph)<br/>citations, provenance, claims, conflicts, clusters, asOf"]
PDF --> Graph
Office --> Graph
Graph --> Ctx["Context (pagesift-graph + pagesift-extract + pagesift-stat)<br/>KnowledgeUnit (BoundedUot), ContextDelta, Search, ContextPack, VOI Action & Learn"]
Graph --> Out["pagesift-sink & pagesift-store<br/>graph.jsonl, offloaded snapshots, SyncStore, ContextStore (ns::CONTEXT + chis), Experience"]
Ctx <--> Out
Ctx -.->|VOI + 4D shadow-price bounded acquisition| Engine
Out -.->|priors, URL rewrites, checkpoints| Plan
Continuous Sync closed loop (Observe → Learn → Decide → Execute → Verify)
Long-lived sources registered via pagesift sync or /v2/sources run an adaptive closed loop per tick without discarding prior verified evidence on transient uncertainty, and optionally reconcile changed documents incrementally into ContextStore:```mermaid
flowchart LR
Obs["Observe & Schedule<br/>pagesift-store + pagesift-stat<br/>Non-aliased periodicity, Gamma-Poisson<br/>rates, Lagrange water-filling, Hub pull-forward"]
Dec["Decide Modality<br/>pagesift-plan<br/>304/HTTP/Browser cost + tab shadow price,<br/>bounded recovery probes, tenant transfer"]
Exe["Execute & Screen<br/>pagesift-engine/sync/capture<br/>HTTP / JSON synthesis / CDP / 304;<br/>censor decode:, truncation & bad 304s"]
Ver["Verify & Commit<br/>pagesift-graph + pagesift-store<br/>Active/Superseded/Withdrawn/Unverified claims,<br/>ContextDelta & unit lineage, snapshot offload"]
Obs --> Dec --> Exe --> Ver --> Obs
### Evidence-checked `/v2/agent` and multi-server fleet topology```mermaid
flowchart LR
subgraph Agent["Bounded /v2/agent research loop"]
Q[Question + Schema] --> M[User LLM step]
M --> T[search / map / scrape / extract<br/>enforced robots & SSRF policy]
T --> Ev[Archived evidence]
Ev --> Check{Complete?}
Check -->|budget left| M
Check -->|done| Verify[Schema match + field grounding + warnings]
end
``````mermaid
flowchart LR
subgraph Fleet["Single binary multi-node cluster (pagesift-cluster)"]
C[Clients] --> G[Gateway<br/>API, jobs, webhooks]
G -->|rendezvous host hash| W1[Worker]
G --> W2[Worker]
G -.->|fenced leases & checkpoints| S[Standby]
end
Workspace crate map
| Layer | Crates |
|---|---|
| Entry surfaces & FFI | pagesift (CLI & library), pagesift-api (HTTP server & schemas), pagesift-mcp (MCP stdio/HTTP), pagesift-py, pagesift-node, pagesift-ffi (C ABI) |
| Orchestration, strategy & graph | pagesift-engine, pagesift-plan, pagesift-hooks, pagesift-llm, pagesift-search, pagesift-graph |
| Network, policy & browser | pagesift-net, pagesift-policy, pagesift-rustls, pagesift-cdp, pagesift-media, pagesift-itna |
| Web extraction & discovery | pagesift-html (yolowhtml-dom), pagesift-content (yolowhtml), pagesift-extract, pagesift-diff, pagesift-frontier |
| Native document & OCR readers | pagesift-pdf (yolowpdf), pagesift-office (yolowdoc), pagesift-syntax (yolowsyntax), pagesift-ocr, pagesift-infer, pagesift-gpu |
| Runtime, math & persistence | pagesift-flow (NUMA workers & cgroup admission), pagesift-stat, pagesift-base64 (yolow64), pagesift-compress (yolowzstd), pagesift-sys, pagesift-store, pagesift-sink, pagesift-cluster |
See Architecture reference for cgroup sizing, statistical defaults (δ = 0.05 USL/survival/Thompson tuning), incremental checkpoint semantics, and byte-range verified media archives.
When to use it, and when not
Good fit
- Feeding live web pages, PDFs, Office suites, archives, or source code into AI agents, RAG pipelines, or downstream analytics with zero external runtime workers
- Compiling token-budgeted, provenance-cited
ContextPackprompts that preserve table headers, qualifiers, units, temporal validity, and opposing multi-source conflicts - Self-hosting Firecrawl- or Jina Reader-compatible APIs and shared MCP tooling without per-page SaaS fees
- Continuous monitoring/sync of changing sources where cost-aware browser escalation, lineage tracking, and claim-level conflict detection matter
Pick something else when
- You want a managed SaaS cloud.
pagesiftruns self-hosted on your infrastructure. - You need automatic CAPTCHA solving. Interactive challenges are classified and paused for human resolution (
[itna]recovery is opt-in and does not bypass CAPTCHAs). - You only process standalone image URLs without local uploads or OCR. Uploaded/local images require
--ocr([ocr] enabled = true); image URLs and OCR-off uploads returnUNSUPPORTED_OPTION.
License
Apache-2.0. For dedicated support, SLAs, or enterprise agreements, see commercial license and support or write to licensing@pagesift.dev.
What's new in 1.6.0
PageSift 1.6.0 introduces Context (docs/context.md) across CLI (pagesift context), HTTP (/v2/context), MCP (pagesift_context), Python (Scraper.context / acontext), Node.js (Scraper.context), and C ABI (pagesift_context), alongside upgrading upstream yolowhtml / yolowhtml-dom to 0.1.2 (` table headings,hidden="until-found"article retention, andinput_truncatedcapture reliability screening): adds cross-versionKnowledgeUnitidentity and explicit ambiguity/split/merge lineage (boundedUot,greedyDice,exactAnchorOnly); persists atomic dependency-governedContextDeltaupdates,IndexVisibilitylag metrics, and idempotent anti-ABA consumer checkpoints inpagesift-store(ns::CONTEXT, immune to cache eviction); provides multi-strategy lexical/symbol/temporal (asOf+effectiveAt)/lineageSearchwith co-retrieved conflict counterparts; compiles token- andMustKeep-constrainedContextPackpayloads (submodularClosure,boundedDpp,mmrGreedy,topKGreedy,edgeSalience) withprovenanceManifest; and adds boundedAction(context,research,verify,watch,changes) driven by gated Good–Turing / Chapman coverage diagnosis, Howard Value of Information, 4D admission shadow prices, and CUSUM /BernoulliBocpddrift-guarded isotonicLearn` calibration.
Community
Questions or feedback: Discord · Issues · Discussions
Citation```bibtex
@software{pagesift2026,
title = {PageSift: Rust Web and Document Extraction Engine for AI Pipelines},
author = {YOLOW92},
year = {2026},
url = {https://github.com/YOLOW92/pagesift},
note = {Version 1.6.0}
}
```
