RAG-BUILD-SKILLS

1. Role Definition

You are a RAG Pipeline Builder — you construct production-grade RAG systems end-to-end.

• Build offline pipeline first (one-time, no loop).
• Build online pipeline with evaluation loop (iterative optimization).
• All decisions must be backed by evaluation data, not intuition.
• Every component is independently swappable and evaluable.

2. Intake & Planning

On receiving a RAG building task:

1. Gather requirements:
   - What domain? (industrial, medical, legal, financial, etc.)
   - What document types? (PDF, HTML, Markdown, etc.)
   - How many knowledge bases? (single vs. multi-product-line)
   - What LLM will be used for generation?
   - Is there an existing evaluation set? If not, plan to build one.
   - Does the knowledge base contain significant visual content (diagrams, schematics, photos) that users may need to search by image? If yes, plan for the optional multimodal module (Step 3.5 + Step 4.6).

2. Output a pipeline plan before writing any code:
   Offline: [documents] → [OCR/parse] → [chunk] → [index] → [prompt] Online: [query] → [rewrite] → [route] → [retrieve] → [rerank] → [generate] Eval: [Q-DOC] + [Q-DOC-A] → loop optimization

3. Confirm with user which modules are needed before proceeding.

3. Offline Pipeline (One-Time Build, No Loop)

Step 1: Keyword Dictionary Construction

Goal: Build a domain term dictionary covering standard terms, abbreviations, colloquial expressions, and common typos.

Data sources (ask user for access):
- User query logs (if available — reflects real expression habits)
- Domain documents (extract professional terms)
- Domain expert input

Dictionary schema:

{
  "standard_term": "canonical form",
  "abbreviation": "short form",
  "full_form": "expanded form",
  "colloquial": ["informal expressions", "common misspellings"],
  "tag": "subsystem/category label"
}

Actions:
1. Extract candidate terms from documents (TF-IDF, domain NER, or LLM-assisted extraction).
2. Merge with query log terms (if available).
3. Have domain expert validate and supplement.
4. Save as keyword_dict.json.

This dictionary serves three purposes: tag annotation, query rewriting, and tokenizer custom vocabulary.

Step 2: Document Parsing & Chunking

Parsing:
- Use PaddleOCR-VL (Vision-Language model) for PDF/image document parsing.
- PaddleOCR-VL handles heterogeneous content types: text, tables, figures, and mixed layouts.
- Unlike text-only OCR, it preserves the visual structure and semantic meaning of non-text elements.
- Output structured format (Markdown preferred), preserving document hierarchy (chapters, sections).
- Maintain page number mapping for traceability.

Image/Figure Handling:
- When PaddleOCR-VL encounters an image or figure in the document:
1. Extract the image and store it as a separate asset (e.g., images/{doc}_{page}_{fig_id}.png).
2. In the parsed Markdown, replace the image's original layout position with:
- An image link (e.g., ![figure_description](images/{doc}_{page}_{fig_id}.png))
- A text description of the image (generated by the VL model's image understanding capability)
3. This ensures images are discoverable during retrieval — both the link and the description become part of the chunk text, indexable by BM25 and Dense retrieval.
- For tables: convert to Markdown table format, preserving structure.
- For mixed text+figure layouts: maintain reading order as it appears in the original document.

Why PaddleOCR-VL over plain OCR:
| Content Type | Plain OCR | PaddleOCR-VL |
|---|---|---|
| Text | ✅ Extracted | ✅ Extracted |
| Tables | ⚠️ Loses structure | ✅ Preserves as structured table |
| Figures/Images | ❌ Ignored or garbage | ✅ Extracted + described + linked |
| Mixed layouts | ⚠️ Reading order broken | ✅ Layout-aware reading order |

Chunking: Parent-Child Strategy:

Chunking rules:
- Never chunk across chapter/section boundaries.
- Preserve document hierarchy in metadata.

Required metadata per chunk:

{
  "chunk_id": "unique identifier",
  "parent_id": "parent chunk ID (null if parent)",
  "child_ids": ["child chunk IDs (null if child)"],
  "chapter": "chapter title",
  "section": "section title",
  "page": "page number",
  "tag": "subsystem label from keyword dictionary",
  "doc_source": "source document filename",
  "doc_id": "document unique identifier (inherited from document level)",
  "doc_version": "document version number (inherited from document level)",
  "product_model": "product model (inherited from document level)",
  "status": "document lifecycle status: active / legacy / superseded (inherited from document level)",
  "effective_date": "document effective date (inherited from document level)",
  "content": "chunk text"
}

Two-layer metadata architecture: Chunk-level metadata redundantly stores key document-level fields (doc_id, doc_version, product_model, status, effective_date). This is intentional denormalization — retrieval can filter by version/model/status directly without joining the document registry. Full document-level metadata (including supersedes version chains, etc.) is stored in a relational database. See Section 6 for details.

Ingestion principle: Chunks missing required fields are REJECTED — never write first and backfill later. Metadata must be fully defined at offline chunking time.

Output: chunked_corpus.json with all chunks and metadata.

Step 3: Index Construction

Build indexes for hybrid retrieval:

Critical: BM25 and Char-BM25 are mutually exclusive — pick ONE, not both. They are both lexical retrieval with overlapping function; using both only adds noise without improving recall. Default to BM25 (domain tokenizer). Switch to Char-BM25 only when user queries contain heavy typos or colloquial expressions that query rewriting cannot fully cover.

Critical: Tokenizer Separation Principle
- BM25 uses a domain tokenizer with the custom dictionary → ensures domain abbreviations stay intact.
- Dense/Sparse uses the embedding model's built-in tokenizer (e.g., BPE) → cannot be changed.
- Do NOT try to make them share a tokenizer.

Actions:
1. Load keyword dictionary into jieba (or equivalent): jieba.load_userdict("keyword_dict.txt").
2. Build one lexical index on child chunks — either BM25 (default) or Char-BM25.
3. Encode all child chunks with the embedding model → store Dense vectors.

Step 3.5 (OPTIONAL): Multimodal Image Index Construction

> This step is OPTIONAL. Only execute when:
> - The knowledge base contains significant visual content (diagrams, schematics, photographs).
> - Users need to query with images (e.g., "this error screenshot shows what problem?").
> - If the knowledge base is text-only, SKIP this step.

Architecture: Metadata as Cross-Modal Bridge

Images are NOT embedded into the text vector space. Instead:
- Images are encoded by a multimodal embedding model (e.g., Qwen3-VL-Embedding) into a SEPARATE image vector index.
- Each image stores metadata (page, chapter, section, parent_chunk_id, description) that bridges to the text pipeline.
- Image search results use metadata to locate the corresponding text chunks, which are then fed to the LLM.

This approach ensures zero-invasion to the existing text pipeline (BGE-M3, Reranker, Prompt all remain unchanged).

Offline workflow:

Document images (from Step 2 PaddleOCR-VL extraction)
  → Qwen3-VL-Embedding (image encoding only, NOT text)
  → Image vector index (separate from Dense/BM25)
  + Image metadata storage

Image metadata schema:

{
  "image_id": "unique identifier",
  "image_path": "path to stored image file",
  "image_embedding": "vector from Qwen3-VL-Embedding",
  "page": "page number",
  "chapter": "chapter title",
  "section": "section title",
  "parent_chunk_id": "ID of the parent text chunk containing this image",
  "description": "text description generated during PaddleOCR-VL parsing (Step 2)",
  "tag": "version/model label (same tagging system as text chunks)"
}

Key design decisions:
1. Qwen3-VL-Embedding is used ONLY for image encoding — text encoding continues to use BGE-M3.
2. The parent_chunk_id is the bridge: image search result → metadata → parent chunk → LLM context.
3. The description field enables text-to-image search (fallback path when image similarity is low).
4. The tag field enables version/model filtering for image queries.

Step 4: LLM Prompt Engineering (Initial Version)

Prompt structure (six elements):

## Identity
[Define LLM's role — e.g., "You are a domain expert assistant for {domain}"]

## Function
[Define task — e.g., "Answer user questions based on provided document chunks"]

## Goal
[Define output goal — e.g., "Provide accurate, well-structured answers with document citations"]

## Answer Specification
[Format rules, citation format, structure requirements]
- Use ^1,2,3^ citation format referencing document chunk numbers
- Structure answers with headers and tables when appropriate
- Answer all sub-questions if the query contains multiple questions

## Prohibitions
[What the LLM must NOT do]
- Do not fabricate information not present in the provided documents
- Do not cite documents that are not relevant to the answer
- Do not refuse to answer if the information is present in the documents

## Examples
[Few-shot examples — 2-3 high-quality Q-A pairs with citations]

This is the initial version. It will be iteratively optimized in the online evaluation loop (Section 5).

Step 5 (Offline): Q-DOC-Q Data Pair Generation

> The evaluation dataset is now auto-generated by the offline pipeline, no longer requiring expert annotation. LLM generates Q-DOC-Q evaluation pairs from chunked_corpus, supporting 1:N:1 relationships (one Query maps to multiple document chunks, with one standard answer).

Q-DOC-Q Pairs (LLM auto-generated):

{
  "query": "user question",
  "gt_doc_ids": ["ground truth document chunk ID"],  // supports 1:N (multiple chunks)
  "gt_pages": [45],
  "gt_answer": "expected answer text",
  "gt_keywords": ["keyword1", "keyword2"],
  "query_type": "factual | procedural | troubleshooting | definitional",
  "gen_source": "single_chunk | multi_chunk",
  "source_chunk_ids": ["original chunk IDs used for generation"]
}

Generation strategy:
- gen_single_chunk_queries: single-chunk query generation (1:1), generates multiple queries of different types per child chunk.
- gen_multi_chunk_queries: cross-chunk query generation (1:N), selects adjacent chunk groups to generate queries requiring synthesized information.
- merge_dedup: merge + dedup + quality filtering, uses BGE-M3 for semantic similarity dedup.
- Expert annotations can be manually merged into eval_set.json before evaluation to enhance coverage.

Step 6 (Offline Eval): Offline Pipeline Evaluation

> This step runs BEFORE online evaluation (Q-DOC / Q-DOC-A). It evaluates the
> offline pipeline artifacts — index completeness, raw retrieval recall, chunking
> quality, keyword dict quality — using the eval_set.json produced by Step 5.
> It uses RAW retrieval (no rewriting, no routing, no reranking) to isolate
> data-processing issues. Must pass before proceeding to online evaluation.

Reference implementation: scripts/offline_eval/offline_eval.py + the
offline_eval section of config/default_config.yaml.

Step 6.5 (OPTIONAL): Embedding Fine-Tuning

> This step is OPTIONAL. Only execute when Q-DOC evaluation shows consistently low Dense recall, AND dictionary updates, retrieval parameter tuning, and other optimizations have been tried without improvement.

Goal: Fine-tune the embedding model (e.g., BGE-M3) on domain-specific data to improve Dense retrieval's matching capability for domain terms and semantics.

Two Fine-Tuning Modes:

> Note: If you fine-tune in Unified mode but only deploy the Dense part, the fine-tuned Sparse/ColBERT weights are wasted. Start with Dense-only.

Critical Constraints:
1. After fine-tuning, you MUST re-encode ALL chunks with the new model — fine-tuned vectors and original vectors are NOT in the same space; mixing them produces garbage similarity results.
2. Deployment decision is based on Q-DOC evaluation comparison: pre-fine-tuning vs. post-fine-tuning chunk recall@K.
3. Deploy threshold: improvement ≥ 5pp to deploy; revert threshold: regression ≥ 2pp to revert to baseline.

Training Data Generation:
- Mine hard negatives from evaluation Bad Cases (chunks retrieved but not in GT).
- Generate 7 hard negatives per query (configurable).
- Training triplets: (query, positive chunk, hard negative chunks).

Actions:
1. Filter Dense recall failures from Q-DOC evaluation report.
2. Generate training data (hard negatives mining).
3. Fine-tune embedding model (Dense-only or Unified).
4. Re-encode ALL chunks with fine-tuned model → rebuild Dense index.
5. Re-run Q-DOC evaluation on the same evaluation set.
6. Compare pre/post chunk recall@K:
- Improvement ≥ 5pp → deploy fine-tuned model.
- Improvement < 5pp → do NOT deploy; gains not worth operational cost.
- Regression → revert to baseline model.

4. Online Pipeline (With Evaluation Loop)

Step 4.1: Query Rewriting

Based on the keyword dictionary, rewrite user queries:

Implementation: Rule-based matching on keyword dictionary + optional LLM-assisted rewriting for complex cases.

Step 4.2: Multi-Knowledge-Base Routing

Query → Layer 1: Knowledge base type identification (product line / doc type)
       → Layer 2: Model/version filtering
       → [Optional] Layer 3: Subsystem tag filtering

WARNING — Routing Risk:
- Routing errors shrink the candidate pool → relevant documents get excluded.
- If routing accuracy < 50%, REMOVE routing — the harm outweighs the benefit.
- With a Reranker in the pipeline, routing pre-filtering steals the Reranker's operating space.
- Always validate routing with evaluation data before deploying.

Step 4.3: Hybrid Retrieval

Query → BM25 or Char-BM25 (pick ONE) ──→ lexical scores
     → Dense (embedding model)        ──→ Dense scores
                                            ↓
                           RRF Fusion → Top-N candidates (typically N=50~64)

Lexical Index Selection (pick ONE):

Do NOT use both simultaneously — they are both lexical retrieval with overlapping function; using both only adds noise without improving recall. Default to BM25.

Fusion Strategy:

Always use RRF (Reciprocal Rank Fusion, k=60), regardless of whether a Reranker exists in the pipeline.

Why not Score-Level: BM25 scores (TF-IDF based, typically 0-30 range) and Dense scores (cosine similarity, typically 0-1 range) have completely different scales and distributions. Even with normalization, the normalization method itself (Min-Max, Z-Score, max-normalize) introduces bias. RRF only uses rank information, naturally avoiding the score incomparability problem.

Step 4.4: Reranking

Top-N candidates → Cross-Encoder Reranker → Top-M (typically M=5~12)

Configuration:
- Model: Cross-Encoder (e.g., bge-reranker-v2-m3)
- Input: Top-N child chunks (NOT parent chunks — most Cross-Encoders have 512 token limits)
- Output: Top-M child chunks

Top-N selection method: Run Recall@K sweep on the evaluation set:

for K in [20, 30, 40, 50, 64, 80, 100, 128, 200]:
    recall = evaluate_recall(hybrid_results, gt_docs, top_k=K)
    rerank_latency = measure_rerank_latency(K)
    # Find the K where recall plateaus — that's your Top-N

• Typical sweet spot: N=50~100 for general use, N=20~50 for narrow-domain RAG.
• Gains plateau by N=200 — beyond that, you pay latency for zero accuracy gain.

Top-M selection method: Experiment with different M values on the evaluation set:
- Too small M → LLM lacks context → incomplete answers.
- Too large M → context noise → LLM distracted by irrelevant chunks.
- Typical range: M=5~12 for domain knowledge bases.

Parent-child mapping after reranking:

Reranked Top-M child chunks → map to parent chunks → feed parent chunks to LLM

This is the core value of parent-child chunking: precise retrieval on child, complete context from parent.

Step 4.5: LLM Generation

Input: Top-M parent chunks (mapped from reranked child chunks) + Prompt (from Step 4 of offline).

Output: Structured answer with document citations.

LLM failure modes to watch for:
- JSON parse errors (if using structured output) → add retry + fallback parsing.
- Hallucination → strengthen Prohibitions in Prompt.
- Refusal to answer when info exists → strengthen Answer Specification in Prompt.

Step 4.6 (OPTIONAL): Multimodal Query Handling

> This step is OPTIONAL. Only activate when the offline multimodal index (Step 3.5) was built.

Query type detection:

Pure image query path:

User image (does NOT go through PaddleOCR)
  → Qwen3-VL-Embedding (encode user image)
  → Image vector search → Top-5 images
  → Check similarity threshold
    ├─ Above threshold → Use image metadata (parent_chunk_id) to locate text chunks
    │                   → Feed parent chunks + image context to LLM
    └─ Below threshold → Degrade to text-search-image
                          (use image description as text query → BM25/Dense → Rerank → find chunks)

Critical: User image queries bypass PaddleOCR — user-uploaded images are directly encoded by Qwen3-VL-Embedding. PaddleOCR-VL is only for offline document parsing, not for user query images.

Dual-path parallel for mixed queries:

Mixed query (image + text)
  ├─ Image path: image → Qwen3-VL-Embedding → image search → Top-5 images → metadata → parent chunks
  └─ Text path:  text  → existing pipeline (rewrite → route → retrieve → rerank) → parent chunks
                    ↓
              Merge results (deduplicate by parent_chunk_id)
                    ↓
              Feed merged parent chunks to LLM

Tag filtering for version/model differentiation:
- Apply the same tag filtering system used in text routing (Step 4.2) to image queries.
- This ensures image search results are filtered to the correct product version/model, preventing cross-version confusion when different models share similar visual content.

Similarity threshold design:
- Set a minimum cosine similarity threshold for image search results (e.g., 0.65, tune on evaluation data).
- When the top result's similarity is below threshold → degrade to text-search-image mode:
- Use the image's description field (from metadata) as a text query.
- Run through the standard text retrieval pipeline (BM25 + Dense → Rerank).
- This provides a graceful fallback when visual similarity is insufficient.

5. Evaluation & Optimization Loop

This is the core iterative process for the online pipeline.

5.1 Three-Layer Evaluation

Evaluation is structured in three layers, each isolating a different part of the pipeline:

Scoring dimensions expanded from 3 to 6: The original 3 dimensions (keyword hit rate, semantic relevance, credibility) are expanded to 6. The new 3 dimensions (content completeness, expression quality, logical coherence) are derived from the 评价点 field in the expert evaluation set. Expanding dimensions covers more generation quality issues (e.g., missing info, verbosity, illogical structure) that 3-dimension scoring would miss. Dimensions can be further expanded (4+) per domain.

Dual evaluation sets coexist:
- LLM-generated eval set (eval_set.json): Large-scale automated evaluation, chunk_id-level GT, for daily iteration.
- Expert-labeled eval set (expert_eval_set.json): Document name + page-level GT (mapped to chunk_id via chunked_corpus.json), contains human scores, for calibrating LLM Judge, covering Bad Cases, and final acceptance.

Human 3-level scoring: 0=completely wrong (wrong source, answer completely mismatched), 1=partially correct/incomplete (incomplete content, verbose/confusing, missing key info), 2=completely correct (accurate, complete, clearly expressed).

Human-LLM Judge dual-track alignment: Human scores serve as calibration baseline. LLM Judge continuous scores are mapped to 3-level scale ([0.8,1.0]→2, [0.4,0.8)→1, [0,0.4)→0). Cohen's Kappa measures agreement. Kappa ≥ 0.6 indicates Judge is sufficiently aligned with humans to trust large-scale automated scoring.

Execution order: Offline Pipeline evaluation → Q-DOC → Q-DOC-A. The offline pipeline evaluation uses raw retrieval (no rewriting, no routing, no reranking) to isolate data processing issues. Only after it passes should you proceed to Q-DOC (which tests the full online retrieval pipeline) and Q-DOC-A (which tests end-to-end generation).

CRITICAL: Retrieval metrics (document-level) and generation metrics (answer-level) are NOT directly comparable. Do NOT subtract them. They measure different things.

5.2 Bottleneck Localization Decision Tree

Q-DOC chunk recall low?
├─ YES → Retrieval stage problem
│   ├─ BM25 recall also low? → Tokenizer/dictionary problem → update keyword dictionary
│   ├─ Dense recall also low? → Embedding problem → adjust retrieval parameters or switch embedding model; if no improvement → enter Loop 1 (Embedding Fine-Tuning)
│   ├─ Fusion recall low? → Fusion strategy problem → check RRF k value or Top-K setting
│   └─ Routing accuracy low? → Routing problem → remove or fix routing
│
└─ NO → Generation stage problem
    ├─ Rerank ranking poor? → Reranker problem → check model, Top-N
    ├─ LLM keyword hit rate low? → Prompt problem → enter Loop 2 (Prompt optimization)
    ├─ LLM content accuracy low? → Prompt problem → enter Loop 2
    ├─ LLM content completeness low? → Prompt or retrieval problem → check if GT sources missing; if retrieval OK → enter Loop 2
    ├─ LLM expression quality low? → Prompt problem → enter Loop 2 (add expression guidelines)
    ├─ LLM logical coherence low? → Prompt problem → enter Loop 2 (add logic guidelines + examples)
    └─ JSON parse errors? → Engineering problem → fix parsing logic FIRST

Rule: Always fix engineering problems (parse errors, API failures) BEFORE optimizing models or prompts. Technical failures masquerade as quality problems.

5.3 Loop 1: Embedding Fine-Tuning Optimization (Retrieval Layer)

> Trigger condition: Q-DOC evaluation shows consistently low Dense recall, AND Loop 3 has ruled out tokenizer/dictionary issues, fusion strategy issues, and routing issues — confirming the bottleneck is the embedding model itself.

1. Filter Dense recall failures from Q-DOC evaluation report
2. Generate training data (hard negatives mining)
   ├─ For each query, retrieve Top-K candidates
   ├─ Exclude GT documents → remaining are hard negatives
   └─ Generate 7 hard negatives per query (configurable)
3. Fine-tune embedding model (Dense-only or Unified)
4. Re-encode ALL chunks with fine-tuned model → rebuild Dense index
5. Re-run Q-DOC evaluation on the same evaluation set
6. Deployment decision:
   ├─ Improvement ≥ 5pp → deploy fine-tuned model
   ├─ Improvement < 5pp → do NOT deploy (gains not worth operational cost)
   └─ Regression ≥ 2pp → revert to baseline model
7. If deployed → re-run Q-DOC-A evaluation to confirm end-to-end effect
8. If improvement < 5pp for 2 consecutive rounds → STOP, embedding has reached fine-tuning ceiling

Key principles:
- Fine-tuning is the LAST resort — try dictionary updates, parameter tuning, and fusion strategy adjustments first.
- After fine-tuning, you MUST re-encode ALL chunks — cannot encode only new chunks.
- Deployment decisions must be backed by evaluation data, not intuition.
- Fine-tuned models increase operational complexity — only worth it when gains are significant (≥ 5pp).

5.4 Loop 2: Prompt Engineering Optimization (Generation Layer)

1. Run current Prompt version on full Q-DOC-A evaluation set
2. Score: 6 dimensions (keyword hit rate + content accuracy + credibility + content completeness + expression quality + logical coherence)
3. Identify Bad Cases and classify:
   - Format issue (citation errors, structure chaos) → add Answer Specification
   - Hallucination (fabricated info) → add Prohibition
   - Refusal (answer exists in docs but LLM didn't extract) → add Answer Spec + Example
   - Over-citation (too many irrelevant docs cited) → add Prohibition
   - Incomplete answer (only partial question answered) → add Answer Spec (content completeness dimension)
   - Verbose/confusing expression → add Answer Spec (expression quality dimension)
   - Illogical/poorly organized → add Answer Spec + Example (logical coherence dimension)
   - Redundant rules (too many rules confusing LLM) → REMOVE redundant items
4. Update Prompt (modify Specification / Prohibitions / Examples)
5. Regression test:
   - Verify Bad Cases are fixed
   - Verify previously-correct cases didn't regress
6. If expert eval set available: check if Cohen's Kappa improved (Judge alignment)
7. If improvement < 0.5pp for 2 consecutive rounds → STOP, Prompt has converged

Key principles:
- Every iteration MUST include regression testing.
- Prompt rules are not "more is better" — redundant rules confuse the LLM, prune regularly.
- The goal is "domain-task-level overfitting" — the Prompt should precisely fit the specific domain's answering requirements.

5.5 Loop 3: Full-Pipeline Evaluation & Component Tuning (System Layer)

1. Run full pipeline (rewrite → route → retrieve → rerank → generate)
2. Evaluate Q-DOC (retrieval) + Q-DOC-A (generation)
3. Use bottleneck decision tree (Section 5.2) to locate the bottleneck
4. Route to appropriate Loop:
   - Retrieval problem → retrieval parameter adjustment or fusion strategy adjustment; if bottleneck is Embedding → enter Loop 1 (Embedding Fine-Tuning)
   - Generation problem → Loop 2 (Prompt optimization)
   - Routing problem → routing strategy adjustment
   - Engineering problem → fix directly
5. Make ONE change at a time → re-evaluate → verify impact
6. Repeat

Component-level evaluation principle: Change ONE component at a time, evaluate its isolated contribution. Never change multiple components simultaneously — you won't know which change caused the improvement or regression.

6. Incremental Knowledge Management & Version Evolution

Once a RAG system goes live, the knowledge base is not static. New documents flow in continuously, old documents get updated. This section defines the engineering approach for incremental ingestion and version management.

6.1 Two Incremental Scenarios

6.2 Scenario 1: New Document Incremental Ingestion

New document arrives
  ├─ 1. Parse + chunk (same as offline pipeline Step 2)
  ├─ 2. Update BM25 index (incremental write + update df/N statistics)
  ├─ 3. Encode new chunks with Embedding model → incremental write to Dense index
  ├─ 4. Chunk metadata written to vector DB scalar fields alongside chunks
  └─ 5. Keyword dictionary: append new terms only, do NOT delete existing entries

Trigger methods: Manual trigger + scheduled incremental tasks (e.g., daily scan of document source, compare against document registry to identify new documents).

Keyword dictionary principle: The dictionary is a globally shared resource serving query rewriting and tokenization for ALL documents. When ingesting a new document, only append new terms — never delete existing entries. Even if an old document is deleted, its terms remain in the dictionary because those terms may still be used by user queries.

IDF (Inverse Document Frequency) drift: When new documents enter, BM25's IDF statistics naturally change — this is normal algorithm behavior, not an anomaly. Modern BM25 implementations (Elasticsearch, Lucene, Milvus) support automatic statistics updates during incremental writes. What actually requires attention:

6.3 Scenario 2: Old Document Update — Document-Level Three-Class Triage

When a new version of the same doc_id arrives, a decision must be made about the old version. The decision granularity is at the document level, not the chunk level — one ingestion decision settles it, no need for per-chunk diffing.

Three-class triage flow:

New document arrives
  ├─ 1. Extract document-level metadata (doc_id, version, product_model, effective_date)
  ├─ 2. Check document registry: does an old version with same doc_id already exist?
  │    ├─ NO → Parallel (class 2)
  │    └─ YES → Enter three-class triage
  ├─ 3. Three-class triage (automated signals + domain expert confirmation):
  │    ├─ Same product_model + version upgrade
  │    │    ├─ Old content obsolete → Replace (class 1)
  │    │    └─ Old content has reference value → Legacy (class 3)
  │    └─ Different product_model → Parallel (class 2)
  └─ 4. Update document registry + update indexes

Automated signals (system provides judgment basis): version number comparison, supersedes field, effective_date.

Domain expert decision (final classification): The system suggests a class based on signals; the expert confirms whether it's "replace" or "legacy". Whether the old version has reference value is a business judgment, not an algorithmic one.

6.4 Specific Operations for Each Class

Class 1: Replace (Full Replacement)

Full replacement flow:
  1. Locate old document via new document metadata (doc_id + version chain)
  2. Mark old document status = "superseded" in document registry (PostgreSQL)
  3. Delete old chunks:
     ├─ Vector DB: delete old chunk vectors + scalar fields
     ├─ Elasticsearch: delete old chunk BM25 index entries
     └─ Local images: delete old document's associated image files (images/{doc}_{page}_*.png)
  4. Write new document: run through standard offline pipeline (parse → chunk → index)
  5. Keyword dictionary: do NOT delete old entries, only append new terms from the new document

Key principle: There is no such thing as a "document-specific keyword dictionary". The dictionary is global; deleting an old document does not touch the dictionary. Deleting dictionary entries requires extremely strict conditions — you must confirm the term no longer appears in ANY document (including future ones). In practice, dictionary entries are almost never deleted.

Class 2: Parallel (New Addition)

Parallel addition flow:
  1. New document ingested, metadata annotated:
     doc_id = "new_doc_id"
     product_model = "Model-B" (different from old document)
     status = "active"
     version = "v1.0"
  2. Chunks inherit above document-level fields and are written to vector DB
  3. Old document: completely untouched, continues with status = "active" in retrieval
  4. During retrieval: user specifies model → routing layer filters product_model → each hits its own chunks

Key principle: The old document's metadata does NOT need modification. Parallel addition means "each minds its own business" — differentiation happens via the product_model field at retrieval time, not by "updating" anything in the old document.

Class 3: Legacy (Retain as Reference)

Legacy is the safest, most flexible version management strategy — old chunks are not deleted, they stay in the index and are filtered out by default via metadata during retrieval. When a user explicitly wants to query historical versions, the filter is relaxed to surface them.

Legacy handling flow:
  Ingestion phase:
    1. Mark old document status = "legacy" in document registry
    2. Update old chunks' status field in vector DB to "legacy"
       (NOTE: this requires updating old chunks' metadata because the state changed)
    3. New document ingested normally, status = "active"

  Retrieval phase (default):
    4. Vector DB retrieval filter: expr = 'status == "active"'
    5. Legacy chunks excluded by default, user is unaware

  Retrieval phase (querying historical versions):
    6. User explicitly requests old version → routing layer relaxes filter
       expr = 'status == "active" or status == "legacy"'
    7. Legacy chunks enter candidate pool, participate in Rerank normally

  Generation phase:
    8. When legacy chunks enter LLM context, prepend version marker to chunk text:
       "[WARNING: The following content is from v1.0 legacy document, for reference only. Current active version is v2.0]"
       + chunk original text
    9. Add Prompt guidance: "Chunks marked as legacy in the context are for reference only.
       Prioritize content from the current active version when answering."

Legacy handling has three layers: ingestion (tag as legacy) → retrieval (default filter out) → generation (prepend version marker + Prompt guidance). It's not just a generation-phase concern.

6.5 Two-Layer Metadata Architecture

Version management and incremental updates rely on the division of labor between two layers of metadata.

Relationship: chunk-level inherits document-level. Chunk-level metadata redundantly stores key document-level fields (e.g., doc_version, product_model, status). This is intentional denormalization — retrieval can filter by version/model directly without joining another database.

Why document-level metadata does NOT go in the vector DB:

Vector DB support for chunk-level metadata: Mainstream vector databases (Milvus scalar fields, Qdrant payloads, Weaviate properties, pgvector plain SQL columns) all natively support metadata storage and filtering. During retrieval, scalar field indexes narrow the candidate pool first, then ANN (Approximate Nearest Neighbor) search runs on the subset — filtered retrieval latency is nearly identical to unfiltered.

Data flow in practice:

During new document ingestion:
  1. Write to document registry (PostgreSQL) first → obtain doc_id, determine status
  2. Chunk document → each chunk inherits doc_version, product_model, status
  3. Chunk vectors + payload written to vector DB (Milvus/Qdrant)
  4. Chunk text written to Elasticsearch (for BM25 + traceability)

During retrieval:
  Query → routing layer determines product_model, version
       → vector DB retrieval with payload filter: status = "active" AND product_model = X
       → legacy excluded by default; relax filter when historical versions are needed

6.6 Scheduled Incremental Update Engineering Safeguards

Scheduled incremental task (e.g., daily at midnight):
  1. Scan document source, compare against document registry, identify new/changed documents
  2. Acquire incremental update lock (prevent concurrent write conflicts)
  3. Create index version snapshot (for rollback)
  4. Process each document: new doc → class 2; old doc update → three-class triage
  5. Update document registry
  6. Release lock
  7. Run regression test on evaluation set (optional)
  8. If evaluation metrics drop significantly → rollback to snapshot version

6.7 Full Rebuild Trigger Conditions

Day-to-day incremental updates only require incremental writes + statistics updates — no full rebuild needed. Full rebuild is a low-frequency maintenance operation:

6.8 Metadata vs. Keyword Dictionary — Division of Labor

These two are the twin pillars of a RAG system — different responsibilities, but both indispensable:

Keyword dictionary determines "can it be retrieved at all"
Metadata            determines "is the retrieved content the right version"

These two are in series — the keyword dictionary acts at the retrieval entry (query rewriting + tokenization), while metadata acts throughout retrieval (routing filter + version management). Without the keyword dictionary, recall is zero; without metadata, the wrong version is retrieved — it looks like "retrieval succeeded" but the answer is wrong.

7. Knowledge Sedimentation

After each optimization cycle, sediment the learnings:

Sedimentation rules:
- Abstract to methodology level — do NOT overfit to specific project data.
- Update continuously with each loop iteration.
- Output should be directly reusable for new projects.

8. Key Decision Principles

9. Common Pitfalls

10. Standard Build Sequence

[OFFLINE — One-time, no loop]

Step 1: Keyword dictionary construction
  → query logs + documents → extract terms → expert validate

Step 2: Document parsing & chunking
  → OCR → Markdown → parent-child chunk (preserve hierarchy + metadata)

Step 3: Index construction
  → BM25 (domain tokenizer + custom dict) + Dense (embedding model)
  → [recommended] character-level BM25 (near-zero cost)

Step 3.5: [OPTIONAL] Multimodal image index construction
  → Qwen3-VL-Embedding encodes images only → separate image vector index + metadata (parent_chunk_id bridge)

Step 4: LLM Prompt engineering (initial version)
  → six-element structure → will be optimized in online loop

Step 5 (Offline): Q-DOC-Q Data Pair Generation
  → LLM auto-generates eval pairs from chunked_corpus → supports 1:N:1 → eval_set.json

Step 6 (Offline Eval): Offline Pipeline Evaluation
  → index completeness + raw retrieval recall + chunking quality + dict quality
  → uses raw retrieval (no rewrite/routing/rerank) to isolate data processing issues
  → must pass before proceeding to online evaluation

Step 6.5: [OPTIONAL] Embedding fine-tuning
  → only triggered when Q-DOC shows consistently low Dense recall
  → hard negatives mining → fine-tune → re-encode all chunks → Q-DOC comparison → deploy/revert decision

[ONLINE — With evaluation loop]

Step 7: Online pipeline assembly
  → Query rewrite → Routing → Hybrid retrieval → Rerank → LLM generation
  → [OPTIONAL] Step 4.6: multimodal query handling (image search with threshold + dual-path parallel for mixed queries)

Step 8: Evaluation & optimization loop
  → Offline evaluation Q-DOC (retrieval quality, no LLM call)
  → Online evaluation Q-DOC-A (generation quality, end-to-end)
  → Loop 1: Embedding fine-tuning (Q-DOC driven, triggered when Dense recall is bottleneck)
  → Loop 2: Prompt engineering (Q-DOC-A driven, Bad Case iteration)
  → Loop 3: Full-pipeline evaluation (bottleneck localization + component tuning)

Step 9: Knowledge sedimentation
  → sediment workflow + methodology + decisions + pitfalls → reusable assets

[OPERATIONS — Incremental Knowledge Management]

Step 10: Incremental knowledge management (see Section 6)
  → New document ingestion: incremental index writes + append new terms to dictionary
  → Old document update: document-level three-class triage (replace/parallel/legacy)
  → Scheduled incremental task: lock + snapshot + regression test
  → Full rebuild: only triggered when corpus doubles / model upgrades / dictionary major change / metrics drop

11. Execution Rules

1. Always start with offline pipeline — do not build online pipeline without indexes.
2. Evaluation set must exist before online optimization — no eval = blind optimization.
3. Reranker scoring uses child chunks — never parent chunks (512 token limit).
4. LLM input uses parent chunks — map from reranked child chunks for complete context.
5. One change per evaluation iteration — isolate component contributions.
6. Fix engineering bugs before model optimization — parse errors are not quality problems.
7. Sediment after every loop — keep knowledge assets up-to-date.
8. Multimodal module is optional (Step 3.5 + Step 4.6) — only build when visual content needs image search.
9. User image queries bypass PaddleOCR — user images go directly to Qwen3-VL-Embedding; PaddleOCR-VL is offline document parsing only.
10. Image and text indexes are separate — Qwen3-VL-Embedding for images, BGE-M3 for text; metadata (parent_chunk_id) bridges them.
11. Always set image similarity threshold — below threshold, degrade to text-search-image using image description as fallback.
12. Metadata must be complete at ingestion — define full metadata schema (including document-level inherited fields) at offline chunking time; reject chunks missing required fields, never write-first-backfill-later.
13. Old document updates use document-level three-class triage — replace/parallel/legacy is one decision per ingestion, not per-chunk diffing.
14. Keyword dictionary is append-only — deleting a document does not touch the dictionary; the dictionary is a globally shared resource serving query rewriting and tokenization for all documents.
15. Legacy does not delete data — mark old versions status=legacy, filter out by default at retrieval; relax filter when querying historical versions.
16. Document-level metadata goes in a relational database — version chains, transactions, and complex queries are relational DB strengths; chunk-level inherits key fields into vector DB scalar fields for retrieval filtering.
17. Incremental updates require lock and snapshot — scheduled incremental tasks must acquire a write lock + create an index version snapshot for rollback on failure.
18. Use Top-K, not fixed score thresholds — IDF drift is normal behavior; fixed thresholds will fail; Top-K is immune to drift.

12. Reference Scripts

These scripts are reference implementations, not a production framework. They demonstrate the core algorithm logic and data structures for each pipeline step. Agents should read them to understand the methodology, then adapt the code to their specific project environment (tokenizer, model paths, storage backend, etc.).

Script conventions:
- # CORE comments mark methodology-critical logic that must not be changed.
- # ADAPT comments mark sections that agents must customize for their project.
- Scripts omit engineering-layer concerns (error handling, logging, monitoring) — add those in production.

Directory Structure

rag-builder/
├── SKILL.md
├── SKILL_CN.md
├── config/
│   └── default_config.yaml
├── data/
│   └── expert_eval_set.json          # expert-validated eval pairs (chunk_id / page level)
├── prompts/
│   └── initial_prompt.md             # LLM prompt template (six-element structure)
└── scripts/
    ├── offline/
    ├── online/
    ├── eval/
    ├── offline_eval/
    └── finetune/

Offline Scripts

Online Scripts

Evaluation Scripts

Fine-Tuning Scripts

Configuration

Script Total

How Agents Should Use These Scripts

1. Read the script to understand the algorithm logic and data structure for a given step.
2. Identify # ADAPT sections — these require project-specific customization.
3. Implement a project-adapted version using the reference as a blueprint.
4. Do NOT copy-paste directly — adapt tokenizer, model paths, storage backend, and error handling to the actual environment.