RAG Chat Engine Hardening

Overview

Local/offline RAG assistants (llama_index + Chroma/bge + Ollama qwen2.5:3b-class
CPU models) are extremely prone to silent retrieval degradation in the chat
path and small-model hallucination. This skill captures two hard-won lessons
from a production government Q&A RAG:

1. The as_chat_engine trap — the single most common reason a carefully
   configured Advanced RAG (hybrid retrieval + rerank) "works in eval but not in
   chat". It fails silently, so answers look plausible while retrieval is actually
   broken.
2. The anti-hallucination hardening pattern — a set of rule-based guards and
   prompt/fact-table/data-organization changes that cost zero extra LLM calls and
   stop a 3B model from fabricating policy numbers.

Apply this skill whenever a RAG chat assistant returns wrong/garbled answers,
especially after "I added hybrid retrieval + rerank but it still answers badly",
or when a known-correct document exists in the knowledge base yet is never cited.

When To Use

• User reports the chat bot "makes things up", answers off-topic, or cites the
  wrong policy.
• Retrieval was upgraded (dense → hybrid, added rerank/compression) but answer
  quality did not improve — suspect the chat engine is not using it.
• Building a new offline RAG chat assistant from scratch and want the guards baked
  in from day one.
• Debugging "why does the standalone query engine find the right doc but the chat
  engine doesn't".

The Core Pitfall: as_chat_engine Discards Your Retriever

Symptom

index.as_chat_engine(query_engine=...) (or query_engine.as_chat_engine(...))
returns a chat engine, but at runtime chat_engine._retriever is a plain
VectorIndexRetriever (default similarity_top_k, no hybrid, no rerank).
Your carefully built retriever + node_postprocessors (rerank, compression,
LongContextReorder) are ignored. Advanced RAG only runs in the standalone
query_engine.query(...) / eval path — never in multi-turn chat. Result: relevant
special-topic docs get crowded out of top-k and the small LLM fabricates from
irrelevant snippets. This passes eval (which uses the query engine directly) and
fails in production (which uses the chat engine).

Root Cause

In the installed llama_index version, passing query_engine= to
as_chat_engine does not forward the engine's retriever / node_postprocessors
into the chat engine's retrieval step. The chat engine reconstructs a default
retriever from the index instead.

The Fix — Build The Chat Engine Directly

Do not go through as_chat_engine. Construct the chat engine with the
configured retriever and postprocessors passed explicitly:

from llama_index.core.chat_engine import CondensePlusContextChatEngine

def get_or_create_chat_engine(session_id="default"):
    index = _get_index()                      # your VectorStoreIndex
    retriever, postprocessors, rerank_ok = _build_query_kwargs(index)
    # retriever == your HybridRetriever (possibly wrapped by PolicyFilterRetriever)
    # postprocessors == [MetadataReplacement, ConditionalRerank, Compression, LongContextReorder]
    engine = CondensePlusContextChatEngine.from_defaults(
        retriever=retriever,                  # <-- honored in chat now
        node_postprocessors=postprocessors,   # <-- rerank/compression honored now
        llm=LlamaSettings.llm,
        memory=memory,                        # SimpleChatMemory / externalized store
        system_prompt=SYSTEM_PROMPT,          # your hardened prompt
    )
    return engine

CondensePlusContextChatEngine keeps multi-turn context (it condenses prior
turns into the retrieval query) while respecting the injected retriever and
postprocessors. Verify immediately after building: assert
type(engine._retriever).__name__ != "VectorIndexRetriever" (or equals your
custom retriever class).

> Note: RetrieverQueryEngine has no .as_chat_engine() method — calling it
> raises AttributeError. Always use CondensePlusContextChatEngine.from_defaults.

How To Detect The Trap Fast

Run this diagnostic against a live engine and inspect the retriever type:

ce = get_or_create_chat_engine()
print(type(ce._retriever).__name__)        # expect your Hybrid/PolicyFilter retriever
print([type(p).__name__ for p in getattr(ce, "node_postprocessors", [])])
# If it prints "VectorIndexRetriever" with an empty postprocessor list,
# the trap is active — apply the fix above.

Then stream a known-bad query and confirm the sources event lists the correct
policy document and that no anchor guard false-fires (see references).

Anti-Hallucination Hardening Pattern (zero extra LLM calls)

Layer these so a small model cannot fabricate. All are rule-based and add no
latency. Implement in order; each is independent and can be toggled.

1. System-prompt hard constraint. Replace soft "don't fabricate" wording with
   three strict rules: answer only from retrieved content; if retrieved material
   lacks the authoritative clause, treat as "not found"; when unable, reply with
   exactly one fixed fallback sentence (e.g. "知识库中暂无相关政策的权威信息，建议拨打 12345 或到政务窗口咨询。").
   See references/anti_hallucination_guards.md for the exact prompt.

2. rerank threshold guard + lexical-overlap anti-false-kill. After rerank, if
   top1 score < threshold (e.g. 0.2) and question↔snippet lexical overlap is
   weak → degrade to the fallback. The overlap check prevents误杀 of legitimately
   low-scoring in-domain hits (bge-reranker-base scores run low on Chinese).

3. Policy anchor-term guard (third gate). Maintain an anchor-term list
   (产假/婚假/社保/工伤/补贴/退休/落户/营业执照 …). If the query names an anchor
   term but none of the retrieved corpus snippets contain it → force degrade.
   This catches the case where rerank thinks it's "related" by word overlap but the
   actual policy is absent. Always-on (no config flag); complements rule 1.

4. Answer-backtrace guard (soft). After generation, check whether key numbers
   in the answer appear in the retrieved snippets (or fact-table hits). If none
   match, emit a gentle "核对官方最新政策" hint. Do not suppress the answer —
   it is a warning, not a block. Skip when a fact-table hit already supplied the
   number.

5. Fact-table injection (SQL). Keep a SQLite table of precise facts
   (hotlines, deadlines, leave days, subsidy amounts). At query time, query it in
   parallel with vector retrieval; if hit, inject the authoritative text into the
   LLM context ("优先直接采用、不要改写数字") and show it as a distinct "📊 事实库"
   source. Small models extract numbers from prose poorly; a fact table makes them
   exact. Schema + loader in references.

6. Widen top-k candidate pools. Hybrid VECTOR/BM25/FUSION_TOP_K 4 → 8,
   RERANK_TOP_N 4 → 6. A relevant special-topic doc that ranks #6 by dense
   similarity is otherwise excluded before rerank even sees it.

7. Organize the knowledge base by month/topic. Name policy docs
   YYYY年M月_<主题>.md (e.g. 2026年8月_产假与生育津贴.md); add a
   README_知识库组织说明.md that is excluded from indexing (skip files whose
   name starts with README/说明/知识库组织说明 in the loader) so the human
   index is never answered as policy. Clear naming also makes maintenance auditable.

Detailed pseudocode for guards 1–5, the fact-table schema, the SYSTEM_PROMPT text,
and the KB loader skip rule live in references/anti_hallucination_guards.md.

Verification Recipe

After applying the fix, prove it end-to-end (do not trust eval alone):

1. Run the retriever-type diagnostic above — confirm custom retriever + non-empty
   postprocessors.
2. Pick a query whose correct doc previously ranked low (e.g. a special-topic
   question). Stream the answer and inspect the sources event: the correct
   policy doc must appear, and anchor_missing must be empty (no false degrade).
3. Confirm the answer contains the authoritative number from the fact table (if
   applicable) and contains no numbers absent from sources.
4. py_compile all changed modules; clean up temporary verification scripts.

Resources

• references/anti_hallucination_guards.md — exact SYSTEM_PROMPT, guard
  implementations (rerank threshold + overlap, anchor-term, answer-backtrace),
  fact-table schema + query snippet, KB loader skip rule, and the
  CondensePlusContextChatEngine wiring in full.