文档审计引擎

Overview

AuditAgent performs a structured 6-stage audit of any text document. Each stage is an independent LLM-powered module that examines the document from a different angle. The stages run sequentially with context flowing from one to the next. A quantitative scorer aggregates findings into a weighted composite score (0-100) with pass/fail determination, severity distribution, red flag summaries, and traffic-light visualization. Reports are produced in structured text, self-contained HTML, and machine-readable JSON formats.

Capability Boundaries

What this tool does:
- Run a 6-stage audit pipeline on any document:

• Quantitative scoring with:
• Per-stage scores (0-100) computed from finding severities.
• Weighted composite score with configurable pass/fail threshold (default: 70).
• Severity distribution histogram (critical, high, medium, low, info).
• Red flag extraction (critical + high findings across all stages).
• Traffic-light map (green >= 80, yellow 50-79, red < 50).
• Multi-format reporting:
• Structured text report with per-stage findings, severity levels, evidence quotes, locations, and recommendations.
• Executive summary (1-page, character-limited).
• Action items list sorted by priority.
• Self-contained HTML report with color-coded severity badges, traffic-light stage indicators, severity distribution bars, and finding cards with evidence blockquotes.
• Machine-readable JSON export.
• Customizable stage weights and pass threshold via AuditScorer configuration.

What this tool does NOT do:
- It does NOT audit code, financial statements, or legal contracts with domain-specific rules.
- It does NOT guarantee factual correctness of findings -- LLM output is probabilistic.
- It does NOT modify or annotate the original document.
- It does NOT run in real-time -- it is a batch pipeline.

Trigger Conditions

Use this skill when the user:
- Provides a document (text, article, report, paper) and asks for an audit, review, or quality assessment.
- Wants to identify logical inconsistencies, unstated assumptions, or biases in a text.
- Needs stakeholder analysis or methodology evaluation for a policy document or research paper.
- Asks to "audit this document", "review document quality", "check for bias", "extract claims", "identify assumptions", "文档审计", or "审查文档".

Usage

Python API

from audit_agent import AuditAgent

agent = AuditAgent(model="local")  # or model="qwen2.5:7b"
result = agent.audit("path/to/document.txt")

# Check results
print(result.executive_summary())

# Export reports
result.export_html("output/report.html")
result.export_json("output/report.json")
result.export_text("output/report.txt")

Configuration

The 6-stage pipeline is defined in audit_agent/stages.py and audit_agent/prompts.py. Each stage has:
- A unique stage_name mapping to its prompt template.
- An __init__ that accepts an LLM client.
- A run(document, prior_stage_output, model_name) method that executes the stage.

Scoring Configuration

from audit_agent.scoring import AuditScorer

scorer = AuditScorer(
    weights={
        "coherence_check": 0.20,
        "claim_extraction": 0.15,
        "assumption_surfacing": 0.15,
        "stakeholder_analysis": 0.15,
        "methodology_review": 0.20,
        "bias_detection": 0.15,
    },
    pass_threshold=70.0,
)

Dependencies

• LLM backend: OpenAI-compatible endpoint (default: Ollama with qwen2.5:7b)
• PyYAML for rules configuration.

Output Example

Text Report Structure

=======================================================================
  AuditAgent -- 结构化文档审计报告
=======================================================================
  文档: My Document
  审计时间: 2026-07-06T10:00:00
  综合评分: 68.5/100
  审计结果: 未通过 (FAIL)
=======================================================================
-----------------------------------------------------------------------
  各阶段评分
-----------------------------------------------------------------------
  🟡 一致性检查 (coherence_check): 72/100 (加权贡献: 14.4)
  🟢 主张提取 (claim_extraction): 85/100 (加权贡献: 12.8)
  🔴 隐含假设揭示 (assumption_surfacing): 45/100 (加权贡献: 6.8)
  ...
-----------------------------------------------------------------------
  问题严重度分布
-----------------------------------------------------------------------
  严重 (Critical): 3  |  高危 (High): 7  |  中等 (Medium): 12  |  低危 (Low): 5  |  信息 (Info): 2

-----------------------------------------------------------------------
  一致性检查 — 检测文档内部逻辑矛盾与不一致性
-----------------------------------------------------------------------
  [CRITICAL] COH-001 (severity=90)
    Section 2 claims X while Section 5 implies not-X.
    位置: Paragraphs 12-15
    原文: "The policy achieved its stated goals..."
    建议: Clarify the contradiction between Section 2 and Section 5.
  ...
=======================================================================
  审计结束
=======================================================================

HTML Report Features

• Gradient header with document name and timestamp.
• Circular score display with PASS/FAIL badge.
• Severity distribution bar chart.
• Per-stage sections with traffic-light indicators, finding cards with severity badges, evidence blockquotes, and recommendations.

FAQ

Q: What model should I use?
A: Default is Qwen2.5:7b via Ollama. Any OpenAI-compatible model works. Larger models may produce better quality audits, especially for complex documents.

Q: Can I audit documents in any language?
A: Yes. The stage prompts are in Chinese and the system prompt instructs the LLM to work with the document's language. Multi-language documents are supported.

Q: What does a "pass" mean?
A: A composite score of 70 or above (configurable) indicates the document passed the audit. This is a relative quality metric, not an absolute guarantee.

Q: How long does an audit take?
A: Each of the 6 stages makes an LLM call. With a local Ollama model, expect 30-120 seconds total depending on document length and hardware.

Q: Can I skip certain stages?
A: The pipeline expects all 6 stages. To skip a stage, you would need to modify the scoring weights to zero out unwanted stages and adjust the orchestrator.