Build Personal Wiki

Build a persistent, compounding Markdown wiki with a simple local-first default. Keep raw sources immutable, let the agent maintain synthesized pages, and place every durable content change behind a reviewable proposal. Do not require Git by default.

Non-negotiable rules

• Never move, rename, overwrite, or delete a user's original file.
• Treat raw/ as immutable after ingestion. Ingest a changed source as a new version.
• Scan metadata and hashes only. Before extracting or reading document content, explain that the active agent/model will process it and obtain explicit approval.
• Never read outside approved source roots or the knowledge-base root.
• Never upload, sync, configure a remote, or push. Warn that raw/ may contain private material.
• Generate a proposal and show its Markdown summary and diff before changing durable wiki pages or raw/. Apply those content changes only when the user explicitly approves the matching operation ID. Explicit source-root configuration commands may update config directly because they do not read or ingest content.
• Keep source citations traceable to a raw/ path and SHA-256.
• Never recommend bulk-copying an entire personal folder such as Documents, Downloads, or a WeChat export root into raw/ by default. Treat broad personal directories as candidate source roots; ingest only explicitly approved files.

Locate the CLI

The CLI is a Python 3 script with no third-party dependencies (only pypdf and python-docx are optional, for PDF/DOCX parsing). Locate it in this order:

1. Check scripts/wiki_standalone.py — the preferred single-file version. If it exists, set <WIKI_CLI> to its path and skip the rest.
2. Check scripts/wiki.py — the original three-file version. If it exists, wiki_core.py and wiki_docs.py must be in the same directory. Set <WIKI_CLI> to scripts/wiki.py.
3. Self-extract from references/cli-source.md — if the scripts/ directory is missing or empty (common when installed from SkillHub without bundled scripts), read cli-source.md, extract the Python code from the triple-backtick python code block, write it to a temp file (e.g. ~/.wiki-cli/wiki_standalone.py or %TEMP%/wiki_standalone.py), and use that path as <WIKI_CLI>. The extracted file is fully self-contained.

Run the CLI with: python <WIKI_CLI> <command> --root <knowledge-base> .... Pass paths as separate arguments; paths may contain spaces or Unicode. Verify by running python <WIKI_CLI> --help — it should list all subcommands.

Start or resume

1. If .wiki-system/state.json exists, run status. If it reports a non-idle operation, ask whether to resume it before starting unrelated work.
2. For a new wiki, confirm the Wiki Home directory before asking about sources. Recommend a dedicated folder directly under the user's home directory, usually C:\Users\<username>\PersonalWiki on Windows or /Users/<username>/PersonalWiki on macOS. Also accept any user-specified absolute path.
3. Run check-home on the proposed path. Confirm that the parent directory is accessible/writable and the target directory is missing or empty. Show the resolved path and ask the user for an explicit OK before init. Never initialize into a non-empty directory or silently choose a path.
4. Interview briefly for domain, intended uses, page types, naming style, output language, approved source roots, sensitive exclusions, and per-loop limits.
5. Ask for any missing high-impact choice; do not invent domain schema, sensitive exclusions, source roots, or loop limits. During initialization, explicitly explain the two source modes and let the user choose:
   - Reference-root mode (recommended): approve one or more external source roots for scanning; keep the originals in place; copy only selected approved files into raw/ during ingest.
   - Managed-copy mode: intentionally copy user-provided files into raw/ as the main managed source set.
   Recommend Reference-root mode by default for broad personal folders such as Documents, Downloads, and WeChat exports.
6. If the user does not yet have a clean intake area, suggest creating one such as KnowledgeInbox, TrainingMaterials, or another domain-specific folder. Treat it as an optional convenience, not a requirement.
7. Run init only after the user has confirmed the checked Wiki Home path and source decisions. Default to --version-control none; no Git installation or repository is required. Use --version-control git only when the user explicitly requests local version history. Never create a remote.
8. For first-time setup, immediately run the initial page-building flow against the confirmed Desktop, Downloads, and Documents roots: bootstrap-scan, content-read confirmation, bootstrap-ingest, proposal review, and apply after approval. This creates initial source pages for supported files instead of leaving the wiki empty.
9. Read protocol.md for schema decisions. Read source-strategy.md when designing source-root behavior, raw semantics, upload flows, or post-init source expansion. Read compatibility.md only for installation or host-specific questions.

Confirm Wiki Home before initialization

Use the CLI to resolve and validate the destination without reading document contents:

python <WIKI_CLI> check-home
python <WIKI_CLI> check-home --platform windows --home "C:/Users/example"
python <WIKI_CLI> check-home --platform macos --home "/Users/example"
python <WIKI_CLI> check-home --path "D:/Knowledge/MyWiki"

Present the recommended path and a custom-path choice. Ask the user to confirm one path. Proceed only when ready is true, which means the parent is writable and the target is missing or empty. If the target is non-empty, ask for another empty directory; do not delete, move, or overwrite its contents.

Selecting source roots

Source roots are the approved directories the wiki is allowed to scan and ingest from. Always let the user pick multiple roots and manually enter any custom path. Never assume a single root or skip the selection step.

Step 1 — Discover common directories

Run discover-roots first when the CLI is available, then present the returned existing paths to the user. Keep the simplified default small and predictable: on both Windows and macOS, only discover Desktop, Downloads, and Documents under the user's home directory. If the CLI cannot run, manually check only those three folders.

Report only directories that exist on the user's machine. Preserve the detected platform in the result. Prioritize Desktop, Downloads, then Documents. Do not automatically discover WeChat, cloud-sync folders, Obsidian, Notion, Flomo, or other app-specific directories during the simplified bootstrap. If the user later wants WeChat files or another source, use add-root with the user-provided path.

python <WIKI_CLI> discover-roots
python <WIKI_CLI> discover-roots --platform windows --home "C:/Users/example"
python <WIKI_CLI> discover-roots --platform macos --home "/Users/example"

Use --platform and --home only for testing or when automatic platform/home detection is unavailable.

Step 2 — Present a multi-select with custom-path option

Use AskUserQuestion with multiSelect: true to present Desktop, Downloads, and Documents when they exist. The question header should be "Source roots". Always include a "Custom path" option described as "Manually type another folder path; additional roots can also be added later".

When the user selects "Custom path" (or chooses it in addition to listed folders), follow up with a plain-text prompt asking them to type the full path(s), one per line or comma-separated. Accept any valid filesystem path including:

• WeChat chat record export paths (e.g. D:\WeChat Exports\2024, ~/Documents/WeChat Files/wxid_xxx/MsgExport).
• External drive or USB paths.
• Network share paths (\\server\share).
• Any folder the user considers a knowledge source.

Step 3 — Validate and confirm

For every selected or entered path:

1. Verify the path exists and is a directory (or a readable file — files are allowed as single sources). If a path does not exist, report it and ask the user to correct or skip it.
2. Warn if a path is very broad (e.g. the entire home directory, C:\, or a system folder) — suggest narrowing to a subfolder to avoid scanning irrelevant or sensitive files.
3. Warn if a path appears to be inside a cloud-sync folder (OneDrive, iCloud, Dropbox) — note that raw/ copies will be local but the original may sync or change.
4. List the final set of approved source roots back to the user in a bulleted summary and get a final OK before running init.

Step 4 — Explain how raw/ should work

Before init, explain the storage model plainly:

• raw/ is not a mirror of Documents, Downloads, or WeChat export folders.
• raw/ should usually contain only the subset of files that have been explicitly approved, ingested, or uploaded as durable sources.
• Source roots may live elsewhere on disk; the wiki can describe those roots in config and scan them later without copying the entire directory tree.
• Future user uploads are allowed: when the user later supplies standalone files, ingest those files into raw/ as new durable sources.
• Users may continue adding source roots over time, including WeChat export folders, external drives, cloud-sync subfolders, and other knowledge directories.

Step 5 — Pass to init

Supply every confirmed root as a separate --source-root argument. Set --source-mode reference_roots and --version-control none for the simplified default. Initialization may omit source roots when the user wants to begin only with future uploads; add roots later with add-root.

python <WIKI_CLI> init --root <kb> --domain "machine learning" --purpose "compare papers" --source-mode reference_roots --version-control none --source-root "<desktop>" --source-root "<downloads>" --source-root "<documents>"

Build initial pages after setup

Do not stop after creating an empty wiki. For first-time setup, scan the confirmed Desktop, Downloads, and Documents roots and create source pages for supported, non-duplicate files within the agreed file limit.

1. Run metadata-only discovery:

python <WIKI_CLI> bootstrap-scan --root <kb> --max-files 100

2. Show the candidate file list, including unsupported, duplicate, and large-file flags. Ask the user to confirm content access and the files to process. Narrow the run with repeated --path arguments when the user excludes a root.
3. After explicit content-read approval, create the initial source-page proposal:

python <WIKI_CLI> bootstrap-ingest --root <kb> --allow-content-read --max-files 100

4. Read and present the proposal. Run apply --approve <operation-id> only after the user approves it. Pages and selected files appear in wiki/sources/ and raw/ only at this step.
5. Skip unsupported files, known duplicates, and large files by default. Handle scanned PDFs through the OCR workflow. Never mirror all three directories into raw/.

Run the maintenance loop

1. State the goal, scope, maximum files, and stopping condition.
2. Run scan; review duplicates, unsupported types, large files, and exclusions.
3. Obtain content-processing approval, then pass --allow-content-read. Never infer approval from an unrelated turn.
4. Generate a proposal. This writes .wiki-system/proposals/<operation-id>.json and .md but no durable wiki content.
5. Read the proposal Markdown and diff. Improve synthesis with an agent-authored propose; do not edit durable Wiki files directly.
6. Ask for approval naming the operation ID and affected files.
7. Run apply --approve <operation-id> only after approval. It rechecks hashes and boundaries, validates, and logs the operation. It creates a local Git commit only when the wiki was initialized with --version-control git.
8. Run status. Stop when scope is complete, budget is reached, approval is pending, or validation fails.

Ingest sources

Run ingest on approved files. Supply --title and --summary when a synthesis exists. Without a summary, the source page is marked as needing synthesis.

Treat ingestion as file-level approval, not folder mirroring:

• For broad personal folders, scan first and let the user pick files; do not ingest the full folder by default.
• For clean user-managed intake folders, batch ingest may be acceptable if the user explicitly wants that behavior.
• For future uploads, ingest the uploaded file directly into raw/ and update the manifest as a new durable source.
• For newly added source roots such as WeChat export folders, first validate the path, scan metadata, ask for content-read approval, then ingest selected files.

python <WIKI_CLI> ingest --root <kb> --allow-content-read --source <file> --title <title> --summary <summary>

For standalone user-provided files that are not located under an approved source root, use the explicit upload path. It preserves the same proposal, approval, raw immutability, hash, and manifest guarantees while marking the manifest entry as uploaded_file:

python <WIKI_CLI> upload-ingest --root <kb> --allow-content-read --source <uploaded-file> --title <title> --summary <summary>

For richer integration, inspect pages via query, then use propose --spec <json> for complete Markdown updates. Cite the manifest raw_path and sha256. Update an existing page for facts about the same subject; create one for a distinct linkable concept.

Smart date inference (人性化日期推断)

Each ingest, upload-ingest, and bootstrap-ingest command now infers the most accurate creation date for a source page from five priority levels, and exposes the result in a date_hints array alongside the proposal:

1. User-specified — --date YYYY-MM-DD — explicit override, highest priority.
2. Filename extraction — scans the filename stem for common date patterns:
   - 2024-01-15 / 2024_01_15 / 20240115
   - 2024年01月15日 / 2024年1月15日
   - WeChat: 聊天记录_20240115.html / WeChat_2024-01-15.txt
3. Content extraction — scans the first 5,000 characters for:
   - Desktop WeChat: 2024-01-15 14:30:25 姓名:
   - Mobile exports: 2024/1/15 14:30 / 1月15日 14:30
   - General date tokens: 2024年1月15日 etc.
4. File modification time — os.stat().st_mtime, more meaningful than now() for historical records.
5. System clock — now() as the final fallback, preserving backward compatibility.

Each date_hints entry reports the resolved ISO 8601 timestamp, the matched source label (e.g. filename-Ymd, content-Y-m-d HH:MM, user-specified, file-mtime, current-time), and the filename. The agent and user can review these before approving the proposal.

Examples for a user who bulk-ingests WeChat exports (filenames like 微信群聊_20231201.html):

• Without --date: each page automatically gets created: 2023-12-01T00:00:00Z from its filename.
• If a file has no date in filename or content: it falls back to file mtime.
• If the user wants every page from a specific discussion to use a uniform date: --date 2024-03-01 forces it.

Rich-format document conversion (markitdown integration)

The CLI integrates Microsoft's MarkItDown as the preferred rich-format converter. When installed, it handles PDF, DOCX, PPTX, XLSX, HTML, images (OCR), and audio (transcription) with a single unified converter. When markitdown is not installed, the CLI gracefully degrades to legacy parsers (pypdf, python-docx, built-in HTML text extractor) or provides clear installation hints.

Why this matters for a personal wiki

Personal knowledge bases contain diverse file types — meeting minutes (DOCX), slide decks (PPTX), spreadsheets (XLSX), scanned documents (PDF with OCR), and voice memos (MP3/M4A). Without markitdown:

• PPTX and XLSX files cannot be extracted at all.
• Scanned PDFs only report a warning; text is not available.
• Images remain unprocessed.
• Audio recordings cannot be transcribed.

Installation

pip install 'markitdown[all]'  # recommended: PDF + DOCX + PPTX + XLSX + OCR + audio

Conversion priority chain

extract(path)
  ├── .md/.txt → read directly
  └── .pdf/.docx/.pptx/.xlsx/.html/.htm/images/audio
        ├── markitdown available → MarkItDown().convert(path)  ← preferred
        └── markitdown unavailable → fallback to pypdf / python-docx / HtmlText
                                     or: clear install hint in warnings

Supported formats

Agent guidance

• When ingest results show "unsupported type" or "requires markitdown" warnings, suggest installing markitdown.
• When a scanned PDF warning appears, recommend pip install 'markitdown[all]' for OCR capability.
• When the user asks about WeChat voice messages, images, or exported Office files, the wiki can now process them natively.
• No migration or re-ingest is needed for existing sources. Only new ingest runs benefit from markitdown.

Full documentation: references/markitdown-integration.md

Manage source roots after initialization

Use CLI commands instead of manually editing .wiki-system/config.json:

python <WIKI_CLI> add-root --root <kb> --path "<directory>" --note "WeChat export for research group"
python <WIKI_CLI> list-roots --root <kb>
python <WIKI_CLI> describe-root --root <kb> --path "<directory>" --note "Only scan approved reports"

• Use add-root for optional later sources such as WeChat exports, external drives, cloud-sync subfolders, KnowledgeInbox, and other user-approved directories. These are not part of the simplified default discovery.
• Use list-roots to verify existence, notes, source mode, and broad-root warnings.
• Use describe-root to keep a concise human explanation of what the directory contains and how it should be used.
• Scan remains metadata-and-hash-only and reports the matching source-root note with each file.

Query and save durable answers

Run query; it searches index.md first and then matching Wiki pages. The CLI returns evidence, not an invented answer. Synthesize only from that evidence, cite sources, and state gaps. Save durable answers only through propose and apply.

Lint and repair

Run lint periodically. It checks malformed frontmatter, missing citations, broken wikilinks, orphan pages, duplicate IDs/titles, and manifest/raw mismatches. Additionally inspect possible contradictions, supersession, missing concepts, and synthesis drift. Lint never repairs files; use proposals.

Recover safely

• status reports the active operation and checkpoint.
• resume verifies a pending proposal and source hashes; it does not apply.
• Re-running apply for an applied operation is idempotent.
• Replace stale proposals; never bypass hash checks.

Optional parsers

• Markdown, TXT, and saved HTML use the standard library.
• PDF: python -m pip install pypdf. Scanned/image-only PDFs are automatically detected — the CLI reports a scanned PDF detected warning with page count and extracted char count. When this happens:
  1. Install OCR tooling: ocrmypdf (wraps Tesseract) is recommended — pip install ocrmypdf plus the Tesseract system binary.
  2. Alternatively, use a vision-capable model to read the PDF pages as images and synthesize a summary.
  3. Re-ingest the OCR'd file as a new source version.
• DOCX: python -m pip install python-docx.
• Images are copied and reported for optional vision/OCR; missing OCR is not an error.
• A parser failure affects only that file.

Built-in scan exclusions

The scanner automatically skips these directories and files — they are never knowledge sources:

• Directories: .git, .svn, .hg, node_modules, __pycache__, .venv, venv, .idea, .vscode, .obsidian, site-packages, dist, build, .next, .nuxt, .cache, .trash
• Files: .DS_Store, Thumbs.db, desktop.ini, .gitignore, .gitattributes, .gitmodules

User-configured --exclude patterns are applied on top of these built-in exclusions. If a source root is itself a Git repository (e.g. ~/Documents contains a .git folder), the .git internals will not pollute scan results.

Fixing orphan pages

lint reports orphan_page for any wiki page that is never referenced by [[wikilinks]] from other pages (only index.md links don't count). Orphan pages are not errors — the knowledge is still accessible — but cross-referencing improves discoverability. To fix:

1. Run lint and collect the orphan page list.
2. For each orphan, identify related pages by running query with the orphan's title or key terms.
3. Create a propose --spec that adds [[orphan-page-slug]] links from the most related pages' "Related" or "See also" sections.
4. Apply the proposal. Re-run lint to confirm.

A page that is genuinely standalone (no related concepts) can remain an orphan — this is acceptable.

Changing the wiki directory

The wiki root is set during init and stored in .wiki-system/config.json. To relocate or rename the wiki:

• Move the entire directory: Copy or move the whole wiki/ folder to the new location. The Git repo, raw files, and all state travel together. No config changes needed — all paths inside are relative.
• Update source roots: Use add-root, list-roots, and describe-root. Do not manually edit .wiki-system/config.json. Each approved root is stored as an absolute path with an optional note; the scanner only reads from listed roots.
• Support continuing source expansion: Encourage adding new roots over time — for example WeChat export folders, a dedicated KnowledgeInbox, an external drive folder, or other domain-specific repositories. Validate each path before use; do not assume every root should be batch-ingested.
• Use raw/ as the durable evidence store: Keep concise source descriptions in source pages and config; reserve raw/ for approved, ingested, or user-uploaded source files rather than entire mirrored folders.
• Reset and reinitialize: To start fresh in a new location, run init with a different --root. The old wiki remains untouched.

Never manually edit manifest.json or state.json — use the CLI commands. If the wiki root must be referenced in scripts, always use the --root flag rather than relying on the current working directory.

SkillHub distribution notes

When this skill is installed from SkillHub or another marketplace, the scripts/ directory may or may not be distributed alongside SKILL.md and references/. The skill is designed to work in both cases:

• Full mode (scripts available): use scripts/wiki_standalone.py or scripts/wiki.py directly. All deterministic safeguards (hash verification, path containment, idempotent apply, manifest management, optional Git commits) are active.
• Self-extracted mode (scripts missing, references available): read cli-source.md, extract the embedded Python source to a temp file, and use it as the CLI. All safeguards are active — the extracted file is functionally identical to the bundled script.
• Prompt-only mode (neither scripts nor references available): read prompt-only.md. The agent manually follows the protocol layers and approval gates. Disclose that deterministic boundary checks, hash verification, idempotent recovery, and Git integration are unavailable.

Always try the CLI first (full or self-extracted). Only fall back to prompt-only mode if Python 3 itself is unavailable or the host environment cannot execute scripts.

Degraded prompt-only mode

If the CLI cannot run (Python 3 unavailable, scripts/ missing, and references/ also missing), read prompt-only.md. Preserve the layers and approval gates, and disclose that deterministic boundary, hash, idempotency, and Git checks are unavailable.