Memtrust

MCP文档知识

提供方 未标注

文档知识 未标注 未标注 社区 / 第三方 Claude DesktopClaudeCursorVS Code
查看接入方式

独立的、可复现的智能体记忆后端基准测试框架。

一句话选型结论

按资料字段整理

Memtrust 属于文档知识类 MCP 服务器,传输方式未标注,鉴权方式未标注,可直接接入 Claude Desktop、Claude、Cursor 等,有可直接执行的安装命令。接入前先确认权限范围;涉及写入、删除或付款的操作,建议先在测试环境验证。

分类文档知识 传输方式未标注 鉴权未标注 部署方式未标注
GitHub 星标 未标注 仓库热度,无公开仓库时显示未标注
工具数 未标注 暴露给模型的工具数量
资源 / 提示词 0 / 0 资源与提示词模板数量
维护方 社区 / 第三方 核验等级 仅资料核对

工具与能力

文档知识
数量来自来源目录的标注,具体工具清单以官方文档为准
工具数量未标注暴露给模型可调用的工具个数
资源数量0可读取的资源条目
提示词数量0内置提示词模板数量

能力标签

支持客户端

适用场景

记忆

安装与配置

未标注 未标注
上游未给出传输方式
{ "mcpServers": { "memtrust": { "command": "uvx", "args": ["--from", "memtrust-cli", "memtrust-mcp"] } } }

这条命令只是入口,具体启动参数请按官方文档补齐,不要直接照搬到别的服务器。

  1. 1
    准备运行环境

    按官方文档确认运行环境(常见为 Node.js 或 Python),装好后再执行安装命令。

  2. 2
    安装服务

    复制上面的安装命令执行;命令里带占位符的位置,换成自己的密钥或路径。

  3. 3
    写入客户端配置

    在客户端的 MCP 配置里填好启动命令和参数,保存后完全退出客户端再打开,让新配置生效。

  4. 4
    验证是否可用

    连接成功后随便调一个只读工具,确认能返回结果,再接入正式流程。

典型使用场景

3 类
按分类与标签整理的常见用法,具体以官方文档为准
文档检索问答

在你的文档库里按问题找答案。

知识库读写

把新资料写进知识库,随用随取。

内容整理

把零散资料归纳成结构化笔记。

选择建议

选之前先看这几条
按条目字段给出的判断,不替你决定用不用

值得选的地方

  • 有现成的安装命令,不用自己找入口,照着跑就能起服务。
  • 已适配 Claude Desktop、Claude、Cursor,换客户端不用重新研究配置。

先想清楚的地方

  • 资料没标注工具数量,实际能力要装好后确认。
  • 版本更新与行为细节以官方仓库为准。

更适合

  • 需要在文档知识场景里让模型直接调用外部能力的人。
  • 已经在用 Claude Desktop、Claude 的用户,接入路径最短。
  • 想先快速试一下、能接受在本地或自有环境跑服务的人。

不太适合

  • 需要厂商级 SLA 和技术支持的生产环境。

同类对比

6 个同类
同分类里挑最接近的几条,字段以各自详情页为准
MCP 服务器工具数传输方式部署方式安装命令最后核验
Memtrust当前 未标注 未标注 未标注 有 未标注
dify 未标注 stdio · 本地进程 本地部署 有 2026-04-11
open-webui 未标注 stdio · 本地进程 本地部署 有 2026-04-11
JeecgBoot 未标注 stdio · 本地进程 本地部署 有 2026-04-11
Cherry Studio 未标注 未标注 未标注 无 2026-05-17
mempalace 未标注 stdio · 本地进程 本地部署 有 2026-04-11

服务器档案

文档知识
字段来自项目仓库与公开索引,中文说明由本站标注
提供方未标注上游标注的维护方 未标注
分类文档知识本站按上游分类展示
连接方式未标注上游未给出传输方式
鉴权方式未标注远程服务建议用最小权限凭据
维护方社区 / 第三方核验等级 仅资料核对
包名 / 运行时未标注 · 未标注本地安装时用到的包标识与运行时
部署类型未标注上游给出的部署形态
最近核验未标注上游最后核验时间

详细介绍

51.9 KB
来自项目仓库与上游目录的原文,未做删改

memtrust

Agent memory backends each publish their own benchmark numbers, on different tests, measured
different ways. memtrust runs the same evals against all four and publishes the raw logs. Run
against the vendors, not by them.```bash
pip install memtrust-cli
memtrust run --backends mempalace,mem0,zep,openviking --eval all


(For contributing to this repo instead of just running it, see Development --
`pip install -e ".[dev]"` from a clone.)

**Contents:** Why this exists · What it does ·
Commands · How this differs ·
Contradiction detection ·
Compression fidelity ·
Temporal-KG boundary ·
The landscape · Benchmarks ·
GitHub Actions usage · Self-host · Install ·
Hosted layer · Backend coverage ·

## Why this exists

If you've compared agent-memory backends recently, you've probably noticed each one leads with a
different accuracy number, on a different benchmark, measured a different way. MemPalace's own
community already flagged the problem in public. Issue #27
on the MemPalace repository, opened April 7, 2026 and still open, documents that a headline 100%
LongMemEval figure, measured with Haiku reranking, wasn't reproducible from the repository's own
benchmark scripts and was pulled from the README as unverifiable. A separate 96.6% figure people
cite everywhere turns out to be mostly ChromaDB's default embeddings doing the work in raw mode,
not MemPalace's own architecture. A "lossless" compression claim (the "AAAK" mode) drops the same
LongMemEval score from 96.6% to 84.2% in practice, a 12.4 percentage point gap. Two internal pull
requests attempting to fix the reporting problem, #433 and #729, were both closed without merging
on April 12, 2026 -- #729 within seven minutes of being opened. As of this writing, the issue has
232 thumbs-up reactions and 39 comments.

None of that means MemPalace, or any other backend, doesn't work. It means nobody outside the
vendor had run the same test, the same way, against every option, and published the raw logs.

memtrust does that. It runs LongMemEval, LoCoMo, and a growing set of evals built specifically for
this project -- 17 of them as of this writing, all registered in the CLI's `--eval` flag. The two
that matter most for understanding what this project is actually for:
contradiction detection, because neither LongMemEval nor LoCoMo tests the question that actually
matters once a memory system sits underneath a production agent -- what happens when a new fact
contradicts an old one? Does the backend flag the conflict? Silently overwrite the old fact with no
audit trail? Serve whichever version it happens to retrieve first? None of the four backends this
project tracks publish a number for that. And compression/round-trip fidelity, built to directly
test claims like the "lossless" one above: it stores content, retrieves it, and scores literal
reconstruction fidelity rather than semantic accuracy, per operating mode a backend exposes (see
`MemoryBackendAdapter.supported_modes`) -- the mechanism that would let a contributor with live
MemPalace credentials actually reproduce the 12.4-point compressed-mode accuracy drop
mempalace/mempalace#27 documents, instead of just citing it. **Neither has been run against a live
MemPalace instance as of this writing** -- both have, however, been run against a live
self-hosted `mem0ai` install; see "Benchmarks" below. The other evals -- ranking quality,
crash recovery, extraction quality, embedding drift, scale/volume stress, lock contention, stats
accuracy, orphan cleanup, result consistency, migration rollback, filter injection, resource-sync
safety, and temporal-KG boundary detection -- each grew out of a specific real bug report against
one of the four tracked backends; see "Success stories" below for the full list.

**Where this stands right now, in one place:** live benchmark results for one
backend (`mem0_direct`, self-hosted `mem0ai`), including a real bug this project's own attempt to
get those numbers surfaced in mem0's default configuration; and 197 real GitHub issues and PRs
filed against MemPalace, Mem0, Zep/Graphiti, and OpenViking independently root-caused against this
codebase -- 55 (28%) PASS, 16 (8%) PARTIAL, 42 (21%) a genuine capability gap, 84 (43%) not
applicable, every verdict re-verified by a reviewer independent of whoever built the fix.

## What it does

Every command below was actually run against this repo, with zero vendor API keys configured, to
produce the output shown. Nothing here is simulated.```
$ memtrust run --backends mempalace,mem0,zep,openviking --eval all
memtrust 0.3.4 -- run_id=mt_2026-08-04T061759Z
Backends: mempalace, mem0, zep, openviking   Evals: longmemeval, locomo, contradiction,
resource_sync_safety, compression, ranking_quality, scale_stress, embedding_drift, crash_recovery,
extraction_quality, migration_rollback, filter_injection, lock_contention, stats_accuracy,
orphan_cleanup, result_consistency, temporal_kg_boundary

mempalace: SKIPPED (not configured) -- mempalace is not configured: environment variable
MEMPALACE_STORAGE_PATH is not set. Skipping this backend. See docs/methodology.md for setup
instructions.
mem0: SKIPPED (not configured) -- mem0 is not configured: environment variable MEM0_API_KEY is not
set. Skipping this backend. See docs/methodology.md for setup instructions.
zep: SKIPPED (not configured) -- zep is not configured: environment variable ZEP_API_KEY is not set.
Skipping this backend. See docs/methodology.md for setup instructions.
openviking: SKIPPED (not configured) -- openviking is not configured: environment variable
OPENVIKING_API_KEY is not set. Skipping this backend. See docs/methodology.md for setup
instructions.

Cost: $0.00 (no LLM-judged evals ran -- structural evals only, or judge not configured)

Full report: memtrust-report-2026-08-03.json

That's the real, reproducible behavior of a fresh clone with no credentials: every backend reports
SKIPPED, the command exits cleanly, and a valid JSON report is still written. memtrust --version
now correctly prints the installed version, matching pip show memtrust-cli. Earlier releases printed
0.0.0+unknown even when properly installed, because src/memtrust/__init__.py read
importlib.metadata.version("memtrust") while the installed distribution is actually named
memtrust-cli -- kept in the FAQ below for the record rather than deleted, since silently erasing
a bug the moment it's fixed is exactly the kind of curation this project exists to push back on in
other people's benchmarks. Set the relevant environment variable for any backend you want to
actually test (MEM0_API_KEY, ZEP_API_KEY, OPENVIKING_API_KEY, MEMPALACE_STORAGE_PATH) and
that backend runs for real against its live API instead of being skipped.

The eval logic itself is proven offline, against the bundled synthetic fixtures and, for several
adapters, the real installed vendor packages with only the network boundary mocked, by the test
suite:```
$ pytest --cov=memtrust --cov-report=term-missing
... (33 module rows total; the 11 most relevant to this README are shown below)
Name Stmts Miss Cover


src/memtrust/adapters/base.py 290 1 99%
src/memtrust/adapters/mempalace_adapter.py 265 15 94%
src/memtrust/adapters/mem0_adapter.py 140 12 91%
src/memtrust/adapters/mem0_direct_adapter.py 281 34 88%
src/memtrust/adapters/openviking_adapter.py 178 18 90%
src/memtrust/adapters/zep_graphiti_adapter.py 63 3 95%
src/memtrust/adapters/zep_graphiti_selfhosted_adapter.py 165 24 85%
src/memtrust/evals/contradiction.py 127 2 98%
src/memtrust/evals/compression.py 86 1 99%
src/memtrust/evals/temporal_kg_boundary.py 90 3 97%
src/memtrust/receipt.py 118 10 92%


TOTAL 4167 265 94%

590 passed, 8 skipped in 5.92s


This is an excerpt, not the full table -- the weakest-covered module in the repo,
`evals/mempalace_metadata_scale.py` (70%), isn't one of the 11 shown above; run the command
yourself for the complete per-module breakdown.

590 passing tests across 33 source modules, 94% overall statement coverage, 98% on the
contradiction-detection eval, 99% on compression/round-trip fidelity, 97% on the temporal-KG
boundary eval, 85-99% across the adapter layer. The 8 skips are live-`mempalace`-package tests that
only run with the optional `mempalace-direct` extra installed (`pip install -e
'.[dev,mempalace-direct]'`). Every test mocks its HTTP or wire
layer, or uses an in-memory fake backend -- none of them touch a real network, though a meaningful
share of the adapter tests now import and exercise *real installed vendor classes* directly
(`mem0ai==2.0.12`'s embedder and vector-store modules, and -- gated behind the optional
`mempalace-direct` extra -- the real `mempalace.mcp_server` functions), mocking only the outermost
network or wire-client boundary rather than the whole library. `graphiti-core` is not installed in
this environment, so its self-hosted adapter's tests still run against a hand-written Protocol
double built to match the real package's confirmed method signatures, not the real classes -- see
`docs/methodology.md`'s adapter confidence table for exactly which claim rests on which kind of
verification.

## Commands```
$ memtrust --help
Usage: memtrust [OPTIONS] COMMAND [ARGS]...

  memtrust: an independent, reproducible benchmark harness for agent-memory
  backends.

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  keygen  Generate a new Ed25519 keypair for signing `memtrust run`...
  report  Read a prior `memtrust run` JSON report and print a formatted...
  run     Run the eval suite against the requested backends.
  verify  Verify a signed receipt produced by `memtrust run --sign`.
Command Flags What it does
memtrust run --backends TEXT comma-separated list or all (default all) · --eval TEXT comma-separated from longmemeval,locomo,contradiction,resource_sync_safety,compression,ranking_quality,scale_stress,embedding_drift,crash_recovery,extraction_quality,migration_rollback,filter_injection,lock_contention,stats_accuracy,orphan_cleanup,result_consistency,temporal_kg_boundary, or all (default all) · --output FILE (defaults to ./memtrust-report-.json) · --locomo-dataset-path FILE points the LoCoMo eval at a real, downloaded locomo10.json instead of the bundled synthetic fixture (memtrust does not bundle or auto-fetch the real dataset) · --locomo-exclude-question-ids-file FILE excludes known-bad-ground-truth LoCoMo question IDs from scoring · --scale-stress-n-records INTEGER (default 500) sets how many synthetic records the scale-stress eval stores and re-queries · --sign FILE writes a signed .receipt.json alongside the report, proving it was produced by the holder of the given Ed25519 private key Runs the eval suite against the requested backends. A backend without its credential env var set prints SKIPPED and the run continues -- this command never crashes on missing credentials. temporal_kg_boundary only applies to the mempalace backend (the only adapter that wires kg_add/kg_invalidate/kg_query); requesting it against any other backend reports not_applicable, not an error.
memtrust report REPORT_PATH positional path to a prior JSON report · --json prints the parsed report as JSON instead of a formatted summary Reads a report written by memtrust run and prints a formatted summary.
memtrust keygen --private-key-out FILE (default memtrust-key.pem) · --public-key-out FILE (default memtrust-key.pub) · --force overwrites existing output files Generates a new Ed25519 keypair for signing reports with run --sign.
memtrust verify RECEIPT_PATH --public-key FILE (or the MEMTRUST_RECEIPT_PUBLIC_KEY env var) · --json prints the result as JSON Verifies a signed receipt produced by memtrust run --sign; a tampered or mismatched receipt fails verification.
memtrust --version -- Prints the installed version.

Every line above came straight from running memtrust --help, memtrust run --help,
memtrust report --help, memtrust keygen --help, and memtrust verify --help against this
repo. Nothing here is invented.

MCP Server

memtrust ships a Model Context Protocol server so an AI agent
(Claude, Cursor, or any MCP-compatible client) can run a memory-backend benchmark directly, without
a human invoking the CLI by hand.

Install the extra:```bash
pip install "memtrust-cli[mcp]"


Add it to your MCP client's config (for Claude Desktop, `claude_desktop_config.json`):```json
{
  "mcpServers": {
    "memtrust": {
      "command": "uvx",
      "args": ["--from", "memtrust-cli", "memtrust-mcp"]
    }
  }
}

The server exposes one tool, run, that shells out to memtrust run with the given arguments
(memtrust has no --json flag, so the wrapper writes to a private temp --output file and reads
it back) and returns the parsed JSON report:```
run(["--backends", "mempalace", "--eval", "stats_accuracy"])


Transport is stdio, so there is nothing to host: the MCP client spawns the server as a local
subprocess. Source: `src/memtrust/mcp_server.py`.

## How this differs from trusting a vendor's own numbers

Every backend memtrust tracks publishes its own benchmark numbers. None of them publish the same
benchmark, scored the same way, with the same held-out discipline. memtrust doesn't ask you to
trust it instead: it asks you to read the raw logs. Every run's methodology, prompt templates,
dataset versions, and scoring rubric are published in `docs/methodology.md`, versioned alongside
the code that produced them. If the methodology has a flaw, it's a flaw you can point to in a
specific file and line, not something buried in a vendor's internal eval pipeline.

General-purpose LLM eval frameworks (promptfoo, DeepEval, RAGAS, and similar tools) are mature and
widely used, but none of them ship a memory-backend adapter abstraction or a contradiction-
detection eval out of the box -- they're built for RAG quality, red-teaming, and general prompt
evaluation, not for comparing how different memory systems handle a fact that changes over time.
memtrust is narrower and more specific on purpose.

## The landscape (verified, not benchmarked)

Real, publicly checkable numbers as of this writing (`gh api repos//`), not
memtrust-run scores -- accuracy and contradiction-handling comparisons stay in the "Benchmarks"
section below until a live run actually produces them:

| Backend | GitHub stars | Self-reported description |
|---|---|---|
| MemPalace | 58,032 | "The best-benchmarked open-source AI memory system. And it's free." |
| Mem0 | 62,450 | "Universal memory layer for AI Agents" |
| Zep / Graphiti | 29,526 | "Build Real-Time Knowledge Graphs for AI Agents" |
| OpenViking | 27,859 | "Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills." |

None of these numbers say anything about which backend handles a contradicted fact correctly --
that's the whole reason the harness exists. Star count measures adoption, not correctness.

## The eval that actually matters: contradiction detection

LongMemEval and LoCoMo both measure recall: can the backend remember a fact you told it earlier.
That's necessary but not sufficient. The harder question is what a backend does when two facts
conflict: you tell it your meeting is at 2pm, then later say it moved to 3pm. Does it flag the
change? Overwrite silently? Serve whichever one it retrieves first? `memtrust`'s classifier stores
a fact, stores a contradicting fact, queries for it, then checks the actual retrieved content for
both values, rather than trusting whatever conflict signal the adapter itself reports. See
`src/memtrust/evals/contradiction.py` and the scoring-logic section of `docs/methodology.md` for
exactly how that classification works.

## The eval built for the other headline overclaim: compression fidelity

mempalace/mempalace#27 documents two separate overclaims, not one: the LongMemEval score gap
described above, and a "lossless" compression claim that measured 12.4 percentage points lower in
practice under a compressed operating mode. memtrust could not previously reproduce that second
number at all -- there was no way to tell an adapter "run this under mode X vs mode Y" through the
shared interface. `MemoryBackendAdapter.store()`/`query()` now accept an optional `mode: str |
None` parameter, and `MemoryBackendAdapter.supported_modes` lets an adapter declare which mode
strings it actually understands (`MemPalaceAdapter.supported_modes` is `("raw", "AAAK")`, the two
names mempalace/mempalace#27 itself uses -- see `src/memtrust/adapters/mempalace_adapter.py` for
the exact provenance and confidence caveat on those names). Adapters with no mode variants accept
and ignore the parameter, so this is a purely additive, backward-compatible interface change.

`src/memtrust/evals/compression.py` runs the same store-then-retrieve round trip once per mode a
backend reports, and scores each round trip with a direct, deterministic character-level
similarity ratio (`fidelity_ratio()`, via `difflib.SequenceMatcher` -- not an LLM judge, since a
"lossless" claim is a literal-reconstruction claim, not a semantic one). This is what would let a
contributor with live MemPalace credentials point `memtrust run --eval compression` at it and
reproduce a "raw vs AAAK" fidelity gap directly. **As of this writing this eval has not been run
against a live MemPalace instance** -- it has been run against a live self-hosted `mem0ai` install
(mean fidelity 31.6%, see "Benchmarks" below); see `docs/methodology.md` for the same
live-credentials caveat that applies to every other eval and backend not yet measured live.

## The eval built from MemPalace's own bug: temporal-KG boundary detection

MemPalace/mempalace#1913 (fixed by merged PR#1914, contributor ggettert) described a real,
concrete bug: `_temporal_filter_sql`'s `as_of` point-in-time query used a closed interval on both
ends, so a fact whose `valid_to` equaled the query's exact `as_of` instant still matched. Hand-roll
a fact change as `kg_invalidate(ended=T)` immediately followed by `kg_add(valid_from=T)` at the
identical boundary instant -- the exact pattern MemPalace's own pre-fix agent guidance told every
caller to do -- and an `as_of=T` query returns both the just-ended fact and its just-started
successor at once, so a single-valued fact reports two contradictory answers with no error.
`src/memtrust/evals/temporal_kg_boundary.py` reproduces that exact hand-rolled sequence against
`MemPalaceAdapter`'s `kg_add()`/`kg_invalidate()`/`kg_query()` and classifies the result with a new
`TemporalBoundarySignal` taxonomy, distinct from `ConflictSignal` and `RankingSignal` because it
concerns one narrow, structurally different failure: two facts sharing one instant, not a
contradiction across time or a ranking-order question.

Honest scope, stated the same way this project states it for every other eval: the real
`mempalace` PyPI package is not installed in this build environment, and PR#1914's fix had not
shipped in a released `mempalace` version as of this adapter's live-verified 3.5.0 build -- it
lands under the package's `[Unreleased]` changelog section. `tests/test_temporal_kg_boundary.py`
proves the *classification logic* is correct against two hand-written fake implementations that
reproduce the confirmed pre-#1914 (closed-interval) and post-#1914 (half-open-interval) SQL
comparison exactly. **This has not been run against a live MemPalace instance.** It is wired into
`memtrust run --eval temporal_kg_boundary` (see "Commands" above); against any backend other than
`mempalace`, it reports `not_applicable` rather than an error.

## Benchmarks

**Live results: mem0_direct (self-hosted), July 2026.** MemPalace, Zep, and OpenViking are still
not yet measured against a live backend -- see "Backend coverage" below for the confidence level
on each adapter. Mem0 has one real result, produced against the actual `mem0ai` OSS library
running self-hosted -- in-process, via `Mem0DirectAdapter`, backed by a local Qdrant instance and
the OpenAI API for embeddings and extraction.

> [!NOTE]
> This result is from the self-hosted `mem0ai` OSS library, not Mem0's hosted Platform API. Don't
> read it as a claim about the hosted product.```
$ export MEM0_DIRECT_EMBEDDER_PROVIDER=openai
$ export MEM0_DIRECT_VECTOR_STORE_PROVIDER=qdrant
$ export MEM0_DIRECT_VECTOR_STORE_URL=http://localhost:6333
$ memtrust run --backends mem0_direct --eval contradiction,compression,extraction_quality
memtrust 0.3.2 -- run_id=mt_2026-07-20T210918Z
Backends: mem0_direct   Evals: contradiction, compression, extraction_quality

mem0_direct: configured, running evals...
  Running Contradiction-Detection against mem0_direct...
    flagged: 0.0%  silent-overwrite: 100.0%  served-stale: 0.0%  empty-or-lost: 0.0%
  Running Compression/Round-Trip-Fidelity against mem0_direct...
    fidelity by mode -- default: 31.6%
  Running Extraction-Quality against mem0_direct...
    junk-retained: 0.0%  valid-lost: 100.0%
    feedback-loop-duplicate: 0.0%

Cost: $0.00 (no LLM-judged evals ran -- structural evals only, or judge not configured)

The full raw report is committed at leaderboard/mem0_direct-2026-07-20.json, and leaderboard/data.json
carries the contradiction numbers into the static leaderboard site (mempalace/mem0/zep/openviking
still show not_measured there; mem0_direct is the one real row).

What that means case by case, not just the percentage:

  • Contradiction detection, 7/7 cases: every contradicting fact silently overwrote the old one.
    0% were flagged as a conflict, 0% served stale, 0% empty-or-lost. Tell it your meeting moved from
    2pm to 3pm and it stores the new fact with no signal that anything changed -- this is exactly the
    question LongMemEval and LoCoMo don't test, and exactly what "Why this exists" above is about.
  • Compression/round-trip fidelity, 5 cases: 31.6% mean literal character-level reconstruction.
    This is expected, not a defect -- mem0's design goal is semantic fact extraction, not verbatim
    storage, so a literal-reconstruction score was never going to be high. It quantifies what "not
    built for lossless storage" concretely means for this backend: ask it what you said and you get
    the gist back, not your words.
  • Extraction quality, 15 cases (12 deliberately junk, 3 deliberately valid): 0% junk retained,
    100% of the valid cases lost.
    All 12 junk inputs (boot-file restating, cron heartbeat noise,
    system dumps, hallucinated-profile bait) were correctly rejected. All 3 valid-content cases were
    also dropped -- stored but never came back on retrieval. The valid-side sample is small (n=3);
    treat this as a signal worth digging into further, not a settled number.

> [!WARNING]
> A real bug this run surfaced in mem0ai itself, not in memtrust: a fresh mem0ai==2.0.12 install
> with nothing but OPENAI_API_KEY set fails every single LLM-based extraction call, out of the
> box, for anyone.

Getting any of the numbers above required a fix first. mem0's own default model
(mem0/llms/openai.py: self.config.model = "gpt-5-mini") is a reasoning-tier model that only
accepts the API's default temperature, but mem0's own reasoning-model detection
(mem0/llms/base.py's reasoning_models set) checks for the string "gpt-5o-mini", not
"gpt-5-mini" -- two different strings, so the check never fires, and mem0 sends temperature=0.1
on every call regardless. The result is a 400 Unsupported value: 'temperature' does not support 0.1 with this model error on every extraction call, silently caught by mem0 and reported by
memtrust as N/A (no scoreable cases) rather than a real result. Mem0DirectAdapter now works
around it by passing is_reasoning_model=True explicitly -- mem0's own documented override for
exactly this situation -- see src/memtrust/adapters/mem0_direct_adapter.py's "Default LLM
extraction is broken out of the box" section for the full citation. No upstream mem0ai issue filed
for this as of this writing.

To reproduce this or measure the remaining three backends:```bash
export MEM0_API_KEY=... # and/or
export ZEP_API_KEY=...
export OPENVIKING_API_KEY=...
export MEMPALACE_STORAGE_PATH=...
export MEMTRUST_JUDGE_API_KEY=... # needed for LongMemEval/LoCoMo grading; contradiction-detection doesn't need it

memtrust run --backends mempalace,mem0,zep,openviking --eval all
memtrust report memtrust-report-<date>.json


The command prints per-backend accuracy and contradiction-handling rates, writes a full JSON
report, and prints an estimated cost for any LLM-judged evals that ran. `MemPalaceAdapter`&#x27;s
drawer and knowledge-graph calls are now live-verified against a real installed instance (see
&quot;Backend coverage&quot; below), but OpenViking&#x27;s memory-write/query paths, and parts of the
self-hosted Mem0 and Zep/Graphiti adapters, are still built against best-effort interpretations of
documented or source-read product concepts rather than a live-confirmed API -- see the confidence
table in `docs/methodology.md` before treating any adapter&#x27;s output as authoritative, and consider
that table&#x27;s gaps a standing invitation to contribute a fix.

**Labeling requirement for any future `accuracy` figure published here.** LongMemEval and LoCoMo
`accuracy` grades the LLM judge&#x27;s verdict on raw retrieved-record content directly -- there is no
answer-generation step in either eval runner. This is not the same measurement as the official
LongMemEval/LoCoMo leaderboards&#x27; generate-then-judge QA-accuracy scores. Any `accuracy` number
this project publishes for those two evals must be labeled &quot;retrieval-graded accuracy,&quot; not bare
&quot;accuracy,&quot; and must not be directly compared to leaderboard figures without that caveat. See
`docs/methodology.md`&#x27;s &quot;Retrieval-graded accuracy vs. generated-answer accuracy&quot; section.

## GitHub Actions usage

Run the suite on a schedule and publish results to the leaderboard:```yaml
name: memtrust-leaderboard
on:
  schedule:
    - cron: &quot;0 9 * * 1&quot;  # weekly
  workflow_dispatch: {}

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: &quot;3.12&quot;
      - run: pip install memtrust-cli
      - run: memtrust run --backends mempalace,mem0,zep,openviking --eval all --output leaderboard/data.json
        env:
          MEM0_API_KEY: ${{ secrets.MEM0_API_KEY }}
          ZEP_API_KEY: ${{ secrets.ZEP_API_KEY }}
          OPENVIKING_API_KEY: ${{ secrets.OPENVIKING_API_KEY }}
          MEMTRUST_JUDGE_API_KEY: ${{ secrets.MEMTRUST_JUDGE_API_KEY }}
      - run: git add leaderboard/data.json &amp;&amp; git commit -m &quot;Update leaderboard&quot; &amp;&amp; git push

This repo's own CI (.github/workflows/ci.yml) runs lint, type-check, test, and a dependency
security audit on every push and pull request -- no vendor credentials required, since every test
runs fully offline.

Self-host```bash

git clone https://github.com/RudrenduPaul/memtrust
cd memtrust
pip install -e ".[dev]"
export MEM0_API_KEY=...
memtrust run --backends mem0 --eval all


Point an adapter at your own backend, or run the suite against your own conversation data instead
of the bundled synthetic fixtures (see `docs/methodology.md`&#x27;s note on swapping in the real
LongMemEval/LoCoMo datasets). Nothing leaves your machine unless you choose to publish it.

## Install

`pip install memtrust-cli` is the verified, working install path -- confirmed against a clean
virtualenv as of this writing. `pip show memtrust-cli` and `memtrust --version` both report
`0.3.4`.

### npx (currently broken -- tracked, not hidden)

The npm package (`memtrust-cli`) is live and no longer 404s, and its source on `main`
(`npm/memtrust-cli/bin/memtrust.js`) correctly runs `uv tool run --from
memtrust-cli== memtrust `. The published `0.3.4` npm tarball, however, still
ships the earlier, broken build of that same file, which runs `uv tool run --from
memtrust== memtrust ` instead -- pointed at a PyPI project named `memtrust` that
has never existed (`pypi.org/pypi/memtrust/json` returns 404, same as the FAQ below already
documents). The source fix landed on `main`; the npm publish that would ship it has not gone
out yet. Confirmed live, today, by downloading the actual published tarball
(`npm pack memtrust-cli@0.3.4`) and inspecting `bin/memtrust.js` directly, not by reading source
and assuming it matches what&#x27;s published:```bash
$ npx -y memtrust-cli --version
npm error could not determine executable to run
  × No solution found when resolving tool dependencies:
  ╰─▶ Because memtrust was not found in the package registry and you
      require memtrust==0.3.4, we can conclude that your requirements are
      unsatisfiable.

Until a new npm version ships with the fixed wrapper, use pip install memtrust-cli (above) --
it is unaffected, since the bug is only in the npm wrapper script, not the PyPI package it
bootstraps. For CI and agent runners that have Node.js but not Python: hold off on npx memtrust-cli until this section no longer carries this notice, or provision Python and use pip install memtrust-cli directly.

…(正文过长,页面只展示前面部分,完整内容见仓库)

常见问题

6 问
答案由本站依据条目字段整理,不引用上游原文
Memtrust 怎么安装?

在客户端按下面的安装命令接入:{ "mcpServers": { "memtrust": { "command": "uvx", "args": ["--from", "memtrust-cli", "memtrust-mcp"] } } }。装完在客户端配置里引用同一个包或端点。

Memtrust 怎么在 Claude Desktop、Claude、Cursor 里配置?

它的传输方式未标注,在客户端的 MCP 配置里填好启动命令和参数,保存后完全退出客户端再打开,配置才会生效。

Memtrust 收费吗?

本站不代收费用,也没有标注具体价格。是否收费取决于服务本身和背后的接口额度,正式使用前按官方说明确认。

Memtrust 需要 API Key 吗?

资料没有标注鉴权方式,是否需要密钥取决于服务本身,接入前按官方文档确认。

Memtrust 支持哪些客户端?

资料标注支持 Claude Desktop、Claude、Cursor、VS Code、Gemini CLI、Codex;其他客户端只要能配置 MCP Server,也能用同样的命令或地址接入。

Memtrust 能提供哪些能力?

资料显示能力标签包括 记忆。具体工具清单以官方文档为准。

热门专题

按标签挑服务器
全部专题 →

服务器信息以作者仓库与官方文档为准;分类、传输方式与鉴权方式的中文说明为中文整理,最近核验 未标注。