MCP Memory Server
A self-hosted semantic memory layer for Claude Code and other MCP clients. Store memories as text, search them by meaning, and capture thoughts from your phone — all on your own infrastructure.
Built with FastMCP, Postgres + pgvector, and OpenAI or local (Ollama) embeddings.
Architecture```
Claude Code ──HTTPS──▶ Reverse proxy ──▶ FastMCP server ──▶ Postgres + pgvector
▲ │
Phone/browser ──HTTPS───────┘ ▼
(capture form) OpenAI API
(embeddings + metadata)
- **Database**: Postgres 17 with pgvector — stores text alongside vector embeddings (1536 dims for OpenAI, 768 for nomic-embed-text) and the model that produced each one
- **Server**: Python 3.13 + FastMCP — Streamable HTTP transport with Bearer token auth
- **Embeddings**: OpenAI `text-embedding-3-small` by default, configurable to any OpenAI-compatible API or to Ollama's native API (see Local embeddings). The server refuses to start if the configured model doesn't match the stored vectors
- **Search**: Hybrid — a vector candidate list fused by reciprocal rank fusion (`RRF_K`, default 60) with a keyword list that admits only strong matches: memories containing the query's identifier-like tokens verbatim (hostnames, config keys, hashes, error codes; weighted 2×), otherwise memories matching every query term, provided they are also in the vector top 10 (so long memories can't get in on word count alone). Each result's `keyword_mode` says which applied
- **Metadata**: the calling client can pass a memory's `action_items` and `memory_type`. Otherwise, if `OPENAI_API_KEY` is set, GPT-4o-mini extracts them (plus tags and entities); it's best-effort and never blocks storage
- **Duplicate detection**: Cosine similarity check before insert (default threshold 0.95, configurable via `DUPLICATE_THRESHOLD`; the right value depends on the embedding model)
- **Related on store**: a successful store also returns `related_earlier`: up to 5 earlier memories at or above `RELATED_ON_STORE_THRESHOLD` (default 0.85), each with its open action items. Memories go stale when a session saves "PR1 shipped" and leaves "PR1 not started" reading as open; the session that wrote the outcome is the cheapest place to close the plan, so the server instructions ask it to resolve the items (`evidence_id` = the new memory) or add a dated STATUS line. Best-effort: a failed lookup never fails the store
- **Indexing**: HNSW (not IVFFlat) — works on empty tables
## Tools
| Tool | Description |
|---|---|
| `store_memory` | Save text with auto-generated embedding, plus optional `action_items` / `memory_type` (or AI-extracted metadata when no client metadata is given and OpenAI is configured). Detects near-duplicates (configurable threshold) — pass `force: true` to skip. |
| `store_memories` | Batch store up to 20 memories in one call. Each is processed concurrently with independent duplicate detection. |
| `search_memory` | Hybrid semantic + full-text search. Optionally filter by `tags` (all must match) and/or `source`. |
| `list_recent` | List the last N memories, optionally filtered by source |
| `delete_memory` | Remove a memory by UUID |
| `update_memory` | Update content, tags, metadata, `action_items` or `memory_type` on an existing memory. Re-embeds automatically if content changes. |
| `find_related` | Find clusters of semantically similar memories — candidates for consolidation. Checks each memory's nearest neighbours via the HNSW index, then union-find clusters the pairs. |
| `weekly_review` | Summarize the last N days: grouped by date, type/tag distribution, open action items. Pass `include_resolved=true` to also see resolved items. |
| `resolve_action_items` | Mark action items on a memory as done (by 0-based index into `metadata.ai.action_items`). Persists to `metadata.ai.resolved_indices`; an optional `reason` and `evidence_id` are kept per item in `metadata.ai.resolutions`. |
| `unresolve_action_items` | Reverse a previous `resolve_action_items` call — remove indices from the resolved list. |
| `memory_stats` | Aggregate dashboard: totals, sources, top tags, 30-day activity |
## Prompts
| Prompt | Description |
|---|---|
| `memory_migration` | Instructs the LLM to extract everything worth keeping from the current conversation and store each piece as a separate memory |
| `quick_capture` | Takes raw text, has the LLM determine optimal phrasing/tags/metadata, then stores it |
## Setup
### Prerequisites
- Docker and Docker Compose
- An OpenAI API key, unless you use local embeddings (Ollama) and client-supplied metadata
- A reverse proxy that handles TLS (e.g. Caddy, Nginx, Traefik)
### 1. Configure environment```bash
cp .env.example .env
Edit .env:```env
POSTGRES_USER=memory
POSTGRES_PASSWORD=<generate-a-strong-password>
POSTGRES_DB=memory
DATABASE_URL=postgresql://memory:<password>@mcp-memory-db:5432/memory
OPENAI_API_KEY=sk-...
MCP_API_KEY=<generate-with-openssl-rand-hex-32>
Optional — scoped API keys that force a source name. Define these in a
gitignored docker-compose.override.yml rather than here (see "Scoped API keys"):
MCP_API_KEY_<NAME>=<token> # Memories stored with this key get source="<name>"
Optional — embedding provider (defaults to OpenAI):
EMBEDDING_PROVIDER=openai # or "ollama" (native /api/embed)
EMBEDDING_API_URL=https://api.openai.com/v1/embeddings
EMBEDDING_API_KEY=sk-... # Defaults to OPENAI_API_KEY; never sent to Ollama
EMBEDDING_MODEL=text-embedding-3-small
EMBEDDING_DOC_PREFIX= # task prefixes for asymmetric models (nomic, E5, BGE)
EMBEDDING_QUERY_PREFIX=
Optional — similarity thresholds. These are model-specific (see Local embeddings):
DUPLICATE_THRESHOLD=0.95 # store rejects a near-duplicate at or above this
FIND_RELATED_THRESHOLD=0.85 # find_related default
FIND_RELATED_NEIGHBORS=10 # nearest neighbours find_related checks per memory (1-100)
RELATED_ON_STORE_THRESHOLD=0.85 # store returns earlier memories at or above this (0 = off)
Optional — reciprocal rank fusion constant for hybrid search (default 60):
RRF_K=60
The `DATABASE_URL` must use the container name (`mcp-memory-db`), not the service name (`db`), to avoid DNS collisions if the server container is on a shared Docker network.
### 2. Start the stack```bash
docker compose up -d
This starts two containers:
- mcp-memory-db — Postgres with pgvector, initialised by init.sql
- mcp-memory-server — FastMCP server on port 8000
3. Configure your reverse proxy
Point your domain at the mcp-memory-server container on port 8000. The server exposes:
/mcp— MCP endpoint (Bearer token auth handled by the server)/webhook/capture— REST endpoint for external capture (Bearer token auth handled by the server)
4. Connect Claude Code```bash
claude mcp add memory-server \
--transport http \
--scope user \
--header "Authorization: Bearer <your-MCP_API_KEY>" \
-- https://your-domain.example.com/mcp
Restart Claude Code. The eleven tools and two prompts will be available in every project.
## Webhook / Capture Form
The `/webhook/capture` endpoint accepts POST requests for storing memories from external sources:```bash
curl -X POST https://your-domain.example.com/webhook/capture \
-H "Authorization: Bearer <your-MCP_API_KEY>" \
-H "Content-Type: application/json" \
-d '{"content": "Remember this", "source": "curl", "tags": ["test"]}'
You can build a mobile-friendly web form that POSTs to this endpoint. If you use a reverse proxy like Caddy, you can handle authentication at the proxy layer and inject the Bearer token server-side so the API key never reaches the browser.
Backups
The included backup.sh script dumps the database with 14-day retention:```bash
Add to cron (e.g. daily at 3am). Invoke via bash so a missing execute bit
on a fresh checkout can't silently break backups.
0 3 * * * /bin/bash /path/to/mcp-memory-server/backup.sh >> /path/to/backups/cron.log 2>&1
**Failure alerting (dead-man's switch).** Because the most common silent failure is "the cron never ran," `backup.sh` sends a success heartbeat to an Uptime Kuma **Push** monitor instead of only alerting from inside the script. Create a Push monitor (heartbeat interval slightly longer than your backup cadence, e.g. 25h), then drop its push URL into a gitignored `.backup-env` next to the script:```bash
# .backup-env (gitignored — never committed)
KUMA_PUSH_URL="https://<your-kuma>/api/push/<token>?status=up&msg=OK"
If the backup ever fails to run or complete, no heartbeat arrives and Kuma alerts via its own notification channels. If .backup-env is absent, no heartbeat is sent and backups still work.
Local embeddings (Ollama)
Embeddings can run on your own hardware with Ollama, so memory text never leaves your network for embedding. (Leave OPENAI_API_KEY unset as well and nothing goes to OpenAI: clients supply action items themselves, see Metadata.)
Settings for nomic-embed-text (768 dims), with its task prefixes and thresholds calibrated on a real store of about 1,700 memories:```env
EMBEDDING_PROVIDER=ollama
EMBEDDING_API_URL=http://ollama:11434/api/embed # the default for ollama
EMBEDDING_MODEL=nomic-embed-text
EMBEDDING_DOC_PREFIX="search_document: "
EMBEDDING_QUERY_PREFIX="search_query: "
EMBEDDING_NUM_THREAD=2 # match to your CPU; omit to let Ollama choose
EMBEDDING_MAX_TOKENS=2048 # nomic's context; longer inputs are truncated
DUPLICATE_THRESHOLD=0.975
FIND_RELATED_THRESHOLD=0.90
Why these thresholds: nomic's cosine scale runs higher than OpenAI's (unrelated memories score about 0.63, against 0.34). Light edits of a memory (typo, added line, dropped sentence) all scored at least 0.981, while distinct memories about closely related facts reached 0.962. 0.975 sits between the two.
Memories longer than the model's context are embedded from their start only, and are flagged `metadata.embedding.truncated`. Keyword search still indexes their full text.
The Ollama container must share a Docker network with the server. Pin the model: re-pulling a tag can change its weights, and with them every vector.
### Switching an existing store to another model
Vectors from different models can't be compared, so switching means re-embedding every memory. `migrations/` has the steps for OpenAI → nomic:
1. `001-embedding-model.sql` adds the `embedding_model` column. Run it before deploying a server version that writes it.
2. `002-nomic-768.sql` phase A adds staging columns while the server keeps running. `backfill_embeddings.py` fills them; it can reuse vectors from an earlier run where the content is unchanged.
3. Stop the server, rerun the backfill to catch late writes, then run phase B. It swaps the columns in one transaction and aborts, changing nothing, if any row lacks a current vector. Set the Ollama env above and start the server.
The old vectors stay in `embedding_openai`. To roll back: stop the server, rename the columns back, re-embed any memories created since the switch with the old model, and restore the old env. Once you're sure, drop the column: `ALTER TABLE memories DROP COLUMN embedding_openai; VACUUM FULL memories;`
## Metadata
Metadata lives under `metadata.ai`, separate from user-supplied metadata. `resolve_action_items`, `unresolve_action_items` and `weekly_review` read it.
**Client-supplied (recommended).** The calling model passes `action_items` (concrete follow-ups the user still has to do; most memories have none) and optionally `memory_type` (`observation`, `task`, `decision`, `idea`, `reference`, `person_note`, `meeting`) to `store_memory`, `store_memories` or `update_memory`. The convention reaches clients through the tool descriptions and server instructions, so no per-client setup is needed. These are stored with `extracted_by: "client"`. The caller has the conversation's context, so it can tell a real to-do from a plan it's only describing, and the text goes to no one new.
**Server-side extraction (fallback).** If the client passes neither field and `OPENAI_API_KEY` is set, GPT-4o-mini extracts the type, 1–3 topic tags (merged into `tags`), entities and action items. It's skipped for near-duplicates and never blocks storage. Leave `OPENAI_API_KEY` unset to turn it off.
Replacing a memory's `action_items` through `update_memory` resets which of them are resolved.
## Authentication
Authentication uses a Starlette `BaseHTTPMiddleware` that validates the Bearer token at the HTTP layer, before FastMCP processes the request. This works around a known FastMCP bug where `get_http_headers()` returns stale or missing headers during tool execution with the Streamable HTTP transport. If FastMCP's own middleware sees your headers correctly in a future release, you could switch back — but the Starlette approach is arguably more correct anyway since auth belongs at the transport layer.
### Scoped API keys
You can create additional API keys that force a `source` value on all memories stored with that key. This is useful for external integrations where you want to identify the origin without trusting the caller to set it.
Keep these keys out of version control: define them in a gitignored `docker-compose.override.yml`, which Compose merges automatically over the base file:```yaml
# docker-compose.override.yml (gitignored — do not commit)
services:
server:
environment:
MCP_API_KEY_<NAME>: ${MCP_API_KEY_<NAME>} # forces source="<name>"
The suffix after MCP_API_KEY_ becomes the forced source name (lowercased). The token values themselves live only in your untracked .env.
MCP client compatibility
Some MCP clients (including Claude Code) double-serialise structured parameters — sending '["a","b"]' (a JSON string) instead of ["a","b"] (a native array). The server handles this transparently using Pydantic BeforeValidator on all list and dict parameters, so tags and metadata work regardless of client behaviour.
Docker networking note
The server container needs to be on your reverse proxy's Docker network so the proxy can reach it. The database container should stay on the default (internal) network only. If both containers share a network with other stacks, use explicit container_name values and reference those in DATABASE_URL to avoid DNS collisions with common service names like db.
Credit
Inspired by Nate B Jones and his Open Brain guide. His approach uses Supabase and OpenRouter — managed services that are faster to set up. This project takes the same concept and makes it fully self-hosted.
License
MIT
