Local LLM MCP Proxy
A fast and efficient Model Context Protocol (MCP) server that acts as a proxy for local LLMs (like Qwen or LLaMA) providing an OpenAI-compatible API backend.
By utilizing this project, you can seamlessly connect robust MCP clients (e.g., Claude Desktop, Cline) to your self-hosted LLMs, exposing them to advanced task orchestration and allowing AI coding assistants to invoke localized reasoning pipelines.
Project Structure
proxy.py
The core FastMCP application. It provides an HTTP-based MCP server containing tools that allow clients to interface with the local LLM. Key tools include:create_response: Constructs and fires prompt payloads (with tunable parameters liketop_k,top_p,temperature, etc.) to the local generating backend. It can also auto-attach a matching persona fromagency-agents/and infer a defaulttemperaturefrom the task if one is not supplied.list_models: Interrogates the upstream API (/v1/models) to determine what locally hosted models are currently available.health: Returns static status and configuration parameters to ensure everything is connected properly.-
workspace_tool: A single MCP dispatcher for file, search, todo, and memory operations -
tool_expansion.py
Workspace-scoped implementations for file IO, glob/grep-style search, and a simple JSON-backed todo store. -
query_engine.py
Handles upstream retry logic, lightweight JSON validation, and file-backed response caching for GET and POST requests. -
memory_architecture.py
Implements a three-layer local memory layout: MEMORY.mdas the always-available index- topic files under
.proxy_state/topics/ -
raw transcript logs under
.proxy_state/transcripts/
The index is only updated after confirmed topic writes. -
mcp_config.json
An example MCP client configuration snippet defining how clients can bridge onto the HTTP MCP proxy usingmcp-remote. -
json_schema.json
A strictly optimized, token-efficient JSON format schema. It is used to enforce structured outputs from the local model (e.g., separating textual reasoning from discrete tool execution objects), significantly speeding up parsing and preventing hallucinated fields. -
rules.txt
A set of specialized system instructions and guardrails. It dictates the architectural pattern used when pairing a primary agent (like Claude) with this proxy (acting as a secondary refinement/query-checker agent), specifically prohibiting the local LLM from generating final code directly.
Getting Started
Prerequisites
- Python 3.8+
fastmcppackage: Install viapip install fastmcp- standard libraries (
requests,starlette) - A locally running LLM inferencing engine (e.g., Unsloth, vLLM, llama.cpp) offering an OpenAI-compatible REST API
/v1/chat/completions.
Configuration
You can configure the server via environment variables:
* BACKEND_BASE_URL: The OpenAI-compatible API root URL of your local LLM (default: http://192.168.137.45:2345)
* MCP_HOST: The host to bind the proxy server to.
* MCP_PORT: The port to run the MCP server on (default: 9999)
* PROXY_TIMEOUT: Timeout parameter for upstream inferencing requests.
* QUERY_MAX_RETRIES: Number of upstream retries before surfacing a failure.
Persona Routing
create_response now supports prompt enrichment from agency-agents/:
- If
temperatureis omitted, the proxy infers one from the detected task type. - If no persona is explicitly set, the proxy scores the request against the available agent files and injects the best matching persona into
instructions. - You can force a persona with
agent_personality, for example:```json
{
"input": "Review this feature for production readiness",
"agent_personality": "Reality Checker"
}
* You can disable the automatic persona logic with:```json
{
"input": "Summarize this",
"metadata": {
"disable_auto_personality": true
}
}
Workspace Tool
The additional utility features are exposed through one MCP tool, workspace_tool, using an action plus params payload.
Examples:```json
{
"action": "read_file",
"params": {
"path": "Readme.md"
}
}
``````json
{
"action": "memory_write_topic",
"params": {
"topic": "repo-overview",
"content": "# Repo Overview\n...",
"summary": "High-level structure of the MCP proxy repository."
}
}
Supported actions:
* `read_file`
* `write_file`
* `glob_search`
* `grep_search`
* `todo_read`
* `todo_write`
* `memory_index`
* `memory_read_topic`
* `memory_write_topic`
* `memory_append_transcript`
* `memory_search_transcripts`
### Running
To boot the proxy server, execute:```bash
python proxy.py
Afterward, use the configuration provided within mcp_config.json in your chosen MCP client platform to bridge the connection.
