Local LLM with MCP Tools - Complete Local Setup
A 100% local AI system with hot-swappable LLM models, MCP tool calling, and local chat interfaces. No external services required!
🎯 What This Is
- ✅ Local LLM models (Mistral, Llama 3, Phi-3)
- ✅ Hot-swappable (switch models without restart)
- ✅ MCP tools (6 tools the LLM can call)
- ✅ Local chat interfaces (CLI + Web)
- ✅ No external services (no Claude, OpenAI, etc.)
🚀 Quick Start
Step 1: Install Ollama
Ollama is the local LLM runtime that runs the models.
macOS```bash
brew install ollama
#### Linux```bash
curl -fsSL https://ollama.com/install.sh | sh
Windows
Download and install from ollama.com/download
Step 2: Start Ollama Service
macOS/Linux:```bash
ollama serve
**Windows:**
Ollama runs as a service automatically after installation. If not, run:```cmd
ollama serve
Keep this terminal/window running! This is the LLM engine.
Step 3: Install Python Dependencies
macOS/Linux:```bash
cd /Users/oseekero/robotics_stuff/local_mcp
pip install -r requirements.txt
**Windows:**```cmd
cd C:\path\to\local_mcp
pip install -r requirements.txt
Step 4: Run Chat Interface
Open a new terminal/command prompt (keep Ollama running in the first one).
Option A: Simple Chat (Recommended) ⭐
macOS/Linux:```bash
python simple_chat.py
**Windows:**```cmd
python simple_chat.py
Features:
- Direct slash commands for tools
- /models - List available models
- /switch - Change model (e.g., /switch llama3.1:8b)
- /info - Get current model info
- /clear - Clear conversation
- Just type normally to chat with the LLM
Option B: Auto Tool-Calling Chat
macOS/Linux:```bash
python chat.py
**Windows:**```cmd
python chat.py
Features:
- LLM automatically calls tools when needed
- Works best with Llama 3.1 8B or Mistral 7B
- Smaller models (phi3:mini) may not follow JSON format well
Option C: Web Interface
macOS/Linux:```bash
python web_chat.py
**Windows:**```cmd
python web_chat.py
Then open your browser to: http://localhost:8000
Features:
- Beautiful gradient UI
- Real-time WebSocket chat
- Typing indicators
- Model switching in browser
💬 Using the Chat
Simple Chat Commands```
You: /models
Assistant: Available Models:
• mistral:7b (active)
• llama3.1:8b
• phi3:mini
...
You: /switch llama3.1:8b
Assistant: Switched to llama3.1:8b
You: /info
Assistant: Model Information:
Name: llama3.1:8b
Size: 4.7GB
Context: 8192 tokens
...
You: Hello! How are you?
Assistant: I'm doing well! How can I help you today?
### Auto Tool-Calling Chat
The LLM automatically detects when to use tools:```
You: What models are available?
🔧 Calling tool: list_models
✓ Tool result: Available Models: mistral:7b, llama3.1:8b...
Assistant: Here are the available models: Mistral 7B (currently active), Llama 3.1 8B, Phi-3 Mini...
You: Switch to llama
🔧 Calling tool: switch_model
✓ Tool result: Successfully switched to llama3.1:8b
Assistant: I've switched to Llama 3.1 8B for you!
🤖 Pre-configured Models
| Model | Size | Best For | Tool Calling |
|---|---|---|---|
| mistral:7b ⭐ | 4.1GB | Balanced, good tool calling | ✅ Excellent |
| llama3.1:8b ⭐ | 4.7GB | Best quality, great tool calling | ✅ Excellent |
| llama3.2:3b | 2.0GB | Fast, lightweight | ⚠️ Limited |
| phi3:mini | 2.3GB | Very fast, efficient | ⚠️ Limited |
| phi3:medium | 7.9GB | Most capable | ✅ Good |
Recommended: Use mistral:7b or llama3.1:8b for best tool-calling performance.
🔧 Available Tools
The LLM can use these tools:
| Tool | Description | Example |
|---|---|---|
chat |
Chat with conversation history | Automatic |
complete |
Single text completion | Automatic |
switch_model |
Hot-swap to different model | "Switch to mistral" |
list_models |
Show all available models | "What models are available?" |
get_model_info |
Get model details | "Tell me about the current model" |
clear_conversation |
Reset conversation history | "Clear the conversation" |
🔄 Hot-Swapping Models
Change models instantly without restarting:
Method 1: In Chat```
/switch mistral:7b
### Method 2: Edit config.yaml```yaml
active_model: "llama3.1:8b"
Save the file - the system auto-reloads!
Method 3: Programmatically```python
from config_manager import ConfigManager
config = ConfigManager()
config.set_active_model("phi3:mini")
---
## 📁 Project Structure```
local_mcp/
├── simple_chat.py # ⭐ CLI chat with slash commands (RECOMMENDED)
├── chat.py # CLI chat with auto tool-calling
├── web_chat.py # Web UI chat interface
├── local_mcp_client.py # MCP client for tool calling
├── model_manager.py # Model operations (download, switch, generate)
├── llm_tools.py # 6 MCP tools
├── config_manager.py # Configuration management
├── config.yaml # Settings (active model, parameters)
├── main.py # MCP server (for external clients like Claude)
├── requirements.txt # Python dependencies
├── test_server.py # Test suite
├── debug_chat.py # Debug tool calling
└── README.md # This file
🛠️ Platform-Specific Instructions
macOS
Terminal 1:```bash
ollama serve
**Terminal 2:**```bash
cd /Users/oseekero/robotics_stuff/local_mcp
python simple_chat.py
First run: Downloads the active model (~4GB for mistral:7b)
Linux
Terminal 1:```bash
ollama serve
**Terminal 2:**```bash
cd /path/to/local_mcp
python3 simple_chat.py
Note: Use python3 instead of python on most Linux distributions.
Windows
Command Prompt 1:```cmd
ollama serve
(Or Ollama may already be running as a service)
**Command Prompt 2:**```cmd
cd C:\path\to\local_mcp
python simple_chat.py
PowerShell Alternative:```powershell
cd C:\path\to\local_mcp
python simple_chat.py
---
## 🧪 Testing
### Quick Test```bash
python debug_chat.py
Shows exactly what's happening with tool calls.
Full Test Suite```bash
python test_server.py
Tests all components.
### Simple Example```bash
python quick_test.py
Quick functionality test.
⚙️ Configuration
Edit config.yaml to customize:```yaml
Active model (hot-swappable)
active_model: "mistral:7b"
Model parameters
model_params:
temperature: 0.7 # Randomness (0.0-2.0)
top_p: 0.9 # Nucleus sampling
num_ctx: 4096 # Context window
num_predict: 512 # Max tokens to generate
Ollama settings
ollama:
host: "http://localhost:11434"
auto_pull: true # Auto-download models
---
## 🐛 Troubleshooting
### "ollama not found"
**Solution:** Install Ollama (see Step 1 above)
### "Connection refused"
**Solution:** Make sure `ollama serve` is running
### Model download slow
**Solution:** First download takes time (2-5GB per model). Be patient!
### Tools not being called
**Solution:**
1. Use `simple_chat.py` with slash commands (always works)
2. Or switch to `mistral:7b` or `llama3.1:8b` (better at tool calling)
3. Run `debug_chat.py` to see what's happening
### Port 8000 already in use (web chat)
**Solution:** Edit `web_chat.py` and change the port:```python
uvicorn.run(app, host="0.0.0.0", port=8001) # Change to 8001
Windows: "python not found"
Solution: Use py instead of python:```cmd
py simple_chat.py
---
## 🔌 Using as MCP Server (Advanced)
To use with external MCP clients like Claude Desktop:
**Start MCP server:**```bash
python main.py
Configure Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):json
{
"mcpServers": {
"local-llm": {
"command": "python",
"args": ["/Users/oseekero/robotics_stuff/local_mcp/main.py"]
}
}
}
Restart Claude Desktop to connect.
🎉 Summary
You now have a fully local AI system:
✅ No API keys needed
✅ No external services
✅ Complete privacy
✅ Works offline (after model download)
✅ Hot-swappable models
✅ Tool calling capability
✅ Multiple interfaces (CLI + Web)
Recommended workflow:
1. ollama serve (Terminal 1)
2. python simple_chat.py (Terminal 2)
3. Use /models, /switch, /info commands
4. Chat normally with the LLM
Enjoy your local AI! 🚀
