🧠 MCP-Toolkit
MCP-Toolkit is a Python-based retrieval-augmented generation (RAG) system that answers natural-language questions about a Python codebase — specifically built to analyze the provided grip-no-tests code (assumed to be included separately by the interview reviewer).
It performs:
- Codebase chunking
- Embedding + semantic indexing
- Retrieval and Q&A via OpenAI
- Benchmark evaluation using grip_qa.json
- High-level analysis via an agent
🚀 Getting Started
1. Install Python 3.12
Make sure you're using Python 3.12.
-
You can download it from the official site:
https://www.python.org/downloads/release/python-3120/ -
After installing, verify with:```bash
python3 --version
Should print: Python 3.12.x
---
### 2. Add the `grip-no-tests` Python codebase
Place the provided `grip-no-tests` codebase in the MCP-Toolkit directory:```
MCP-Toolkit/
├── main.py
├── grip-no-tests/ ← manually included folder
├── grip_qa/
│ ├── 0001.q.md
│ ├── 0001.a.md
├── results/ ← contains example output files
├── .env
> ⚠️ This repo is not hosted on GitHub. It is assumed to be included manually as part of the assignment.
3. Install dependencies```bash
pip install -r requirements.txt
Or manually:```bash
pip install openai fastapi uvicorn chromadb python-dotenv pydantic
4. Set your API key
Create a .env file in the project root:```
OPEN_API_KEY=sk-...
---
### 5. Generate `grip_qa.json` from markdown
This converts your `.q.md` and `.a.md` files into a benchmark dataset.```bash
python main.py --generate-grip
✅ This creates grip_qa.json from files in the grip_qa/ folder.
💬 Run the Question-Answering API
Start the FastAPI server:```bash
python main.py
Open your browser at:
http://localhost:8000/docs
Test the `/qa` endpoint with a POST request like:```json
{
"question": "How do I run grip from the command line?"
}
✅ The system will:
- Embed your question
- Retrieve the most relevant code chunks
- Feed them to GPT
- Return a helpful answer
🧪 Run Evaluation
To evaluate against benchmark answers from grip_qa.json:```bash
python main.py --evaluate
✅ This:
- Runs all grip QA questions through the system
- Compares answers to reference ones using `difflib.SequenceMatcher`
- Prints a similarity score per question and an overall average
---
## 🧠 Run Repository Analysis
To generate a markdown-style report about the codebase:```bash
python main.py --analyze
✅ This uses the LLM to answer high-level questions like:
- What’s the architecture?
- What design patterns are used?
- What are the key dependencies?
📂 Example Results
The project includes a results/ folder with example output files from:
--evaluateruns--analyzereports- Sample
/qaresponses
These illustrate what typical runs should look like for validation and testing purposes.
🛠 Next Steps / Improvements
Here are ways the project could be improved with minimal effort:
1. Modularize the code
- Right now everything is in
main.py - Splitting into modules like
retriever.py,evaluator.py, etc. would improve readability and testability
2. Make repo selection dynamic
- Currently the system assumes the
grip-no-testsrepo is manually added - Add a CLI flag or config option to analyze any repo path dynamically
3. Improve evaluation metric
- The current evaluator uses
difflib.SequenceMatcher, which is character-level - Switching to a semantic similarity model (e.g.,
sentence-transformers) would better capture paraphrased but accurate answers
🔁 Optional: Validate with a Shell Script
To test the system end-to-end with one command, use the included validate.sh script:
🧪 Run all steps at once:```bash
chmod +x validate.sh
./validate.sh
```
This will:
- Generate grip_qa.json from markdown files
- Run evaluation and save output to results/eval_test.txt
- Run analysis and save output to results/analysis_test.md
✅ Use this to quickly verify your setup matches the expected behavior.
