ArXiv RAG MCP代理
使用的容器化代理管道 模型上下文协议(MCP) 用于学术文献检索和问答的服务器。
🎯 项目概述
第1阶段-摄入:搜索ArXiv论文→ 废料内容→ 块状文本→ 嵌入松果并储存\
第2阶段-查询:检索相关上下文→ 使用GPT-4生成答案→ 登录到Notion→ 本地保存
🔧 使用的MCP服务器
1. ArXiv MCP服务器
搜索、下载并阅读ArXiv的学术论文。```json
{
"command": "uv",
"args": ["tool", "run", "arxiv-mcp-server", "--storage-path", "/app/data/arxiv_papers"]
}
**工具**: `search_papers`, `download_paper`, `read_paper`, `list_papers`
______________________________________________________________________
### 2. **松果体MCP**
集成嵌入的语义搜索矢量数据库。```json
{
"command": "npx",
"args": ["-y", "@pinecone-database/mcp"],
"env": {
"PINECONE_API_KEY": "${PINECONE_API_KEY}"
}
}
索引配置:
- 名字:
arxiv-papers - 模型:
llama-text-embed-v2(NVIDIA托管,免费版) - 维度: 1024
- 指标:余弦
- 云:AWS(美国东部-1)
工具: create-index-for-model, upsert-records, query-index, describe-index-stats
3. MCP概念
将查询交互记录到Notion数据库。```json
{
"command": "npx",
"args": ["-y", "@notionhq/notion-mcp-server"],
"env": {
"NOTION_TOKEN": "${NOTION_TOKEN}"
}
}
**数据库模式**:
- **查询** (标题)-用户的问题
- **时间戳** (日期)-查询时
- **回答** (富格文本)-GPT-4生成的答案
- **来源** (富格文本)-检索到的顶部纸块
**设置**:在以下位置创建集成https://www.notion.so/my-integrations并与之共享数据库。
______________________________________________________________________
### 4. **文件系统MCP**
将最终输出保存在本地。```json
{
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-filesystem", "/app/outputs"]
}
工具: read_file, write_file, list_directory, move_file
🚀 快速开始
1.先决条件
- Docker&Docker编写
- API密钥:
- OpenAI(GPT-4)
- 松果
- 概念集成令牌
2.设置```bash
Clone repository
git clone
cd arxiv-rag-mcp
Create .env file
touch .env
Edit .env with your API keys
Create Pinecone index (via web UI):
- Name: arxiv-papers
- Model: llama-text-embed-v2
- Dimension: 1024, Metric: cosine
Create Notion database with 4 columns:
Query (Title), Timestamp (Date), Answer (Rich Text), Sources (Rich Text)
Share it with your integration
### 3.跑步```bash
# Build and run both phases
docker compose build --no-cache
docker compose up
# Or run phases separately:
# Phase 1 only (ingestion)
docker compose run --rm -e PHASE=ingestion arxiv-rag-agent
# Phase 2 only (query)
docker compose run --rm -e PHASE=query -e USER_QUERY="Your question here" arxiv-rag-agent
4.查看结果```bash
Check generated answer
cat outputs/answer.md
View logs
docker compose logs -f
______________________________________________________________________
## ⚙️ 配置
编辑 `.env` 文件:```bash
# Required
OPENAI_API_KEY=sk-...
PINECONE_API_KEY=...
PINECONE_INDEX_NAME=arxiv-papers
NOTION_TOKEN=ntn_...
NOTION_DATABASE_ID=...
# Optional
SEARCH_TOPIC="Large Language Model Reasoning"
MAX_PAPERS=10
ARXIV_CATEGORIES=cs.AI,cs.CL,cs.LG # Leave empty for all categories
PHASE=both # ingestion, query, or both
类别示例:
- AI/ML:
cs.AI,cs.CL,cs.LG - 物理学:
hep-ph,hep-th,hep-ex - 生物学:
q-bio.BM,q-bio.NC - 留空以搜索所有ArXiv类别
📁 项目结构```
arxiv-rag-mcp/
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
├── main.py
├── src/
│ ├── config.py
│ ├── mcp_manager.py
│ ├── phase1_ingestion.py
│ └── phase2_query.py
├── data/arxiv_papers/ # Downloaded papers
├── outputs/ # Generated answers
└── logs/ # Application logs
```
📝 许可证
该项目根据 MIT许可证.
看 许可证 文件以获取详细信息。
