用于日常研究+新闻报纸的MCP服务器
自动化 模型上下文协议(MCP)服务器 它发现、抓取、处理和总结研究论文和新闻文章,以HTML、PDF和JSON格式生成精美的日报。
🌟 特性
-
多源发现:自动从以下位置发现内容:
-
arXiv、PubMed、Crossref、bioRxiv(研究论文)
- 期刊RSS提要(自然、科学等)
-
新闻来源(麻省理工学院技术评论、Ars Technica、谷歌新闻)
-
AI驱动的总结:用途 谷歌双子座 和 OpenAI GPT-4 生成:
-
引人注目的标题
- TL;DR摘要
- 关键要点
- 显著性分析
-
局限性
-
智能处理:
-
跨源重复数据删除
- 使用嵌入进行相关性评分
- 时效性和可信度权重
-
自动分段分配
-
精美的报纸输出:
-
专业的HTML设计
- PDF生成
- JSON元数据导出
-
多个部分(热门研究、新闻、工具等)
-
法律与道德:
-
尊重
robots.txt - 每个域的速率限制
- 合理使用合规性
- 元数据仅适用于付费内容
📋 需求
- Python 3.8+
- OpenAI API密钥(用于嵌入和GPT-4)
- Google Gemini API密钥(用于摘要)
- 可选:PostgreSQL、Elasticsearch、wkhtmltopdf
🚀 快速开始
1.克隆和设置```bash
git clone https://github.com/abhishek7467/mcp-research-bot.git
cd "mcp-research-bot"
Create virtual environment
python3 -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
Install dependencies
pip install -r requirements.txt
### 2.配置API密钥```bash
# Copy environment template
cp .env.example .env
# Edit .env and add your API keys
nano .env
所需API密钥:```env
OPENAI_API_KEY=sk-your-openai-key-here
GEMINI_API_KEY=your-gemini-key-here
CROSSREF_EMAIL=your-email@example.com
### 3.配置主题和源
编辑 `config/config.yaml`:```yaml
topics:
- "machine learning"
- "quantum computing"
- "CRISPR gene editing"
4.运行你的第一代```bash
Single run for today
python mcp_orchestrator.py --topics "machine learning" "AI" --max-items 50
Backfill last 7 days
python mcp_orchestrator.py --backfill 7
Run on schedule (daily at 2 AM UTC)
python mcp_orchestrator.py --schedule
## 📁 项目结构```
MCP Server for daily/
├── config/
│ └── config.yaml # Main configuration
├── src/
│ ├── ai/
│ │ ├── summarizer.py # Gemini/OpenAI summarizer
│ │ └── headline_generator.py # Headline generation
│ ├── discovery/
│ │ └── discovery_manager.py # API discovery (arXiv, PubMed, etc.)
│ ├── scrapers/
│ │ └── scraper_manager.py # Web scraping with robots.txt
│ ├── extractors/
│ │ └── extractor_manager.py # Content extraction
│ ├── processors/
│ │ ├── deduplicator.py # Deduplication logic
│ │ └── scorer.py # Relevance scoring
│ ├── generators/
│ │ └── newspaper_generator.py # HTML/PDF generation
│ ├── storage/
│ │ └── database.py # SQLite/PostgreSQL
│ └── utils/
│ ├── logger.py # Logging setup
│ ├── config_loader.py # Config management
│ └── notifier.py # Slack/Email notifications
├── data/ # Generated data
│ ├── newspapers/ # Daily newspapers
│ │ └── YYYY-MM-DD/
│ │ ├── newspaper.json
│ │ ├── newspaper.html
│ │ └── newspaper.pdf
│ ├── cache/ # Cached data
│ └── mcp.db # SQLite database
├── logs/ # Log files
├── mcp_orchestrator.py # Main entry point
├── requirements.txt # Dependencies
├── .env.example # Environment template
└── README.md # This file
🎯 使用示例
基本用法```bash
Single topic, today's date
python mcp_orchestrator.py --topics "protein folding"
Multiple topics, specific date
python mcp_orchestrator.py --topics "NLP" "transformers" --date 2025-10-15
Limit items processed
python mcp_orchestrator.py --topics "robotics" --max-items 30
### 高级用法```bash
# Backfill last 5 days
python mcp_orchestrator.py --backfill 5 --topics "climate change"
# Run scheduled job (keeps running)
python mcp_orchestrator.py --schedule
# Custom config file
python mcp_orchestrator.py --config my_config.yaml --topics "blockchain"
程序化使用```python
from mcp_orchestrator import MCPOrchestrator
Initialize
orchestrator = MCPOrchestrator(config_path='config/config.yaml')
Run pipeline
newspaper = orchestrator.run_pipeline(
topics=['deep learning', 'computer vision'],
date='2025-10-29',
max_items=100
)
print(f"Generated: {newspaper['paths']['html']}")
## ⚙️ 配置
### AI模型选择
在 `config/config.yaml`:```yaml
ai_models:
summarizer: "gemini" # or "openai"
headline_generator: "gemini" # or "openai"
embeddings: "openai" # for relevance scoring
gemini_model: "gemini-1.5-pro-latest"
openai_chat_model: "gpt-4o"
openai_embedding_model: "text-embedding-3-small"
评分权重```yaml
scoring:
weights:
relevance: 0.35 # Semantic similarity to topics
recency: 0.25 # How recent the publication
credibility: 0.20 # Source reputation
novelty: 0.20 # Uniqueness
relevance_threshold: 0.6
### 速率限制```yaml
crawling:
rate_limit_per_second: 1.0
max_crawl_per_source: 200
respect_robots_txt: true
timeout_seconds: 30
📊 输出格式
1.JSON(newspaper.json)```json
{
"date": "2025-10-29",
"title": "Daily Research Bulletin: Machine Learning",
"sections": [
{
"id": "top_research",
"title": "Top Research Picks",
"items": [
{
"id": "arXiv:2501.01234",
"headline": "Scaling GNNs to Trillion-edge Graphs",
"tldr": "Introduces sparse attention for GNNs...",
"bullets": ["...", "..."],
"score": 0.92
}
]
}
]
}
### 2.HTML(`newspaper.html`)
专业报纸风格布局,包括:
- 标题和日期
- 目录
- 有文章的章节
- 可点击的源链接
- 元数据页脚
### 3.PDF(`newspaper.pdf`)
从HTML生成的可打印PDF。
## 🔧 高级功能
### 数据库选项
**SQLite(默认):**```yaml
storage:
database:
type: "sqlite"
path: "./data/mcp.db"
PostgreSQL:```yaml
storage:
database:
type: "postgresql"
host: "localhost"
port: 5432
database: "mcp_research"
user: "mcp_user"
password: "your_password"
### 通知
**松弛:**```yaml
notifications:
enabled: true
slack:
webhook_url: "https://hooks.slack.com/services/YOUR/WEBHOOK"
电子邮件:```yaml
notifications:
email:
smtp_host: "smtp.gmail.com"
smtp_port: 587
username: "your_email@gmail.com"
password: "your_app_password"
### 调度
**Cron表达式:**```yaml
schedule:
enabled: true
cron: "0 2 * * *" # Daily at 2 AM UTC
timezone: "UTC"
📝 日志记录
日志将写入:
- 控制台(彩色,信息级别)
logs/mcp.log(详细,调试级别,以10MB旋转)```python
Adjust log level in config/config.yaml
logging:
level: "INFO" # DEBUG, INFO, WARNING, ERROR
## 🤝 贡献
欢迎投稿!拜托:
1. 分叉存储库
1. 创建要素分支
1. 进行更改
1. 如果适用,添加测试
1. 提交拉取请求
## 📜 许可证
MIT许可证-有关详细信息,请参阅许可证文件
## ⚠️ 法律与道德
- **Robots.txt合规性**:始终遵守robots.txt指令
- **速率限制**:强制礼貌爬行(默认为1个请求/秒)
- **合理使用**:仅存储付费墙内容的元数据
- **归因**:所有来源均正确引用和链接
- **DMCA合规性**:根据有效的删除请求删除内容
## 🐛 故障排除
### API密钥错误```bash
# Verify keys are set
source .env
echo $OPENAI_API_KEY
echo $GEMINI_API_KEY
PDF生成问题```bash
Install wkhtmltopdf
sudo apt-get install wkhtmltopdf # Ubuntu/Debian
brew install wkhtmltopdf # macOS
Or use weasyprint (pure Python)
Set in config.yaml:
output:
pdf_engine: "weasyprint"
### 缺少的依赖```bash
# Reinstall all dependencies
pip install --upgrade -r requirements.txt
📞 支持
- 问题: GitHub问题
- 电子邮件: abhishek746781@gmail.com
- 文档:参见
docs/文件夹
🙏 致谢
- 用于GPT-4和嵌入API的OpenAI
- Google for Gemini API
- arXiv、PubMed、Crossref用于研究API
- 开源社区
由以下材料制成❤️ 面向研究人员和知识寻求者
