URL2Markdown — 网页转Markdown转换器
Overview | 概述
Converts any webpage URL into a clean, readable Markdown document with all images saved locally.
将任何网页URL转换为干净、易读的Markdown文档，所有图片会保存到本地。
Features | 功能:
- Extracts main content and removes noise (ads, navigation, etc.) | 提取主要内容，去除噪音（广告、导航等）
- Preserves document structure (headings, paragraphs, lists, code blocks, tables, blockquotes) | 保留文档结构（标题、段落、列表、代码块、表格、引用）
- Downloads all images and embeds them as relative paths | 下载所有图片并保存为相对路径引用
- Extracts metadata (title, description, author, date) and saves as YAML frontmatter | 提取元数据（标题、描述、作者、日期）并保存为YAML frontmatter
- Handles srcset for high-resolution images | 处理srcset以获取高分辨率图片
- Auto-generates timestamped filenames | 自动生成带时间戳的文件名
Usage | 使用方法
Invoke | 调用
python url2markdown.py <url> [output_directory]
Examples | 示例
Basic usage | 基本用法:
python url2markdown.py https://www.0xkato.xyz/how-llms-actually-work
With custom output directory | 自定义输出目录:
python url2markdown.py https://www.0xkato.xyz/how-llms-actually-work ./my_articles
Without https:// prefix | 不带https://前缀:
python url2markdown.py www.example.com/my-article
Output | 输出
The script generates a directory structure like this:
output/
├── how-llms-actually-work_20260615_143022.md
└── images/
├── img_000.png
├── img_001.jpg
└── img_002.webp
The Markdown file includes:
- YAML frontmatter with metadata | 带元数据的YAML frontmatter
- Clean Markdown content | 干净的Markdown内容
- Image references as relative paths | 图片引用为相对路径
- Footer with source URL and generation info | 包含来源URL和生成信息的页脚
Metadata in Frontmatter | Frontmatter中的元数据
---
title: "Page Title | 页面标题"
url: "https://example.com/page"
date: "2026-06-15 14:30:22"
description: "Page description | 页面描述"
author: "Author Name | 作者"
published: "2026-01-15T10:00:00Z"
---
Dependencies | 依赖
- Python 3.6+
requests— HTTP requestsbeautifulsoup4— HTML parsing
Auto-installs on first run if missing. 首次运行时会自动安装所需依赖。
How It Works | 工作原理
- Fetch the webpage with a browser-like User-Agent | 使用浏览器User-Agent获取网页
- Parse the HTML and extract metadata (title, description, author, dates) | 解析HTML并提取元数据
- Locate main content area (main, article, or content div) | 定位主内容区域（main、article或content div）
- Extract all images, handling srcset for higher resolution | 提取所有图片，处理srcset以获取更高分辨率
- Download images to local
images/directory | 下载图片到本地images目录 - Convert content to Markdown, preserving structure | 将内容转换为Markdown，保留结构
- Generate the final Markdown file with frontmatter and footer | 生成包含frontmatter和页脚的最终Markdown文件
Supported HTML Elements | 支持的HTML元素
| Element | Markdown | 说明 |
|---|---|---|
<h1>-<h6> |
#-###### |
标题 |
<p> |
(text) | 段落 |
<ul>, <ol> |
- item, 1. item |
列表 |
<pre><code> |
python ... |
代码块 |
<blockquote> |
> text |
引用 |
<table> |
col1 \| col2 |
表格 |
<img> |
 |
图片 |
<hr> |
--- |
分隔线 |
<a> |
(text) | 链接（保留文字，去除链接） |
Tips | 技巧
- For JavaScript-rendered pages (SPA/React/Vue), content may be incomplete. Use browser dev tools to get the API response or use a headless browser alternative. | 对于JavaScript渲染的页面（SPA/React/Vue），内容可能不完整。使用浏览器开发者工具获取API响应或使用无头浏览器替代方案。
- Images from CDNs are downloaded and saved locally with sequential numbering. | 来自CDN的图片会被下载并保存到本地，按顺序编号。
- Large pages with many images may take longer. The script handles errors gracefully and continues. | 图片较多的长页面可能需要更长时间。脚本会优雅地处理错误并继续执行。
- Default output directory is
./output. You can change it by providing a second argument. | 默认输出目录为./output，可以通过提供第二个参数来更改。
Error Handling | 错误处理
- Network errors: prints error and exits | 网络错误：打印错误并退出
- Missing images: prints warning and continues | 缺失图片：打印警告并继续
- Invalid URLs: auto-adds
https://prefix | 无效URL：自动添加https://前缀 - Failed downloads: skips the image, continues processing | 下载失败：跳过该图片，继续处理
