用于LLM管道的非结构化文档分区和ETL库
https://github.com/your-repo/unstructured
pip install unstructured
from unstructured.partition.pdf import partition_pdf
from unstructured.cleaning.clean import clean_text
Example usage
elements = partition_pdf("document.pdf")
cleaned_text = clean_text(elements)
class DocumentProcessor:
def init(self):
self.pipeline = [
"partition",
"clean",
"transform"
]
def process(self, file_path):
elements = partition_pdf(file_path)
cleaned = clean_text(elements)
return cleaned
Usage
processor = DocumentProcessor()
result = processor.process("example.pdf")
Unstructured is an open-source library for ingesting and partitioning PDFs, HTML, Office documents, emails, and other unstructured inputs into structured elements and metadata. It is commonly used as a preprocessing layer for RAG, search, extraction, and downstream AI pipelines.
Prerequisites
Python 3.11+
Installation
Use the upstream install or setup path that matches your environment:
- docker pull downloads.unstructured.io/unstructured-io/unstructured:latest
- docker run -dt --name unstructured downloads.unstructured.io/unstructured-io/unstructured:latest
- docker exec -it unstructured bash
- make docker-build
Requirements and caveats from upstream:
- <a href="https://github.com/1991513ccie-png/skills"></a>
- <a href="https://pypi.python.org/pypi/unstructured/"></a>
- <a href="https://pypi.python.org/pypi/unstructured/">
