Mini-Rag项目构建-9文档分块和入库
1077 字
5 分钟
Mini-Rag项目构建-9文档分块和入库
项目Github地址: https://github.com/gaodaxiu0406/Mini-Rag
⭐ 如果这个项目对你有帮助,欢迎Github点一个 Star——你的一个星,是对独立开发者最直接的鼓励;发现问题或有想法,也欢迎提 Issue。
文档分块和入库
storage.py追加文档索引存储:上传的文档被切成若干chunk,每个chunk带一段TF-IDF向量,全部存在data/docstore.json中。在storage.py顶部常量区和文件末尾各补一段。
顶部追加(在SESSION_DIR下方):
DOCSTORE_PATH = os.path.join(DATA_DIR, "docstore.json")文件末尾追加:
def get_docstore(): return _load_json(DOCSTORE_PATH, {"documents": {}, "chunks": []})
def save_docstore(docstore): _save_json(DOCSTORE_PATH, docstore)
# 添加文档def add_document(filename, chunks): """添加一个文档及其所有chunk。chunks是列表,每一项 = {text, embedding}""" ds = get_docstore() # 当前文件id 随机生成 doc_id = str(uuid.uuid4())[:8] ds["documents"][doc_id] = { "filename": filename, "chunk_count": len(chunks), "uploaded_at": datetime.now().isoformat() } for i, chunk in enumerate(chunks): ds["chunks"].append({ "doc_id": doc_id, "filename": filename, "chunk_index": i, "text": chunk["text"], "embedding": chunk["embedding"] }) save_docstore(ds) return doc_id
# 删除def remove_document(filename): """按文件名删除文档及其所有chunk""" ds = get_docstore() ds["chunks"] = [c for c in ds["chunks"] if c["filename"] != filename] ds["documents"] = { k: v for k, v in ds["documents"].items() if v["filename"] != filename } save_docstore(ds)
def list_documents(): """列出所有已入库的文档""" ds = get_docstore() return list(ds["documents"].values())
def get_all_chunks(): """获取全部chunk,供检索使用""" ds = get_docstore() return ds["chunks"]数据结构示意
{ "document": { "a1b2c3d4": { "filename": "1.txt", "chunk_count": 12, "uploaded_at": "2026-11-01T18:34:00" } }, "chunks": [ { "doc_id": "a1b2c3d4", "filename": "1.txt", "chunk_index": 0, "text": "详细概述,...", "embedding": [0.012, 0.0, 0.034, ...] } ]}TF-IDF向量器分块
新建rag.py文件
import pickleimport osimport numpy as npfrom sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.metrics.pairwise import cosine_similarityfrom config import CHUNK_SIZE, CHUNK_OVERLAP, RETRIEVAL_TOP_K, DATA_DIRimport storage
VECTORIZER_PATH = os.path.join(DATA_DIR, "vectorizer.pkl")
_vectorizer = None
def get_vectorizer(): """获取全局TF-IDF向量器。首次调用从磁盘加载,没有则创建新的""" global _vectorizer if _vectorizer is not None: return _vectorizer if os.path.exists(VECTORIZER_PATH): with open(VECTORIZER_PATH, "rb") as f: _vectorizer = pickle.load(f) else: _vectorizer = TfidfVectorizer( analyzer = "char_wb", ngram_range = (2, 4), max_features = 10000 ) return _vectorizer关键参数:
analyzer=”char_wb“:字符级n-gram,且只在词边界内取。这是中英文混合场景的关键选择————中文没有空格分词,char_wb直接在字符上切n-gram,中英文都适用ngram——rang=(2, 4):同时提取2-gram、3-gram、4-gram。既覆盖短词(“产品”),也覆盖长短语(“用户手册”)max_features=10000:词汇表上限,控制内存占用和计算速度
分块函数
def split_text(text, chunk_size=CHUNK_SIZE, overlap=CHUNK_OVERLAP): """滑动窗口分块:每chunk_size字符一个块,相邻块重叠overlap字符""" text = text.strip() if not text: return [] chunks = [] start = 0 while start < len(text): end = start + chunk_size chunk = text[start:end] chunks.append(chunk) start = end - overlap if start >= len(text): break return chunks为什么要重叠?举例:
原文:"...第三季度营收增长15%,超出市场预期。其中海外业务贡献了..."
无重叠分块:块1:"...第三季度营收增长15%,超出市场预期。"块2:"其中海外业务贡献了..."
有重叠(100字符):
块1:"...第三季度营收增长15%,超出市场预期。"块2:"营收增长15%,超出市场预期。其中海外业务贡献了..." ← 保留了衔接重叠让相邻块之间有”共同区域”,避免关键信息恰好落在分块边界上被切断。
文档入库
def index_document(filename, content): """分块 → TF-IDF向量化 → 存入DocStore。返回chunk数量""" chunks_texts = split_text(content) if not chunks_texts: return 0
# 用所有新chunk更新词汇表 vec = get_vectorizer() vec.fit(chunks_texts) # embeddings = vec.transfore(chunks_texts) embeddings = vec.transform(chunks_texts)
# 组装chunk对象 chunks = [] for i, (text, emb) in enumerate(zip(chunks_texts, embeddings)): chunks.append({ "text": text, "embedding": emb.toarray()[0].tolist() })
storage.add_document(filename, chunks)
# 用全量语料重建向量器,保证所有文档的向量在同一词汇空间 rebuild_vectorizer() _re_encode_all()
return len(chunks)
def rebuild_vectorizer(): """在全量chunk上重新训练向量器""" global _vectorizer all_chunks = storage.get_all_chunks() texts = [c["text"] for c in all_chunks] _vectorizer = TfidfVectorizer( analyzer = "char_wb", ngram_rang=(2, 4), max_features = 10000 ) _vectorizer.fit(texts) with open(VECTORIZER_PATH, "wb") as f: pickle.dump(_vectorizer, f) return _vectorizer
def _re_encode_all(): """用新的全局向量器重新编码所有chunk""" all_chunks = storage.get_all_chunks() if not all_chunks: return vec = get_vectorizer() texts = [c["text"] for c in all_chunks] matrices = vec.transform(texts) ds = storage.get_docstore() for i, row in enumerate(matrices): ds["chunks"][i]["embedding"] = row.toarray()[0].tolist storage.save_docstore(ds)运行效果
在Python交互环境中入库文档并检查data/目录,打开终端运行以下命令
pythonimport ragcount = rag.index_document("测试.txt", "Python是一种解释型编程语言。Flash是Python的Web框架。Deepseek是一个国产大模型")print(f"入库{count}个chunk")
data/目录下应出现docstore.json和vectorize.pkl。docstore.json里能看到分块结果和向量数据。
支持与分享
如果这篇文章对你有帮助,欢迎分享给更多人或打赏支持!
Mini-Rag项目构建-9文档分块和入库
https://gaodaxiu-blog.netlify.app/posts/guide/7项目构建-9文档分块和入库/











