【RAG框架】GoMate：RAG Framework within Reliable input,Trusted output
【项目链接】：https://github.com/gomate-community/GoMate
一、赛题背景
RAG（检索增强生成）是一种结合了检索模型和生成模型的技术，它通过检索大量外部知识来辅助文本生成，从而提高大型语言模型（LLMs）的准确度和可靠性。
RAG特别适合于需要不断更新知识的知识密集型场景或特定领域应用，它通过引入外部信息源，有效缓解了大语言模型在领域知识缺乏、信息准确性问题以及生成虚假内容等方面的挑战。本次挑战赛旨在探索RAG技术的极限，鼓励开发者、研究人员和爱好者利用RAG技术解决实际问题，推动人工智能领域的进步。
二、赛题任务
赛题需要参赛选手设计并实现一个RAG模型，该模型能够从给定的问题出发，检索知识库中的相关信息。利用检索到的信息，结合问题本身，生成准确、全面、权威的回答。
三、评审规则
1.数据说明
数据集还可能包括一些未标注的文本，需要参赛者使用RAG技术中的检索增强方法来找到相关信息，并生成答案。这要求参赛者不仅要有强大的检索能力，还要能够生成准确、连贯且符合上下文的文本。
测试集为模拟生成的用户提问，需要参赛选手结合提问和语料完成回答。需注意，在问题中存在部分问题无法回答，需要选手设计合适的策略进行拒绝回答的逻辑。
• corpus.txt.zip：语料库，每行为一篇新闻
• test_question.csv：测试提问
- 评审规则
对于测试提问的回答，采用字符重合比例进行评价，分数最高为1。
四、数据分析
-
检索语料
-
文本长度
五、RAG基线实现
import pickle
import pandas as pd
from tqdm import tqdm
from gomate.modules.document.chunk import TextChunker
from gomate.modules.document.txt_parser import TextParser
from gomate.modules.document.utils import PROJECT_BASE
from gomate.modules.generator.llm import GLM4Chat
from gomate.modules.reranker.bge_reranker import BgeRerankerConfig, BgeReranker
from gomate.modules.retrieval.bm25s_retriever import BM25RetrieverConfig
from gomate.modules.retrieval.dense_retriever import DenseRetrieverConfig
from gomate.modules.retrieval.hybrid_retriever import HybridRetriever, HybridRetrieverConfig
def generate_chunks():
tp = TextParser()
tc = TextChunker()
paragraphs = tp.parse(r'H:/2024-Xfyun-RAG/data/corpus.txt', encoding="utf-8")
print(len(paragraphs))
chunks = []
for content in tqdm(paragraphs):
chunk = tc.chunk_sentences([content], chunk_size=1024)
chunks.append(chunk)
with open(f'{PROJECT_BASE}/output/chunks.pkl', 'wb') as f:
pickle.dump(chunks, f)
if __name__ == '__main__':
# test_path="H:/2024-Xfyun-RAG/data/test_question.csv"
# embedding_model_path="H:/pretrained_models/mteb/bge-m3"
# llm_model_path="H:/pretrained_models/llm/Qwen2-1.5B-Instruct"
test_path = "/data/users/searchgpt/yq/GoMate_dev/data/competitions/xunfei/test_question.csv"
embedding_model_path = "/data/users/searchgpt/pretrained_models/bge-large-zh-v1.5"
llm_model_path = "/data/users/searchgpt/pretrained_models/glm-4-9b-chat"
# ====================文件解析+切片=========================
generate_chunks()
with open(f'{PROJECT_BASE}/output/chunks.pkl', 'rb') as f:
chunks = pickle.load(f)
corpus = []
for chunk in chunks:
corpus.extend(chunk)
# ====================检索器配置=========================
# BM25 and Dense Retriever configurations
bm25_config = BM25RetrieverConfig(
method='lucene',
index_path='indexs/description_bm25.index',
k1=1.6,
b=0.7
)
bm25_config.validate()
print(bm25_config.log_config())
dense_config = DenseRetrieverConfig(
model_name_or_path=embedding_model_path,
dim=1024,
index_path='indexs/dense_cache'
)
config_info = dense_config.log_config()
print(config_info)
# Hybrid Retriever configuration
# 由于分数框架不在同一维度，建议可以合并
hybrid_config = HybridRetrieverConfig(
bm25_config=bm25_config,
dense_config=dense_config,
bm25_weight=0.7, # bm25检索结果权重
dense_weight=0.3 # dense检索结果权重
)
hybrid_retriever = HybridRetriever(config=hybrid_config)
# 构建索引
# hybrid_retriever.build_from_texts(corpus)
# 保存索引
# hybrid_retriever.save_index()
# 加载索引
hybrid_retriever.load_index()
# ====================检索测试=========================
query = "新冠肺炎疫情"
results = hybrid_retriever.retrieve(query, top_k=5)
# Output results
for result in results:
print(f"Text: {result['text']}, Score: {result['score']}")
# ====================排序配置=========================
reranker_config = BgeRerankerConfig(
model_name_or_path="/data/users/searchgpt/pretrained_models/bge-reranker-large"
)
bge_reranker = BgeReranker(reranker_config)
# ====================生成器配置=========================
# qwen_chat = QwenChat(llm_model_path)
glm4_chat = GLM4Chat(llm_model_path)
# ====================检索问答=========================
test = pd.read_csv(test_path)
answers = []
for question in tqdm(test['question'], total=len(test)):
search_docs = hybrid_retriever.retrieve(question)
search_docs = bge_reranker.rerank(
query=question,
documents=[doc['text'] for idx, doc in enumerate(search_docs)]
)
# print(search_docs)
content = '/n'.join([f'信息[{idx}]：' + doc['text'] for idx, doc in enumerate(search_docs)])
answer = glm4_chat.chat(prompt=question, content=content)
answers.append(answer[0])
print(question)
print(answer[0])
print("************************************/n")
test['answer'] = answers
test[['answer']].to_csv(f'{PROJECT_BASE}/output/gomate_baseline.csv', index=False)
