网站建设的目标定位,wordpress 登录 显示,网站seo优化服务商,企业邮箱域名查询原文地址#xff1a;https://pub.towardsai.net/advanced-rag-04-re-ranking-85f6ae8170b1
2024 年 2 月 14 日
重新排序在检索增强生成#xff08;RAG#xff09;过程中起着至关重要的作用。在简单的 RAG 方法中#xff0c;可以检索大量上下文#xff0c;但并非所有上下…原文地址https://pub.towardsai.net/advanced-rag-04-re-ranking-85f6ae8170b1
2024 年 2 月 14 日
重新排序在检索增强生成RAG过程中起着至关重要的作用。在简单的 RAG 方法中可以检索大量上下文但并非所有上下文都一定与问题相关。重新排序允许对文档进行重新排序和过滤将相关文档置于最前面从而提高 RAG 的有效性。
本文介绍了 RAG 的重新排名技术并演示了如何使用两种方法合并重新排名功能。
重新排序Re-ranking简介
图 1RAG 中的重新排序重新排序的任务是评估这些上下文的相关性并优先考虑最有可能提供准确且相关答案的上下文红色框。
如图1所示重新排序的任务就像一个智能过滤器。当检索器从索引集合中检索多个上下文时这些上下文可能与用户的查询具有不同的相关性。有些上下文可能非常相关在图 1 中以红色框突出显示而另一些上下文可能只是轻微相关甚至不相关在图 1 中以绿色和蓝色框突出显示。
重新排名的任务是评估这些上下文的相关性并优先考虑最有可能提供准确且相关答案的上下文。这使得LLMs能够在生成答案时优先考虑这些排名靠前的上下文从而提高响应的准确性和质量。
简单来说重新排名就像开卷考试时帮助你从一堆学习材料中选择最相关的参考文献以便你更高效、更准确地回答问题。
本文介绍的重排序方法主要可以分为以下两种
重新排序模型这些模型考虑文档和查询之间的交互特征以更准确地评估它们的相关性。LLMLLM的出现为重新排名开辟了新的可能性。通过彻底理解整个文档和查询可以更全面地捕获语义信息。
使用重新排序模型作为重排器
重新排序模型与嵌入模型不同它将查询和上下文作为输入直接输出相似性得分而不是嵌入得分。值得注意的是重新排序模型是利用交叉熵损失进行优化的因此相关性得分不局限于特定范围甚至可以是负分。
目前可用的重新排序模型并不多。一种选择是 Cohere 提供的在线模型可以通过 API 访问。此外还有一些开源模型如 bge-reranker-base 和 bge-reranker-large 等。
图 2 显示了使用命中率Hit Rate和平均倒数排名Mean Reciprocal RankMRR指标的评估结果 从评估结果可以看出
无论使用哪种嵌入模型重新排序都能显示出更高的命中率和 MRR这表明重新排序具有重大影响。目前最好的重新排名模型是 Cohere但它是一项付费服务。开源的 bge-reranker-large 模型具有与 Cohere 相似的功能。嵌入模型和重新排序模型的组合也会产生影响因此开发人员可能需要在实际过程中尝试不同的组合。
本文将使用 bge-reranker-base 模型。
环境配置
导入相关库设置环境变量和全局变量
import os
os.environ[OPENAI_API_KEY] YOUR_OPENAI_KEYfrom llama_index import VectorStoreIndex, SimpleDirectoryReader
from llama_index.postprocessor.flag_embedding_reranker import FlagEmbeddingReranker
from llama_index.schema import QueryBundledir_path YOUR_DIR_PATH目录中只有一个 PDF 文件即 TinyLlama: An Open Source Small Language Model。
(py) Florian:~ Florian$ ls /Users/Florian/Downloads/pdf_test/ tinyllama.pdf
使用 LlamaIndex 构建一个简单的检索器
documents SimpleDirectoryReader(dir_path).load_data()
index VectorStoreIndex.from_documents(documents)
retriever index.as_retriever(similarity_top_k 3)基本检索
query Can you provide a concise description of the TinyLlama model?
nodes retriever.retrieve(query)
for node in nodes:print(----------------------------------------------------)display_source_node(node, source_length 500)display_source_node 函数改编自 llama_index 源代码。原始函数是为 Jupyter notebook 设计的因此修改如下
from llama_index.schema import ImageNode, MetadataMode, NodeWithScore
from llama_index.utils import truncate_textdef display_source_node(source_node: NodeWithScore,source_length: int 100,show_source_metadata: bool False,metadata_mode: MetadataMode MetadataMode.NONE,
) - None:Display source nodesource_text_fmt truncate_text(source_node.node.get_content(metadata_modemetadata_mode).strip(), source_length)text_md (fNode ID: {source_node.node.node_id} \nfScore: {source_node.score} \nfText: {source_text_fmt} \n)if show_source_metadata:text_md fMetadata: {source_node.node.metadata} \nif isinstance(source_node.node, ImageNode):text_md Image:print(text_md)# display(Markdown(text_md))# if isinstance(source_node.node, ImageNode) and source_node.node.image is not None:# display_image(source_node.node.image)基本检索结果如下表示重新排序前的前 3 个节点
----------------------------------------------------
Node ID: 438b9d91-cd5a-44a8-939e-3ecd77648662
Score: 0.8706055408845863
Text: 4 Conclusion
In this paper, we introduce TinyLlama, an open-source, small-scale language model. To promote
transparency in the open-source LLM pre-training community, we have released all relevant infor-
mation, including our pre-training code, all intermediate model checkpoints, and the details of our
data processing steps. With its compact architecture and promising performance, TinyLlama can
enable end-user applications on mobile devices, and serve as a lightweight platform for testing a
w... ----------------------------------------------------
Node ID: ca4db90f-5c6e-47d5-a544-05a9a1d09bc6
Score: 0.8624531691777889
Text: TinyLlama: An Open-Source Small Language Model
Peiyuan Zhang∗Guangtao Zeng∗Tianduo Wang Wei Lu
StatNLP Research Group
Singapore University of Technology and Design
{peiyuan_zhang, tianduo_wang, sutd.edu.sgluwei}sutd.edu.sg
guangtao_zengmymail.sutd.edu.sg
Abstract
We present TinyLlama, a compact 1.1B language model pretrained on around 1
trillion tokens for approximately 3 epochs. Building on the architecture and tok-
enizer of Llama 2 (Touvron et al., 2023b), TinyLlama leverages various advances
contr... ----------------------------------------------------
Node ID: e2d97411-8dc0-40a3-9539-a860d1741d4f
Score: 0.8346160605298356
Text: Although these works show a clear preference on large models, the potential of training smaller
models with larger dataset remains under-explored. Instead of training compute-optimal language
models, Touvron et al. (2023a) highlight the importance of the inference budget, instead of focusing
solely on training compute-optimal language models. Inference-optimal language models aim for
optimal performance within specific inference constraints This is achieved by training models with
more tokens...重新排序
要对上述节点重新排序请使用 bge-reranker-base 模型。
print(------------------------------------------------------------------------------------------------)
print(Start reranking...)reranker FlagEmbeddingReranker(top_n 3,model BAAI/bge-reranker-base,
)query_bundle QueryBundle(query_strquery)
ranked_nodes reranker._postprocess_nodes(nodes, query_bundle query_bundle)
for ranked_node in ranked_nodes:print(----------------------------------------------------)display_source_node(ranked_node, source_length 500)重新排序后的结果如下
------------------------------------------------------------------------------------------------
Start reranking...
----------------------------------------------------
Node ID: ca4db90f-5c6e-47d5-a544-05a9a1d09bc6
Score: -1.584416151046753
Text: TinyLlama: An Open-Source Small Language Model
Peiyuan Zhang∗Guangtao Zeng∗Tianduo Wang Wei Lu
StatNLP Research Group
Singapore University of Technology and Design
{peiyuan_zhang, tianduo_wang, sutd.edu.sgluwei}sutd.edu.sg
guangtao_zengmymail.sutd.edu.sg
Abstract
We present TinyLlama, a compact 1.1B language model pretrained on around 1
trillion tokens for approximately 3 epochs. Building on the architecture and tok-
enizer of Llama 2 (Touvron et al., 2023b), TinyLlama leverages various advances
contr... ----------------------------------------------------
Node ID: e2d97411-8dc0-40a3-9539-a860d1741d4f
Score: -1.7028117179870605
Text: Although these works show a clear preference on large models, the potential of training smaller
models with larger dataset remains under-explored. Instead of training compute-optimal language
models, Touvron et al. (2023a) highlight the importance of the inference budget, instead of focusing
solely on training compute-optimal language models. Inference-optimal language models aim for
optimal performance within specific inference constraints This is achieved by training models with
more tokens... ----------------------------------------------------
Node ID: 438b9d91-cd5a-44a8-939e-3ecd77648662
Score: -2.904750347137451
Text: 4 Conclusion
In this paper, we introduce TinyLlama, an open-source, small-scale language model. To promote
transparency in the open-source LLM pre-training community, we have released all relevant infor-
mation, including our pre-training code, all intermediate model checkpoints, and the details of our
data processing steps. With its compact architecture and promising performance, TinyLlama can
enable end-user applications on mobile devices, and serve as a lightweight platform for testing a
w...很明显经过重新排序后ID 为 ca4db90f-5c6e-47d5-a544-05a9a1d09bc6 的节点的排名从 2 变为 1这意味着最相关的上下文被排在了第一位。
使用 LLM 作为重排器
现有的涉及 LLM 的重新排序方法大致可分为三类利用重新排序任务对 LLM 进行微调、提示 LLM 进行重新排序以及在训练过程中使用 LLM 进行数据增强。
提示 LLM 重新排序的方法成本较低。下面是使用 RankGPT 进行的演示它已被集成到 LlamaIndex 中。
RankGPT 的理念是使用 LLM如 ChatGPT 或 GPT-4 或其他 LLM执行零样本列表式段落重新排序。它采用排列生成方法和滑动窗口策略来有效地对段落重新排序。
如图 3 所示本文提出了三种可行的方法。 前两种方法是传统方法即给每篇文档打分然后根据分数对所有段落进行排序。
本文提出了第三种方法即排列生成法。具体来说该模型不依赖外部评分而是直接对段落进行端到端排序。换句话说它直接利用 LLM 的语义理解能力对所有候选段落进行相关性排序。
然而候选文档的数量通常非常大而 LLM 的输入却很有限。因此通常无法一次性输入所有文本。 因此如图 4 所示我们引入了一种滑动窗口法它沿用了冒泡排序的思想。每次只对前 4 个文本进行排序然后移动窗口对后面 4 个文本进行排序。在对整个文本进行反复排序后我们就可以得到性能最好的文本。
请注意要使用 RankGPT您需要安装较新版本的 LlamaIndex。我之前安装的版本0.9.29不包含 RankGPT 所需的代码。因此我用 LlamaIndex 0.9.45.post1 版本创建了一个新的 conda 环境。
代码非常简单基于上一节的代码只需将 RankGPT 设置为重选器即可。
from llama_index.postprocessor import RankGPTRerank
from llama_index.llms import OpenAI
reranker RankGPTRerank(top_n 3,llm OpenAI(modelgpt-3.5-turbo-16k),# verboseTrue,
)总体结果如下
(llamaindex_new) Florian:~ Florian$ python /Users/Florian/Documents/rerank.py
----------------------------------------------------
Node ID: 20de8234-a668-442d-8495-d39b156b44bb
Score: 0.8703492815379594
Text: 4 Conclusion
In this paper, we introduce TinyLlama, an open-source, small-scale language model. To promote
transparency in the open-source LLM pre-training community, we have released all relevant infor-
mation, including our pre-training code, all intermediate model checkpoints, and the details of our
data processing steps. With its compact architecture and promising performance, TinyLlama can
enable end-user applications on mobile devices, and serve as a lightweight platform for testing a
w... ----------------------------------------------------
Node ID: 47ba3955-c6f8-4f28-a3db-f3222b3a09cd
Score: 0.8621633467539512
Text: TinyLlama: An Open-Source Small Language Model
Peiyuan Zhang∗Guangtao Zeng∗Tianduo Wang Wei Lu
StatNLP Research Group
Singapore University of Technology and Design
{peiyuan_zhang, tianduo_wang, sutd.edu.sgluwei}sutd.edu.sg
guangtao_zengmymail.sutd.edu.sg
Abstract
We present TinyLlama, a compact 1.1B language model pretrained on around 1
trillion tokens for approximately 3 epochs. Building on the architecture and tok-
enizer of Llama 2 (Touvron et al., 2023b), TinyLlama leverages various advances
contr... ----------------------------------------------------
Node ID: 17cd9896-473c-47e0-8419-16b4ac615a59
Score: 0.8343984516104476
Text: Although these works show a clear preference on large models, the potential of training smaller
models with larger dataset remains under-explored. Instead of training compute-optimal language
models, Touvron et al. (2023a) highlight the importance of the inference budget, instead of focusing
solely on training compute-optimal language models. Inference-optimal language models aim for
optimal performance within specific inference constraints This is achieved by training models with
more tokens... ------------------------------------------------------------------------------------------------
Start reranking...
----------------------------------------------------
Node ID: 47ba3955-c6f8-4f28-a3db-f3222b3a09cd
Score: 0.8621633467539512
Text: TinyLlama: An Open-Source Small Language Model
Peiyuan Zhang∗Guangtao Zeng∗Tianduo Wang Wei Lu
StatNLP Research Group
Singapore University of Technology and Design
{peiyuan_zhang, tianduo_wang, sutd.edu.sgluwei}sutd.edu.sg
guangtao_zengmymail.sutd.edu.sg
Abstract
We present TinyLlama, a compact 1.1B language model pretrained on around 1
trillion tokens for approximately 3 epochs. Building on the architecture and tok-
enizer of Llama 2 (Touvron et al., 2023b), TinyLlama leverages various advances
contr... ----------------------------------------------------
Node ID: 17cd9896-473c-47e0-8419-16b4ac615a59
Score: 0.8343984516104476
Text: Although these works show a clear preference on large models, the potential of training smaller
models with larger dataset remains under-explored. Instead of training compute-optimal language
models, Touvron et al. (2023a) highlight the importance of the inference budget, instead of focusing
solely on training compute-optimal language models. Inference-optimal language models aim for
optimal performance within specific inference constraints This is achieved by training models with
more tokens... ----------------------------------------------------
Node ID: 20de8234-a668-442d-8495-d39b156b44bb
Score: 0.8703492815379594
Text: 4 Conclusion
In this paper, we introduce TinyLlama, an open-source, small-scale language model. To promote
transparency in the open-source LLM pre-training community, we have released all relevant infor-
mation, including our pre-training code, all intermediate model checkpoints, and the details of our
data processing steps. With its compact architecture and promising performance, TinyLlama can
enable end-user applications on mobile devices, and serve as a lightweight platform for testing a
w...请注意由于使用了 LLM重新排序后的分数并未更新。当然这并不重要。
从结果可以看出经过重新排序后排在第一位的结果是包含答案的正确文本这与前面使用重新排序模型得出的结果一致。
评估
reranker FlagEmbeddingReranker(top_n 3,model BAAI/bge-reranker-base,use_fp16 False
)# or using LLM as reranker
# from llama_index.postprocessor import RankGPTRerank
# from llama_index.llms import OpenAI
# reranker RankGPTRerank(
# top_n 3,
# llm OpenAI(modelgpt-3.5-turbo-16k),
# # verboseTrue,
# )query_engine index.as_query_engine( # add reranker to query_enginesimilarity_top_k 3, node_postprocessors[reranker]
)
# query_engine index.as_query_engine() # original query_engine参考https://ai.plainenglish.io/advanced-rag-03-using-ragas-llamaindex-for-rag-evaluation-84756b82dca7
结论
总之本文介绍了重新排序的原则和两种主流方法。
另一方面使用 LLM 的方法在多个基准测试中表现良好但成本较高而且仅在使用 ChatGPT 和 GPT-4 时表现良好而在使用 FLAN-T5 和 Vicuna-13B 等其他开源模型时表现不佳。
