地址

论文地址：https://arxiv.org/abs/2408.06195

时间

2024年 8月12 日

作者

微软亚洲研究院、哈佛大学

This paper introduces rStar, a self-play mutual reasoning approach that significantly improves reasoning capabilities of small language models (SLMs) without fine-tuning or superior models. rStar decouples reasoning into a self-play mutual generation-discrimination process. First, a target SLM augments the Monte Carlo Tree Search (MCTS) with a rich set of human-like reasoning actions to construct higher quality reasoning trajectories. Next, another SLM, with capabilities similar to the target SLM, acts as a discriminator to verify each trajectory generated by the target SLM. The mutually agreed reasoning trajectories are considered mutual consistent, thus are more likely to be correct. Extensive experiments across five SLMs demonstrate rStar can effectively solve diverse reasoning problems, including GSM8K, GSM-Hard, MATH, SVAMP, and StrategyQA. Remarkably, rStar boosts GSM8K accuracy from 12.51% to 63.91% for LLaMA2-7B, from 36.46% to 81.88% for Mistral-7B, from 74.53% to 91.13% for LLaMA3-8B-Instruct. Code will be available at this https URL.

本文介绍了rStar，这是一种无需微调或使用更高级模型的自我博弈互推理方法，可显著提高小型语言模型（SLMs）的推理能力。rStar将推理分解为一个自我博弈的生成-鉴别过程。首先，一个目标SLM通过向蒙特卡洛树搜索（MCTS）添加一套类似于人类推理的操作来构建更高质量的推理轨迹。然后，另一个与目标SLM能力相似的SLM作为鉴别器，验证目标SLM生成的每个轨迹。相互达成一致的推理轨迹被认为是相互一致的，因此更有可能正确。在五个SLM上的广泛实验表明，rStar可以有效地解决各种推理问题，包括GSM8K、GSM-Hard、MATH、SVAMP和StrategyQA。值得注意的是，rStar将LLaMA2-7B在GSM8K上的准确率从12.51%提高到63.91%，将Mistral-7B在GSM8K上的准确率从36.46%提高到81.88%，将LLaMA3更令人担忧的事实是，上述两个问题在较小版本的LLMs（即SLMs）中表现得更为明显，因为它们的能力较弱。例如，虽然GPT-4可以通过自我完善输出来提高性能（Madaan等人，2024；Wu等人，2024；Zhou等人，2024），但这些方法在SLMs中的效果较差，甚至可能导致更差的表现（Forsman，2024）。这大大阻碍了神经语言模型的采用。

Despite their success, large language models (LLMs) face significant challenges in complex reasoning (Valmeekam et al., 2022; Weng et al., 2023). For example, state of the art models like Mistral-7B (Jiang et al., 2023) can only achieve 36.5% accuracy on the GSM8K dataset, even with techniques like Chain-of-Throught (CoT) (Wei et al., 2022). Although fine-tuning is shown to be an effective way to improve reasoning capability, most LLMs rely on fine-tuning data distilled or synthesized by superior models like GPT-4 (Wang et al., 2024a; Gou et al., 2023). Meanwhile, the community has been actively working on a complimentary and yet more challenging approach: Reasoning improvements without a superior teacher LLM.

First, LLMs often struggle to effectively explore the solution space during reasoning. The self-exploration often traps in a solution space with low-quality reasoning steps even after many attempts. For example, our experiments reveal that after 32 rounds of self-exploration with RAP (Hao et al., 2023), only 24% of the trajectories generated by LLaMA2-7B on GSM8K are correct. Second, even the self-exploration can find high quality reasoning steps, it is difficult for SLMs to tell which reasoning steps are of higher quality or determine which final answers are correct, thus it is hard to effectively guide the self-exploration. Our study shows that a naïve reward-based self-exploration guidance can lead to results no better than random guesses (see Appendix A.1).

尽管它们已经取得了成功，但大型语言模型（LLMs）在复杂推理方面仍面临重大挑战（Valmeekam et al.， 2022; Weng et al.， 2023）。例如，像Mistral-7B这样的先进模型（Jiang et al.， 2023）在GSM8K数据集上的准确率只有36.5%，即使使用了链式推理（Chain-of-Throught）等技术（Wei et al.， 2022）。尽管微调被证明是提高推理能力的有效方法，但大多数LLMs依赖于由GPT-4等更优越的模型提炼或合成的数据进行微调（Wang et al.， 2024a; Gou et al.， 2023）。与此同时，社区一直在积极探索一种补充性且更具挑战性的方法：无需优越的教师LLM的推理改进。

首先，LLMs在推理过程中往往难以有效探索解决方案空间。自我探索往往在低质量的推理步骤的解决方案空间中陷入困境，即使经过多次尝试也是如此。例如，我们的实验表明，在32轮自我探索（Hao et al.， 2023）之后，LLaMA2-7B在GSM8K上的生成轨迹中只有24%是正确的。第二，即使自我探索可以找到高质量的推理步骤，SLMs也很难区分哪些推理步骤质量更高或确定哪些最终答案是正确的，因此很难有效引导自我探索。我们的研究表明，一种简单的基于奖励的自我探索指导可能导致的结果甚至不如随机猜测（参见附录A.1）。

A more troublesome fact is that the above two issues are more pronounced in the smaller version of LLMs, i.e., SLMs, due to their weaker capabilities. For instance, while GPT-4 can improve by self-refining its output (Madaan et al., 2024; Wu et al., 2024; Zhou et al., 2024), the approaches are less effective in SLMs and may even lead to worse performance (Forsman, 2024). This significantly hinders the adoption of neural language models.

This paper introduces Self-play muTuAl Reasoning (rStar), a novel approach that boosts SLMs’ reasoning capability during inference without fine-tuning or superior models. To address the aforementioned challenges, rStar decouples reasoning into a self-play mutual generation-discrimination process as illustrated in Fig. 2. Specifically, rStar is unique in the following approaches. First, although relying on a conventional Monte Carlo Tree Search (MCTS) for SLMs to self-generate reasoning steps, rStar advocates a richer set of reasoning actions in the self-exploration. The new proposed actions simulate human reasoning behaviors given the current reasoning state, such as decomposing and searching for a specific reasoning step, proposing a new sub-question, or rephrasing the given question. This enables SLMs to generate high-quality candidate reasoning trajectories during self-exploration. Second, to effectively guide the exploration among the generated reasoning trajectories, rStar augments the MCTS process with a new discrimination process called mutual consistency. In particular, rStar employs a second SLM with the similar capability, acting as a discriminator to provide unsupervised feedback on each candidate reasoning trajectory generated by MCTS. To improve the accuracy of the feedback, rStar hints the second SLM with sampled partial reasoning trajectories, asking it to complete the remaining reasoning steps. And rStar deems the mutually agreed reasoning trajectories of higher quality. Mutual consistency mirrors the common human practice in the absence of supervision, where agreement among peers (i.e., two SLMs) on derived answers suggests a higher likelihood of correctness. As a result, mutual consistency offers more effective reasoning across diverse tasks than other approaches like self-consistency (Wang et al., 2023) and avoids the risk of overfitting when training a reward model (Chen et al., 2024a; Wang et al., 2024b).

更令人担忧的是，上述两个问题在较小版本的LLMs（即SLMs）中更为明显，因为它们的能力较弱。例如，虽然GPT-4可以通过自我完善输出来改进（Madaan等人，2024；Wu等人，2024；Zhou等人，2024），但这些方法在SLMs中的效果较差，甚至可能导致更差的表现（Forsman，2024）。这大大阻碍了神经语言模型的采用。

本文介绍了Self-play muTuAl Reasoning（rStar），这是一种新颖的方法，可以在推理过程中提升SLMs的推理能力，无需微调或使用高级模型。为了解决上述挑战，rStar将推理分解为一个自我博弈的生成-鉴别过程，如图2所示。具体来说，rStar在以下方面具有独特之处。首先，尽管依赖传统的蒙特卡洛树搜索（MCTS）算法来指导SLMs进行自我推理，但rStar在自我探索中提倡更丰富的推理动作集。新提出的动作模拟了在当前推理状态下的人类推理行为，例如分解并搜索特定的推理步骤、提出新的子问题或重述给定的问题。这使得SLMs能够在自我探索期间生成高质量的候选推理轨迹。其次，为了有效地引导生成的推理轨迹之间的探索，rStar在MCTS过程中引入了一个新的鉴别过程，称为相互一致性。特别是，rStar使用了一个具有类似能力的第二SLM，作为鉴别器，为由MCTS生成的每个候选推理轨迹提供无监督反馈。为了提高反馈的准确性，rStar向第二SLM提供了部分推理轨迹样本，要求它完成剩余的推理步骤。rStar认为相互一致的推理轨迹质量更高。相互一致性反映了在缺乏监督的情况下，人类的常见做法，即同行（即两个SLM）对衍生答案的一致意见表明正确性的可能性更高。因此，相互一致性比其他方法（如自一致性（Wang 等人，2023））能够更有效地在各种任务之间进行推理，并且避免了在训练奖励模型时出现过拟合的风险（Chen 等人，2024a；Wang 等人，2024b）。

Extensive experiments across five SLMs and five diverse reasoning tasks demonstrate the effectiveness of rStar. With just 32 rounds of MCTS inference, rStar significantly enhances SLMs’ reasoning capabilities, matching or even surpassing the accuracy achieved after fine-tuning. For example, rStar boosts GSM8K accuracy from 12.51% to 63.91% for LLaMA2-7B, from 36.46% to 81.88% for Mistral, and from 47.23% to 85.52% for LLaMA3-8B. Furthermore, we conduct comprehensive experiments to verify rStar’s superiority over state-of-the-art baselines, including single-round inference techniques like few-shot CoT, multi-round prompting approaches such as self-consistency, and self-improvement techniques such as RAP, ToT, self-evaluation and self-verification.

在五个SLM和五个不同的推理任务中进行的大量实验证明了rStar的有效性。通过32轮MCTS推理，rStar显著提高了slm的推理能力，达到甚至超过微调后的精度。例如，rStar将LLaMA2-7B的GSM8K精度从12.51%提高到63.91%，Mistral的GSM8K精度从36.46%提高到81.88%，LLaMA3-8B的GSM8K精度从47.23%提高到85.52%。此外，我们进行了全面的实验来验证rStar相对于最先进基线的优势，包括单轮推理技术，如少样本CoT，多轮提示方法，如自一致性，以及自我完善技术，如RAP， ToT，自我评估和自我验证。

In this work, we present rStar, a generator-discriminator self-play approach that significantly grow the reasoning capabilities for SLMs at the inference time. Our approach reveals that SLMs, such as LLaMA2-7B, already exhibit strong reasoning capabilities prior to domain specialized supervised fine-tuning. rStar achieves state-of-the-art performance across five SLMs and five diverse reasoning tasks, substantially outperforming existing multi-round prompting and self-improvement approaches. Furthermore, we conduct extensive ablation studies and analysis, contributing to the development of more advanced SLM self-improved reasoning.

在这项工作中，我们提出了rStar，这是一种生成器-判别器自玩方法，可以在推理时显著提高SLM（序列学习模型）的推理能力。我们的方法表明，在进行针对特定领域的监督微调之前，例如LLaMA2-7B，SLM已经具有很强的推理能力。rStar在五个SLM和五个不同推理任务上实现了当前最佳性能，显著优于现有的多轮提示和自我改进方法。此外，我们进行了广泛的实验分析，为开发更先进的SLM自我改进推理技术做出了贡献。