Beyond Basic Training

超越基础训练

The majority of the effort in training ChatGPT is spent “showing it” large amounts of existing text from the web, books, etc. But it turns out there’s another—apparently rather important—part too.

在训练 ChatGPT 的过程中，大部分努力都花在了让它“阅读”大量现有的网络、书籍等文本上。但事实证明，还有另一个看似非常重要的部分。

As soon as it’s finished its “raw training” from the original corpus of text it’s been shown, the neural net inside ChatGPT is ready to start generating its own text, continuing from prompts, etc. But while the results from this may often seem reasonable, they tend—particularly for longer pieces of text—to “wander off” in often rather non-human-like ways. It’s not something one can readily detect, say, by doing traditional statistics on the text. But it’s something that actual humans reading the text easily notice.

在完成了原始文本语料库的“原始训练”后，ChatGPT 内部的神经网络就准备好开始生成自己的文本，接着完成提示等任务。尽管这样的结果看起来很多时候是合理的，但它们往往会以非常不像人类的方式“漫游”，特别是在生成较长的文本时。这不是我们通过对文本进行传统统计就能轻易发现的东西，但实际阅读文本的人类可以很容易地注意到。

And a key idea in the construction of ChatGPT was to have another step after “passively reading” things like the web: to have actual humans actively interact with ChatGPT, see what it produces, and in effect give it feedback on “how to be a good chatbot”. But how can the neural net use that feedback? The first step is just to have humans rate results from the neural net. But then another neural net model is built that attempts to predict those ratings. But now this prediction model can be run—essentially like a loss function—on the original network, in effect allowing that network to be “tuned up” by the human feedback that’s been given. And the results in practice seem to have a big effect on the success of the system in producing “human-like” output.

在构建 ChatGPT 时，关键的想法是在“被动阅读”网络等内容之后增加一个步骤：让真正的人类积极地与 ChatGPT 互动，查看它生成的内容，并实际上给它提供关于“如何成为一个好的聊天机器人”的反馈。但是神经网络如何利用这种反馈呢？第一步就是让人类对神经网络的结果进行评分。然后，构建另一个神经网络模型，试图预测这些评分。但现在，这个预测模型可以像损失函数一样在原始网络上运行，实际上允许这个网络通过已经给出的人类反馈进行“调优”。实践中的结果似乎对系统产生“人类般”的输出的成功产生了很大的影响。

In general, it’s interesting how little “poking” the “originally trained” network seems to need to get it to usefully go in particular directions. One might have thought that to have the network behave as if it’s “learned something new” one would have to go in and run a training algorithm, adjusting weights, and so on.

总的来说，让“原始训练”过的网络在特定方向上有用的“调整”似乎需要的程度很小。人们可能会认为，要让网络表现得像是“学到了新东西”，就必须进入并运行训练算法，调整权重等。

But that’s not the case. Instead, it seems to be sufficient to basically tell ChatGPT something one time—as part of the prompt you give—and then it can successfully make use of what you told it when it generates text. And once again, the fact that this works is, I think, an important clue in understanding what ChatGPT is “really doing” and how it relates to the structure of human language and thinking.

然而事实并非如此。相反，基本上只需告诉 ChatGPT 一次，作为给出提示的一部分，然后它就可以在生成文本时成功利用你告诉它的内容。再次强调，这种方法行之有效，我认为这是理解 ChatGPT 的“真正功能”以及它与人类语言和思维结构之间关系的一个重要线索。

There’s certainly something rather human-like about it: that at least once it’s had all that pre-training you can tell it something just once and it can “remember it”—at least “long enough” to generate a piece of text using it. So what’s going on in a case like this? It could be that “everything you might tell it is already in there somewhere”—and you’re just leading it to the right spot. But that doesn’t seem plausible. Instead, what seems more likely is that, yes, the elements are already in there, but the specifics are defined by something like a “trajectory between those elements” and that’s what you’re introducing when you tell it something.

这确实具有相当人类化的特点：至少在完成所有预先训练后，你只需告诉它一次，它就能“记住”——至少“足够长的时间”来生成使用它的文本。那么，在这种情况下到底发生了什么？可能是“你可能告诉它的一切已经存在于某个地方”——而你只是引导它到正确的地方。但这似乎不太可能。相反，更有可能的是，是的，元素已经存在，但是具体细节是由类似于“这些元素之间的轨迹”所定义的，这就是你在告诉它某件事时所引入的东西。

And indeed, much like for humans, if you tell it something bizarre and unexpected that completely doesn’t fit into the framework it knows, it doesn’t seem like it’ll successfully be able to “integrate” this. It can “integrate” it only if it’s basically riding in a fairly simple way on top of the framework it already has.

实际上，与人类非常相似，如果你告诉它一些奇怪而出乎意料的东西，完全不符合它所知道的框架，它似乎就无法成功地“整合”这些内容。只有当它基本上是以非常简单的方式基于已有的框架时，它才能“整合”这些内容。

It’s also worth pointing out again that there are inevitably “algorithmic limits” to what the neural net can “pick up”. Tell it “shallow” rules of the form “this goes to that”, etc., and the neural net will most likely be able to represent and reproduce these just fine—and indeed what it “already knows” from language will give it an immediate pattern to follow. But try to give it rules for an actual “deep” computation that involves many potentially computationally irreducible steps and it just won’t work. (Remember that at each step it’s always just “feeding data forward” in its network, never looping except by virtue of generating new tokens.)

同时，再次指出，神经网络能够“学到”的东西在算法上是有限的。告诉它“浅层”的规则，例如“这个变成那个”等，神经网络很可能能够很好地表示和复制这些规则——实际上，它从语言中“已经知道”的东西会立即给出一种模式。但是，尝试为涉及许多可能的计算不可约步骤的实际“深层”计算提供规则，那就行不通了。（请记住，在每一步中，它总是只在网络中“向前传递数据”，除了通过生成新的标记之外，从不循环。）

Of course, the network can learn the answer to specific “irreducible” computations. But as soon as there are combinatorial numbers of possibilities, no such “table-lookup-style” approach will work. And so, yes, just like humans, it’s time then for neural nets to “reach out” and use actual computational tools. (And, yes, Wolfram|Alpha and Wolfram Language are uniquely suitable, because they’ve been built to “talk about things in the world”, just like the language-model neural nets.)

当然，网络可以学习特定的“不可约”的计算答案。但是，一旦存在组合数数量级的可能性，那么任何类似“查表法”的方法都将行不通。因此，是的，就像人类一样，神经网络现在需要“拓展”并使用实际的计算工具。（而且，是的，Wolfram|Alpha 和 Wolfram 语言是独特适合的，因为它们是为了“谈论世界中的事物”而构建的，就像语言模型神经网络一样。）

“点赞有美意，赞赏是鼓励”