The Concept of Embeddings 

嵌入的概念

Neural nets—at least as they’re currently set up—are fundamentally based on numbers. So if we’re going to to use them to work on something like text we’ll need a way to represent our text with numbers. And certainly we could start (essentially as ChatGPT does) by just assigning a number to every word in the dictionary. But there’s an important idea—that’s for example central to ChatGPT—that goes beyond that. And it’s the idea of “embeddings”. One can think of an embedding as a way to try to represent the “essence” of something by an array of numbers—with the property that “nearby things” are represented by nearby numbers.

神经网络——至少在目前的设置中——是基于数字的。所以，如果我们要使用它们来处理像文本这样的东西，我们需要一种用数字表示文本的方法。当然，我们可以开始（就像 ChatGPT 所做的那样）为字典中的每个单词分配一个数字。但是，还有一个重要的想法——例如，这是 ChatGPT 的核心思想——超越了这一点。这个想法就是“嵌入”。我们可以将嵌入看作是一种尝试通过一组数字来表示事物“本质”的方法——其特性是“相邻的事物”由相邻的数字表示。

And so, for example, we can think of a word embedding as trying to lay out words in a kind of “meaning space” in which words that are somehow “nearby in meaning” appear nearby in the embedding. The actual embeddings that are used—say in ChatGPT—tend to involve large lists of numbers. But if we project down to 2D, we can show examples of how words are laid out by the embedding:

因此，例如，我们可以将词嵌入视为尝试在一种“意义空间”中布局单词，其中在意义上“相近”的单词在嵌入中也相邻。实际使用的嵌入——比如在 ChatGPT 中——倾向于涉及大量的数字列表。但如果我们将其投影到二维空间，我们可以展示单词在嵌入中的布局示例：

And, yes, what we see does remarkably well in capturing typical everyday impressions. But how can we construct such an embedding? Roughly the idea is to look at large amounts of text (here 5 billion words from the web) and then see “how similar” the “environments” are in which different words appear. So, for example, “alligator” and “crocodile” will often appear almost interchangeably in otherwise similar sentences, and that means they’ll be placed nearby in the embedding. But “turnip” and “eagle” won’t tend to appear in otherwise similar sentences, so they’ll be placed far apart in the embedding.

是的，我们看到的确实在很大程度上捕捉到了典型的日常印象。但是我们如何构建这样的嵌入呢？大致的想法是查看大量的文本（这里是来自网络的 50 亿个单词），然后看看不同单词出现的“环境”有多“相似”。因此，例如，“alligator”和“crocodile”通常会在其他相似的句子中几乎可以互换地出现，这意味着它们在嵌入中会被放置得很近。但“turnip”和“eagle”不会出现在其他类似的句子中，因此它们在嵌入中的位置会相距较远。

But how does one actually implement something like this using neural nets? Let’s start by talking about embeddings not for words, but for images. We want to find some way to characterize images by lists of numbers in such a way that “images we consider similar” are assigned similar lists of numbers.

但是，如何使用神经网络实际实现这样的功能呢？让我们先谈谈关于图像而不是单词的嵌入。我们希望建立某种方法，通过数字列表来描述图像，使得“我们认为相似的图像”被分配相似的数字列表。

How do we tell if we should “consider images similar”? Well, if our images are, say, of handwritten digits we might “consider two images similar” if they are of the same digit. Earlier we discussed a neural net that was trained to recognize handwritten digits. And we can think of this neural net as being set up so that in its final output it puts images into 10 different bins, one for each digit.

我们如何判断我们是否应该“认为图像相似”呢？好吧，如果我们的图像是，比如说，手写数字，我们可能会“认为两个图像相似”，如果它们是相同的数字。之前我们讨论了一个经过训练的神经网络，用于识别手写数字。我们可以将这个神经网络看作是设置成在其最终输出中将图像放入 10 个不同的箱子，每个数字一个。

But what if we “intercept” what’s going on inside the neural net before the final “it’s a ‘4’” decision is made? We might expect that inside the neural net there are numbers that characterize images as being “mostly 4-like but a bit 2-like” or some such. And the idea is to pick up such numbers to use as elements in an embedding.

但是，如果我们在最终做出“这是‘4’”的决定之前，“拦截”神经网络内部正在进行的操作会怎样呢？我们可能会期望神经网络内部有一些数字来表征图像，比如“大部分像 4，但有点像 2 ”之类的。这个想法就是利用这些数字作为嵌入中的元素。

So here’s the concept. Rather than directly trying to characterize “what image is near what other image”, we instead consider a well-defined task (in this case digit recognition) for which we can get explicit training data—then use the fact that in doing this task the neural net implicitly has to make what amount to “nearness decisions”. So instead of us ever explicitly having to talk about “nearness of images” we’re just talking about the concrete question of what digit an image represents, and then we’re “leaving it to the neural net” to implicitly determine what that implies about “nearness of images”.

这里是概念。我们不是直接尝试描述“哪个图像靠近哪个图像”，而是考虑一个可以获得明确训练数据的明确定义任务（在这种情况下是数字识别），然后利用神经网络在执行这个任务时必须隐含地进行类似“靠近决策”的事实。所以我们不需要明确地讨论“图像的靠近性”，我们只是讨论一个具体问题，即图像代表的数字是什么，然后我们“把它留给神经网络”来隐含地确定这对“图像的靠近性”意味着什么。

So how in more detail does this work for the digit recognition network? We can think of the network as consisting of 11 successive layers, that we might summarize iconically like this (with activation functions shown as separate layers):

那么，对于数字识别网络，这个方法具体是如何工作的呢？我们可以将网络看作是由 11 个连续的层组成，我们可以用图标表示如下（激活函数显示为单独的层）：

At the beginning we’re feeding into the first layer actual images, represented by 2D arrays of pixel values. And at the end—from the last layer—we’re getting out an array of 10 values, which we can think of saying “how certain” the network is that the image corresponds to each of the digits 0 through 9.

在开始时，我们将实际图像（由像素值的二维数组表示）输入到第一层。在最后，从最后一层中，我们得到一个包含 10 个值的数组，我们可以认为它表示网络确定图像对应 0 到 9 的每个数字的“确定程度”。

Feed in the image 

 and the values of the neurons in that last layer are:

输入图像：最后一层神经元的值为：

In other words, the neural net is by this point “incredibly certain” that this image is a 4—and to actually get the output “4” we just have to pick out the position of the neuron with the largest value.

换句话说，到目前为止，神经网络“非常确定”这个图像是 4——要实际得到输出“4”，我们只需要找到数值最大的神经元的位置。

But what if we look one step earlier? The very last operation in the network is a so-called softmax which tries to “force certainty”. But before that’s been applied the values of the neurons are:

但是，如果我们再往前看一步会怎样呢？网络中的最后一个操作是所谓的“softmax”，它试图“强制确定”。但在应用“softmax”之前，神经元的值是：

The neuron representing “4” still has the highest numerical value. But there’s also information in the values of the other neurons. And we can expect that this list of numbers can in a sense be used to characterize the “essence” of the image—and thus to provide something we can use as an embedding. And so, for example, each of the 4’s here has a slightly different “signature” (or “feature embedding”)—all very different from the 8’s:

表示“4”的神经元仍然具有最高的数值。但是其他神经元的值中也有信息。我们可以期望这个数字列表在某种程度上用来描述图像的“本质”，从而提供我们可以用作嵌入的东西。因此，例如，这里的每个 4 都有一个略有不同的“签名”（或“特征嵌入”）——与 8 完全不同：

Here we’re essentially using 10 numbers to characterize our images. But it’s often better to use much more than that. And for example in our digit recognition network we can get an array of 500 numbers by tapping into the preceding layer. And this is probably a reasonable array to use as an “image embedding”.

在这里，我们实质上使用了 10 个数字来描述我们的图像。但通常使用更多的数字会更好。例如，在我们的数字识别网络中，我们可以通过连接到前一层来获得一个包含 500 个数字的数组。这可能是一个合理的“图像嵌入”。

If we want to make an explicit visualization of “image space” for handwritten digits we need to “reduce the dimension”, effectively by projecting the 500-dimensional vector we’ve got into, say, 3D space:

如果我们想为手写数字的“图像空间”制作一个明确的可视化，我们需要“降低维度”，实际上是将我们得到的 500 维向量投影到，例如，3D 空间：

We’ve just talked about creating a characterization (and thus embedding) for images based effectively on identifying the similarity of images by determining whether (according to our training set) they correspond to the same handwritten digit. And we can do the same thing much more generally for images if we have a training set that identifies, say, which of 5000 common types of object (cat, dog, chair, …) each image is of. And in this way we can make an image embedding that’s “anchored” by our identification of common objects, but then “generalizes around that” according to the behavior of the neural net. And the point is that insofar as that behavior aligns with how we humans perceive and interpret images, this will end up being an embedding that “seems right to us”, and is useful in practice in doing “human-judgement-like” tasks.

我们刚刚讨论了如何根据确定图像是否（根据我们的训练集）对应于相同的手写数字来识别图像的相似性，从而为图像创建一个特征（因此嵌入）。对于图像，如果我们有一个训练集，该训练集可以识别出，例如，每个图像是 5000 种常见类型的对象（猫、狗、椅子等）中的哪一种，我们可以做同样的事情。通过这种方式，我们可以制作一个以我们识别常见物体为“锚点”的图像嵌入，然后根据神经网络的行为“在此基础上进行推广”。关键在于，只要该行为与我们人类感知和解释图像的方式相一致，这将最终成为一个“对我们来说看似正确”的嵌入，并在实践中用于执行“类似人类判断”的任务。

OK, so how do we follow the same kind of approach to find embeddings for words? The key is to start from a task about words for which we can readily do training. And the standard such task is “word prediction”. Imagine we’re given “the ___ cat”. Based on a large corpus of text (say, the text content of the web), what are the probabilities for different words that might “fill in the blank”? Or, alternatively, given “___ black ___” what are the probabilities for different “flanking words”?

好的，那么我们如何采用相同的方法来找到单词的嵌入呢？关键是从一个可以轻松进行训练的单词任务开始。标准的这类任务是“单词预测”。想象一下，我们得到了“the ___ cat”。基于大量的文本语料库（比如，互联网的文本内容），可能填入空白处的不同单词的概率是多少？或者，换个说法，给定“___ black ___”，不同的“外围词”的概率是多少？

How do we set this problem up for a neural net? Ultimately we have to formulate everything in terms of numbers. And one way to do this is just to assign a unique number to each of the 50,000 or so common words in English. So, for example, “the” might be 914, and “ cat” (with a space before it) might be 3542. (And these are the actual numbers used by GPT-2.) So for the “the ___ cat” problem, our input might be {914, 3542}. What should the output be like? Well, it should be a list of 50,000 or so numbers that effectively give the probabilities for each of the possible “fill-in” words. And once again, to find an embedding, we want to “intercept” the “insides” of the neural net just before it “reaches its conclusion”—and then pick up the list of numbers that occur there, and that we can think of as “characterizing each word”.

我们如何为神经网络设置这个问题？归根结底，我们必须用数字来表达一切。实现这一目标的一种方法就是为英语中大约 50000 个常用单词中的每一个分配一个唯一的数字。例如，“the”可能是 914，“cat”（前面有一个空格）可能是 3542。（这些实际上是 GPT-2 使用的数字。）所以对于“the ___ cat”问题，我们的输入可能是{914, 3542}。输出应该是什么样的呢？嗯，它应该是一个包含大约 50000 个数字的列表，有效地给出每个可能的“填空”单词的概率。再次强调，要找到嵌入，我们希望在神经网络“得出结论”之前的某个时刻“拦截”它的“内部”，然后获取那里出现的数字列表，我们可以认为这个列表是“描述每个单词”的。

OK, so what do those characterizations look like? Over the past 10 years there’ve been a sequence of different systems developed (word2vec, GloVe, BERT, GPT, …), each based on a different neural net approach. But ultimately all of them take words and characterize them by lists of hundreds to thousands of numbers.

那么，这些描述看起来是什么样的呢？在过去的 10 年里，已经有一系列不同的系统（word2vec、GloVe、BERT、GPT等）被开发出来，每一个都基于不同的神经网络方法。但最终，它们都会将单词表示为包含数百到数千个数字的列表。

In their raw form, these “embedding vectors” are quite uninformative. For example, here’s what GPT-2 produces as the raw embedding vectors for three specific words:

在其原始形式下，这些“嵌入向量”并不包含太多信息。例如，这是 GPT-2 对三个特定单词生成的原始嵌入向量：

If we do things like measure distances between these vectors, then we can find things like “nearnesses” of words. Later we’ll discuss in more detail what we might consider the “cognitive” significance of such embeddings. But for now the main point is that we have a way to usefully turn words into “neural-net-friendly” collections of numbers.

如果我们测量这些向量之间的距离，那么我们可以找到诸如单词之间的“接近度”。稍后我们将更详细地讨论这种嵌入的“认知”意义。但目前最主要的一点是，我们有一种方法可以将单词有效地转换成适合神经网络的数字集合。

But actually we can go further than just characterizing words by collections of numbers; we can also do this for sequences of words, or indeed whole blocks of text. And inside ChatGPT that’s how it’s dealing with things. It takes the text it’s got so far, and generates an embedding vector to represent it. Then its goal is to find the probabilities for different words that might occur next. And it represents its answer for this as a list of numbers that essentially give the probabilities for each of the 50,000 or so possible words.

实际上，我们不仅可以通过数字集合来描述单词，还可以用它们来描述单词序列，甚至整个文本块。在 ChatGPT 中，就是这样处理事物的。它接收到目前为止的文本，并生成一个嵌入向量来表示它。然后它的目标是找到接下来可能出现的不同单词的概率。它将这个答案表示为一个数字列表，这个列表基本上给出了大约 50000 个可能单词的概率。

(Strictly, ChatGPT does not deal with words, but rather with “tokens”—convenient linguistic units that might be whole words, or might just be pieces like “pre” or “ing” or “ized”. Working with tokens makes it easier for ChatGPT to handle rare, compound and non-English words, and, sometimes, for better or worse, to invent new words.)

(严格来说，ChatGPT 处理的不是单词，而是“标记”——便捷的语言单位，可能是整个单词，也可能仅仅是像“pre”、“ing”或“ized”这样的部分。使用标记使 ChatGPT 更容易处理罕见、复合和非英语单词，有时候，或好或坏地，创造新词。)

“点赞有美意，赞赏是鼓励”