1. NLP Basics Distributed Word Representation词表示

Word representation: a process that transform the symbols to the machine understandable meanings

1.1 How to represent the meaning so that the machine can understand

• Compute word similarity
  计算词相似度
  • WR(Star) ≃ WR(Sun)
  • WR(Motel) ≃ WR(Hotel)

• Infer word relation
  推断词之间的语义关系
  • WR(China) − WR(Beijing) ≃ WR(Japan) - WR(Tokyo)
  • WR(Man) ≃ WR(King) − WR(Queen) + WR(Woman)
  • WR(Swimming) ≃ WR(Walking) − WR(Walk) + WR(Swim

1.2 Synonym and Hypernym 同义词和上位词

过去怎么表示一个词的词义呢?
By Using set of related words, such as synonyms and hypernyms to represent a word

譬如说我们想表示GOOD这个词

Synonyms of “Good” in WordNet:
(n)good,goodness
(n)commodity,trade_good,good
(s)full,good
(s)adept,expert,good,practiced,proficient,skillful
(s)estimable,good,honorable,respectable

但这种方法存在问题

1. Missing nuance
   e.g. (“proficient”, “good”) are synonyms only in some contexts
2. Missing new meanings of words
   e.g. Apple (fruit → IT company)
3. Subjective主观性问题,受限于原本的词源标注
4. Data sparsity
5. Requires human labor to create and adapt

1.3 One-Hot Representation

对于计算机来说, 更好的办法仍然是将一个词表示为一个唯一的符号(向量)
每个词对应的向量都是独一无二的

• Vector dimension = # words in vocabulary
• Order is not important

但这种方法存在问题

• All the vectors are orthogonal. No natural notion of similarity for one-hot vectors.
  因此, 根本无法计算相似度
  

1.4 Represent Word by Context

核心思想是用词的上下文来表示这个词

• The meaning of a word is given by the words that frequently appear close-by
• One of the most successful ideas of modern statistical NLP

e.g. Use context words to represent stars

1.5 Count-Based Representation

在Represent Word by Context 的基础上, 仍然对 n 个词的文本创建一个 n 维向量 ,
并且对其他词与词 A 同时出现的次数进行计数( Co-Occurrence Counts ), 写入A的向量中
这样我们能得到一个稠密向量, 对稠密向量之间进行相似度计算是可行的
但这种方法存在问题

• Increase in size with vocabulary, require a lot of storage
• sparsity issues for those less frequent words

1.6 Word Embedding

运用分布式表达的方法Distributed Representation

尝试用一个低维的空间就将文本全集装载, 然后在这个低维空间中进行相似度运算

• Build a dense vector for each word learned from large-scale text corpora
• 一个比较知名的方法 : Word2Vec (We will learn it in the next class)\

1.7 Language Model

• Language Modeling is the task of predicting the upcoming word
  
• A language model is a probability distribution over a sequence of words\

语言模型的两个任务
由此引出一个问题 : 如何计算概率?

引入一个假设
Assumption: the probability of an upcoming word is only determined by all its previous words
以此就能将句子的概率拆解为条件概率
e.g.
即对于语言模型来说
一个句子的联合概率 = 每个词相对于整体的条件概率再取积

1.8 N-gram Model

Collect statistics about how frequent different ngrams are, and use these to predict next word.

例如 , 对于 4-gram, 统计三个词too late to 之后接不同的词的概率

但这种方法存在问题

• Need to store count for all possible n-grams. So model size is O ( e^n )
• Not considering contexts farther than 1 or 2 words
• Not capturing the similarity between words

最简单的例子, 如果以整个互联网的文本去统计, 而每次仅统计两三个词连在一起的概率, 最终统计结果会相当稀疏
e.g.
• The cat is walking in the bedroom
• A dog was running in a room
3-gram 也无法认识到 cat 和 dog 的相似度, walking 和 running 的相似度

1.9 Neural Language Model

A neural language model is a language model based on neural networks to learn distributed representadons of words

• Associate words with distributed vectors
• Compute the joint probability of word sequences in terms of the feature vectors
• Optimize the word feature vectors (embedding matrix E) and the parameters of the loss function (map matrix W)

求Wt在Context下的条件概率, 可以利用前几个词( 这里取3 )的向量, 拼成一个高维的上下文向量, 再经过非线性转换tanh , 就可以预测下一个词.

整个的匹配过程是通过 神经网络 , 在可调的过程中完成的.

2. Big Model Basics Development

3 Paradigms behind Big Models 大模型背后的范式

1. 对于预训练模型来说, 很关键的一点是 模型会从无标注的数据中学习, 通过自监督的任务获取通用知识.
2. 在预训练完毕的模型上引入任务相关数据, 帮助具体的任务做适配
3. 最终得到解决具体任务的模型
   
   The breakthrough of NLP: Transformer
   Based on Transformer, a series of deep pretraining models are developed instead of shallow RNNs, which is more powerful