1 Neural Network

1.1 Neural Network Components

• Simple Neuron单个神经元
  A neuron is a computational unit with n inputs and 1 output and parameters W(权重) , b

具体来说, 输入向量( Xi )和权重向量( Wi )进行点乘得到标量值, 标量值加上偏置值b后送入激活函数activation function中, 最后得到一个神经元的输出

• Single Layer Neural Network
  A single layer neural network: Hooking together many simple neurons
  在下面的例子中, 单层内有3个神经元, 且3个神经元之间并行计算.
  对每个神经元来说, 都要分别计算3次点乘, 与偏置值b相加后再送入各自的激活函数
  
  对于Wi, 可以直接用一个W矩阵表示

• Stacking multiple layers of neural networks

• Feedforward Computation 前馈计算
  

• Output Layer
  In Order to get specific outputs, add an output layer in the end
  
  与神经元层有多种激活函数类似, 输出层也可以选择多种输出函数

1.1.1 Why use non-linearities activation function ?

Without non-linearities, deep neural networks cannot do anything more than a linear transform

因此我们假设现有两层

可以看到, 若仅仅经过线性变换, 多层的神经网络和单层的神经网络表达能力是完全一致的,并没有扩展.

1.1.2 Choices of non-linearities

1.2 How to Train Neural Network

• Objective; Gradients; Backpropogation

1.3 Word Representation: Word2Vec(Based on Neural Network)

• Common Neural Networks
• RNN
• Sequential Memory; Language Model
• Gradient Problem for RNN
• Variants: GRU; LSTM; Bidirectional;
• CNN

1.4 NLP Pipeline Tutorial DEMO (Based on PyTorch)