引言

上一节介绍了循环神经网络反向传播中存在的梯度消失问题，并以此为引介绍了长短期记忆神经网络$(\text{Long-Short Term Memory,LSTM})$。本节将从反向传播角度观察为什么$\text{LSTM}$能够抑制梯度消失的情况。

回顾加补充：通过时间反向传播

回顾上一节中针对$\text{RNN}$中$\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(1)}\Rightarrow h^{(1)}}}\end{array}$的反向传播过程：

$$\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(1)}\Rightarrow h^{(1)}}}=({\mathcal{O}}^{(\mathcal{T})}-y^{(\mathcal{T})})\cdot {\mathcal{W}}_{h^{(T)}\Rightarrow {\mathcal{O}}^{(\mathcal{T})}}\cdot \left\{\prod _{k=1}^{\mathcal{T}}\text{Diag}\left[1-{\text{Tanh}}^2({\mathcal{Z}}_1^{(k)})\right]\right\}\cdot \left\{\prod _{k=2}^{\mathcal{T}}{\mathcal{W}}_{h^{(k-1)}\Rightarrow h^{(k)}}\right\}\cdot x^{(1)}$$
如果仅仅是描述$\mathcal{T}$时刻损失结果${\mathcal{L}}^{(\mathcal{T})}$对权重分量${\mathcal{W}}_{x^{(1)}\Rightarrow h^{(1)}}$的梯度信息(红色路径)，使用上述公式即可；但实际上，该网络层是一个循环过程，我们需要求解${\mathcal{L}}^{(\mathcal{T})}$对整个权重${\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{H}}$的梯度进行求解。

这意味着：每更新到一个时刻$t(t=1,2,\cdots ,\mathcal{T})$,都会将当前时刻${\mathcal{W}}_{x^{(t)}\Rightarrow h^{(t)}}$的梯度累加在${\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{H}}$的梯度中。因而关于${\mathcal{L}}^{(\mathcal{T})}$对${\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{H}}$的梯度$\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{H}}}\end{array}$表示为如下形式：
将上式$1\Rightarrow t$代入。
下面公式中最后一项的大括号不是矩阵，而是为方便表达，描述$\mathcal{T}$项累加和的过程。
$$\begin{array}{rl}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{H}}}& =\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T})}\Rightarrow h^{(\mathcal{T})}}}+\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow h^{(\mathcal{T}-1)}}}+\cdots +\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(1)}\Rightarrow h^{(1)}}}\\ & =\sum _{t=1}^{\mathcal{T}}\frac{\partial {\mathcal{L}}^{(t)}}{\partial {\mathcal{W}}_{x^{(t)}\Rightarrow h^{(t)}}}\\ & =\sum _{t=1}^{\mathcal{T}}\left[({\mathcal{O}}^{(\mathcal{T})}-y^{(\mathcal{T})})\cdot {\mathcal{W}}_{h^{(T)}\Rightarrow {\mathcal{O}}^{(\mathcal{T})}}\cdot \left\{\prod _{k=t}^{\mathcal{T}}\text{Diag}\left[1-{\text{Tanh}}^2({\mathcal{Z}}_1^{(k)})\right]\right\}\cdot \left\{\prod _{k=t+1}^{\mathcal{T}}{\mathcal{W}}_{h^{(k-1)}\Rightarrow h^{(k)}}\right\}\cdot x^{(t)}\right]\\ & =({\mathcal{O}}^{(\mathcal{T})}-y^{(\mathcal{T})})\cdot {\mathcal{W}}_{h^{(T)}\Rightarrow {\mathcal{O}}^{(\mathcal{T})}}\cdot \left\{\begin{array}{c}\text{Diag}\left[1-{\text{Tanh}}^2({\mathcal{Z}}_1^{(\mathcal{T})})\right]\cdot x^{(\mathcal{T})}\\ +{\prod }_{k=\mathcal{T}-1}^{\mathcal{T}}\text{Diag}\left[1-{\text{Tanh}}^2({\mathcal{Z}}_1^{(k)})\right]\cdot {\mathcal{W}}_{h^{(\mathcal{T}-1)}\Rightarrow h^{(\mathcal{T})}}\cdot x^{(\mathcal{T}-1)}\\ ⋮\\ +{\prod }_{k=1}^{\mathcal{T}}\text{Diag}\left[1-{\text{Tanh}}^2({\mathcal{Z}}_1^{(k)})\right]\cdot {\prod }_{k=2}^{\mathcal{T}}{\mathcal{W}}_{h^{(k-1)}\Rightarrow h^{(k)}}\cdot x^{(1)}\end{array}\right\}\end{array}$$
很明显，上述公式中大括号内共包含$\mathcal{T}$项的累加结果。可以通过观察发现：越后面的累加项，梯度消失的越厉害。也就是说：梯度$\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{H}}}\end{array}$的结果，主要贡献来源于其反向传播最开始的若干个时刻。而这种计算代价为$\mathcal{O}(\mathcal{T})$的反向传播算法也被称作通过时间反向传播$(\text{Back-Propagation Through Time,BPTT})$。

$\text{LSTM}$的反向传播过程

场景构建

关于$t$时刻$\text{LSTM}$的前馈计算过程表示如下：
这里$y^{(t)}$表示当前时刻的最终输出，后与对应时刻的损失函数结果${\mathcal{L}}^{(t)}$相衔接。
$$\begin{array}{rl}& f^{(t)}=\sigma \left[{\mathcal{W}}_{x^{(t)}\Rightarrow f^{(t)}}\cdot x^{(t)}+{\mathcal{W}}_{h^{(t-1)}\Rightarrow h^{(t)}}\cdot h^{(t-1)}+b_f\right]\\ & i^{(t)}=\sigma \left[{\mathcal{W}}_{x^{(t)}\Rightarrow i^{(t)}}\cdot x^{(t)}+{\mathcal{W}}_{h^{(t-1)}\Rightarrow i^{(t)}}\cdot h^{(t-1)}+b_i\right]\\ & {\mathcal{O}}^{(t)}=\sigma \left[{\mathcal{W}}_{x^{(t)}\Rightarrow {\mathcal{O}}^{(t)}}\cdot x^{(t)}+{\mathcal{W}}_{h^{(t-1)}\Rightarrow {\mathcal{O}}^{(t)}}\cdot h^{(t-1)}+b_{\mathcal{O}}\right]\\ & {\tilde{\mathcal{C}}}^{(t)}=\text{Tanh}\left[{\mathcal{W}}_{x^{(t)}\Rightarrow {\tilde{\mathcal{C}}}^{(t)}}\cdot x^{(t)}+{\mathcal{W}}_{h^{(t-1)}\Rightarrow {\tilde{\mathcal{C}}}^{(t)}}\cdot h^{(t-1)}+b_{\tilde{\mathcal{C}}}\right]\\ & {\mathcal{C}}^{(t)}={\mathcal{C}}^{(t-1)}\ast f^{(t)}+{\tilde{\mathcal{C}}}^{(t)}\ast i^{(t)}\\ & h^{(t)}={\mathcal{O}}^{(t)}\ast \text{Tanh}({\mathcal{C}}^{(t)})\\ & y^{(t)}={\mathcal{W}}_{h^{(t)}\Rightarrow y^{(t)}}\cdot h^{(t)}+b_y{\mathcal{W}}_{h^{(t)}\Rightarrow y^{(t)}}\Rightarrow {\mathcal{W}}_{\mathcal{H}\Rightarrow \mathcal{Y}}\end{array}$$
和上面的$\text{BPTT}$思路相同，上述公式中的权重参数如${\mathcal{W}}_{x^{(t)}\Rightarrow f^{(t)}},{\mathcal{W}}_{h^{(t-1)}\Rightarrow i^{(t)}}$等等，它们均是某时刻$t$关于输入变量$x^{(t)}$或者上一时刻隐变量$h^{(t-1)}$在各个门结构中的权重信息。在反向传播过程中：每一时刻的梯度均会存放在对应的权重参数中。我们将其对应设定为：
$$\begin{array}{rl}& {\mathcal{W}}_{\mathcal{X}\Rightarrow }:\{\begin{array}{l}{\mathcal{W}}_{x^{(t)}\Rightarrow f^{(t)}}\Rightarrow {\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{F}}{\mathcal{W}}_{x^{(t)}\Rightarrow i^{(t)}}\Rightarrow {\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{I}}\\ {\mathcal{W}}_{x^{(t)}\Rightarrow {\mathcal{O}}^{(t)}}\Rightarrow {\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{O}}{\mathcal{W}}_{x^{(t)}\Rightarrow {\tilde{\mathcal{C}}}^{(t)}}\Rightarrow {\mathcal{W}}_{\mathcal{X}\Rightarrow \tilde{\mathcal{C}}}\end{array}\\ & {\mathcal{W}}_{\mathcal{H}\Rightarrow }:\{\begin{array}{l}{\mathcal{W}}_{h^{(t-1)}\Rightarrow f^{(t)}}\Rightarrow {\mathcal{W}}_{\mathcal{H}\Rightarrow \mathcal{F}}{\mathcal{W}}_{h^{(t-1)}\Rightarrow i^{(t)}}\Rightarrow {\mathcal{W}}_{\mathcal{H}\Rightarrow \mathcal{I}}\\ {\mathcal{W}}_{h^{(t-1)}\Rightarrow {\mathcal{O}}^{(t)}}\Rightarrow {\mathcal{W}}_{\mathcal{H}\Rightarrow \mathcal{O}}{\mathcal{W}}_{h^{(t-1)}\Rightarrow {\tilde{\mathcal{C}}}^{(t)}}\Rightarrow {\mathcal{W}}_{\mathcal{H}\Rightarrow \tilde{\mathcal{C}}}\end{array}\end{array}$$

示例：求解梯度$\begin{array}{r}\frac{{\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{F}}}\end{array}$

假设序列长度$\mathcal{T}$，并且$\mathcal{T}$时刻输出的损失结果为${\mathcal{L}}^{(\mathcal{T})}$，我们想要求解${\mathcal{L}}^{(\mathcal{T})}$对权重矩阵${\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{F}}$的梯度结果$\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{F}}}\end{array}$：
和‘循环神经网络’逻辑相同，描述各时刻的梯度累加。
$$\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{F}}}=\sum _{t=1}^{\mathcal{T}}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(t)}\Rightarrow f^{(t)}}}$$

反向传播过程$\mathcal{T}$时刻梯度$\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T})}\Rightarrow f^{(\mathcal{T})}}}\end{array}$ 求解

这里先观察最后一个时刻的梯度$\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T})}\Rightarrow f^{(\mathcal{T})}}}\end{array}$：

• 它的梯度传播路径可表示为：
  $$\begin{array}{r}\{\begin{array}{ll}& {\tilde{f}}^{(\mathcal{T})}={\mathcal{W}}_{x^{(\mathcal{T})}\Rightarrow f^{(\mathcal{T})}}\cdot x^{(\mathcal{T})}+\underset{梯度无关}{\underbrace{{\mathcal{W}}_{h^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T})}}\cdot h^{(\mathcal{T}-1)}+b_f}}\\ & f^{(\mathcal{T})}=\text{Sigmoid}({\tilde{f}}^{(\mathcal{T})})\\ & {\mathcal{C}}^{(\mathcal{T})}={\mathcal{C}}^{(\mathcal{T}-1)}\ast f^{(\mathcal{T})}+\underset{梯度无关}{\underbrace{{\tilde{\mathcal{C}}}^{(\mathcal{T})}\ast i^{(\mathcal{T})}}}\\ & m^{(\mathcal{T})}=\text{Tanh}({\mathcal{C}}^{(\mathcal{T})})\\ & h^{(\mathcal{T})}={\mathcal{O}}^{(\mathcal{T})}\ast m^{(\mathcal{T})}\\ & y^{(\mathcal{T})}={\mathcal{W}}_{h^{(\mathcal{T})}\Rightarrow y^{(\mathcal{T})}}\cdot h^{(\mathcal{T})}+b_y\end{array}\end{array}$$
• 对应传播路径图像表示为(红色箭头路径)：
  
  因此，梯度$\begin{array}{r}\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T})}\Rightarrow f^{(\mathcal{T})}}}\end{array}\end{array}$可表示为：
  $$\begin{array}{rl}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T})}\Rightarrow f^{(\mathcal{T})}}}& =\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial y^{(\mathcal{T})}}\cdot \frac{\partial y^{(\mathcal{T})}}{\partial h^{(\mathcal{T})}}\cdot \frac{\partial h^{(\mathcal{T})}}{\partial m^{(\mathcal{T})}}\cdot \frac{\partial m^{(\mathcal{T})}}{\partial {\mathcal{C}}^{(\mathcal{T})}}\cdot \frac{\partial {\mathcal{C}}^{(\mathcal{T})}}{\partial f^{(\mathcal{T})}}\cdot \frac{\partial f^{(\mathcal{T})}}{\partial {\tilde{f}}^{(\mathcal{T})}}\cdot \frac{\partial {\tilde{f}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T})}\Rightarrow f^{(\mathcal{T})}}}\end{array}$$

反向传播过程$\mathcal{T}-1$时刻梯度$\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}}\end{array}$求解

至此，距离$\mathcal{T}$时刻最近的，关于${\mathcal{W}}_{\mathcal{X}\Rightarrow \mathcal{F}}$的的梯度信息已经求解出来。那么$\mathcal{T}-1$时刻呢？和$\mathcal{T}$时刻相比有什么区别呢$?$损失结果${\mathcal{L}}^{(\mathcal{T})}$关于$\mathcal{T}-1$时刻${\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}$的梯度结果$\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}}\end{array}$进行表示：
关于$W_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}$的梯度必然会经过$\mathcal{T}$时刻，它的反向传播过程包含几类路径：

• 从输出门将$h^{(\mathcal{T})}$直接反向传播至$h^{(\mathcal{T}-1)}$，再从$h^{(\mathcal{T}-1)}$反向传播至$W_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}$。它的梯度传播路径可表示为：
  省略号部分的与$\mathcal{T}$时刻相同，仅需将对应的$\mathcal{T}$改成$\mathcal{T}-1$即可，下面省略号同理。
  $$\{\begin{array}{ll}& y^{(\mathcal{T})}={\mathcal{W}}_{h^{(\mathcal{T})}\Rightarrow y^{(\mathcal{T})}}\cdot h^{(\mathcal{T})}+b_y\\ & h^{(\mathcal{T})}={\mathcal{O}}^{(\mathcal{T})}\ast m^{(\mathcal{T})}\\ & {\mathcal{O}}^{(\mathcal{T})}=\sigma \left[\underset{h^{(\mathcal{T}-1)}无关}{\underbrace{{\mathcal{W}}_{x^{(\mathcal{T})}\Rightarrow {\mathcal{O}}^{(\mathcal{T})}}\cdot x^{(\mathcal{T})}+b_{\mathcal{O}}}}+{\mathcal{W}}_{h^{(\mathcal{T}-1)}\Rightarrow {\mathcal{O}}^{(\mathcal{T})}}\cdot h^{(\mathcal{T}-1)}\right]\\ & h^{(\mathcal{T}-1)}={\mathcal{O}}^{(\mathcal{T}-1)}\ast m^{(\mathcal{T}-1)}\\ & ⋮\end{array}$$
  因此，该路径的梯度可表示为：
  $$\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial y^{(\mathcal{T})}}\cdot \frac{\partial y^{(\mathcal{T})}}{\partial h^{(\mathcal{T})}}\cdot \left[\frac{\partial h^{(\mathcal{T})}}{\partial {\mathcal{O}}^{(\mathcal{T})}}\cdot \frac{\partial {\mathcal{O}}^{(\mathcal{T})}}{\partial h^{(\mathcal{T}-1)}}\cdot \frac{\partial h^{(\mathcal{T}-1)}}{\partial m^{(\mathcal{T}-1)}}\cdot \frac{\partial m^{(\mathcal{T}-1)}}{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}\cdot \frac{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}{\partial f^{(\mathcal{T}-1)}}\cdot \frac{\partial f^{(\mathcal{T}-1)}}{\partial {\tilde{f}}^{(\mathcal{T}-1)}}\cdot \frac{\partial {\tilde{f}}^{(\mathcal{T}-1)}}{\partial {\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}}\right]$$
• 从细胞状态${\mathcal{C}}^{(\mathcal{T})}$角度反向传播至${\mathcal{C}}^{(\mathcal{T}-1)}$，再从${\mathcal{C}}^{(\mathcal{T}-1)}$反向传播至$f^{(\mathcal{T}-1)}$，直至$W_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}$。它的梯度传播路径可表示为：
  $$\begin{array}{r}\{\begin{array}{ll}& y^{(\mathcal{T})}={\mathcal{W}}_{h^{(\mathcal{T})}\Rightarrow y^{(\mathcal{T})}}\cdot h^{(\mathcal{T})}+b_y\\ & h^{(\mathcal{T})}={\mathcal{O}}^{(\mathcal{T})}\ast m^{(\mathcal{T})}\\ & m^{(\mathcal{T})}=\text{Tanh}({\mathcal{C}}^{(\mathcal{T})})\\ & {\mathcal{C}}^{(\mathcal{T})}={\mathcal{C}}^{(\mathcal{T}-1)}\ast f^{(\mathcal{T})}+{\tilde{\mathcal{C}}}^{(\mathcal{T})}\ast i^{(\mathcal{T})}\\ & {\mathcal{C}}^{(\mathcal{T}-1)}={\mathcal{C}}^{(\mathcal{T}-2)}\ast f^{(\mathcal{T}-1)}+{\tilde{\mathcal{C}}}^{(\mathcal{T}-1)}\ast i^{(\mathcal{T}-1)}\\ & f^{(\mathcal{T}-1)}=\text{Sigmoid}({\tilde{f}}^{(\mathcal{T}-1)})\\ & {\tilde{f}}^{(\mathcal{T}-1)}={\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}\cdot x^{(\mathcal{T}-1)}+\underset{梯度无关}{\underbrace{{\mathcal{W}}_{h^{(\mathcal{T}-2)}\Rightarrow f^{(\mathcal{T}-1)}}\cdot h^{(\mathcal{T}-2)}+b_f}}\end{array}\end{array}$$
  因此，该路径的梯度可表示为：
  $$\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial y^{(\mathcal{T})}}\cdot \frac{\partial y^{(\mathcal{T})}}{\partial h^{(\mathcal{T})}}\cdot \left[\frac{\partial h^{(\mathcal{T})}}{\partial m^{\mathcal{T}}}\cdot \frac{\partial m^{(\mathcal{T})}}{\partial {\mathcal{C}}^{(\mathcal{T})}}\cdot \frac{\partial {\mathcal{C}}^{(\mathcal{T})}}{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}\cdot \frac{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}{\partial f^{(\mathcal{T}-1)}}\cdot \frac{\partial f^{(\mathcal{T}-1)}}{\partial {\tilde{f}}^{(\mathcal{T}-1)}}\cdot \frac{\partial {\tilde{f}}^{(\mathcal{T}-1)}}{\partial {\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}}\right]$$
• 在细胞状态${\mathcal{C}}^{(\mathcal{T})}$的基础上，通过遗忘门反向传播至$h^{(\mathcal{T}-1)}$，后续与第一种情况相同。它的传播路径可表示为：
  $$\begin{array}{r}\{\begin{array}{ll}& y^{(\mathcal{T})}={\mathcal{W}}_{h^{(\mathcal{T})}\Rightarrow y^{(\mathcal{T})}}\cdot h^{(\mathcal{T})}+b_y\\ & h^{(\mathcal{T})}={\mathcal{O}}^{(\mathcal{T})}\ast m^{(\mathcal{T})}\\ & m^{(\mathcal{T})}=\text{Tanh}({\mathcal{C}}^{(\mathcal{T})})\\ & {\mathcal{C}}^{(\mathcal{T})}={\mathcal{C}}^{(\mathcal{T}-1)}\ast f^{(\mathcal{T})}+{\tilde{\mathcal{C}}}^{(\mathcal{T})}\ast i^{(\mathcal{T})}\\ & f^{(\mathcal{T})}=\text{Sigmoid}({\tilde{f}}^{(\mathcal{T})})\\ & {\tilde{f}}^{(\mathcal{T})}={\mathcal{W}}_{x^{(\mathcal{T})}\Rightarrow f^{(\mathcal{T})}}\cdot x^{(\mathcal{T})}+{\mathcal{W}}_{h^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T})}}\cdot h^{(\mathcal{T}-1)}+b_f\\ & h^{(\mathcal{T}-1)}={\mathcal{O}}^{(\mathcal{T}-1)}\ast m^{(\mathcal{T}-1)}\\ & ⋮\end{array}\end{array}$$
  因此，该路径的梯度可表示为：
  从$h^{(\mathcal{T}-1)}$到${\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}$的梯度路径是固定的，见情况$1$(后$5$个梯度),这里使用省略号表示，下同。
  $$\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial y^{(\mathcal{T})}}\cdot \frac{\partial y^{(\mathcal{T})}}{\partial h^{(\mathcal{T})}}\cdot \left[\frac{\partial h^{(\mathcal{T})}}{\partial m^{\mathcal{T}}}\cdot \frac{\partial m^{(\mathcal{T})}}{\partial {\mathcal{C}}^{(\mathcal{T})}}\cdot \frac{\partial {\mathcal{C}}^{(\mathcal{T})}}{\partial f^{(\mathcal{T})}}\cdot \frac{\partial f^{(\mathcal{T})}}{\partial {\tilde{f}}^{(\mathcal{T})}}\cdot \frac{\partial {\tilde{f}}^{(\mathcal{T})}}{\partial h^{(\mathcal{T}-1)}}\cdots \right]$$
• 在细胞状态${\mathcal{C}}^{(\mathcal{T})}$的基础上，通过输入门反向传播至$h^{(\mathcal{T}-1)}$，后续与第一种情况相同。它的传播路径可表示为：
  新出现的符号：${\tilde{i}}^{(\mathcal{T})}$表示输入门的线性计算过程。
  $$\begin{array}{r}\{\begin{array}{ll}& y^{(\mathcal{T})}={\mathcal{W}}_{h^{(\mathcal{T})}\Rightarrow y^{(\mathcal{T})}}\cdot h^{(\mathcal{T})}+b_y\\ & h^{(\mathcal{T})}={\mathcal{O}}^{(\mathcal{T})}\ast m^{(\mathcal{T})}\\ & m^{(\mathcal{T})}=\text{Tanh}({\mathcal{C}}^{(\mathcal{T})})\\ & {\mathcal{C}}^{(\mathcal{T})}={\mathcal{C}}^{(\mathcal{T}-1)}\ast f^{(\mathcal{T})}+{\tilde{\mathcal{C}}}^{(\mathcal{T})}\ast i^{(\mathcal{T})}\\ & i^{(\mathcal{T})}=\text{Sigmoid}({\tilde{i}}^{(\mathcal{T})})\\ & {\tilde{i}}^{(\mathcal{T})}={\mathcal{W}}_{x^{(\mathcal{T})}\Rightarrow i^{(\mathcal{T})}}\cdot x^{(\mathcal{T})}+{\mathcal{W}}_{h^{(\mathcal{T}-1)}\Rightarrow i^{(\mathcal{T})}}\cdot h^{(\mathcal{T}-1)}+b_i\\ & h^{(\mathcal{T}-1)}={\mathcal{O}}^{(\mathcal{T}-1)}\ast m^{(\mathcal{T}-1)}\\ & ⋮\end{array}\end{array}$$
  对应路径梯度可表示为：
  $$\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial y^{(\mathcal{T})}}\cdot \frac{\partial y^{(\mathcal{T})}}{\partial h^{(\mathcal{T})}}\cdot \left[\frac{\partial h^{(\mathcal{T})}}{\partial m^{\mathcal{T}}}\cdot \frac{\partial m^{(\mathcal{T})}}{\partial {\mathcal{C}}^{(\mathcal{T})}}\cdot \frac{\partial {\mathcal{C}}^{(\mathcal{T})}}{\partial i^{(\mathcal{T})}}\cdot \frac{\partial i^{(\mathcal{T})}}{\partial {\tilde{i}}^{(\mathcal{T})}}\cdot \frac{\partial {\tilde{i}}^{(\mathcal{T})}}{\partial h^{(\mathcal{T}-1)}}\cdots \right]$$
• 在细胞状态${\mathcal{C}}^{(\mathcal{T})}$的基础上，通过候选状态${\tilde{\mathcal{C}}}^{(\mathcal{T})}$反向传播至$h^{(\mathcal{T}-1)}$，后续与第一种情况相同。它的传播路径可表示为：
  $$\begin{array}{r}\{\begin{array}{ll}& y^{(\mathcal{T})}={\mathcal{W}}_{h^{(\mathcal{T})}\Rightarrow y^{(\mathcal{T})}}\cdot h^{(\mathcal{T})}+b_y\\ & h^{(\mathcal{T})}={\mathcal{O}}^{(\mathcal{T})}\ast m^{(\mathcal{T})}\\ & m^{(\mathcal{T})}=\text{Tanh}({\mathcal{C}}^{(\mathcal{T})})\\ & {\mathcal{C}}^{(\mathcal{T})}={\mathcal{C}}^{(\mathcal{T}-1)}\ast f^{(\mathcal{T})}+{\tilde{\mathcal{C}}}^{(\mathcal{T})}\ast i^{(\mathcal{T})}\\ & {\tilde{\mathcal{C}}}^{(\mathcal{T})}=\text{Tanh}\left[{\mathcal{W}}_{x^{(\mathcal{T})}\Rightarrow {\tilde{\mathcal{C}}}^{(\mathcal{T})}}\cdot x^{(\mathcal{T})}+{\mathcal{W}}_{h^{(\mathcal{T}-1)}\Rightarrow {\tilde{\mathcal{C}}}^{(\mathcal{T})}}\cdot h^{(\mathcal{T}-1)}+b_{\tilde{\mathcal{C}}}\right]\\ & h^{(\mathcal{T}-1)}={\mathcal{O}}^{(\mathcal{T}-1)}\ast m^{(\mathcal{T}-1)}\\ & ⋮\end{array}\end{array}$$
  对应路径的梯度可表示为：
  $$\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial y^{(\mathcal{T})}}\cdot \frac{\partial y^{(\mathcal{T})}}{\partial h^{(\mathcal{T})}}\cdot \left[\frac{\partial h^{(\mathcal{T})}}{\partial m^{\mathcal{T}}}\cdot \frac{\partial m^{(\mathcal{T})}}{\partial {\mathcal{C}}^{(\mathcal{T})}}\cdot \frac{\partial {\mathcal{C}}^{(\mathcal{T})}}{\partial {\tilde{\mathcal{C}}}^{(\mathcal{T})}}\cdot \frac{\partial {\tilde{\mathcal{C}}}^{(\mathcal{T})}}{\partial h^{(\mathcal{T}-1)}}\cdots \right]$$

至此，我们将所有关于$\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}}\end{array}$的全部梯度路径查找完毕。将这些梯度结果进行累加：
其中大括号内的所有部分同上，仅表示各项的累加结果，并非矩阵;其中有$4$条路径是从细胞状态${\mathcal{C}}^{(\mathcal{T})}$得到。将其进行简写。
$$\begin{array}{rl}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}}& =\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial y^{(\mathcal{T})}}\cdot \frac{\partial y^{(\mathcal{T})}}{\partial h^{(\mathcal{T})}}\cdot \left\{\begin{array}{c}\frac{\partial h^{(\mathcal{T})}}{\partial {\mathcal{O}}^{(\mathcal{T})}}\cdot \frac{\partial {\mathcal{O}}^{(\mathcal{T})}}{\partial h^{(\mathcal{T}-1)}}\cdot \underset{\cdots }{\underbrace{\frac{\partial h^{(\mathcal{T}-1)}}{\partial m^{(\mathcal{T}-1)}}\cdot \frac{\partial m^{(\mathcal{T}-1)}}{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}\cdot \frac{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}{\partial f^{(\mathcal{T}-1)}}\cdot \frac{\partial f^{(\mathcal{T}-1)}}{\partial {\tilde{f}}^{(\mathcal{T}-1)}}\cdot \frac{\partial {\tilde{f}}^{(\mathcal{T}-1)}}{\partial {\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}}}}\\ \\ +\frac{\partial h^{(\mathcal{T})}}{\partial m^{\mathcal{T}}}\cdot \frac{\partial m^{(\mathcal{T})}}{\partial {\mathcal{C}}^{(\mathcal{T})}}\cdot \left[\begin{array}{c}\frac{\partial {\mathcal{C}}^{(\mathcal{T})}}{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}\cdot \frac{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}{\partial f^{(\mathcal{T}-1)}}\cdot \frac{\partial f^{(\mathcal{T}-1)}}{\partial {\tilde{f}}^{(\mathcal{T}-1)}}\cdot \frac{\partial {\tilde{f}}^{(\mathcal{T}-1)}}{\partial {\mathcal{W}}_{x^{(\mathcal{T}-1)}\Rightarrow f^{(\mathcal{T}-1)}}}\\ +\frac{\partial {\mathcal{C}}^{(\mathcal{T})}}{\partial f^{(\mathcal{T})}}\cdot \frac{\partial f^{(\mathcal{T})}}{\partial {\tilde{f}}^{(\mathcal{T})}}\cdot \frac{\partial {\tilde{f}}^{(\mathcal{T})}}{\partial h^{(\mathcal{T}-1)}}\cdots \\ +\frac{\partial {\mathcal{C}}^{(\mathcal{T})}}{\partial i^{(\mathcal{T})}}\cdot \frac{\partial i^{(\mathcal{T})}}{\partial {\tilde{i}}^{(\mathcal{T})}}\cdot \frac{\partial {\tilde{i}}^{(\mathcal{T})}}{\partial h^{(\mathcal{T}-1)}}\cdots \\ +\frac{\partial {\mathcal{C}}^{(\mathcal{T})}}{\partial {\tilde{\mathcal{C}}}^{(\mathcal{T})}}\cdot \frac{\partial {\tilde{\mathcal{C}}}^{(\mathcal{T})}}{\partial h^{(\mathcal{T}-1)}}\cdots \end{array}\right]\end{array}\right\}\end{array}$$

$\mathcal{T}-2$时刻与$\mathcal{T}-1$时刻关于${\mathcal{W}}_{x^{(t)}\Rightarrow f^{(t)}}$梯度的比较

$\mathcal{T}-2$时刻的梯度结果$\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial {\mathcal{W}}_{x^{(\mathcal{T}-2)}\Rightarrow f^{(\mathcal{T}-2)}}}\end{array}$是否与$\mathcal{T}-1$时刻的情况相同呢？不相同。原因在于：$\mathcal{T}\Rightarrow \mathcal{T}-1$时刻仅包含$h^{(\mathcal{T})}$的相关路径，也就是说，它均是从$\begin{array}{r}\frac{\partial {\mathcal{L}}^{(\mathcal{T})}}{\partial y^{(\mathcal{T})}}\cdot \frac{\partial y^{(\mathcal{T})}}{\partial h^{(\mathcal{T})}}\end{array}$执行下来的。但是：$\mathcal{T}-1\Rightarrow \mathcal{T}-2$时刻不仅存在$h^{(\mathcal{T}-1)}$的相关路径，并且还包含${\mathcal{C}}^{(\mathcal{T}-1)}$的相关路径：
其中关于$h^{(\mathcal{T}-1)}$的相关路径与$h^{(\mathcal{T})}$相同，不再赘述;与${\mathcal{C}}^{(\mathcal{T}-1)}$的相关路径存在如下几种形式。可以看出它们之间确实存在重合的部分，但需要分开进行梯度计算。因为${\mathcal{C}}^{(\mathcal{T}-1)}$和$h^{(\mathcal{T}-1)}$不是一个东西。
$${\mathcal{C}}^{(\mathcal{T}-1)}\Rightarrow \{\begin{array}{l}\begin{array}{rl}& \frac{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}{\partial {\mathcal{C}}^{(\mathcal{T}-2)}}\cdot \frac{\partial {\mathcal{C}}^{(\mathcal{T}-2)}}{\partial f^{(\mathcal{T}-2)}}\cdot \frac{\partial f^{(\mathcal{T}-2)}}{\partial {\tilde{f}}^{(\mathcal{T}-2)}}\cdot \frac{\partial {\tilde{f}}^{(\mathcal{T}-2)}}{\partial {\mathcal{W}}_{x^{(\mathcal{T}-2)}\Rightarrow f^{(\mathcal{T}-2)}}}\\ & \frac{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}{\partial f^{(\mathcal{T}-1)}}\cdot \frac{\partial f^{(\mathcal{T}-1)}}{\partial {\tilde{f}}^{(\mathcal{T}-1)}}\cdot \frac{\partial {\tilde{f}}^{(\mathcal{T}-1)}}{\partial h^{(\mathcal{T}-2)}}\cdots \\ & \frac{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}{\partial i^{(\mathcal{T}-1)}}\cdot \frac{\partial i^{(\mathcal{T}-1)}}{\partial {\tilde{i}}^{(\mathcal{T}-1)}}\cdot \frac{\partial {\tilde{i}}^{(\mathcal{T}-1)}}{\partial h^{(\mathcal{T}-2)}}\cdots \\ & \frac{\partial {\mathcal{C}}^{(\mathcal{T}-1)}}{\partial {\tilde{\mathcal{C}}}^{(\mathcal{T}-1)}}\cdot \frac{\partial {\tilde{\mathcal{C}}}^{(\mathcal{T}-1)}}{\partial h^{(\mathcal{T}-2)}}\cdots \end{array}\end{array}$$
最终，$\mathcal{T}\Rightarrow \mathcal{T}-2$一共包含$4\times 5+4=24$条路经。
这个$+4$是指输出门路径，因为该路径没有经过‘细胞状态’${\mathcal{C}}^{(t)}$,因此每一次达到$h^{(\mathcal{T})},h^{(\mathcal{T}-1)}$时，它仅存在唯一一条路径向对应的$h^{(\mathcal{T}-1)},h^{(\mathcal{T}-2)}$传播。

为什么$\text{LSTM}$能够抑制梯度消失

随着反向传播深度的增加，反向传播路径的数量呈指数级别增长。即便可能出现梯度消失，也可以从数量的角度进行补充；
例如：关于细胞状态的梯度$\begin{array}{r}\frac{\partial {\mathcal{C}}^{(t)}}{\partial {\mathcal{C}}^{(t-1)}}\end{array}$可以通过各门结构权重参数进行调节：
$$\begin{array}{rl}\frac{\partial {\mathcal{C}}^{(t)}}{\partial {\mathcal{C}}^{(t-1)}}& =f^{(t)}+\left(\begin{array}{c}\begin{array}{rl}& \frac{\partial {\mathcal{C}}^{(t)}}{\partial f^{(t)}}\cdot \frac{\partial f^{(t)}}{\partial h^{(t-1)}}\cdot \frac{\partial h^{(t-1)}}{\partial {\mathcal{C}}^{(t-1)}}\\ & +\frac{\partial {\mathcal{C}}^{(t)}}{\partial i^{(t)}}\cdot \frac{\partial i^{(t)}}{\partial h^{(t-1)}}\cdot \frac{\partial h^{(t-1)}}{\partial {\mathcal{C}}^{(t-1)}}\\ & +\frac{\partial {\mathcal{C}}^{(t)}}{\partial {\tilde{\mathcal{C}}}^{(t)}}\cdot \frac{\partial {\tilde{\mathcal{C}}}^{(t)}}{\partial h^{(t-1)}}\cdot \frac{\partial h^{(t-1)}}{\partial {\mathcal{C}}^{(t-1)}}\end{array}\end{array}\right)\\ & =f^{(t)}+\left(\begin{array}{c}{\mathcal{C}}^{(t-1)}\cdot \left\{{\left[\text{Sigmoid}(\cdot )\right]}^{\prime }\cdot {\mathcal{W}}_{h^{(t-1)}\Rightarrow f^{(t)}}\right\}\cdot \left\{{\mathcal{O}}^{(t-1)}\ast {\left[\text{Tanh}({\mathcal{C}}^{(t-1)})\right]}^{\prime }\right\}\\ +{\tilde{\mathcal{C}}}^{(t)}\cdot \left\{{\left[\text{Sigmoid}(\cdot )\right]}^{\prime }\cdot {\mathcal{W}}_{h^{(t-1)}\Rightarrow i^{(t)}}\right\}\cdot \left\{{\mathcal{O}}^{(t-1)}\ast {\left[\text{Tanh}({\mathcal{C}}^{(t-1)})\right]}^{\prime }\right\}\\ +i^{(t)}\cdot \left\{{\left[\text{Tanh}(\cdot )\right]}^{\prime }\cdot {\mathcal{W}}_{h^{(t-1)}\Rightarrow {\tilde{\mathcal{C}}}^{(t)}}\right\}\cdot \left\{{\mathcal{O}}^{(t-1)}\ast {\left[\text{Tanh}({\mathcal{C}}^{(t-1)})\right]}^{\prime }\right\}\end{array}\right)\end{array}$$
可以发现：每向前反向传播一个梯度，都回出现$4$项偏导伴随着该时刻梯度的出现，并且其中三项是由当前时刻遗忘门、输入门、输出门的权重参数相互调节决定的。

可以理解为：

• 整个反向传播过程中，所有时刻门结构的权重均参与到了$\begin{array}{r}\frac{\partial {\mathcal{C}}^{(t)}}{\partial {\mathcal{C}}^{(t-1)}}\end{array}$的调节中，相比于循环神经网络中仅有一个权重矩阵的描述，它的鲁棒性会强很多；
• 并且循环神经网络中的权重矩阵是纯纯的累积，而$\text{LSTM}$是各项累加，即便是其中一个时刻某门结构梯度消失，剩余门结构也会做出相应调整，来维持当前时刻梯度。

相关参考：
LSTM如何缓解梯度消失(公式推导)