矩阵的导数运算(理解分子布局、分母布局)

1、分子布局和分母布局

请思考这样一个问题，一个维度为m的向量y对一个标量x的求导，那么结果也是一个m维的向量，那么这个结果向量是行向量，还是列向量呢？

答案是：行向量或者列向量皆可！ 求导的本质只是把标量求导的结果排列起来，至于是按行排列还是按列排列都是可以的。但是这样也有问题，在我们机器学习算法优化过程中，如果行向量或者列向量随便写，那么结果就不唯一，乱套了。

为了解决矩阵向量求导的结果不唯一，我们引入求导布局。最基本的求导布局有两个：分子布局(numerator layout)和分母布局(denominator layout )。

• 对于分子布局来说，我们求导结果的维度以分子为主。

• 对于分母布局来说，我们求导结果的维度以分母为主。

2、标量方程对向量的导数

标量方程中的未知量是标量，而不是矢量或矩阵。

通常情况下，标量方程可以是各种类型的代数方程，包括线性方程、二次方程、多项式方程等。这些方程中的未知量都是标量，通常表示为一个变量，例如 x、y、z 等。
$$已知标量方程f(y)=f(y_1,y_2,...,y_m)，我们求解标量方程f(y)对向量\overrightarrow{y}=\left(\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right)的导数$$

分母为向量y，维度为m×1，求导结果的行数和分母相同，都为m，因此为分母布局。

分子为标量，维度为1×1，求导结果的行数和分子相同，都为1，因此为分子布局。

具体案例如下：
$$\begin{gathered} 已知标量方程f(y)=y_1^2+y_2^2，我们求解标量方程f(y)对向量\overrightarrow{y}=\left(\begin{array}{c}{y}_1\\ y_2\end{array}\right)的导数 \\ 按照分母布局(行数和分母相同)，则\frac{\partial f(\overrightarrow{y})}{\partial \overrightarrow{y}}=\left(\begin{array}{c}\frac{\partial f(\overrightarrow{y})}{\partial y_1}\\ \frac{\partial f(\overrightarrow{y})}{\partial y_2}\end{array}\right)=\left(\begin{array}{c}2y_1\\ 2y_2\end{array}\right) \\ 按照分子布局(行数和分子相同)，则\frac{\partial f(\overrightarrow{y})}{\partial \overrightarrow{y}}=(\frac{\partial f(\overrightarrow{y})}{\partial y_1},\frac{\partial f(\overrightarrow{y})}{\partial y_2})=(2y_1,2y_2) \end{gathered}$$
注意：分子布局结果和分母布局结果互为转置。

3、向量方程对向量的导数

3.1 公式

$$已知\overrightarrow{y}=\left(\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right)，求向量方程\overrightarrow{f}(\overrightarrow{y})=\left(\begin{array}{c}{f}_1(\overrightarrow{y})\\ f_2(\overrightarrow{y})\\ ⋮\\ f_n(\overrightarrow{y})\end{array}\right)对\overrightarrow{y}的导数$$

利用分母布局：
$$\frac{\partial \overrightarrow{f}(\overrightarrow{y})}{\partial \overrightarrow{y}}=\left(\begin{array}{c}\frac{\partial f(\overrightarrow{y})}{\partial y_1}\\ \frac{\partial f(\overrightarrow{y})}{\partial y_2}\\ ⋮\\ \frac{\partial f(\overrightarrow{y})}{\partial y_m}\end{array}\right)=\left(\begin{array}{cccc}\frac{\partial f_1(\overrightarrow{y})}{\partial y_1}& \frac{\partial f_2(\overrightarrow{y})}{\partial y_1}& \cdots & \frac{\partial f_n(\overrightarrow{y})}{\partial y_1}\\ \frac{\partial f_1(\overrightarrow{y})}{\partial y_2}& \frac{\partial f_2(\overrightarrow{y})}{\partial y_2}& \cdots & \frac{\partial f_n(\overrightarrow{y})}{\partial y_2}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial f_1(\overrightarrow{y})}{\partial y_m}& \frac{\partial f_2(\overrightarrow{y})}{\partial y_m}& \cdots & \frac{\partial f_n(\overrightarrow{y})}{\partial y_m}\end{array}\right)$$
利用分子布局：

$$\frac{\partial \overrightarrow{f}(\overrightarrow{y})}{\partial \overrightarrow{y}}=\left(\begin{array}{c}\frac{\partial f_1(\overrightarrow{y})}{\partial \overrightarrow{y}}\\ \frac{\partial f_2(\overrightarrow{y})}{\partial \overrightarrow{y}}\\ ⋮\\ \frac{\partial f_n(\overrightarrow{y})}{\partial \overrightarrow{y}}\end{array}\right)=\left(\begin{array}{cccc}\frac{\partial f_1(\overrightarrow{y})}{\partial y_1}& \frac{\partial f_1(\overrightarrow{y})}{\partial y_2}& \cdots & \frac{\partial f_1(\overrightarrow{y})}{\partial y_m}\\ \frac{\partial f_2(\overrightarrow{y})}{\partial y_1}& \frac{\partial f_2(\overrightarrow{y})}{\partial y_2}& \cdots & \frac{\partial f_2(\overrightarrow{y})}{\partial y_m}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{\partial f_n(\overrightarrow{y})}{\partial y_1}& \frac{\partial f_n(\overrightarrow{y})}{\partial y_2}& \cdots & \frac{\partial f_n(\overrightarrow{y})}{\partial y_m}\end{array}\right)$$

3.2 具体示例

$$已知\overrightarrow{y}=\left(\begin{array}{c}{y}_1\\ y_2\\ y_3\end{array}\right)，求向量方程\overrightarrow{f}(\overrightarrow{y})=\left(\begin{array}{c}{f}_1(\overrightarrow{y})\\ f_2(\overrightarrow{y})\end{array}\right)=\left(\begin{array}{c}{y}_1^2+y_2^2+y_3\\ y_3^2+2y_1\end{array}\right)对\overrightarrow{y}的导数$$

我们按照分母布局来求(得到结果为m×n的矩阵，即3×2)：
$$\frac{\partial \overrightarrow{f}(\overrightarrow{y})}{\partial \overrightarrow{y}}=\left(\begin{array}{c}\frac{\partial f(\overrightarrow{y})}{\partial y_1}\\ \frac{\partial f(\overrightarrow{y})}{\partial y_2}\\ \frac{\partial f(\overrightarrow{y})}{\partial y_3}\end{array}\right)=\left(\begin{array}{ccc}\frac{\partial f_1(\overrightarrow{y})}{\partial y_1}& \frac{\partial f_2(\overrightarrow{y})}{\partial y_1}& \\ \frac{\partial f_1(\overrightarrow{y})}{\partial y_2}& \frac{\partial f_2(\overrightarrow{y})}{\partial y_2}& \\ \frac{\partial f_1(\overrightarrow{y})}{\partial y_3}& \frac{\partial f_2(\overrightarrow{y})}{\partial y_3}& \end{array}\right)=\left(\begin{array}{ccc}2y_1& 2& \\ 2y_2& 0& \\ 1& 2y_3& \end{array}\right)$$

3.3 常用特例

常用特例1：
$$已知\overrightarrow{y}=\left(\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right)，方阵A=\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1m}\\ a_{21}& a_{22}& \cdots & a_{2m}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mm}\end{array}\right),证明\frac{\partial A\overrightarrow{y}}{\partial \overrightarrow{y}}=A^T，\frac{\partial \overrightarrow{y^T}A}{\partial \overrightarrow{y}}=A(分母布局)$$
我们使用分母布局来求
$$\begin{gathered} A\overrightarrow{y}=\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1m}\\ a_{21}& a_{22}& \cdots & a_{2m}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mm}\end{array}\right).\left(\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right)=\left(\begin{array}{c}{a}_{11}{y}_1+a_{12}{y}_2+\cdots +a_{1m}{y}_m\\ a_{21}{y}_1+a_{22}{y}_2+\cdots +a_{2m}{y}_m\\ ⋮\\ a_{m1}{y}_1+a_{m2}{y}_2+\cdots +a_{mm}{y}_m\end{array}\right) \\ 按照分母布局，我们可以得到： \\ \frac{\partial A\overrightarrow{y}}{\partial \overrightarrow{y}}=\left(\begin{array}{c}\frac{\partial A\overrightarrow{y}}{\partial y_1}\\ \frac{\partial A\overrightarrow{y}}{\partial y_2}\\ ⋮\\ \frac{\partial A\overrightarrow{y}}{\partial y_m}\end{array}\right)=\left(\begin{array}{cccc}\frac{a_{11}{y}_1+a_{12}{y}_2+\cdots +a_{1m}{y}_m}{\partial y_1}& \frac{a_{21}{y}_1+a_{22}{y}_2+\cdots +a_{2m}{y}_m}{\partial y_1}& \cdots & \frac{a_{m1}{y}_1+a_{m2}{y}_2+\cdots +a_{mm}{y}_m}{\partial y_1}\\ \frac{a_{11}{y}_1+a_{12}{y}_2+\cdots +a_{1m}{y}_m}{\partial y_2}& \frac{a_{21}{y}_1+a_{22}{y}_2+\cdots +a_{2m}{y}_m}{\partial y_2}& \cdots & \frac{a_{m1}{y}_1+a_{m2}{y}_2+\cdots +a_{mm}{y}_m}{\partial y_2}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{a_{11}{y}_1+a_{12}{y}_2+\cdots +a_{1m}{y}_m}{\partial y_m}& \frac{a_{21}{y}_1+a_{22}{y}_2+\cdots +a_{2m}{y}_m}{\partial y_m}& \cdots & \frac{a_{m1}{y}_1+a_{m2}{y}_2+\cdots +a_{mm}{y}_m}{\partial y_m}\end{array}\right) \\ =\left(\begin{array}{cccc}{a}_{11}& a_{21}& \cdots & a_{m1}\\ a_{12}& a_{22}& \cdots & a_{m2}\\ ⋮& ⋮& \ddots & ⋮\\ a_{1m}& a_{2m}& \cdots & a_{mm}\end{array}\right)=A^T \end{gathered}$$

$$\begin{gathered} 同理，我们知道\overrightarrow{y^T}A=(y_1,y_2,\cdots ,y_m).\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1m}\\ a_{21}& a_{22}& \cdots & a_{2m}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mm}\end{array}\right)=\left(\begin{array}{c}{a}_{11}{y}_1+a_{21}{y}_2+\cdots +a_{m1}{y}_m\\ a_{12}{y}_1+a_{22}{y}_2+\cdots +a_{m2}{y}_m\\ ⋮\\ a_{1m}{y}_1+a_{2m}{y}_2+\cdots +a_{mm}{y}_m\end{array}\right) \\ \frac{\partial \overrightarrow{y^T}A}{\partial \overrightarrow{y}}=\left(\begin{array}{cccc}\frac{a_{11}{y}_1+a_{21}{y}_2+\cdots +a_{m1}{y}_m}{\partial y_1}& \frac{a_{12}{y}_1+a_{22}{y}_2+\cdots +a_{m2}{y}_m}{\partial y_1}& \cdots & \frac{a_{1m}{y}_1+a_{2m}{y}_2+\cdots +a_{mm}{y}_m}{\partial y_1}\\ \frac{a_{11}{y}_1+a_{12}{y}_2+\cdots +a_{m1}{y}_m}{\partial y_2}& \frac{a_{12}{y}_1+a_{22}{y}_2+\cdots +a_{m2}{y}_m}{\partial y_2}& \cdots & \frac{a_{1m}{y}_1+a_{2m}{y}_2+\cdots +a_{mm}{y}_m}{\partial y_2}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{a_{11}{y}_1+a_{12}{y}_2+\cdots +a_{m1}{y}_m}{\partial y_m}& \frac{a_{12}{y}_1+a_{22}{y}_2+\cdots +a_{m2}{y}_m}{\partial y_m}& \cdots & \frac{a_{1m}{y}_1+a_{2m}{y}_2+\cdots +a_{mm}{y}_m}{\partial y_m}\end{array}\right) \\ =\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1m}\\ a_{21}& a_{22}& \cdots & a_{2m}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mm}\end{array}\right)=A \end{gathered}$$

常用特例2：
$$\begin{gathered} 已知\overrightarrow{y}=\left(\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right)，方阵A=\left(\begin{array}{cccc}{a}_{11}& a_{12}& \cdots & a_{1m}\\ a_{21}& a_{22}& \cdots & a_{2m}\\ ⋮& ⋮& \ddots & ⋮\\ a_{m1}& a_{m2}& \cdots & a_{mm}\end{array}\right),证明\frac{\partial {\overrightarrow{y}}^{T}A\overrightarrow{y}}{\partial \overrightarrow{y}}=A\overrightarrow{y}+A^T\overrightarrow{y}(分母布局) \\ 另外，当A对称时，A^T=A,左式=2A\overrightarrow{y} \end{gathered}$$
我们A为2阶方阵，那么：

我们再利用分母布局：

3.4 利用常用特例求解线性回归的解析解

$$\begin{gathered} 线性回归可以用y=Xw+b进行表示 \\ 我们将偏置b合并到参数w中，合并⽅法是在包含所有参数的矩阵中附加⼀列 \\ 那么，线性回归的代价函数可以表示为： \\ {E}_w=(y-Xw{)}^T(y-Xw) \\ =(y^T-w^{T}X)(y-Xw) \\ =y^{T}y-y^{T}Xw-w^T{X}^{T}y+w^T{X}^{T}Xw \\ 因此\frac{\partial E_w}{\partial W}=\frac{\partial (y^{T}y)}{\partial w}-\frac{\partial (y^{T}Xw)}{\partial w}-\frac{\partial (w^T{X}^{T}y)}{\partial w}+\frac{\partial (w^T{X}^{T}Xw)}{\partial w} \\ =0-X^{T}y(常用特例1)-X^{T}y(常用特例1)+2X^{T}Xw(常用特例2，X^{T}X为对称阵) \\ =2X^{T}Xw-2X^{T}y \\ 我们将损失关于w的导数设置为0，那么可以得到解析解：w=(X^{T}X{)}^{-1}{X}^{T}y \end{gathered}$$

4、向量求导的链式法则

$$举例证明链式求导法则为：\frac{\partial J}{\partial \overrightarrow{u}}=\frac{\partial \overrightarrow{y}(\overrightarrow{u})}{\partial \overrightarrow{u}}.\frac{\partial J}{\partial \overrightarrow{y}(\overrightarrow{u})}$$