本文仅供学习使用
本文参考：
B站：DR_CAN

---

1. 标量向量方程对向量求导，分母布局，分子布局

1.1 标量方程对向量的导数

• $y$ 为 一元向量 或 二元向量
  
• $y$为多元向量
  $$\vec{y}=\left[y_1,y_2,\cdots ,y_{\mathrm{n}}\right]\Rightarrow \frac{\partial f\left(\vec{y}\right)}{\partial \vec{y}}$$
  其中：$f\left(\vec{y}\right)$ 为标量 $1\times 1$, $\vec{y}$为向量 $1\times n$

1. 分母布局 Denominator Layout——行数与分母相同
   $\frac{\partial f\left(\vec{y}\right)}{\partial \vec{y}}={\left[\begin{array}{c}\frac{\partial f\left(\vec{y}\right)}{\partial y_1}\\ ⋮\\ \frac{\partial f\left(\vec{y}\right)}{\partial y_{\mathrm{n}}}\end{array}\right]}_{n\times 1}$
2. 分子布局 Nunerator Layout——行数与分子相同
   $\frac{\partial f\left(\vec{y}\right)}{\partial \vec{y}}={\left[\begin{array}{ccc}\frac{\partial f\left(\vec{y}\right)}{\partial y_1}& \cdots & \frac{\partial f\left(\vec{y}\right)}{\partial y_{\mathrm{n}}}\end{array}\right]}_{1\times n}$

1.2 向量方程对向量的导数

$\vec{f}\left(\vec{y}\right)={\left[\begin{array}{c}{\vec{f}}_1\left(\vec{y}\right)\\ ⋮\\ {\vec{f}}_{\mathrm{n}}\left(\vec{y}\right)\end{array}\right]}_{n\times 1},\vec{y}={\left[\begin{array}{c}{y}_1\\ ⋮\\ y_{\mathrm{m}}\end{array}\right]}_{\mathrm{m}\times 1}$
$\frac{\partial \vec{f}{\left(\vec{y}\right)}_{n\times 1}}{\partial {\vec{y}}_{\mathrm{m}\times 1}}={\left[\begin{array}{c}\frac{\partial \vec{f}\left(\vec{y}\right)}{\partial y_1}\\ ⋮\\ \frac{\partial \vec{f}\left(\vec{y}\right)}{\partial y_{\mathrm{m}}}\end{array}\right]}_{\mathrm{m}\times 1}={\left[\begin{array}{ccc}\frac{\partial f_1\left(\vec{y}\right)}{\partial y_1}& \cdots & \frac{\partial f_{\mathrm{n}}\left(\vec{y}\right)}{\partial y_1}\\ ⋮& \ddots & ⋮\\ \frac{\partial f_1\left(\vec{y}\right)}{\partial y_{\mathrm{m}}}& \cdots & \frac{\partial f_{\mathrm{n}}\left(\vec{y}\right)}{\partial y_{\mathrm{m}}}\end{array}\right]}_{\mathrm{m}\times \mathrm{n}}$, 为分母布局

若： $\vec{y}={\left[\begin{array}{c}{y}_1\\ ⋮\\ y_{\mathrm{m}}\end{array}\right]}_{\mathrm{m}\times 1},A=\left[\begin{array}{ccc}{a}_{11}& \cdots & a_{1\mathrm{n}}\\ ⋮& \ddots & ⋮\\ a_{\mathrm{m}1}& \cdots & a_{\mathrm{mn}}\end{array}\right]$, 则有：

• $\frac{\partial A\vec{y}}{\partial \vec{y}}=A^{\mathrm{T}}$(分母布局)
• $\frac{\partial {\vec{y}}^{\mathrm{T}}A\vec{y}}{\partial \vec{y}}=A\vec{y}+A^{\mathrm{T}}\vec{y}$, 当$A=A^{\mathrm{T}}$时, $\frac{\partial {\vec{y}}^{\mathrm{T}}A\vec{y}}{\partial \vec{y}}=2A\vec{y}$

若为分子布局，则有：$\frac{\partial A\vec{y}}{\partial \vec{y}}=A$

2. 案例分析，线性回归

• $\frac{\partial A\vec{y}}{\partial \vec{y}}=A^{\mathrm{T}}$(分母布局)
• $\frac{\partial {\vec{y}}^{\mathrm{T}}A\vec{y}}{\partial \vec{y}}=A\vec{y}+A^{\mathrm{T}}\vec{y}$, 当$A=A^{\mathrm{T}}$时, $\frac{\partial {\vec{y}}^{\mathrm{T}}A\vec{y}}{\partial \vec{y}}=2A\vec{y}$

Linear Regression 线性回归
$$\hat{z}=y_1+y_{2}x\Rightarrow J=\sum _{i=1}^n{\left[z_i-\left(y_1+y_2{x}_i\right)\right]}^2$$
找到 $y_1,y_2$ 使得 $J$最小

$\vec{z}=\left[\begin{array}{c}{z}_1\\ ⋮\\ z_{\mathrm{n}}\end{array}\right],\left[\vec{x}\right]=\left[\begin{array}{lc}1& x_1\\ ⋮& ⋮\\ 1& x_{\mathrm{n}}\end{array}\right],\vec{y}=\left[\begin{array}{c}{y}_1\\ y_2\end{array}\right]\Rightarrow \hat{\vec{z}}=\left[\vec{x}\right]\vec{y}=\left[\begin{array}{c}{y}_1+y_2{x}_1\\ ⋮\\ y_1+y_2{x}_{\mathrm{n}}\end{array}\right]$
$J={\left[\vec{z}-\hat{\vec{z}}\right]}^{\mathrm{T}}\left[\vec{z}-\hat{\vec{z}}\right]={\left[\vec{z}-\left[\vec{x}\right]\vec{y}\right]}^{\mathrm{T}}\left[\vec{z}-\left[\vec{x}\right]\vec{y}\right]=\vec{z}{\vec{z}}^{\mathrm{T}}-{\vec{z}}^{\mathrm{T}}\left[\vec{x}\right]\vec{y}-{\vec{y}}^{\mathrm{T}}{\left[\vec{x}\right]}^{\mathrm{T}}\vec{z}+{\vec{y}}^{\mathrm{T}}{\left[\vec{x}\right]}^{\mathrm{T}}\left[\vec{x}\right]\vec{y}$
其中： ${\left({\vec{z}}^{\mathrm{T}}\left[\vec{x}\right]\vec{y}\right)}^{\mathrm{T}}={\vec{y}}^{\mathrm{T}}{\left[\vec{x}\right]}^{\mathrm{T}}\vec{z}$， 则有：
$$J=\vec{z}{\vec{z}}^{\mathrm{T}}-2{\vec{z}}^{\mathrm{T}}\left[\vec{x}\right]\vec{y}+{\vec{y}}^{\mathrm{T}}{\left[\vec{x}\right]}^{\mathrm{T}}\left[\vec{x}\right]\vec{y}$$
进而：
$\frac{\partial J}{\partial \vec{y}}=0-2{\left({\vec{z}}^{\mathrm{T}}\left[\vec{x}\right]\right)}^{\mathrm{T}}+2{\left[\vec{x}\right]}^{\mathrm{T}}\left[\vec{x}\right]\vec{y}=\nabla \vec{y}⟹\frac{\partial J}{\partial {\vec{y}}^{\ast }}=0,{\vec{y}}^{\ast }={\left({\left[\vec{x}\right]}^{\mathrm{T}}\left[\vec{x}\right]\right)}^{-1}{\left[\vec{x}\right]}^{\mathrm{T}}\vec{z}$
其中：${\left({\left[\vec{x}\right]}^{\mathrm{T}}\left[\vec{x}\right]\right)}^{-1}$不一定有解，则${\vec{y}}^{\ast }$无法得到解析解——定义初始${\vec{y}}^{\ast }$， ${\vec{y}}^{\ast }={\vec{y}}^{\ast }-\alpha \nabla ,\alpha =\left[\begin{array}{cc}{\alpha }_1& 0\\ 0& {\alpha }_2\end{array}\right]$
其中： $\alpha$称为学习率，对$x$而言则需进行归一化

3. 矩阵求导的链式法则

标量函数： $J=f\left(y\left(u\right)\right),\frac{\partial J}{\partial u}=\frac{\partial J}{\partial y}\frac{\partial y}{\partial u}$

标量对向量求导：$J=f\left(\vec{y}\left(\vec{u}\right)\right),\vec{y}={\left[\begin{array}{c}{y}_1\left(\vec{u}\right)\\ ⋮\\ y_{\mathrm{m}}\left(\vec{u}\right)\end{array}\right]}_{m\times 1},\vec{u}={\left[\begin{array}{c}{\vec{u}}_1\\ ⋮\\ {\vec{u}}_{\mathrm{n}}\end{array}\right]}_{\mathrm{n}\times 1}$

分析：${\frac{\partial J_{1\times 1}}{\partial u_{\mathrm{n}\times 1}}}_{\mathrm{n}\times 1}={\frac{\partial J}{\partial y_{m\times 1}}}_{m\times 1}{\frac{\partial y_{m\times 1}}{\partial u_{\mathrm{n}\times 1}}}_{\mathrm{n}\times \mathrm{m}}$ 无法相乘

$\vec{y}={\left[\begin{array}{c}{y}_1\left(\vec{u}\right)\\ y_2\left(\vec{u}\right)\end{array}\right]}_{2\times 1},\vec{u}={\left[\begin{array}{c}{\vec{u}}_1\\ {\vec{u}}_2\\ {\vec{u}}_3\end{array}\right]}_{3\times 1}$
$$\begin{gathered} J=f\left(\vec{y}\left(\vec{u}\right)\right),\frac{\partial J}{\partial \vec{u}}={\left[\begin{array}{c}\frac{\partial J}{\partial {\vec{u}}_1}\\ \frac{\partial J}{\partial {\vec{u}}_2}\\ \frac{\partial J}{\partial {\vec{u}}_3}\end{array}\right]}_{3\times 1}⟹\begin{array}{c}\frac{\partial J}{\partial {\vec{u}}_1}=\frac{\partial J}{\partial y_1}\frac{\partial y_1\left(\vec{u}\right)}{\partial {\vec{u}}_1}+\frac{\partial J}{\partial y_2}\frac{\partial y_2\left(\vec{u}\right)}{\partial {\vec{u}}_1}\\ \frac{\partial J}{\partial {\vec{u}}_2}=\frac{\partial J}{\partial y_1}\frac{\partial y_1\left(\vec{u}\right)}{\partial {\vec{u}}_2}+\frac{\partial J}{\partial y_2}\frac{\partial y_2\left(\vec{u}\right)}{\partial {\vec{u}}_2}\\ \frac{\partial J}{\partial {\vec{u}}_3}=\frac{\partial J}{\partial y_1}\frac{\partial y_1\left(\vec{u}\right)}{\partial {\vec{u}}_3}+\frac{\partial J}{\partial y_2}\frac{\partial y_2\left(\vec{u}\right)}{\partial {\vec{u}}_3}\end{array} \\ ⟹\frac{\partial J}{\partial \vec{u}}={\left[\begin{array}{lc}\frac{\partial y_1\left(\vec{u}\right)}{\partial {\vec{u}}_1}& \frac{\partial y_2\left(\vec{u}\right)}{\partial {\vec{u}}_1}\\ \frac{\partial y_1\left(\vec{u}\right)}{\partial {\vec{u}}_2}& \frac{\partial y_2\left(\vec{u}\right)}{\partial {\vec{u}}_2}\\ \frac{\partial y_1\left(\vec{u}\right)}{\partial {\vec{u}}_3}& \frac{\partial y_2\left(\vec{u}\right)}{\partial {\vec{u}}_3}\end{array}\right]}_{3\times 2}{\left[\begin{array}{c}\frac{\partial J}{\partial y_1}\\ \frac{\partial J}{\partial y_2}\end{array}\right]}_{2\times 2}=\frac{\partial \vec{y}\left(\vec{u}\right)}{\partial \vec{u}}\frac{\partial J}{\partial \vec{y}} \end{gathered}$$

$$\frac{\partial J}{\partial \vec{u}}=\frac{\partial \vec{y}\left(\vec{u}\right)}{\partial \vec{u}}\frac{\partial J}{\partial \vec{y}}$$

eg:
$$\vec{x}\left[k+1\right]=A\vec{x}\left[k\right]+B\vec{u}\left[k\right],J={\vec{x}}^{\mathrm{T}}\left[k+1\right]\vec{x}\left[k+1\right]$$
$$\frac{\partial J}{\partial \vec{u}}=\frac{\partial \vec{x}\left[k+1\right]}{\partial \vec{u}}\frac{\partial J}{\partial \vec{x}\left[k+1\right]}=B^{\mathrm{T}}\cdot 2\vec{x}\left[k+1\right]=2B^{\mathrm{T}}\vec{x}\left[k+1\right]$$