SVM Summary

Example

Suppose the dataset contains two positive samples $x^{(1)}=[1,1{]}^T$ and$x^{(2)}=[2,2{]}^T$, and two negative samples $x^{(3)}=[0,0{]}^T$ and $x^{(4)}=[-1,0{]}^T$. Please calculate the SVM decision hyperplane.

Calculate

$$\underset{\lambda }{\min }\mathcal{J}(\lambda )=\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\lambda }_i{\lambda }_j{y}^{(i)}{y}^{(j)}(x^{(i)}{)}^T{x}^{(j)}-\sum _{i=1}^N{\lambda }_i$$
$$s.t.{\lambda }_i⩾0,\sum _{i=1}^N{\lambda }_i{y}^{(i)}=0$$
由 D a t a s e t   D : { x : { [ 1 , 1 ] , [ 2 , 2 ] , [ 0 , 0 ] , [ − 1 , 0 ] } , y : { 1 , 1 , − 1 , − 1 } } Dataset\ D:\{x:\{[1,1],[2,2],[0,0],[-1,0]\},y:\{1,1,-1,-1\}\} Dataset D:{x:{[1,1],[2,2],[0,0],[−1,0]},y:{1,1,−1,−1}}可得下式：
$$\begin{gathered} \underset{\lambda }{\min }\mathcal{J}(\lambda )=\frac{1}{2}(2{\lambda }_1^2+8{\lambda }_2^2+{\lambda }_4^2+8{\lambda }_1{\lambda }_2+2{\lambda }_1{\lambda }_4+4{\lambda }_2{\lambda }_4) \\ -{\lambda }_1-{\lambda }_2-{\lambda }_3-{\lambda }_4 \\ s.t{\lambda }_1⩾0,{\lambda }_2⩾0,{\lambda }_3⩾0,{\lambda }_4⩾0 \\ {\lambda }_1+{\lambda }_2-{\lambda }_3-{\lambda }_4=0 \end{gathered}$$
since ${\lambda }_1+{\lambda }_2={\lambda }_3+{\lambda }_4\to {\lambda }_3={\lambda }_1+{\lambda }_2-{\lambda }_4$:
$$\begin{gathered} \underset{\lambda }{\min }\mathcal{J}(\lambda )={\lambda }_1^2+4{\lambda }_2^2+\frac{1}{2}{\lambda }_4^2+4{\lambda }_1{\lambda }_2+{\lambda }_1{\lambda }_4+2{\lambda }_2{\lambda }_4-2{\lambda }_1-2{\lambda }_2 \\ s.t{\lambda }_1⩾0,{\lambda }_2⩾0 \\ {⟹}^{求偏导} \\ \{\begin{array}{c}\frac{\partial \mathcal{J}}{\partial {\lambda }_1}=2{\lambda }_1+4{\lambda }_2+{\lambda }_4-2=0\\ \frac{\partial \mathcal{J}}{\partial {\lambda }_2}=4{\lambda }_1+8{\lambda }_2+2{\lambda }_4-2=0\\ \frac{\partial \mathcal{J}}{\partial {\lambda }_4}={\lambda }_1+2{\lambda }_2+{\lambda }_4=0\end{array} \end{gathered}$$
Lagrange无解，所以极小值在边界上：

• 令${\lambda }_1=0，{\lambda }_3={\lambda }_1+{\lambda }_2-{\lambda }_4$带入$\mathcal{J}(\lambda )$中，得：
  $$\begin{gathered} \mathcal{J}(\lambda )=4{\lambda }_2^2+\frac{1}{2}{\lambda }_4^2++2{\lambda }_2{\lambda }_4-2{\lambda }_2 \\ {⟹}^{求偏导} \\ \{\begin{array}{c}\frac{\partial \mathcal{J}}{\partial {\lambda }_2}=8{\lambda }_2+2{\lambda }_4-2=0\\ \frac{\partial \mathcal{J}}{\partial {\lambda }_4}=2{\lambda }_2+{\lambda }_4=0\end{array}⟹\{\begin{array}{c}{\lambda }_2=\frac{1}{2}\\ {\lambda }_4=-1(\le 0不满足s.t.)\end{array} \\ 再令： \\ {\lambda }_2=0,则{\lambda }_4=0，\mathcal{J}(\lambda )=0； \\ 或{\lambda }_4=0,则{\lambda }_2=\frac{1}{4}，\mathcal{J}(\lambda )=-\frac{1}{4}； \end{gathered}$$

同理可得：

• ${\lambda }_2=0$
  $$\begin{gathered} {\lambda }_1=0,则{\lambda }_4=0，\mathcal{J}(\lambda )=0； \\ 或{\lambda }_4=0,则{\lambda }_1=1，\mathcal{J}(\lambda )=-1； \end{gathered}$$
• ${\lambda }_3=0$
  $$\begin{gathered} {\lambda }_1=0,则{\lambda }_2=\frac{2}{13}，\mathcal{J}(\lambda )=-\frac{2}{13}； \\ 或{\lambda }_2=0,则{\lambda }_1=\frac{2}{5}，\mathcal{J}(\lambda )=-\frac{2}{5}； \end{gathered}$$
• ${\lambda }_4=0$
  $$\begin{gathered} {\lambda }_1=0,则{\lambda }_2=\frac{1}{4}，\mathcal{J}(\lambda )=-\frac{1}{4}； \\ 或{\lambda }_2=0,则{\lambda }_1=1，\mathcal{J}(\lambda )=-1； \end{gathered}$$
  综上： λ 1 , 2 , 3 , 4 = { 1 , 0 , 1 , 0 } \lambda_{1,2,3,4} =\{1,0,1,0\} λ1,2,3,4​={1,0,1,0}
  $$\begin{gathered} \{\begin{array}{c}W={\sum }_{i=1}^N{\lambda }_i{y}^{(i)}{\mathbf{x}}^{(i)}\\ b=y^{(j)}-{\sum }_{i=1}^N{\lambda }_i{y}^{(i)}{\left(x^{(i)}\right)}^T{x}^{(j)}\end{array}⟹\{\begin{array}{c}W=[1,1{]}^T\\ b=-1\end{array} \\ ⟹x^{(1)}+x^{(2)}-1=0 \end{gathered}$$