PumpkinBook/docs/chapter6/chapter6.md

6.1

$$\boldsymbol{w}^{\mathrm{T}}\boldsymbol{x}+b=0$$ [解析]：略

6.2

$$r=\frac{|\boldsymbol{w}^{\mathrm{T}}\boldsymbol{x}+b|}{|\boldsymbol{w}|}$$ [解析]：略

6.3

$$\left{\begin{array}{ll}{\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}+b \geqslant+1,} & {y{i}=+1} \ {\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}+b \leqslant-1,} & {y{i}=-1}\end{array}\right.$$ [解析]：略

6.4

$$\gamma=\frac{2}{|\boldsymbol{w}|}$$ [解析]：略

6.5

$$\begin{array}{l} \underset{\boldsymbol{w}, b}{\max} \frac{2}{|\boldsymbol{w}|} \ \text { s.t. } y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}+b\right) \geqslant 1, \quad i=1,2, \ldots, m \end{array}$$ [解析]：略

6.6

$$\begin{array}{l} \underset{\boldsymbol{w}, b}{\max} \frac{1}{2}|\boldsymbol{w}|^2 \ \text { s.t. } y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}+b\right) \geqslant 1, \quad i=1,2, \ldots, m \end{array}$$ [解析]：略

6.8

$$ L(\boldsymbol{w}, b, \boldsymbol{\alpha})=\frac{1}{2}|\boldsymbol{w}|^{2}+\sum{i=1}^{m} \alpha{i}\left(1-y{i}\left(\boldsymbol{w}^{\top} \boldsymbol{x}{i}+b\right)\right) $$ [解析]：略

6.9

$$\boldsymbol{w} = \sum_{i=1}^m\alpha_iy_i\boldsymbol{x}i$$ [推导]：公式(6.8)可作如下展开 $$\begin{aligned} L(\boldsymbol{w},b,\boldsymbol{\alpha}) &= \frac{1}{2}||\boldsymbol{w}||^2+\sum{i=1}^m\alpha_i(1-y_i(\boldsymbol{w}^T\boldsymbol{x}i+b)) \ & = \frac{1}{2}||\boldsymbol{w}||^2+\sum{i=1}^m(\alpha_i-\alpha_iy_i \boldsymbol{w}^T\boldsymbol{x}_i-\alpha_iyib)\ & =\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}+\sum{i=1}^m\alphai -\sum{i=1}^m\alpha_iy_i\boldsymbol{w}^T\boldsymbol{x}i-\sum{i=1}^m\alpha_iyib \end{aligned}​$$ 对$\boldsymbol{w}$和$b$分别求偏导数​并令其等于0 $$\frac {\partial L}{\partial \boldsymbol{w}}=\frac{1}{2}\times2\times\boldsymbol{w} + 0 - \sum{i=1}^{m}\alpha_iy_i \boldsymbol{x}i-0= 0 \Longrightarrow \boldsymbol{w}=\sum{i=1}^{m}\alpha_iy_i \boldsymbol{x}_i$$

$$\frac {\partial L}{\partial b}=0+0-0-\sum_{i=1}^{m}\alpha_iyi=0 \Longrightarrow \sum{i=1}^{m}\alpha_iy_i=0$$ 值得一提的是，上述求解过程遵循的是西瓜书附录B中公式(B.7)左边的那段话“在推导对偶问题时，常通过将拉格朗日函数$L(\boldsymbol{x},\boldsymbol{\lambda},\boldsymbol{\mu})$对$\boldsymbol{x}$求导并令导数为0，来获得对偶函数的表达形式”。那么这段话背后的缘由是啥呢？在这里我猜测可能有两点理由：

1. 对于强对偶性成立的优化问题，其主问题的最优解$\boldsymbol{x}^$一定满足附录①给出的KKT条件（证明参见参考文献[3]的§ 5.5），而KKT条件中的条件(1)就要求最优解$\boldsymbol{x}^$能使得拉格朗日函数$L(\boldsymbol{x},\boldsymbol{\lambda},\boldsymbol{\mu})$关于$\boldsymbol{x}$的一阶导数等于0；
2. 对于任意优化问题，若拉格朗日函数$L(\boldsymbol{x},\boldsymbol{\lambda},\boldsymbol{\mu})$是关于$\boldsymbol{x}$的凸函数，那么此时对$L(\boldsymbol{x},\boldsymbol{\lambda},\boldsymbol{\mu})$关于$\boldsymbol{x}$求导并令导数等于0解出来的点一定是最小值点。根据对偶函数的定义可知，将最小值点代回$L(\boldsymbol{x},\boldsymbol{\lambda},\boldsymbol{\mu})$即可得到对偶函数。

显然，对于SVM来说，它同时满足上述两种情形。

6.10

$$0=\sum_{i=1}^m\alpha_iy_i$$ [解析]：参见公式(6.9)

6.11

$$\begin{aligned} \max{\boldsymbol{\alpha}} & \sum{i=1}^m\alphai - \frac{1}{2}\sum{i = 1}^m\sum_{j=1}^m\alpha_i \alpha_j y_iy_j\boldsymbol{x}_i^T\boldsymbol{x}j \ \text { s.t. } & \sum{i=1}^m \alpha_i y_i =0 \ & \alphai \geq 0 \quad i=1,2,\dots ,m \end{aligned}$$
[推导]：将公式(6.9)和公式(6.10)代入公式(6.8)即可将$L(\boldsymbol{w},b,\boldsymbol{\alpha})$中的$\boldsymbol{w}$和$b$消去，再考虑公式(6.10)的约束，就得到了公式(6.6)的对偶问题 $$\begin{aligned} \inf{\boldsymbol{w},b} L(\boldsymbol{w},b,\boldsymbol{\alpha}) &=\frac{1}{2}\boldsymbol{w}^T\boldsymbol{w}+\sum_{i=1}^m\alphai -\sum{i=1}^m\alpha_iy_i\boldsymbol{w}^T\boldsymbol{x}i-\sum{i=1}^m\alpha_iy_ib \ &=\frac {1}{2}\boldsymbol{w}^T\sum _{i=1}^m\alpha_iy_i\boldsymbol{x}_i-\boldsymbol{w}^T\sum _{i=1}^m\alpha_iy_i\boldsymbol{x}_i+\sum {i=1}^m\alpha i -b\sum _{i=1}^m\alpha_iy_i \ & = -\frac {1}{2}\boldsymbol{w}^T\sum _{i=1}^m\alpha_iy_i\boldsymbol{x}_i+\sum _{i=1}^m\alpha_i -b\sum _{i=1}^m\alpha_iyi \end{aligned}$$ 由于$\sum\limits{i=1}^{m}\alpha_iyi=0$，所以上式最后一项可化为0，于是得 $$\begin{aligned} \inf{\boldsymbol{w},b} L(\boldsymbol{w},b,\boldsymbol{\alpha}) &= -\frac {1}{2}\boldsymbol{w}^T\sum _{i=1}^m\alpha_iy_i\boldsymbol{x}_i+\sum _{i=1}^m\alphai \ &=-\frac {1}{2}(\sum{i=1}^{m}\alpha_iy_i\boldsymbol{x}_i)^T(\sum _{i=1}^m\alpha_iy_i\boldsymbol{x}_i)+\sum _{i=1}^m\alphai \ &=-\frac {1}{2}\sum{i=1}^{m}\alpha_iy_i\boldsymbol{x}_i^T\sum _{i=1}^m\alpha_iy_i\boldsymbol{x}_i+\sum _{i=1}^m\alpha_i \ &=\sum _{i=1}^m\alphai-\frac {1}{2}\sum{i=1 }^{m}\sum_{j=1}^{m}\alpha_i\alpha_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}j \end{aligned}$$ 所以 $$\max{\boldsymbol{\alpha}}\inf{\boldsymbol{w},b} L(\boldsymbol{w},b,\boldsymbol{\alpha})=\max{\boldsymbol{\alpha}} \sum_{i=1}^m\alphai - \frac{1}{2}\sum{i = 1}^m\sum_{j=1}^m\alpha_i \alpha_j y_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j $$

6.12

$$\begin{aligned} f(\boldsymbol{x}) &=\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}+b \ &=\sum{i=1}^{m} \alpha{i} y{i} \boldsymbol{x}{i}^{\mathrm{T}} \boldsymbol{x}+b \end{aligned}$$ [解析]：略

6.13

$$\left{\begin{array}{l}\alpha{i} \geqslant 0 \ y{i} f\left(\boldsymbol{x}{i}\right)-1 \geqslant 0 \ \alpha{i}\left(y{i} f\left(\boldsymbol{x}{i}\right)-1\right)=0\end{array}\right.$$ [解析]：参见公式(6.9)中给出的第1点理由

6.14

$$\alpha{i} y{i}+\alpha{j} y{j}=c, \quad \alpha{i} \geqslant 0, \quad \alpha{j} \geqslant 0$$ [解析]：略

6.15

$$c=-\sum{k \neq i, j} \alpha{k} y_{k}$$ [解析]：略

6.16

$$\alpha{i} y{i}+\alpha{j} y{j}=c$$ [解析]：略

6.17

$$y{s}\left(\sum{i \in S} \alpha{i} y{i} \boldsymbol{x}{i}^{\mathrm{T}} \boldsymbol{x}{s}+b\right)=1$$ [解析]：略

6.18

$$b=\frac{1}{|S|} \sum{s \in S}\left(y{s}-\sum{i \in S} \alpha{i} y{i} \boldsymbol{x}{i}^{\mathrm{T}} \boldsymbol{x}_{s}\right)$$ [解析]：略

6.19

$$f(\boldsymbol{x})=\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x})+b$$ [解析]：略

6.20

$$\begin{array}{l} \underset{\boldsymbol{w}, b}{\max} \frac{1}{2}|\boldsymbol{w}|^2 \ \text { s.t. } y{i}\left(\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x}{i})+b\right) \geqslant 1, \quad i=1,2, \ldots, m \end{array}$$ [解析]：略

6.21

$$\begin{aligned} \max{\boldsymbol{\alpha}} & \sum{i=1}^m\alphai - \frac{1}{2}\sum{i = 1}^m\sum_{j=1}^m\alpha_i \alpha_j y_iy_j\phi(\boldsymbol{x}_i)^T\phi(\boldsymbol{x}j) \ \text { s.t. } & \sum{i=1}^m \alpha_i y_i =0 \ & \alpha_i \geq 0 \quad i=1,2,\dots ,m \end{aligned}$$ [解析]：略

6.22

$$\kappa\left(\boldsymbol{x}{i}, \boldsymbol{x}{j}\right)=\left\langle\phi\left(\boldsymbol{x}{i}\right), \phi\left(\boldsymbol{x}{j}\right)\right\rangle=\phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}} \phi\left(\boldsymbol{x}{j}\right)$$ [解析]：略

6.23

$$\begin{aligned} \max{\boldsymbol{\alpha}} & \sum{i=1}^m\alphai - \frac{1}{2}\sum{i = 1}^m\sum_{j=1}^m\alpha_i \alpha_j y_iyj\kappa\left(\boldsymbol{x}{i}, \boldsymbol{x}{j}\right) \ \text { s.t. } & \sum{i=1}^m \alpha_i y_i =0 \ & \alpha_i \geq 0 \quad i=1,2,\dots ,m \end{aligned}$$ [解析]：略

6.24

$$\begin{aligned} f(\boldsymbol{x}) &=\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x})+b \ &=\sum{i=1}^{m} \alpha{i} y{i}\phi(\boldsymbol{x}{i})^{\mathrm{T}}\phi(\boldsymbol{x})+b \ &=\sum{i=1}^{m} \alpha{i} y{i}\kappa\left(\boldsymbol{x}, \boldsymbol{x}{i}\right)+b \ \end{aligned}$$ [解析]：略

6.25

$$\gamma_1\kappa_1+\gamma_2\kappa_2$$ [解析]：略

6.26

$$\kappa_1\otimes\kappa_2\left(\boldsymbol{x}, \boldsymbol{z}\right)=\kappa_1\left(\boldsymbol{x}, \boldsymbol{z}\right)\kappa_2\left(\boldsymbol{x}, \boldsymbol{z}\right)$$ [解析]：略

6.27

$$\kappa\left(\boldsymbol{x}, \boldsymbol{z}\right)=g(\boldsymbol{x})\kappa_1\left(\boldsymbol{x}, \boldsymbol{z}\right)g(\boldsymbol{z})$$ [解析]：略

6.28

$$y_i(\boldsymbol{w}^{\mathrm{T}}\boldsymbol{x}_i+b)\geqslant 1$$ [解析]：略

6.28

$$y_i(\boldsymbol{w}^{\mathrm{T}}\boldsymbol{x}_i+b)\geqslant 1$$ [解析]：略

6.29

$$\min {\boldsymbol{w}, b} \frac{1}{2}|\boldsymbol{w}|^{2}+C \sum{i=1}^{m} \ell{0 / 1}\left(y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}_{i}+b\right)-1\right)$$ [解析]：略

6.30

$$\ell_{0 / 1}(z)=\left{\begin{array}{ll}{1,} & {\text { if } z < 0} \ {0,} & {\text { otherwise }}\end{array}\right.$$ [解析]：略

6.31

$$\ell_{hinge}(z)=\max(0,1-z)$$ [解析]：略

6.32

$$\ell_{exp}(z)=\exp(-z)$$ [解析]：略

6.33

$$\ell_{log}(z)=\log(1+\exp(-z))$$ [解析]：略

6.34

$$\min {\boldsymbol{w}, b} \frac{1}{2}|\boldsymbol{w}|^{2}+C \sum{i=1}^{m} \max \left(0,1-y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}+b\right)\right)$$ [解析]：略

6.35

$$\begin{aligned} \min {\boldsymbol{w}, b, \xi{i}} & \frac{1}{2}|\boldsymbol{w}|^{2}+C \sum{i=1}^{m} \xi{i} \ \text { s.t. } & y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}+b\right) \geqslant 1-\xi{i} \ & \xi{i} \geqslant 0, i=1,2, \ldots, m \end{aligned}$$ [解析]：令 $$\max \left(0,1-y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}+b\right)\right)=\xi_{i}$$ 显然$\xii\geq 0$，而且当$1-y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}+b\right)>0$时 $$1-y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}_{i}+b\right)=\xii$$ 当$1-y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}_{i}+b\right)\leq 0$时 $$\xii = 0$$ 所以综上可得 $$1-y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}_{i}+b\right)\leq\xii\Rightarrow y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}+b\right) \geqslant 1-\xi{i}$$

6.36

$$\begin{aligned} L(\boldsymbol{w}, b, \boldsymbol{\alpha}, \boldsymbol{\xi}, \boldsymbol{\mu})=& \frac{1}{2}|\boldsymbol{w}|^{2}+C \sum{i=1}^{m} \xi{i} \ &+\sum{i=1}^{m} \alpha{i}\left(1-\xi{i}-y{i}\left(\boldsymbol{w}^{\mathrm{T}} \boldsymbol{x}{i}+b\right)\right)-\sum{i=1}^{m} \mu{i} \xi{i} \end{aligned}$$ [解析]：略

6.37

$$\boldsymbol{w}=\sum{i=1}^{m}\alpha{i}y{i}\boldsymbol{x}{i}$$ [解析]：参见公式(6.9)

6.38

$$0=\sum{i=1}^{m}\alpha{i}y_{i}$$ [解析]：参见公式(6.10)

6.39

$$ C=\alpha_i +\mu_i $$ [推导]：对式（6.36）关于$\xi_i$求偏导并令其等于0可得： $$\frac{\partial L}{\partial \xi_i}=0+C \times 1 - \alpha_i \times 1-\mu_i \times 1 =0\Longrightarrow C=\alpha_i +\mu_i$$

6.40

$$\begin{aligned} \max_{\boldsymbol{\alpha}}&\sum _{i=1}^m\alphai-\frac {1}{2}\sum{i=1 }^{m}\sum_{j=1}^{m}\alpha_i\alpha_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}j \ s.t. &\sum{i=1}^m \alpha_i y_i=0 \ & 0 \leq\alphai \leq C \quad i=1,2,\dots ,m \end{aligned}$$ 将式6.37-6.39代入6.36可以得到6.35的对偶问题： $$\begin{aligned} \min{\boldsymbol{w},b,\boldsymbol{\xi}}L(\boldsymbol{w},b,\boldsymbol{\alpha},\boldsymbol{\xi},\boldsymbol{\mu}) &= \frac{1}{2}||\boldsymbol{w}||^2+C\sum_{i=1}^m \xii+\sum{i=1}^m \alpha_i(1-\xi_i-y_i(\boldsymbol{w}^T\boldsymbol{x}i+b))-\sum{i=1}^m\mu_i \xii \ &=\frac{1}{2}||\boldsymbol{w}||^2+\sum{i=1}^m\alpha_i(1-y_i(\boldsymbol{w}^T\boldsymbol{x}i+b))+C\sum{i=1}^m \xii-\sum{i=1}^m \alpha_i \xii-\sum{i=1}^m\mu_i \xii \ & = -\frac {1}{2}\sum{i=1}^{m}\alpha_iy_i\boldsymbol{x}_i^T\sum _{i=1}^m\alpha_iy_i\boldsymbol{x}_i+\sum _{i=1}^m\alphai +\sum{i=1}^m C\xii-\sum{i=1}^m \alpha_i \xii-\sum{i=1}^m\mu_i \xii \ & = -\frac {1}{2}\sum{i=1}^{m}\alpha_iy_i\boldsymbol{x}_i^T\sum _{i=1}^m\alpha_iy_i\boldsymbol{x}_i+\sum _{i=1}^m\alphai +\sum{i=1}^m (C-\alpha_i-\mu_i)\xi_i \ &=\sum _{i=1}^m\alphai-\frac {1}{2}\sum{i=1 }^{m}\sum_{j=1}^{m}\alpha_i\alpha_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}j \end{aligned}$$
所以 $$\begin{aligned} \max{\boldsymbol{\alpha},\boldsymbol{\mu}} \min{\boldsymbol{w},b,\boldsymbol{\xi}}L(\boldsymbol{w},b,\boldsymbol{\alpha},\boldsymbol{\xi},\boldsymbol{\mu})&=\max{\boldsymbol{\alpha},\boldsymbol{\mu}}\sum _{i=1}^m\alphai-\frac {1}{2}\sum{i=1 }^{m}\sum_{j=1}^{m}\alpha_i\alpha_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}j \ &=\max{\boldsymbol{\alpha}}\sum _{i=1}^m\alphai-\frac {1}{2}\sum{i=1 }^{m}\sum_{j=1}^{m}\alpha_i\alpha_jy_iy_j\boldsymbol{x}_i^T\boldsymbol{x}_j \end{aligned}$$ 又 $$\begin{aligned} \alpha_i &\geq 0 \ \mu_i &\geq 0 \ C &= \alpha_i+\mu_i \end{aligned}$$ 消去$\mu_i$可得等价约束条件为： $$0 \leq\alpha_i \leq C \quad i=1,2,\dots ,m$$

6.41

$$\left{\begin{array}{l}\alpha{i} \geqslant 0, \quad \mu{i} \geqslant 0 \ y{i} f\left(\boldsymbol{x}{i}\right)-1+\xi{i} \geqslant 0 \ \alpha{i}\left(y{i} f\left(\boldsymbol{x}{i}\right)-1+\xi{i}\right)=0 \ \xi{i} \geqslant 0, \mu{i} \xi{i}=0\end{array}\right.$$ [解析]：参见公式(6.13)

6.42

$$\min {f} \Omega(f)+C \sum{i=1}^{m} \ell\left(f\left(\boldsymbol{x}{i}\right), y{i}\right)$$ [解析]：略

6.43

$$\min {\boldsymbol{w}, b} \frac{1}{2}|\boldsymbol{w}|^{2}+C \sum{i=1}^{m} \ell{\epsilon}\left(f\left(\boldsymbol{x}{i}\right)-y_{i}\right)$$ [解析]：略

6.44

$$\ell_{\epsilon}(z)=\left{\begin{array}{cc}{0,} & {\text { if }|z| \leqslant \epsilon} \ {|z|-\epsilon,} & {\text { otherwise }}\end{array}\right.$$ [解析]：略

6.45

$$\begin{array}{ll} \underset{\boldsymbol{w}, b, \xi{i}, \hat{\xi}{i}}{\min} & \frac{1}{2}|\boldsymbol{w}|^{2}+C \sum{i=1}^{m}\left(\xi{i}+\hat{\xi}{i}\right) \ {\text { s.t. }} & {f\left(\boldsymbol{x}{i}\right)-y{i} \leqslant \epsilon+\xi{i}} \ {} & {y{i}-f\left(\boldsymbol{x}{i}\right) \leqslant \epsilon+\hat{\xi}{i}} \ {} & {\xi{i} \geqslant 0, \hat{\xi}_{i} \geqslant 0, i=1,2, \ldots, m}\end{array}$$ [解析]：略

6.46

$$\begin{array}{l}L(\boldsymbol{w}, b, \boldsymbol{\alpha}, \hat{\boldsymbol{\alpha}}, \boldsymbol{\xi}, \hat{\boldsymbol{\xi}}, \boldsymbol{\mu}, \hat{\boldsymbol{\mu}}) \ =\frac{1}{2}|\boldsymbol{w}|^{2}+C \sum\limits{i=1}^{m}\left(\xi{i}+\hat{\xi}{i}\right)-\sum\limits{i=1}^{m} \mu{i} \xi{i}-\sum\limits{i=1}^{m} \hat{\mu}{i} \hat{\xi}{i} \ +\sum\limits{i=1}^{m} \alpha{i}\left(f\left(\boldsymbol{x}{i}\right)-y{i}-\epsilon-\xi{i}\right)+\sum\limits{i=1}^{m} \hat{\alpha}{i}\left(y{i}-f\left(\boldsymbol{x}{i}\right)-\epsilon-\hat{\xi}_{i}\right)\end{array}$$ [解析]：略

6.47

$$\boldsymbol{w}=\sum{i=1}^{m}(\hat{\alpha}{i}-\alpha{i})\boldsymbol{x}{i}$$ [解析]：略

6.48

$$0=\sum{i=1}^{m}(\hat{\alpha}{i}-\alpha_{i})$$ [解析]：略

6.49

$$C=\alpha{i}+\mu{i}$$ [解析]：略

6.50

$$C=\hat{\alpha}{i}+\hat{\mu}{i}$$ [解析]：略

6.51

$$\begin{aligned} \max {\boldsymbol{\alpha}, \hat{\boldsymbol{\alpha}}} & \sum{i=1}^{m} y{i}\left(\hat{\alpha}{i}-\alpha{i}\right)-\epsilon\left(\hat{\alpha}{i}+\alpha{i}\right) \ &-\frac{1}{2} \sum{i=1}^{m} \sum{j=1}^{m}\left(\hat{\alpha}{i}-\alpha{i}\right)\left(\hat{\alpha}{j}-\alpha{j}\right) \boldsymbol{x}{i}^{\mathrm{T}} \boldsymbol{x}{j} \ \text { s.t. } & \sum{i=1}^{m}\left(\hat{\alpha}{i}-\alpha{i}\right)=0 \ & 0 \leqslant \alpha{i}, \hat{\alpha}{i} \leqslant C \end{aligned}$$ [解析]：略

6.52

$$ \left{\begin{array}{l} {\alpha{i}\left(f\left(\boldsymbol{x}{i}\right)-y{i}-\epsilon-\xi{i}\right)=0} \ {\hat{\alpha}{i}\left(y{i}-f\left(\boldsymbol{x}{i}\right)-\epsilon-\hat{\xi}{i}\right)=0} \ {\alpha{i} \hat{\alpha}{i}=0, \xi{i} \hat{\xi}{i}=0} \ {\left(C-\alpha{i}\right) \xi{i}=0,\left(C-\hat{\alpha}{i}\right) \hat{\xi}{i}=0} \end{array}\right. $$ [推导]： 将式（6.45）的约束条件全部恒等变形为小于等于0的形式可得： $$ \left{\begin{array}{l} {f\left(\boldsymbol{x}{i}\right)-y{i}-\epsilon-\xi{i} \leq 0 } \ {y{i}-f\left(\boldsymbol{x}{i}\right)-\epsilon-\hat{\xi}{i} \leq 0 } \ {-\xi{i} \leq 0} \ {-\hat{\xi}{i} \leq 0} \end{array}\right. $$ 由于以上四个约束条件的拉格朗日乘子分别为$\alpha_i,\hat{\alpha}_i,\mu_i,\hat{\mu}_i$，所以由西瓜书附录式（B.3）可知，以上四个约束条件可相应转化为以下KKT条件： $$ \left{\begin{array}{l} {\alphai\left(f\left(\boldsymbol{x}{i}\right)-y{i}-\epsilon-\xi{i} \right) = 0 } \ {\hat{\alpha}i\left(y{i}-f\left(\boldsymbol{x}{i}\right)-\epsilon-\hat{\xi}{i} \right) = 0 } \ {-\mui\xi{i} = 0 \Rightarrow \mui\xi{i} = 0 } \ {-\hat{\mu}i \hat{\xi}{i} = 0 \Rightarrow \hat{\mu}i \hat{\xi}{i} = 0 } \end{array}\right. $$ 由式（6.49）和式（6.50）可知： $$ \begin{aligned} \mu_i=C-\alpha_i \ \hat{\mu}_i=C-\hat{\alpha}_i \end{aligned} $$ 所以上述KKT条件可以进一步变形为： $$ \left{\begin{array}{l} {\alphai\left(f\left(\boldsymbol{x}{i}\right)-y{i}-\epsilon-\xi{i} \right) = 0 } \ {\hat{\alpha}i\left(y{i}-f\left(\boldsymbol{x}{i}\right)-\epsilon-\hat{\xi}{i} \right) = 0 } \ {(C-\alphai)\xi{i} = 0 } \ {(C-\hat{\alpha}i) \hat{\xi}{i} = 0 } \end{array}\right. $$ 又因为样本$(\boldsymbol{x}_i,yi)$只可能处在间隔带的某一侧，那么约束条件$f\left(\boldsymbol{x}{i}\right)-y{i}-\epsilon-\xi{i}=0$和$y{i}-f\left(\boldsymbol{x}{i}\right)-\epsilon-\hat{\xi}_{i}=0$不可能同时成立，所以$\alpha_i$和$\hat{\alpha}_i$中至少有一个为0，也即$\alpha_i\hat{\alpha}_i=0$。在此基础上再进一步分析可知，如果$\alpha_i=0$的话，那么根据约束$(C-\alphai)\xi{i} = 0$可知此时$\xi_i=0$，同理，如果$\hat{\alpha}_i=0$的话，那么根据约束$(C-\hat{\alpha}i)\hat{\xi}{i} = 0$可知此时$\hat{\xi}_i=0$，所以$\xi_i$和$\hat{\xi}i$中也是至少有一个为0，也即$\xi{i} \hat{\xi}_{i}=0$。将$\alpha_i\hat{\alpha}i=0,\xi{i} \hat{\xi}_{i}=0$整合进上述KKT条件中即可得到式（6.52）。

6.53

$$f(\boldsymbol{x})=\sum{i=1}^{m}(\hat{\alpha}{i}-\alpha{i})\boldsymbol{x}{i}^{\mathrm{T}}\boldsymbol{x}+b$$ [解析]：略

6.54

$$b=yi+\epsilon-\sum{i=1}^{m}(\hat{\alpha}{i}-\alpha{i})\boldsymbol{x}_{i}^{\mathrm{T}}\boldsymbol{x}$$ [解析]：略

6.55

$$\boldsymbol{w}=\sum{i=1}^{m}(\hat{\alpha}{i}-\alpha{i})\phi(\boldsymbol{x}{i})$$ [解析]：略

6.56

$$f(\boldsymbol{x})=\sum{i=1}^{m}(\hat{\alpha}{i}-\alpha{i})\kappa(\boldsymbol{x},\boldsymbol{x}{i})+b$$ [解析]：略

6.57

$$\min {h \in \mathbb{H}} F(h)=\Omega\left(|h|{\mathbb{H}}\right)+\ell\left(h\left(\boldsymbol{x}{1}\right), h\left(\boldsymbol{x}{2}\right), \ldots, h\left(\boldsymbol{x}_{m}\right)\right)$$ [解析]：略

6.58

$$h^{*}(\boldsymbol{x})=\sum{i=1}^{m} \alpha{i} \kappa\left(\boldsymbol{x}, \boldsymbol{x}_{i}\right)$$ [解析]：略

6.59

$$h(\boldsymbol{x})=\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x})$$ [解析]：由于书上已经交代公式(6.60)是公式(3.35)引入核函数后的形式，而公式(3.35)是二分类LDA的损失函数，并且此式为直线方程，所以此时讨论的KLDA应当也是二分类KLDA。那么此公式就类似于第3章图3.3里的$y=\boldsymbol{w}^{\mathrm{T}}\boldsymbol{x}$，表示的是二分类KLDA中所要求解的那条投影直线。

6.60

$$\max {\boldsymbol{w}} J(\boldsymbol{w})=\frac{\boldsymbol{w}^{\mathrm{T}} \mathbf{S}{b}^{\phi} \boldsymbol{w}}{\boldsymbol{w}^{\mathrm{T}} \mathbf{S}_{w}^{\phi} \boldsymbol{w}}$$ [解析]：类似于第3章的公式(3.35)。

6.61

$$\boldsymbol{\mu}{i}^{\phi}=\frac{1}{m{i}} \sum{\boldsymbol{x} \in X{i}} \phi(\boldsymbol{x})$$ [解析]：略

6.62

$$\mathbf{S}{b}^{\phi}=\left(\boldsymbol{\mu}{1}^{\phi}-\boldsymbol{\mu}{0}^{\phi}\right)\left(\boldsymbol{\mu}{1}^{\phi}-\boldsymbol{\mu}_{0}^{\phi}\right)^{\mathrm{T}}$$ [解析]：类似于第3章的公式(3.34)。

6.63

$$\mathbf{S}{w}^{\phi}=\sum{i=0}^{1} \sum{\boldsymbol{x} \in X{i}}\left(\phi(\boldsymbol{x})-\boldsymbol{\mu}{i}^{\phi}\right)\left(\phi(\boldsymbol{x})-\boldsymbol{\mu}{i}^{\phi}\right)^{\mathrm{T}}$$ [解析]：类似于第3章的公式(3.33)。

6.64

$$h(\boldsymbol{x})=\sum{i=1}^{m} \alpha{i} \kappa\left(\boldsymbol{x}, \boldsymbol{x}_{i}\right)$$ [解析]：略

6.65

$$\boldsymbol{w}=\sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right)$$ [推导]：由表示定理可知，此时二分类KLDA最终求得的投影直线方程总可以写成如下形式 $$h(\boldsymbol{x})=\sum{i=1}^{m} \alpha{i} \kappa\left(\boldsymbol{x}, \boldsymbol{x}{i}\right)$$ 又因为直线方程的固定形式为 $$h(\boldsymbol{x})=\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x})$$ 所以 $$\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x})=\sum{i=1}^{m} \alpha{i} \kappa\left(\boldsymbol{x}, \boldsymbol{x}{i}\right)$$ 将$\kappa\left(\boldsymbol{x}, \boldsymbol{x}{i}\right)=\phi(\boldsymbol{x})^{\mathrm{T}}\phi(\boldsymbol{x}i)$代入可得 $$\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x})=\sum{i=1}^{m} \alpha_{i} \phi(\boldsymbol{x})^{\mathrm{T}}\phi(\boldsymbol{x}i)$$ $$\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x})=\phi(\boldsymbol{x})^{\mathrm{T}}\cdot\sum{i=1}^{m} \alpha_{i} \phi(\boldsymbol{x}i)$$ 由于$\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x})$的计算结果为标量，而标量的转置等于其本身，所以 $$\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x})=\left(\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x})\right)^{\mathrm{T}}=\phi(\boldsymbol{x})^{\mathrm{T}}\cdot\sum{i=1}^{m} \alpha_{i} \phi(\boldsymbol{x}i)$$ $$\boldsymbol{w}^{\mathrm{T}}\phi(\boldsymbol{x})=\phi(\boldsymbol{x})^{\mathrm{T}}\boldsymbol{w}=\phi(\boldsymbol{x})^{\mathrm{T}}\cdot\sum{i=1}^{m} \alpha_{i} \phi(\boldsymbol{x}i)$$ $$\boldsymbol{w}=\sum{i=1}^{m} \alpha_{i} \phi(\boldsymbol{x}_i)$$

6.66

$$\hat{\boldsymbol{\mu}}{0}=\frac{1}{m{0}} \mathbf{K} \mathbf{1}_{0}$$ [解析]：为了详细地说明此公式的计算原理，下面首先先举例说明，然后再在例子的基础上延展出其一般形式。假设此时仅有4个样本，其中第1和第3个样本的标记为0，第2和第4个样本的标记为1，那么此时： $$m=4$$ $$m_0=2,m_1=2$$ $$X_0={\boldsymbol{x}_1,\boldsymbol{x}_3},X_1={\boldsymbol{x}_2,\boldsymbol{x}_4}$$ $$\mathbf{K}=\left[ \begin{array}{cccc} \kappa\left(\boldsymbol{x}_1, \boldsymbol{x}_1\right) & \kappa\left(\boldsymbol{x}_1, \boldsymbol{x}_2\right) & \kappa\left(\boldsymbol{x}_1, \boldsymbol{x}_3\right) & \kappa\left(\boldsymbol{x}_1, \boldsymbol{x}_4\right)\ \kappa\left(\boldsymbol{x}_2, \boldsymbol{x}_1\right) & \kappa\left(\boldsymbol{x}_2, \boldsymbol{x}_2\right) & \kappa\left(\boldsymbol{x}_2, \boldsymbol{x}_3\right) & \kappa\left(\boldsymbol{x}_2, \boldsymbol{x}_4\right)\ \kappa\left(\boldsymbol{x}_3, \boldsymbol{x}_1\right) & \kappa\left(\boldsymbol{x}_3, \boldsymbol{x}_2\right) & \kappa\left(\boldsymbol{x}_3, \boldsymbol{x}_3\right) & \kappa\left(\boldsymbol{x}_3, \boldsymbol{x}_4\right)\ \kappa\left(\boldsymbol{x}_4, \boldsymbol{x}_1\right) & \kappa\left(\boldsymbol{x}_4, \boldsymbol{x}_2\right) & \kappa\left(\boldsymbol{x}_4, \boldsymbol{x}_3\right) & \kappa\left(\boldsymbol{x}_4, \boldsymbol{x}4\right)\ \end{array} \right]\in \mathbb{R}^{4\times 4}$$ $$\mathbf{1}{0}=\left[ \begin{array}{c} 1\ 0\ 1\ 0\ \end{array} \right]\in \mathbb{R}^{4\times 1}$$ $$\mathbf{1}{1}=\left[ \begin{array}{c} 0\ 1\ 0\ 1\ \end{array} \right]\in \mathbb{R}^{4\times 1}$$ 所以 $$\hat{\boldsymbol{\mu}}{0}=\frac{1}{m{0}} \mathbf{K} \mathbf{1}{0}=\frac{1}{2}\left[ \begin{array}{c} \kappa\left(\boldsymbol{x}_1, \boldsymbol{x}_1\right)+\kappa\left(\boldsymbol{x}_1, \boldsymbol{x}_3\right)\ \kappa\left(\boldsymbol{x}_2, \boldsymbol{x}_1\right)+\kappa\left(\boldsymbol{x}_2, \boldsymbol{x}_3\right)\ \kappa\left(\boldsymbol{x}_3, \boldsymbol{x}_1\right)+\kappa\left(\boldsymbol{x}_3, \boldsymbol{x}_3\right)\ \kappa\left(\boldsymbol{x}_4, \boldsymbol{x}_1\right)+\kappa\left(\boldsymbol{x}_4, \boldsymbol{x}3\right)\ \end{array} \right]\in \mathbb{R}^{4\times 1}$$ $$\hat{\boldsymbol{\mu}}{1}=\frac{1}{m{1}} \mathbf{K} \mathbf{1}{1}=\frac{1}{2}\left[ \begin{array}{c} \kappa\left(\boldsymbol{x}_1, \boldsymbol{x}_2\right)+\kappa\left(\boldsymbol{x}_1, \boldsymbol{x}_4\right)\ \kappa\left(\boldsymbol{x}_2, \boldsymbol{x}_2\right)+\kappa\left(\boldsymbol{x}_2, \boldsymbol{x}_4\right)\ \kappa\left(\boldsymbol{x}_3, \boldsymbol{x}_2\right)+\kappa\left(\boldsymbol{x}_3, \boldsymbol{x}_4\right)\ \kappa\left(\boldsymbol{x}_4, \boldsymbol{x}_2\right)+\kappa\left(\boldsymbol{x}_4, \boldsymbol{x}4\right)\ \end{array} \right]\in \mathbb{R}^{4\times 1}$$ 根据此结果易得$\hat{\boldsymbol{\mu}}{0},\hat{\boldsymbol{\mu}}{1}$的一般形式为 $$\hat{\boldsymbol{\mu}}{0}=\frac{1}{m{0}} \mathbf{K} \mathbf{1}{0}=\frac{1}{m{0}}\left[ \begin{array}{c} \sum{\boldsymbol{x} \in X_{0}}\kappa\left(\boldsymbol{x}1, \boldsymbol{x}\right)\ \sum{\boldsymbol{x} \in X_{0}}\kappa\left(\boldsymbol{x}2, \boldsymbol{x}\right)\ \vdots\ \sum{\boldsymbol{x} \in X_{0}}\kappa\left(\boldsymbol{x}m, \boldsymbol{x}\right)\ \end{array} \right]\in \mathbb{R}^{m\times 1}$$ $$\hat{\boldsymbol{\mu}}{1}=\frac{1}{m{1}} \mathbf{K} \mathbf{1}{1}=\frac{1}{m{1}}\left[ \begin{array}{c} \sum{\boldsymbol{x} \in X_{1}}\kappa\left(\boldsymbol{x}1, \boldsymbol{x}\right)\ \sum{\boldsymbol{x} \in X_{1}}\kappa\left(\boldsymbol{x}2, \boldsymbol{x}\right)\ \vdots\ \sum{\boldsymbol{x} \in X_{1}}\kappa\left(\boldsymbol{x}_m, \boldsymbol{x}\right)\ \end{array} \right]\in \mathbb{R}^{m\times 1}$$

6.67

$$\hat{\boldsymbol{\mu}}{1}=\frac{1}{m{1}} \mathbf{K} \mathbf{1}_{1}$$ [解析]：参见公式(6.66)的解析。

6.68

$$\mathbf{M}=\left(\hat{\boldsymbol{\mu}}{0}-\hat{\boldsymbol{\mu}}{1}\right)\left(\hat{\boldsymbol{\mu}}{0}-\hat{\boldsymbol{\mu}}{1}\right)^{\mathrm{T}}$$ [解析]：略

6.69

$$\mathbf{N}=\mathbf{K} \mathbf{K}^{\mathrm{T}}-\sum{i=0}^{1} m{i} \hat{\boldsymbol{\mu}}{i} \hat{\boldsymbol{\mu}}{i}^{\mathrm{T}}$$ [解析]：略

6.70

$$\max {\boldsymbol{\alpha}} J(\boldsymbol{\alpha})=\frac{\boldsymbol{\alpha}^{\mathrm{T}} \mathbf{M} \boldsymbol{\alpha}}{\boldsymbol{\alpha}^{\mathrm{T}} \mathbf{N} \boldsymbol{\alpha}}$$ [推导]：此公式是将公式(6.65)代入公式(6.60)后推得而来的，下面给出详细地推导过程。首先将公式(6.65)代入公式(6.60)的分子可得： $$\begin{aligned} \boldsymbol{w}^{\mathrm{T}} \mathbf{S}{b}^{\phi} \boldsymbol{w}&=\left(\sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right)\right)^{\mathrm{T}}\cdot\mathbf{S}{b}^{\phi}\cdot \sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right) \ &=\sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}}\cdot\mathbf{S}{b}^{\phi}\cdot \sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right) \ \end{aligned}$$ 其中 $$\begin{aligned} \mathbf{S}{b}^{\phi} &=\left(\boldsymbol{\mu}{1}^{\phi}-\boldsymbol{\mu}{0}^{\phi}\right)\left(\boldsymbol{\mu}{1}^{\phi}-\boldsymbol{\mu}{0}^{\phi}\right)^{\mathrm{T}} \ &=\left(\frac{1}{m{1}} \sum{\boldsymbol{x} \in X{1}} \phi(\boldsymbol{x})-\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \phi(\boldsymbol{x})\right)\left(\frac{1}{m{1}} \sum{\boldsymbol{x} \in X{1}} \phi(\boldsymbol{x})-\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \phi(\boldsymbol{x})\right)^{\mathrm{T}} \ &=\left(\frac{1}{m{1}} \sum{\boldsymbol{x} \in X{1}} \phi(\boldsymbol{x})-\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \phi(\boldsymbol{x})\right)\left(\frac{1}{m{1}} \sum{\boldsymbol{x} \in X{1}} \phi(\boldsymbol{x})^{\mathrm{T}}-\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \phi(\boldsymbol{x})^{\mathrm{T}}\right) \ \end{aligned}$$ 将其代入上式可得 $$\begin{aligned} \boldsymbol{w}^{\mathrm{T}} \mathbf{S}{b}^{\phi} \boldsymbol{w}&=\sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}}\cdot\left(\frac{1}{m{1}} \sum{\boldsymbol{x} \in X{1}} \phi(\boldsymbol{x})-\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \phi(\boldsymbol{x})\right)\cdot\left(\frac{1}{m{1}} \sum{\boldsymbol{x} \in X{1}} \phi(\boldsymbol{x})^{\mathrm{T}}-\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \phi(\boldsymbol{x})^{\mathrm{T}}\right)\cdot \sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right) \ &=\left(\frac{1}{m{1}} \sum{\boldsymbol{x} \in X{1}}\sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}} \phi(\boldsymbol{x})-\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}}\phi(\boldsymbol{x})\right)\cdot\left(\frac{1}{m{1}} \sum{\boldsymbol{x} \in X{1}} \sum{i=1}^{m} \alpha{i} \phi(\boldsymbol{x})^{\mathrm{T}}\phi\left(\boldsymbol{x}{i}\right)-\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \sum{i=1}^{m} \alpha{i} \phi(\boldsymbol{x})^{\mathrm{T}}\phi\left(\boldsymbol{x}{i}\right)\right) \ \end{aligned}$$ 由于$\kappa\left(\boldsymbol{x}_i, \boldsymbol{x}\right)=\phi(\boldsymbol{x}_i)^{\mathrm{T}}\phi(\boldsymbol{x})$为标量，所以其转置等于本身，也即$\kappa\left(\boldsymbol{x}_i, \boldsymbol{x}\right)=\phi(\boldsymbol{x}_i)^{\mathrm{T}}\phi(\boldsymbol{x})=\left(\phi(\boldsymbol{x}_i)^{\mathrm{T}}\phi(\boldsymbol{x})\right)^{\mathrm{T}}=\phi(\boldsymbol{x})^{\mathrm{T}}\phi(\boldsymbol{x}_i)=\kappa\left(\boldsymbol{x}i, \boldsymbol{x}\right)^{\mathrm{T}}$，将其代入上式可得 $$\boldsymbol{w}^{\mathrm{T}} \mathbf{S}{b}^{\phi} \boldsymbol{w}=\left(\frac{1}{m{1}} \sum{i=1}^{m}\sum{\boldsymbol{x} \in X{1}}\alpha_{i} \kappa\left(\boldsymbol{x}i, \boldsymbol{x}\right)-\frac{1}{m{0}} \sum{i=1}^{m} \sum{\boldsymbol{x} \in X{0}} \alpha{i} \kappa\left(\boldsymbol{x}i, \boldsymbol{x}\right)\right)\cdot\left(\frac{1}{m{1}} \sum{i=1}^{m}\sum{\boldsymbol{x} \in X{1}} \alpha{i} \kappa\left(\boldsymbol{x}i, \boldsymbol{x}\right)-\frac{1}{m{0}}\sum{i=1}^{m} \sum{\boldsymbol{x} \in X{0}} \alpha{i} \kappa\left(\boldsymbol{x}_i, \boldsymbol{x}\right)\right)$$ 令$\boldsymbol{\alpha}=(\alpha_1;\alpha_2;...;\alpham)^{\mathrm{T}}\in \mathbb{R}^{m\times 1}$，同时结合公式(6.66)的解析中得到的$\hat{\boldsymbol{\mu}}{0},\hat{\boldsymbol{\mu}}{1}$的一般形式，上式可以化简为 $$\begin{aligned} \boldsymbol{w}^{\mathrm{T}} \mathbf{S}{b}^{\phi} \boldsymbol{w}&=\left(\boldsymbol{\alpha}^{\mathrm{T}}\hat{\boldsymbol{\mu}}{1}-\boldsymbol{\alpha}^{\mathrm{T}}\hat{\boldsymbol{\mu}}{0}\right)\cdot\left(\hat{\boldsymbol{\mu}}{1}^{\mathrm{T}}\boldsymbol{\alpha}-\hat{\boldsymbol{\mu}}{0}^{\mathrm{T}}\boldsymbol{\alpha}\right)\ &=\boldsymbol{\alpha}^{\mathrm{T}}\cdot\left(\hat{\boldsymbol{\mu}}{1}-\hat{\boldsymbol{\mu}}{0}\right)\cdot\left(\hat{\boldsymbol{\mu}}{1}^{\mathrm{T}}-\hat{\boldsymbol{\mu}}{0}^{\mathrm{T}}\right)\cdot\boldsymbol{\alpha}\ &=\boldsymbol{\alpha}^{\mathrm{T}}\cdot\left(\hat{\boldsymbol{\mu}}{1}-\hat{\boldsymbol{\mu}}{0}\right)\cdot\left(\hat{\boldsymbol{\mu}}{1}-\hat{\boldsymbol{\mu}}{0}\right)^{\mathrm{T}}\cdot\boldsymbol{\alpha}\ &=\boldsymbol{\alpha}^{\mathrm{T}} \mathbf{M} \boldsymbol{\alpha}\ \end{aligned}$$ 以上便是公式(6.70)分子部分的推导，下面继续推导公式(6.70)的分母部分。将公式(6.65)代入公式(6.60)的分母可得： $$\begin{aligned} \boldsymbol{w}^{\mathrm{T}} \mathbf{S}{w}^{\phi} \boldsymbol{w}&=\left(\sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right)\right)^{\mathrm{T}}\cdot\mathbf{S}{w}^{\phi}\cdot \sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right) \ &=\sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}}\cdot\mathbf{S}{w}^{\phi}\cdot \sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right) \ \end{aligned}$$ 其中 $$\begin{aligned} \mathbf{S}{w}^{\phi}&=\sum{i=0}^{1} \sum{\boldsymbol{x} \in X{i}}\left(\phi(\boldsymbol{x})-\boldsymbol{\mu}{i}^{\phi}\right)\left(\phi(\boldsymbol{x})-\boldsymbol{\mu}{i}^{\phi}\right)^{\mathrm{T}} \ &=\sum{i=0}^{1} \sum{\boldsymbol{x} \in X{i}}\left(\phi(\boldsymbol{x})-\boldsymbol{\mu}{i}^{\phi}\right)\left(\phi(\boldsymbol{x})^{\mathrm{T}}-\left(\boldsymbol{\mu}{i}^{\phi}\right)^{\mathrm{T}}\right) \ &=\sum{i=0}^{1} \sum{\boldsymbol{x} \in X{i}}\left(\phi(\boldsymbol{x})\phi(\boldsymbol{x})^{\mathrm{T}}-\phi(\boldsymbol{x})\left(\boldsymbol{\mu}{i}^{\phi}\right)^{\mathrm{T}}-\boldsymbol{\mu}{i}^{\phi}\phi(\boldsymbol{x})^{\mathrm{T}}+\boldsymbol{\mu}{i}^{\phi}\left(\boldsymbol{\mu}{i}^{\phi}\right)^{\mathrm{T}}\right) \ \end{aligned}$$ 由于$\phi(\boldsymbol{x})\left(\boldsymbol{\mu}{i}^{\phi}\right)^{\mathrm{T}}$的计算结果为标量，所以$\phi(\boldsymbol{x})\left(\boldsymbol{\mu}{i}^{\phi}\right)^{\mathrm{T}}=\left[\phi(\boldsymbol{x})\left(\boldsymbol{\mu}{i}^{\phi}\right)^{\mathrm{T}}\right]^{\mathrm{T}}=\boldsymbol{\mu}{i}^{\phi}\phi(\boldsymbol{x})^{\mathrm{T}}$，将其代回上式可得 $$\begin{aligned} \mathbf{S}{w}^{\phi}&=\sum{i=0}^{1} \sum{\boldsymbol{x} \in X{i}}\left(\phi(\boldsymbol{x})\phi(\boldsymbol{x})^{\mathrm{T}}-2\boldsymbol{\mu}{i}^{\phi}\phi(\boldsymbol{x})^{\mathrm{T}}+\boldsymbol{\mu}{i}^{\phi}\left(\boldsymbol{\mu}{i}^{\phi}\right)^{\mathrm{T}}\right) \ &=\sum{i=0}^{1} \sum{\boldsymbol{x} \in X{i}}\phi(\boldsymbol{x})\phi(\boldsymbol{x})^{\mathrm{T}}-\sum{i=0}^{1} \sum{\boldsymbol{x} \in X{i}}2\boldsymbol{\mu}{i}^{\phi}\phi(\boldsymbol{x})^{\mathrm{T}}+\sum{i=0}^{1} \sum{\boldsymbol{x} \in X{i}}\boldsymbol{\mu}{i}^{\phi}\left(\boldsymbol{\mu}{i}^{\phi}\right)^{\mathrm{T}} \ &=\sum{\boldsymbol{x} \in D}\phi(\boldsymbol{x})\phi(\boldsymbol{x})^{\mathrm{T}}-2\boldsymbol{\mu}{0}^{\phi}\sum{\boldsymbol{x} \in X{0}}\phi(\boldsymbol{x})^{\mathrm{T}}-2\boldsymbol{\mu}{1}^{\phi}\sum{\boldsymbol{x} \in X{1}}\phi(\boldsymbol{x})^{\mathrm{T}}+\sum{\boldsymbol{x} \in X{0}}\boldsymbol{\mu}{0}^{\phi}\left(\boldsymbol{\mu}{0}^{\phi}\right)^{\mathrm{T}}+\sum{\boldsymbol{x} \in X{1}}\boldsymbol{\mu}{1}^{\phi}\left(\boldsymbol{\mu}{1}^{\phi}\right)^{\mathrm{T}} \ &=\sum{\boldsymbol{x} \in D}\phi(\boldsymbol{x})\phi(\boldsymbol{x})^{\mathrm{T}}-2m0\boldsymbol{\mu}{0}^{\phi}\left(\boldsymbol{\mu}_{0}^{\phi}\right)^{\mathrm{T}}-2m1\boldsymbol{\mu}{1}^{\phi}\left(\boldsymbol{\mu}_{1}^{\phi}\right)^{\mathrm{T}}+m0 \boldsymbol{\mu}{0}^{\phi}\left(\boldsymbol{\mu}_{0}^{\phi}\right)^{\mathrm{T}}+m1 \boldsymbol{\mu}{1}^{\phi}\left(\boldsymbol{\mu}{1}^{\phi}\right)^{\mathrm{T}} \ &=\sum{\boldsymbol{x} \in D}\phi(\boldsymbol{x})\phi(\boldsymbol{x})^{\mathrm{T}}-m0\boldsymbol{\mu}{0}^{\phi}\left(\boldsymbol{\mu}_{0}^{\phi}\right)^{\mathrm{T}}-m1\boldsymbol{\mu}{1}^{\phi}\left(\boldsymbol{\mu}{1}^{\phi}\right)^{\mathrm{T}}\ \end{aligned}$$ 再将此式代回$\boldsymbol{w}^{\mathrm{T}} \mathbf{S}{b}^{\phi} \boldsymbol{w}$可得 $$\begin{aligned} \boldsymbol{w}^{\mathrm{T}} \mathbf{S}{w}^{\phi} \boldsymbol{w}&=\sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}}\cdot\mathbf{S}{w}^{\phi}\cdot \sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right) \ &=\sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}}\cdot\left(\sum{\boldsymbol{x} \in D}\phi(\boldsymbol{x})\phi(\boldsymbol{x})^{\mathrm{T}}-m0\boldsymbol{\mu}{0}^{\phi}\left(\boldsymbol{\mu}_{0}^{\phi}\right)^{\mathrm{T}}-m1\boldsymbol{\mu}{1}^{\phi}\left(\boldsymbol{\mu}{1}^{\phi}\right)^{\mathrm{T}}\right)\cdot \sum{i=1}^{m} \alpha{i} \phi\left(\boldsymbol{x}{i}\right) \ &=\sum{i=1}^{m}\sum{j=1}^{m}\sum{\boldsymbol{x} \in D}\alpha{i} \phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}}\phi(\boldsymbol{x})\phi(\boldsymbol{x})^{\mathrm{T}}\alpha{j} \phi\left(\boldsymbol{x}{j}\right)-\sum{i=1}^{m}\sum{j=1}^{m}\alpha{i} \phi\left(\boldsymbol{x}_{i}\right)^{\mathrm{T}}m0\boldsymbol{\mu}{0}^{\phi}\left(\boldsymbol{\mu}{0}^{\phi}\right)^{\mathrm{T}}\alpha{j} \phi\left(\boldsymbol{x}{j}\right)-\sum{i=1}^{m}\sum{j=1}^{m}\alpha{i} \phi\left(\boldsymbol{x}_{i}\right)^{\mathrm{T}}m1\boldsymbol{\mu}{1}^{\phi}\left(\boldsymbol{\mu}{1}^{\phi}\right)^{\mathrm{T}}\alpha{j} \phi\left(\boldsymbol{x}{j}\right) \ \end{aligned}$$ 其中，第1项可化简为 $$\begin{aligned} \sum{i=1}^{m}\sum{j=1}^{m}\sum{\boldsymbol{x} \in D}\alpha{i} \phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}}\phi(\boldsymbol{x})\phi(\boldsymbol{x})^{\mathrm{T}}\alpha{j} \phi\left(\boldsymbol{x}{j}\right)&=\sum{i=1}^{m}\sum{j=1}^{m}\sum{\boldsymbol{x} \in D}\alpha{i} \alpha_{j}\kappa\left(\boldsymbol{x}_i, \boldsymbol{x}\right)\kappa\left(\boldsymbol{x}j, \boldsymbol{x}\right)\ &=\boldsymbol{\alpha}^{\mathrm{T}} \mathbf{K} \mathbf{K}^{\mathrm{T}} \boldsymbol{\alpha} \end{aligned}$$ 第2项可化简为 $$\begin{aligned} \sum{i=1}^{m}\sum{j=1}^{m}\alpha{i} \phi\left(\boldsymbol{x}_{i}\right)^{\mathrm{T}}m0\boldsymbol{\mu}{0}^{\phi}\left(\boldsymbol{\mu}{0}^{\phi}\right)^{\mathrm{T}}\alpha{j} \phi\left(\boldsymbol{x}_{j}\right)&=m0\sum{i=1}^{m}\sum{j=1}^{m}\alpha{i}\alpha{j}\phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}}\boldsymbol{\mu}{0}^{\phi}\left(\boldsymbol{\mu}{0}^{\phi}\right)^{\mathrm{T}} \phi\left(\boldsymbol{x}_{j}\right)\ &=m0\sum{i=1}^{m}\sum{j=1}^{m}\alpha{i}\alpha{j}\phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}}\left[\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \phi(\boldsymbol{x})\right]\left[\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \phi(\boldsymbol{x})\right]^{\mathrm{T}} \phi\left(\boldsymbol{x}_{j}\right)\ &=m0\sum{i=1}^{m}\sum{j=1}^{m}\alpha{i}\alpha{j}\left[\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \phi\left(\boldsymbol{x}{i}\right)^{\mathrm{T}}\phi(\boldsymbol{x})\right]\left[\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \phi(\boldsymbol{x})^{\mathrm{T}}\phi\left(\boldsymbol{x}_{j}\right)\right] \ &=m0\sum{i=1}^{m}\sum{j=1}^{m}\alpha{i}\alpha{j}\left[\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \kappa\left(\boldsymbol{x}i, \boldsymbol{x}\right)\right]\left[\frac{1}{m{0}} \sum{\boldsymbol{x} \in X{0}} \kappa\left(\boldsymbol{x}_j, \boldsymbol{x}\right)\right] \ &=m0\boldsymbol{\alpha}^{\mathrm{T}} \hat{\boldsymbol{\mu}}{0} \hat{\boldsymbol{\mu}}{0}^{\mathrm{T}} \boldsymbol{\alpha} \end{aligned}$$ 同理可得，第3项可化简为 $$\sum{i=1}^{m}\sum{j=1}^{m}\alpha{i} \phi\left(\boldsymbol{x}_{i}\right)^{\mathrm{T}}m1\boldsymbol{\mu}{1}^{\phi}\left(\boldsymbol{\mu}{1}^{\phi}\right)^{\mathrm{T}}\alpha{j} \phi\left(\boldsymbol{x}_{j}\right)=m1\boldsymbol{\alpha}^{\mathrm{T}} \hat{\boldsymbol{\mu}}{1} \hat{\boldsymbol{\mu}}{1}^{\mathrm{T}} \boldsymbol{\alpha}$$ 将上述三项的化简结果代回再将此式代回$\boldsymbol{w}^{\mathrm{T}} \mathbf{S}{b}^{\phi} \boldsymbol{w}$可得 $$\begin{aligned} \boldsymbol{w}^{\mathrm{T}} \mathbf{S}_{b}^{\phi} \boldsymbol{w}&=\boldsymbol{\alpha}^{\mathrm{T}} \mathbf{K} \mathbf{K}^{\mathrm{T}} \boldsymbol{\alpha}-m0\boldsymbol{\alpha}^{\mathrm{T}} \hat{\boldsymbol{\mu}}{0} \hat{\boldsymbol{\mu}}_{0}^{\mathrm{T}} \boldsymbol{\alpha}-m1\boldsymbol{\alpha}^{\mathrm{T}} \hat{\boldsymbol{\mu}}{1} \hat{\boldsymbol{\mu}}_{1}^{\mathrm{T}} \boldsymbol{\alpha}\ &=\boldsymbol{\alpha}^{\mathrm{T}} \cdot\left(\mathbf{K} \mathbf{K}^{\mathrm{T}} -m0\hat{\boldsymbol{\mu}}{0} \hat{\boldsymbol{\mu}}_{0}^{\mathrm{T}} -m1\hat{\boldsymbol{\mu}}{1} \hat{\boldsymbol{\mu}}{1}^{\mathrm{T}} \right)\cdot\boldsymbol{\alpha}\ &=\boldsymbol{\alpha}^{\mathrm{T}} \cdot\left(\mathbf{K} \mathbf{K}^{\mathrm{T}}-\sum{i=0}^{1} m{i} \hat{\boldsymbol{\mu}}{i} \hat{\boldsymbol{\mu}}_{i}^{\mathrm{T}} \right)\cdot\boldsymbol{\alpha}\ &=\boldsymbol{\alpha}^{\mathrm{T}} \mathbf{N}\boldsymbol{\alpha}\ \end{aligned}$$

附录

①KKT条件^[1]

对于一般地约束优化问题 $$\begin{array}{ll} {\min } & {f(\boldsymbol x)} \ {\text {s.t.}} & {g{i}(\boldsymbol x) \leq 0 \quad(i=1, \ldots, m)} \ {} & {h{j}(\boldsymbol x)=0 \quad(j=1, \ldots, n)} \end{array}$$ 其中，自变量$\boldsymbol x\in \mathbb{R}^n$。设$f(\boldsymbol x),g_i(\boldsymbol x),h_j(\boldsymbol x)$具有连续的一阶偏导数，$\boldsymbol x^*$是优化问题的局部可行解。若该优化问题满足任意一个约束限制条件（constraint qualifications or regularity conditions）^[2]，则一定存在$\boldsymbol \mu^=(\mu_1^,\mu_2^,...,\mu_m^)^T,\boldsymbol \lambda^=(\lambda_1^,\lambda_2^,...,\lambda_n^)^T,$使得 $$\left{ \begin{aligned} & \nabla{\boldsymbol x} L(\boldsymbol x^* ,\boldsymbol \mu^* ,\boldsymbol \lambda^* )=\nabla f(\boldsymbol x^* )+\sum{i=1}^{m}\mu_i^* \nabla gi(\boldsymbol x^* )+\sum{j=1}^{n}\lambda_j^* \nabla h_j(\boldsymbol x^)=0 &(1) \ & h_j(\boldsymbol x^)=0 &(2) \ & g_i(\boldsymbol x^) \leq 0 &(3) \ & \mu_i^ \geq 0 &(4)\ & \mu_i^* gi(\boldsymbol x^*)=0 &(5) \end{aligned} \right. $$ 其中$L(\boldsymbol x,\boldsymbol \mu,\boldsymbol \lambda)$为拉格朗日函数 $$L(\boldsymbol x,\boldsymbol \mu,\boldsymbol \lambda)=f(\boldsymbol x)+\sum{i=1}^{m}\mu_i gi(\boldsymbol x)+\sum{j=1}^{n}\lambda_j h_j(\boldsymbol x)$$ 以上5条即为KKT条件，严格数学证明参见参考文献[1]的§ 4.2.1。

参考文献

[1] 王燕军. 《最优化基础理论与方法》
[2] https://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions#Regularity_conditions_(or_constraint_qualifications)
[3] 王书宁 译.《凸优化》