- 贝叶斯决策
最小风险：
min R ( α i ∣ x ) = ∑ j = 1 c λ ( α i ∣ ω j ) P ( ω j ∣ x ) \min R\left(\alpha_i \mid \mathrm{x}\right)=\sum_{j=1}^c \lambda\left(\alpha_i \mid \omega_j\right) P\left(\omega_j \mid \mathrm{x}\right) minR(αi∣x)=j=1∑cλ(αi∣ωj)P(ωj∣x)
最小错误率（最大后验概率）： max P ( ω i ∣ x ) \max P\left(\omega_i \mid \mathbf{x}\right) maxP(ωi∣x) - 独立二值特征（binary features）
– 独立
p ( x ) = p ( x 1 x 2 ⋯ x d ) = ∏ i = 1 d p ( x i ) p(\mathbf{x})=p\left(x_1 x_2 \cdots x_d\right)=\prod_{i=1}^d p\left(x_i\right) p(x)=p(x1x2⋯xd)=i=1∏dp(xi)
– 二值，这边假设只有两类，样本是 d d d维的
p i = Prob ( x i = 1 ∣ ω 1 ) i = 1 , … , d q i = Prob ( x i = 1 ∣ ω 2 ) i = 1 , … , d \begin{aligned} & p_i=\operatorname{Prob}\left(x_i=1 \mid \omega_1\right) \quad i=1, \ldots, d \\ & q_i=\operatorname{Prob}\left(x_i=1 \mid \omega_2\right) \quad i=1, \ldots, d \end{aligned} pi=Prob(xi=1∣ω1)i=1,…,dqi=Prob(xi=1∣ω2)i=1,…,d
– 类条件概率密度
P ( x ∣ ω 1 ) = ∏ i = 1 d p i x i ( 1 − p i ) 1 − x i P ( x ∣ ω 2 ) = ∏ i = 1 d q i x i ( 1 − q i ) 1 − x i \begin{aligned} & P\left(\mathbf{x} \mid \omega_1\right)=\prod_{i=1}^d p_i^{x_i}\left(1-p_i\right)^{1-x_i} \\ & P\left(\mathbf{x} \mid \omega_2\right)=\prod_{i=1}^d q_i^{x_i}\left(1-q_i\right)^{1-x_i} \end{aligned} P(x∣ω1)=i=1∏dpixi(1−pi)1−xiP(x∣ω2)=i=1∏dqixi(1−qi)1−xi
x i x_i xi是样本的第 i i i维特征，取值 0 , 1 0,1 0,1。如果 x i = 0 x_i=0 xi=0，则 p i x i ( 1 − p i ) 1 − x i = 1 − p i p_i^{x_i}\left(1-p_i\right)^{1-x_i}=1-p_i pixi(1−pi)1−xi=1−pi；如果 x i = 1 x_i=1 xi=1，则 p i x i ( 1 − p i ) 1 − x i = p i p_i^{x_i}\left(1-p_i\right)^{1-x_i}= p_i pixi(1−pi)1−xi=pi
– 似然比
P ( x ∣ ω 1 ) P ( x ∣ ω 2 ) = ∏ i = 1 d ( p i q i ) x i ( 1 − p i 1 − q i ) 1 − x i \frac{P\left(\mathrm{x} \mid \omega_1\right)}{P\left(\mathrm{x} \mid \omega_2\right)}=\prod_{i=1}^d\left(\frac{p_i}{q_i}\right)^{x_i}\left(\frac{1-p_i}{1-q_i}\right)^{1-x_i} P(x∣ω2)P(x∣ω1)=i=1∏d(qipi)xi(1−qi1−pi)1−xi
– 判别函数
g ( x ) = log p ( x ∣ ω 1 ) P ( ω 1 ) p ( x ∣ ω 2 ) P ( ω 2 ) = ∑ i = 1 d [ x i ln p i q i + ( 1 − x i ) ln 1 − p i 1 − q i ] + ln P ( ω 1 ) P ( ω 2 ) g(\mathbf{x})=\log \frac{p\left(\mathbf{x} \mid \omega_1\right) P\left(\omega_1\right)}{p\left(\mathbf{x} \mid \omega_2\right) P\left(\omega_2\right)}=\sum_{i=1}^d\left[x_i \ln \frac{p_i}{q_i}+\left(1-x_i\right) \ln \frac{1-p_i}{1-q_i}\right]+\ln \frac{P\left(\omega_1\right)}{P\left(\omega_2\right)} g(x)=logp(x∣ω2)P(ω2)p(x∣ω1)P(ω1)=i=1∑d[xilnqipi+(1−xi)ln1−qi1−pi]+lnP(ω2)P(ω1)
上式为线性判别函数
g ( x ) = ∑ i = 1 d w i x i + w 0 g(\mathbf{x})=\sum_{i=1}^d w_i x_i+w_0 g(x)=i=1∑dwixi+w0
w i = ln p i ( 1 − q i ) q i ( 1 − p i ) i = 1 , … , d w 0 = ∑ i = 1 d ln 1 − p i 1 − q i + ln P ( ω 1 ) P ( ω 2 ) \begin{aligned} & w_i=\ln \frac{p_i\left(1-q_i\right)}{q_i\left(1-p_i\right)} \quad i=1, \ldots, d \\ & w_0=\sum_{i=1}^d \ln \frac{1-p_i}{1-q_i}+\ln \frac{P\left(\omega_1\right)}{P\left(\omega_2\right)} \end{aligned} wi=lnqi(1−pi)pi(1−qi)i=1,…,dw0=i=1∑dln1−qi1−pi+lnP(ω2)P(ω1)
【例一】： 若
P
(
ω
1
)
=
0.5
,
P
(
ω
2
)
=
0.5
P\left(\omega_1\right)=0.5, P\left(\omega_2\right)=0.5
P(ω1)=0.5,P(ω2)=0.5，
p
i
=
0.8
,
q
i
=
0.5
,
i
=
1
,
2
,
3
p_i=0.8, q_i=0.5, i=1,2,3
pi=0.8,qi=0.5,i=1,2,3
g
(
x
)
=
∑
i
=
1
d
w
i
x
i
+
w
0
w
i
=
ln
.
8
(
1
−
.
5
)
.
5
(
1
−
.
8
)
=
1.3863
w
0
=
∑
i
=
1
3
ln
1
−
.
8
1
−
.
5
+
ln
.
5
.
5
=
−
2.7489
\begin{aligned} & g(\mathbf{x})=\sum_{i=1}^d w_i x_i+w_0 \\ & w_i=\ln \frac{.8(1-.5)}{.5(1-.8)}=1.3863 \\ & w_0=\sum_{i=1}^3 \ln \frac{1-.8}{1-.5}+\ln \frac{.5}{.5}=-2.7489 \end{aligned}
g(x)=i=1∑dwixi+w0wi=ln.5(1−.8).8(1−.5)=1.3863w0=i=1∑3ln1−.51−.8+ln.5.5=−2.7489
【例二】： 若
P
(
ω
1
)
=
0.5
,
P
(
ω
2
)
=
0.5
P\left(\omega_1\right)=0.5, P\left(\omega_2\right)=0.5
P(ω1)=0.5,P(ω2)=0.5，
p
1
=
p
2
=
0.8
,
p
3
=
0.5
;
q
i
=
0.5
,
i
=
1
,
2
,
3
p_1=p_2=0.8, p_3=0.5 ; q_i=0.5, i=1,2,3
p1=p2=0.8,p3=0.5;qi=0.5,i=1,2,3
w
1
=
w
2
=
ln
.
8
(
1
−
.
5
)
.
5
(
1
−
.
8
)
=
1.3863
w
3
=
0
w
0
=
2
ln
1
−
0.8
1
−
0.5
=
−
1.8326
\begin{aligned} & w_1=w_2=\ln \frac{.8(1-.5)}{.5(1-.8)}=1.3863 \\ & w_3=0 \\ & \mathrm{w}_0=2 \ln \frac{1-0.8}{1-0.5}=-1.8326 \end{aligned}
w1=w2=ln.5(1−.8).8(1−.5)=1.3863w3=0w0=2ln1−0.51−0.8=−1.8326
