给定一个生成序列“The cat sat on the mat”和两个参考序列“The cat is on the mat”“The bird sat on the bush”分别计算BLEU-N和ROUGE-N得分(N=1或N =2时).

• 生成序列 $\mathbf{x}=\text{the cat sat on the mat}$
• 参考序列
  • ${\mathbf{s}}^{(1)}=\text{the cat is on the mat}$
  • ${\mathbf{s}}^{(2)}=\text{the bird sat on the bush}$

一、BLEU-N得分（Bilingual Evaluation Understudy）

  【深度学习】序列生成模型（五）：评价方法计算实例：计算BLEU-N得分

二、ROUGE-N得分（Recall-Oriented Understudy for Gisting Evaluation）

1. 定义

  设 $\mathbf{x}$ 为从模型分布 $p_{\theta }$ 中生成的一个候选序列，${\mathbf{s}}^{(1)},\cdots ,{\mathbf{s}}^{(\mathbf{K})}$ 为从真实数据分布中采样得到的一组参考序列，$\mathcal{W}$ 为从参考序列中提取N元组合的集合，ROUGE-N算法的定义为：

$$\text{ROUGE-N}(\mathbf{x})=\frac{{\sum }_{k=1}^K{\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(k)}))}{{\sum }_{k=1}^K{\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(k))}}$$

其中 $c_w(\mathbf{x})$ 是N元组合 $w$ 在生成序列 $\mathbf{x}$ 中出现的次数，$c_w({\mathbf{s}}^{(k))})$ 是N元组合 $w$ 在参考序列 ${\mathbf{s}}^{(k)}$ 中出现的次数。

2. 计算

N=1

• 生成序列 $\mathbf{x}=\text{the cat sat on the mat}$
• 参考序列
  • ${\mathbf{s}}^{(1)}=\text{the cat is on the mat}$
  • ${\mathbf{s}}^{(2)}=\text{the bird sat on the bush}$
• $\mathcal{W}=\text{ the, cat, is, on, mat, bird, sat, bush }$

$w$	$c_w(\mathbf{x})$	$c_w({\mathbf{s}}^{(1)})$	$c_w({\mathbf{s}}^{(2)})$	$\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(1)})$	$\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(2)})$
the	2	2	2	2	2
cat	1	1	0	1	0
is	0	1	0	0	0
on	1	1	1	1	1
mat	1	1	0	1	0
bird	0	0	1	0	0
sat	1	0	1	0	1
bush	0	0	1	0	0

• 分子${\sum }_{k=1}^K{\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(k)}))$
  • ${\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(1)})=2+1+0+1+1+0+0+0=5$
  • ${\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(2)})=2+0+0+1+0+0+1+0=4$
  • ${\sum }_{k=1}^2{\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(k)}))={\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(1)}))+{\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(2)}))=5+4=9$
• 分母${\sum }_{k=1}^K{\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(k)})$
  • ${\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(1))}=6$
  • ${\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(2)})=6$
  • ${\sum }_{k=1}^2{\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(k)})={\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(1)})+{\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(2)})=12$
• $\text{ROUGE-N}(\mathbf{x})=\frac{{\sum }_{k=1}^K{\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(k)}))}{{\sum }_{k=1}^K{\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(k))}}=\frac{5+4}{6+6}=\frac{9}{12}=0.75$

N=2

• 生成序列 $\mathbf{x}=\text{the cat sat on the mat}$
• 参考序列
  • ${\mathbf{s}}^{(1)}=\text{the cat is on the mat}$
  • ${\mathbf{s}}^{(2)}=\text{the bird sat on the bush}$
• $\mathcal{W}=\text{ the cat, cat is, is on, on the, the mat, the bird, bird sat, sat on, the bush }$

$w$	$c_w(\mathbf{x})$	$c_w({\mathbf{s}}^{(1)})$	$c_w({\mathbf{s}}^{(2)})$	$\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(1)})$	$\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(2)})$
the cat	1	1	0	1	0
cat is	0	1	0	0	0
is on	0	1	0	0	0
on the	1	1	1	1	1
the mat	1	1	0	0	0
the bird	0	0	1	0	0
bird sat	0	0	1	0	0
sat on	1	0	1	1	1
the bush	0	0	1	0	0

• 分子${\sum }_{k=1}^K{\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(k)}))$
  • ${\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(1)})=3$
  • ${\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(2)})=2$
  • ${\sum }_{k=1}^2{\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(k)}))={\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(1)}))+{\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(2)}))=3+2=5$
• 分母${\sum }_{k=1}^K{\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(k)})$
  • ${\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(1))}=5$
  • ${\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(2)})=5$
  • ${\sum }_{k=1}^2{\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(k)})={\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(1)})+{\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(2)})=10$
• $\text{ROUGE-N}(\mathbf{x})=\frac{{\sum }_{k=1}^K{\sum }_{w\in \mathcal{W}}\min (c_w(\mathbf{x}),c_w({\mathbf{s}}^{(k)}))}{{\sum }_{k=1}^K{\sum }_{w\in \mathcal{W}}{c}_w({\mathbf{s}}^{(k))}}=\frac{3+2}{5+5}=\frac{5}{10}=0.5$

3. 程序

main_string = 'the cat sat on the mat'
string1 = 'the cat is on the mat'
string2 = 'the bird sat on the bush'

words = list(set(string1.split(' ')+string2.split(' ')))  # 去除重复元素

total_occurrences, matching_occurrences = 0, 0
for word in words:
    matching_occurrences += min(main_string.count(word), string1.count(word)) + min(main_string.count(word), string2.count(word))
    total_occurrences += string1.count(word) + string2.count(word)

print(matching_occurrences / total_occurrences)

bigrams = []
split1 = string1.split(' ')
for i in range(len(split1) - 1):
    bigrams.append(split1[i] + ' ' + split1[i + 1])

split2 = string2.split(' ')
for i in range(len(split2) - 1):
    bigrams.append(split2[i] + ' ' + split2[i + 1])

bigrams = list(set(bigrams))  # 去除重复元素

total_occurrences, matching_occurrences = 0, 0
for bigram in bigrams:
    matching_occurrences += min(main_string.count(bigram), string1.count(bigram)) + min(main_string.count(bigram), string2.count(bigram))
    total_occurrences += string1.count(bigram) + string2.count(bigram)

print(matching_occurrences / total_occurrences)

输出：

0.75
0.5