算法

参考：Greedy search与beam search

在下面会用到解码的方法选择

• greedy_search：贪心搜索，贪心搜索是一种来自计算机科学的算法，生成第一个词的分布以后，它将会根据你的条件语言模型挑选出最有可能的第一个词进入你的机器翻译模型中，在挑选出第一个词之后它将会继续挑选出最有可能的第二个词，然后继续挑选第三个最有可能的词，这种算法就叫做贪心搜索。
• beam_search：集束搜索，对于语音识别，给定一个输入的语音片段，你不会想要一个随机的文本翻译结果，你想要最好的，最接近原意的翻译结果，集束搜索就是解决这个最常用的算法。

一、环境

1.1 硬件环境–RV1126开发板

• 硬件：RV-1126

[root@RV1126_RV1109:/mnt/nfs/sherpa]# uname -r
4.19.111
[root@RV1126_RV1109:/mnt/nfs/sherpa]# uname -m
armv7l
[root@RV1126_RV1109:/mnt/nfs/sherpa]# cat /proc/cpuinfo
processor       : 0
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 70.52
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 1
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 70.52
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 2
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 70.52
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

processor       : 3
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 70.52
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

Hardware        : Generic DT based system
Revision        : 0000
Serial          : 3ea3a71e7adec418

1.2 交叉编译器

编辑/etc/profile文件将SDK的交叉编译器（gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf）添加到环境。

export PATH=/home/liefyuan/rv1126/rp_rv1126_sdk/prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/bin:$PATH

1.3 需要Cmake版本大于3.1以上

$ cmake -version
cmake version 3.26.0

CMake suite maintained and supported by Kitware (kitware.com/cmake).

二、交叉编译sherpa

2.1 下载sherpa

$ git clone https://github.com/k2-fsa/sherpa-ncnn

2.2 编译sherpa

$ cd sherpa-ncnn/
$ ./build-arm-linux-gnueabihf.sh

编译成功后会生成两个文件：

~/rv1126/kaldi/sherpa-ncnn$ ls -lh build-arm-linux-gnueabihf/install/bin/
total 4.1M
-rwxr-xr-x 1 liefyuan liefyuan 2.1M Apr 25 23:26 sherpa-ncnn
-rwxr-xr-x 1 liefyuan liefyuan 2.1M Apr 25 23:26 sherpa-ncnn-alsa

将这两个可执行文件拷贝到开发板上去。

2.3 运行测试

运行sherpa-ncnn

[root@RV1126_RV1109:/mnt/nfs/sherpa]# ./sherpa-ncnn
./sherpa-ncnn: error while loading shared libraries: libgomp.so.1: cannot open shared object file: No such file or directory

运行提示缺少libgomp.so.1
缺少的文件在SDK里面找一下：

liefyuan@ubuntu:~/rv1126/rp_rv1126_sdk$ find ./ -name libgomp*
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/share/info/libgomp.info
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/libc/usr/lib/libgomp.so.1
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/libc/usr/lib/libgomp.so
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/libc/usr/lib/libgomp.so.1.0.0
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so.1
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.a
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.spec
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so.1.0.0
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/share/doc/libgomp.html
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/share/info/libgomp.info
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/libc/lib/libgomp.so.1
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/libc/lib/libgomp.a
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/libc/lib/libgomp.so
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/libc/lib/libgomp.spec
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/libc/lib/libgomp.so.1.0.0
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so.1
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.a
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.spec
./prebuilts/gcc/linux-x86/arm/gcc-linaro-6.3.1-2017.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/lib/libgomp.so.1.0.0
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so.1
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so.1.0.0
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so.1
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.a
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.spec
./buildroot/output/rockchip_rv1126_rv1109_facial_gate/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so.1.0.0
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so.1
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/usr/lib/libgomp.so.1.0.0
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so.1
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.a
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.spec
./buildroot/output/rockchip_rv1126_rv1109_recovery/host/arm-buildroot-linux-gnueabihf/sysroot/lib/libgomp.so.1.0.0

从这里

./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/libc/usr/lib/libgomp.so.1
./prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/arm-linux-gnueabihf/libc/usr/lib/libgomp.so

拷贝libgomp.so.1和libgomp.so文件到开发板的/usr/lib里面去。

然后运行，出现下面信息代表正常！

[root@RV1126_RV1109:/mnt/nfs/sherpa]# ./sherpa-ncnn

Usage:
  ./bin/sherpa-ncnn \
    /path/to/tokens.txt \
    /path/to/encoder.ncnn.param \
    /path/to/encoder.ncnn.bin \
    /path/to/decoder.ncnn.param \
    /path/to/decoder.ncnn.bin \
    /path/to/joiner.ncnn.param \
    /path/to/joiner.ncnn.bin \
    /path/to/foo.wav [num_threads] [decode_method, can be greedy_search/modified_beam_search]

Please refer to
https://k2-fsa.github.io/sherpa/ncnn/pretrained_models/index.html
for a list of pre-trained models to download.

运行OK！

三、下载模型

使用的模型是小模型，下载地址：Small models

对应的文档：https://k2-fsa.github.io/sherpa/ncnn/pretrained_models/zipformer-transucer-models.html#sherpa-ncnn-streaming-zipformer-small-bilingual-zh-en-2023-02-16

cd /home/rv1126/kaldi/
 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/csukuangfj/sherpa-ncnn-streaming-zipformer-small-bilingual-zh-en-2023-02-16
 
cd sherpa-ncnn-streaming-zipformer-small-bilingual-zh-en-2023-02-16
 
git lfs pull --include "*.bin"

运行最后一步报错

liefyuan@ubuntu:/home/rv1126/kaldi/sherpa-ncnn-streaming-zipformer-small-bilingual-zh-en-2023-02-16$ git lfs pull --include "*.bin"
git: 'lfs' is not a git command. See 'git --help'.

The most similar command is
	log

处理方法：

$ curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash 
$ sudo apt-get install git-lfs

将文件夹里面的.bin，.param，.txt文件拷贝到开发板上去，与可执行文件放一起。
我使用的是nfs，开发板和虚拟机共享一个文件夹我的sherpa库和sherpa可执行文件也都在这个文件夹里面：

四、语音测试

4.1 单个语音文件解码测试

./sherpa-ncnn \
./tokens.txt \
./encoder_jit_trace-pnnx.ncnn.param \
./encoder_jit_trace-pnnx.ncnn.bin \
./decoder_jit_trace-pnnx.ncnn.param \
./decoder_jit_trace-pnnx.ncnn.bin \
./joiner_jit_trace-pnnx.ncnn.param \
./joiner_jit_trace-pnnx.ncnn.bin \
./1.wav \
3 \
greedy_search

3：是指3个线程
greedy_search：贪心搜索算法
modified_beam_search：改进集束搜索算法

4.2 开发板上使用alsa架构从MIC说话测试

./sherpa-ncnn-alsa \
./tokens.txt \
./encoder_jit_trace-pnnx.ncnn.param \
./encoder_jit_trace-pnnx.ncnn.bin \
./decoder_jit_trace-pnnx.ncnn.param \
./decoder_jit_trace-pnnx.ncnn.bin \
./joiner_jit_trace-pnnx.ncnn.param \
./joiner_jit_trace-pnnx.ncnn.bin \
"default" \
4 \
greedy_search

4：是指4个线程
greedy_search：贪心搜索算法
modified_beam_search：改进集束搜索算法

运行起来后对着板子的MIC说话，背诵一个古诗看看。
测试log

[root@RV1126_RV1109:/mnt/nfs/sherpa]# ./sherpa-ncnn-alsa ./tokens.txt ./encoder_
jit_trace-pnnx.ncnn.param ./encoder_jit_trace-pnnx.ncnn.bin ./decoder_jit_trace-
pnnx.ncnn.param ./decoder_jit_trace-pnnx.ncnn.bin ./joiner_jit_trace-pnnx.ncnn.p
aram ./joiner_jit_trace-pnnx.ncnn.bin "default" 4 greedy_search
RecognizerConfig(feat_config=FeatureExtractorConfig(sampling_rate=16000, feature_dim=80), model_config=ModelConfig(encoder_param="./encoder_jit_trace-pnnx.ncnn.param", encoder_bin="./encoder_jit_trace-pnnx.ncnn.bin", decoder_param="./decoder_jit_trace-pnnx.ncnn.param", decoder_bin="./decoder_jit_trace-pnnx.ncnn.bin", joiner_param="./joiner_jit_trace-pnnx.ncnn.param", joiner_bin="./joiner_jit_trace-pnnx.ncnn.bin", tokens="./tokens.txt", encoder num_threads=4, decoder num_threads=4, joiner num_threads=4), decoder_config=DecoderConfig(method="greedy_search", num_active_paths=4), endpoint_config=EndpointConfig(rule1=EndpointRule(must_contain_nonsilence=False, min_trailing_silence=2.4, min_utterance_length=0), rule2=EndpointRule(must_contain_nonsilence=True, min_trailing_silence=1.2, min_utterance_length=0), rule3=EndpointRule(must_contain_nonsilence=False, min_trailing_silence=0, min_utterance_length=300)), enable_endpoint=True)
Disable fp16 for Zipformer encoder
Don't Use GPU. has_gpu: 0, config.use_vulkan_compute: 1
Failed to set number of channels to 1. Invalid argument
Channel count is set to 2. Will use only 1 channel of it.
Current sample rate: 16000
Recording started!
Use recording device: default
0:床前明月光
1:疑似地上霜
2:举头望明月
3:低头思故乡
4:好啦这是一段测试^C
Caught Ctrl + C. Exiting...

很好！中英文都可以识别。

文档写的真不错：
https://k2-fsa.github.io/sherpa/ncnn/pretrained_models/zipformer-transucer-models.html#sherpa-ncnn-streaming-zipformer-small-bilingual-zh-en-2023-02-16