深度学习图像视觉 RKNN Toolkit2 部署 RK3588S边缘端 过程全记录
- 认识RKNN Toolkit2 工程文件
- 学习路线:
- Anaconda Miniconda安装
- .condarc 文件
- 配置镜像源
- 自定义conda虚拟环境路径
- 创建Conda虚拟环境
- 本地训练环境
- 本地转换环境
- 安装 RKNN-Toolkit2:
- 添加 linaro交叉编译工具链
- 安装 Cmake
- rknn_model_zoo 文件
- 1. 使用 rknn_model_zoo 转换模型 model.rknn
- 以转换yolov8n.onnx 为例:
- 测试:
- 2. 编译模型 model.rknn 生成 install 文件
- 以编译 yolov8 生成 install 为例
- 全过程记录转换编译模型
- rknpu2 文件
- 1. 以Aarch64 Linux Demo 编译构建
- 以编译 Yolo-v5 demo 生成 install 为例:
- 全过程记录转换编译模型
- Aarch64设备部署
- rknn_model_zoo examples 文件
- 使用 rknn_model_zoo 编译构建输出文件(install)推送到开发板
- 终端设备执行操作全记录
- rknpu2 examples 文件
- 使用 rknpu2 编译构建输出文件(install)推送到开发板
- 视频演示指南:
- 终端设备执行操作全记录
- aarch64设备部署 C API 推理
------ AArch64是由ARM公司为其ARMv8-A 64位指令集架构推出的一种新的编程模型,它是ARMv8-A架构的一个子集。这种编程模型主要是为了在64位模式下提高程序性能和安全性。与ARM32(即ARMv7-A)的指令集不同,AArch64在寄存器和指令方面有许多改进和增强。
------ 因此,ARM64和AArch64都是指同一个东西,只是名称不同,来源也不同。如果你在不同的地方看到这两个术语,不要感到困惑,它们都是指代同一个处理器架构。
作者:量子孤岛
来源:知乎
认识RKNN Toolkit2 工程文件
RKNN-Toolkit, RKNN Toolkit Lite, 和 RKNPU2 都与Rockchip芯片上的神经网络推理相关,但它们在用途和应用场景上有所不同:
-
RKNN-Toolkit:
这是一个用于PC平台的开发套件,主要用于在Rockchip芯片上进行NPU(神经处理单元)模型的转换、优化和性能评估。
它提供了全面的功能,包括模型导入(如ONNX、TensorFlow、Torchscript等),模型转换为RKNN格式,以及在目标硬件上进行推理的接口。
RKNN-Toolkit通常用于开发环境,允许开发者在强大的PC上进行模型的预处理和调试工作。 -
RKNN Toolkit Lite:
这是轻量级版本的RKNN工具,设计用于在Rockchip的嵌入式设备或开发板上直接运行。
它可能包含了基本的模型加载和推理功能,适合在资源有限的环境中使用。
RKNN Toolkit Lite可能不包含所有PC版本的高级特性,例如详细的性能分析或模型转换工具,但它更专注于在实际硬件上的高效运行。 -
RKNPU2:
RKNPU2似乎是指Rockchip NPU的第二代软件栈或SDK,它提供了与NPU硬件交互的底层接口。
这个SDK通常包含驱动程序、库文件和API,使得开发者能够直接控制NPU进行神经网络计算。
RKNPU2可能被用于实现更底层的性能优化或者在没有完整RKNN-Toolkit的情况下进行定制化开发。
RKNPU2 SDK
提供了C语言编程接口,专门设计用于带有Rockchip神经处理单元(NPU)的芯片平台。这个接口允许开发者直接在目标设备上编写C代码来集成和执行已经通过RKNN-Toolkit2转换和优化的RKNN模型。以下是一些关键点:模型转换:
使用RKNN-Toolkit2,开发者可以将常见的深度学习框架(如TensorFlow, PyTorch等)的模型转换为针对RockchipNPU优化的RKNN格式。
C语言接口:
RKNPU2 SDK 提供的C API使得开发者能够在应用程序中加载、初始化和运行这些RKNN模型。这些API通常包括模型加载函数、推理接口、数据预处理和后处理函数等。
设备交互:
通过这些接口,开发者可以直接控制NPU进行高效的硬件加速计算,充分利用NPU的并行处理能力。API会处理与硬件的低级别通信,包括内存管理和指令调度。
资源管理:
开发者需要管理模型的内存分配和释放,以及在运行时管理输入和输出数据缓冲区。
性能优化:
RKNPU2可能还包括一些工具和指导,帮助开发者进行性能调优,比如批量处理、多线程支持等。
部署和测试:
一旦模型在目标设备上编译和链接,开发者可以通过编写C程序来实现模型的部署,并进行实际的推理任务。
错误处理和调试:
C语言接口也会提供错误处理机制,以便在遇到问题时能够捕获和诊断错误。 通过这种方式,RKNPU2 SDK 和 C语言接口为开发者提供了一种灵活和高效的方法,将AI模型集成到嵌入式系统中,尤其适用于需要实时推理和低功耗要求的应用场景。
总结来说,RKNN-Toolkit是全面的开发和调试工具,适合在PC上进行模型准备;RKNN Toolkit Lite是简化版,适用于嵌入式设备上的推理;而RKNPU2是NPU的软件开发包,提供直接访问硬件的能力。根据开发需求和目标平台的不同,开发者会选择适合的工具。
学习路线:
Anaconda Miniconda安装
清华大学开源软件镜像站 Miniconda下载
Anaconda 镜像使用帮助
linux安装软件:安装过程中根据提示输入enter或yes
bash Miniconda3-py312_24.3.0-0-Linux-x86_64.sh
安装完后,通过conda命令进行使用。
.condarc 文件
Windows 用户无法直接创建名为 .condarc 的文件,可先执行
conda config --set show_channel_urls yes
配置镜像源
通过修改文件添加(推荐)
直接修改.condarc文件是最方便的。
找到系统用户下的 .condarc 的文件,记事本打开并添加镜像源。
channels:
- defaults
show_channel_urls: true
default_channels:
- https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
- https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
- https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
custom_channels:
conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
deepmodeling: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/
自定义conda虚拟环境路径
找到系统用户下的 .condarc 的文件,记事本打开并添加路径,换成自己要保存的位置,建议放在非C盘中。
##windows
envs_dirs:
- E://Miniconda3//envs
##linux
envs_dirs:
- /home/wlj/.conda/envs/
创建Conda虚拟环境
# 列举所有环境
conda env list
# Python创建虚拟训练环境
conda create -n RKNN_yolov8_py3.10 python==3.10
conda activate RKNN_yolov8_py3.10
# Python创建虚拟转换环境
conda create -n RKNN_Toolkit2_py3.10 python==3.10
conda activate RKNN_Toolkit2_py3.10
# 退出虚拟环境
conda deactivate
本地训练环境
本地转换环境
安装 RKNN-Toolkit2:
注意事项:
-
使用大于或等于 1.4.0 的 rknn-toolkit2 版本。
-
使用自己训练的模型时,请对齐anchor等后处理参数,否则会导致后处理分析误差。
-
demo需要librga.so的支持,编译使用请参考
https://github.com/airockchip/librga
。
# 从官方RKNN-Toolkit2仓库拉取最新版本(toolkit2版本是/v2.0.0-beta0)
git clone https://github.com/airockchip/rknn-toolkit2/tree/v2.0.0-beta0
# 配置pip源
pip3 config set global.index-url https://mirror.baidu.com/pypi/simple
# 安装依赖库,根据rknn-toolkit2/packages/requirements_cp310-2.0.0b0.txt
pip3 install -r requirements_cp310-2.0.0b0.txt
# 安装rknn_toolkit2 | rknn-toolkit2/packages/rknn_toolkit2-2.0.0b0+9bab5682-cp310-cp310-linux_x86_64.whl
# 根据系统的python版本选择不同的whl文件安装:
pip3 install rknn_toolkit2-2.0.0b0+9bab5682-cp310-cp310-linux_x86_64.whl
检测是否安装成功:
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:~$ python3
Python 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from rknn.api import RKNN
>>>
添加 linaro交叉编译工具链
环境 :ubnutu20.04.06
使用linaro交叉编译工具链下载
博客
解压交叉编译工具链
解压gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu
tar -xzvf gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar.xz
tar -xvf gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu.tar
sudo gedit ~/.bashrc 或 sudo vim ~/.bashrc
export GCC_COMPILER=/home/ubuntu20/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu
source ~/.bashrc
安装 Cmake
第一种:指令安装
sudo apt install cmake
第二种:解压tar (自定义)
Cmake下载地址
tar -xzvf cmake-3.29.0-linux-x86_64.tar.gz
#打开个人path配置
gedit ~/.bashrc 或 sudo vim ~/.bashrc
#在末尾添加如下的内容
export PATH=/home/ubuntu/cmake-3.29.0-linux-x86_64/bin:$PATH
#接着在终端source一下.bashrc文件让path立即生效
source ~/.bashrc
#安装完毕测试版本
cmake --version
rknn_model_zoo 文件
rknn_model_zoo
├── 3rdparty # 第三方库
├── datasets # 数据集
├── examples # 示例代码
├── utils # 常用方法,如文件操作,画图等
├── build-android.sh # 用于目标为 Android 系统开发板的编译脚本
├── build-linux.sh # 用于目标为 Linux 系统开发板的编译脚本
└── ...
1. 使用 rknn_model_zoo 转换模型 model.rknn
# 从官方rknn_model_zoo仓库拉取最新版本
git clone https://github.com/airockchip/rknn_model_zoo/tree/v2.0.0
Convert to RKNN
以转换yolov8n.onnx 为例:
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo/examples/yolov8$ tree
.
├── README.md
├── cpp
│ ├── CMakeLists.txt
│ ├── main.cc
│ ├── postprocess.cc
│ ├── postprocess.h
│ ├── rknpu1
│ │ └── yolov8.cc
│ ├── rknpu2
│ │ ├── yolov8.cc
│ │ └── yolov8_rv1106_1103.cc
│ └── yolov8.h
├── model
│ ├── bus.jpg
│ ├── coco_80_labels_list.txt
│ ├── dataset.txt
│ ├── download_model.sh
│ └── yolov8n.onnx
├── model_comparison
│ ├── yolov8_graph_comparison.jpg
│ └── yolov8_output_comparison.jpg
├── python
│ ├── convert.py
│ └── yolov8.py
└── result.png
首先导入 GCC_COMPILER ,例如 export GCC_COMPILER=/home/cat/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu
(如系统添加,无需此步) ,然后执行:
cd python
# 运行 convert.py 脚本,将原始的 ONNX 模型转成 RKNN 模型
# 用法: python convert.py model_path [rk3566|rk3588|rk3562] [i8/fp] [output_path]
# output model will be saved as ../model/yolov8.rknn
python convert.py ../model/yolov8n.onnx rk3588 i8 ../model/yolov8n.rknn
测试:
# 如果想先在计算机端运行原始的 onnx 模型,可以参考以下命令
# 用法: python yolov5.py --model_path {onnx_model} --img_show
python yolov8.py --model_path ../model/yolov8n.onnx --img_show
2. 编译模型 model.rknn 生成 install 文件
For Linux develop board:
首先导入 GCC_COMPILER ,例如 export GCC_COMPILER=/home/cat/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu
(如系统添加,无需此步) ,然后执行:
./build-linux.sh -t <target> -a <arch> -d <build_demo_name> [-b <build_type>] [-m]
-t : target (rk356x/rk3588/rk3576/rv1106/rk1808/rv1126)
-a : arch (aarch64/armhf)
-d : demo name
-b : build_type(Debug/Release)
-m : enable address sanitizer, build_type need set to Debug
Note: 'rk356x' represents rk3562/rk3566/rk3568, 'rv1106' represents rv1103/rv1106, 'rv1126' represents rv1109/rv1126
# Here is an example for compiling yolov5 demo for 64-bit Linux RK3566.
./build-linux.sh -t rk356x -a aarch64 -d yolov5
For Android development board:
# For Android develop boards, it's require to set path for Android NDK compilation tool path according to the user environment
export ANDROID_NDK_PATH=~/opts/ndk/android-ndk-r18b
./build-android.sh -t <target> -a <arch> -d <build_demo_name> [-b <build_type>] [-m]
-t : target (rk356x/rk3588/rk3576)
-a : arch (arm64-v8a/armeabi-v7a)
-d : demo name
-b : build_type (Debug/Release)
-m : enable address sanitizer, build_type need set to Debug
# Here is an example for compiling yolov5 demo for 64-bit Android RK3566.
./build-android.sh -t rk356x -a arm64-v8a -d yolov5
以编译 yolov8 生成 install 为例
位置在rknn_model_zoo/examples/yolov8
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/rknn_model_zoo/examples/yolov8$ tree
.
├── README.md
├── cpp # C/C++ 版本的示例代码
│ ├── CMakeLists.txt
│ ├── main.cc
│ ├── postprocess.cc
│ ├── postprocess.h
│ ├── rknpu1
│ │ └── yolov8.cc
│ ├── rknpu2
│ │ ├── yolov8.cc
│ │ └── yolov8_rv1106_1103.cc
│ └── yolov8.h
├── model # 模型、测试图片等文件
│ ├── bus.jpg
│ ├── coco_80_labels_list.txt
│ ├── dataset.txt
│ ├── download_model.sh
│ ├── yolov8.rknn
│ └── yolov8n.onnx
├── model_comparison
│ ├── yolov8_graph_comparison.jpg
│ └── yolov8_output_comparison.jpg
├── python # 模型转换脚本和 Python 版本的示例代码
│ ├── convert.py
│ └── yolov8.py
└── result.png
cd /rknn_model_zoo
./build-linux.sh -t rk3588 -a aarch64 -d yolov8
rknn_model_zoo目录下生成 install
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/rknn_model_zoo/install$ tree
.
└── rk3588_linux_aarch64
└── rknn_yolov8_demo
├── lib # 依赖库
│ ├── librga.so
│ └── librknnrt.so
├── model # 存放模型、测试图片等文件
│ ├── bus.jpg
│ ├── coco_80_labels_list.txt
│ └── yolov8.rknn
└── rknn_yolov8_demo # 可执行文件
全过程记录转换编译模型
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo/examples/yolov8$ ls
README.md cpp model model_comparison python result.png
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo/examples/yolov8$ cd python/
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo/examples/yolov8/python$ python convert.py ../model/yolov8n.onnx rk3588
I rknn-toolkit2 version: 2.0.0b0+9bab5682
--> Config model
done
--> Loading model
I It is recommended onnx opset 19, but your onnx model opset is 12!
I Model converted from pytorch, 'opset_version' should be set 19 in torch.onnx.export for successful convert!
I Loading : 100%|██████████████████████████████████████████████| 136/136 [00:00<00:00, 35716.32it/s]
done
--> Building model
W build: found outlier value, this may affect quantization accuracy
const name abs_mean abs_std outlier value
model.0.conv.weight 2.44 2.47 -17.494
model.22.cv3.2.1.conv.weight 0.09 0.14 -10.215
model.22.cv3.1.1.conv.weight 0.12 0.19 13.361, 13.317
model.22.cv3.0.1.conv.weight 0.18 0.20 -11.216
I GraphPreparing : 100%|████████████████████████████████████████| 161/161 [00:00<00:00, 5348.48it/s]
I Quantizating : 100%|████████████████████████████████████████████| 161/161 [00:05<00:00, 27.53it/s]
W build: The default input dtype of 'images' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of '318' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of 'onnx::ReduceSum_326' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of '331' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of '338' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of 'onnx::ReduceSum_346' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of '350' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of '357' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of 'onnx::ReduceSum_365' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
W build: The default output dtype of '369' is changed from 'float32' to 'int8' in rknn model for performance!
Please take care of this change when deploy rknn model with Runtime API!
I rknn building ...
I rknn buiding done.
done
--> Export rknn model
done
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo/examples/yolov8/python$ cd ..
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo/examples/yolov8$ cd model
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo/examples/yolov8/model$ ls
bus.jpg coco_80_labels_list.txt dataset.txt download_model.sh yolov8.rknn yolov8n.onnx
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo/examples/yolov8/model$ cd ..
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo/examples/yolov8$ cd ..
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo/examples$ cd ..
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo$ ./build-linux.sh -t rk3588 -a aarch64 -d yolov8
./build-linux.sh -t rk3588 -a aarch64 -d yolov8
/home/ubuntu20/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu
===================================
BUILD_DEMO_NAME=yolov8
BUILD_DEMO_PATH=examples/yolov8/cpp
TARGET_SOC=rk3588
TARGET_ARCH=aarch64
BUILD_TYPE=Release
ENABLE_ASAN=OFF
INSTALL_DIR=/mnt/d/Desktop/RK3588/rknn_model_zoo/install/rk3588_linux_aarch64/rknn_yolov8_demo
BUILD_DIR=/mnt/d/Desktop/RK3588/rknn_model_zoo/build/build_rknn_yolov8_demo_rk3588_linux_aarch64_Release
CC=/home/ubuntu20/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc
CXX=/home/ubuntu20/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-g++
===================================
-- !!!!!!!!!!!CMAKE_SYSTEM_NAME: Linux
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/d/Desktop/RK3588/rknn_model_zoo/build/build_rknn_yolov8_demo_rk3588_linux_aarch64_Release
[ 20%] Built target fileutils
[ 40%] Built target imagedrawing
[ 60%] Built target imageutils
[100%] Built target rknn_yolov8_demo
[ 20%] Built target fileutils
[ 40%] Built target imageutils
[ 60%] Built target imagedrawing
[100%] Built target rknn_yolov8_demo
Install the project...
-- Install configuration: "Release"
-- Installing: /mnt/d/Desktop/RK3588/rknn_model_zoo/install/rk3588_linux_aarch64/rknn_yolov8_demo/./rknn_yolov8_demo
-- Set runtime path of "/mnt/d/Desktop/RK3588/rknn_model_zoo/install/rk3588_linux_aarch64/rknn_yolov8_demo/./rknn_yolov8_demo" to "$ORIGIN/../lib"
-- Installing: /mnt/d/Desktop/RK3588/rknn_model_zoo/install/rk3588_linux_aarch64/rknn_yolov8_demo/model/bus.jpg
-- Installing: /mnt/d/Desktop/RK3588/rknn_model_zoo/install/rk3588_linux_aarch64/rknn_yolov8_demo/model/coco_80_labels_list.txt
-- Installing: /mnt/d/Desktop/RK3588/rknn_model_zoo/install/rk3588_linux_aarch64/rknn_yolov8_demo/model/yolov8.rknn
-- Installing: /mnt/d/Desktop/RK3588/rknn_model_zoo/install/rk3588_linux_aarch64/rknn_yolov8_demo/lib/librknnrt.so
-- Installing: /mnt/d/Desktop/RK3588/rknn_model_zoo/install/rk3588_linux_aarch64/rknn_yolov8_demo/lib/librga.so
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo$
rknpu2 文件
注意事项:
-
使用大于或等于 1.4.0 的 rknn-toolkit2 版本。
-
使用自己训练的模型时,请对齐anchor等后处理参数,否则会导致后处理分析误差。
-
官网和rk预训练模型均检测80种目标。如果训练自己的模型,则需要更改 include/postprocess.h 中的OBJ_CLASS_NUM和NMS_THRESH,BOX_THRESH后处理参数。
-
demo需要librga.so的支持,编译使用请参考
https://github.com/airockchip/librga
。 -
由于硬件限制,该demo的模型默认把 yolov5 模型的后处理部分,移至cpu实现。本demo附带的模型均使用relu为激活函数,相比silu激活函数精度略微下降,性能大幅上升。
1. 以Aarch64 Linux Demo 编译构建
首先导入 GCC_COMPILER ,例如 export GCC_COMPILER=/home/cat/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu
(如系统添加,无需此步) ,然后执行:
#进入rknn-toolkit2/rknpu2/examples/rknn_yolov5_demo
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo$ ls
CMakeLists.txt README.md README_CN.md build-android.sh build-linux.sh convert_rknn_demo include model src utils
以编译 Yolo-v5 demo 生成 install 为例:
cat@lubancat:~/rknpu2/examples/rknn_yolov5_demo$ tree
.
├── CMakeLists.txt
├── README.md
├── README_CN.md
├── build-android.sh
├── build-linux.sh
├── convert_rknn_demo
│ └── yolov5
│ ├── README.md
│ ├── README_CN.md
│ ├── bus.jpg
│ ├── dataset.txt
│ ├── onnx2rknn.py
│ └── onnx_models
│ ├── yolov5s_for_apk_demo.onnx
│ └── yolov5s_relu.onnx
├── include
│ ├── drm_func.h
│ ├── postprocess.h
│ ├── preprocess.h
│ └── rga_func.h
├── model
│ ├── RK3562
│ │ └── yolov5s-640-640.rknn
│ ├── RK3566_RK3568
│ │ └── yolov5s-640-640.rknn
│ ├── RK3576
│ │ └── yolov5s-640-640.rknn
│ ├── RK3588
│ │ └── yolov5s-640-640.rknn
│ ├── bus.jpg
│ └── coco_80_labels_list.txt
├── src
│ ├── main.cc
│ ├── main_video.cc
│ ├── postprocess.cc
│ └── preprocess.cc
└── utils
├── drawing.cpp
├── drawing.h
├── mpp_decoder.cpp
├── mpp_decoder.h
├── mpp_encoder.cpp
└── mpp_encoder.h
11 directories, 32 files
# such as:
# ./build-linux.sh -t <target> -a <arch> -b <build_type>]
./build-linux.sh -t rk3588 -a aarch64 -b Release
在rknn_yolov5_demo目录下生成 install
全过程记录转换编译模型
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo$ ls
CMakeLists.txt README.md README_CN.md build-android.sh build-linux.sh convert_rknn_demo include model src utils
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo$ ll
total 20
drwxrwxrwx 1 ubuntu20 ubuntu20 4096 Mar 25 11:53 ./
drwxrwxrwx 1 ubuntu20 ubuntu20 4096 Mar 25 11:53 ../
-rwxrwxrwx 1 ubuntu20 ubuntu20 3948 Mar 25 11:53 CMakeLists.txt*
-rwxrwxrwx 1 ubuntu20 ubuntu20 3561 Mar 25 11:53 README.md*
-rwxrwxrwx 1 ubuntu20 ubuntu20 3496 Mar 25 11:53 README_CN.md*
-rwxrwxrwx 1 ubuntu20 ubuntu20 2632 Mar 25 11:53 build-android.sh*
-rwxrwxrwx 1 ubuntu20 ubuntu20 2722 Mar 25 11:53 build-linux.sh*
drwxrwxrwx 1 ubuntu20 ubuntu20 4096 Mar 25 11:53 convert_rknn_demo/
drwxrwxrwx 1 ubuntu20 ubuntu20 4096 Mar 25 11:53 include/
drwxrwxrwx 1 ubuntu20 ubuntu20 4096 Mar 25 11:53 model/
drwxrwxrwx 1 ubuntu20 ubuntu20 4096 Mar 25 11:53 src/
drwxrwxrwx 1 ubuntu20 ubuntu20 4096 Mar 25 11:53 utils/
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo$ ./build-linux.sh -t rk3588 -a aarch64 -b Release
./build-linux.sh -t rk3588 -a aarch64 -b Release
/home/ubuntu20/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu
===================================
TARGET_SOC=RK3588
TARGET_ARCH=aarch64
BUILD_TYPE=Release
BUILD_DIR=/mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/build/build_RK3588_linux_aarch64_Release
CC=/home/ubuntu20/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc
CXX=/home/ubuntu20/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-g++
===================================
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
-- Check for working C compiler: /home/ubuntu20/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc
-- Check for working C compiler: /home/ubuntu20/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /home/ubuntu20/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-g++
-- Check for working CXX compiler: /home/ubuntu20/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenCV: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/3rdparty/opencv/opencv-linux-aarch64 (found version "3.4.5")
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/build/build_RK3588_linux_aarch64_Release
Scanning dependencies of target rknn_yolov5_demo
Scanning dependencies of target rknn_yolov5_video_demo
[ 10%] Building CXX object CMakeFiles/rknn_yolov5_video_demo.dir/src/main_video.cc.o
[ 30%] Building CXX object CMakeFiles/rknn_yolov5_video_demo.dir/src/postprocess.cc.o
[ 30%] Building CXX object CMakeFiles/rknn_yolov5_video_demo.dir/utils/mpp_decoder.cpp.o
[ 40%] Building CXX object CMakeFiles/rknn_yolov5_video_demo.dir/utils/mpp_encoder.cpp.o
[ 50%] Building CXX object CMakeFiles/rknn_yolov5_video_demo.dir/utils/drawing.cpp.o
[ 60%] Linking CXX executable rknn_yolov5_video_demo
[ 60%] Built target rknn_yolov5_video_demo
[ 70%] Building CXX object CMakeFiles/rknn_yolov5_demo.dir/src/main.cc.o
[ 90%] Building CXX object CMakeFiles/rknn_yolov5_demo.dir/src/postprocess.cc.o
[ 90%] Building CXX object CMakeFiles/rknn_yolov5_demo.dir/src/preprocess.cc.o
[100%] Linking CXX executable rknn_yolov5_demo
[100%] Built target rknn_yolov5_demo
[ 60%] Built target rknn_yolov5_video_demo
[100%] Built target rknn_yolov5_demo
Install the project...
-- Install configuration: "Release"
-- Installing: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/install/rknn_yolov5_demo_Linux/./rknn_yolov5_demo
-- Installing: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/install/rknn_yolov5_demo_Linux/lib/librknnrt.so
-- Installing: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/install/rknn_yolov5_demo_Linux/lib/librga.so
-- Installing: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/install/rknn_yolov5_demo_Linux/./model/RK3588
-- Installing: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/install/rknn_yolov5_demo_Linux/./model/RK3588/yolov5s-640-640.rknn
-- Installing: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/install/rknn_yolov5_demo_Linux/./model/bus.jpg
-- Installing: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/install/rknn_yolov5_demo_Linux/./model/coco_80_labels_list.txt
-- Installing: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/install/rknn_yolov5_demo_Linux/./rknn_yolov5_video_demo
-- Installing: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/install/rknn_yolov5_demo_Linux/lib/librockchip_mpp.so
-- Installing: /mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo/install/rknn_yolov5_demo_Linux/lib/libmk_api.so
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn-toolkit2-2.0.0-beta0/rknpu2/examples/rknn_yolov5_demo$
Aarch64设备部署
rknn_model_zoo examples 文件
使用 rknn_model_zoo 编译构建输出文件(install)推送到开发板
在 rknn_model_zoo 生成 install 文件夹
(RKNN_Toolkit2_py3.10) ubuntu20@DESKTOP-EJ39PBE:/mnt/d/Desktop/RK3588/rknn_model_zoo/install$ tree
.
└── rk3588_linux_aarch64
└── rknn_yolov8_demo
├── lib
│ ├── librga.so
│ └── librknnrt.so
├── model
│ ├── bus.jpg
│ ├── coco_80_labels_list.txt
│ └── yolov8.rknn
└── rknn_yolov8_demo
拷贝 install 到 aarch64设备,再
cat@lubancat:~/rk3588_linux_aarch64/rknn_yolov8_demo$ chmod 777 *
cat@lubancat:~/rk3588_linux_aarch64/rknn_yolov8_demo$ export LD_LIBRARY_PATH=./lib
cat@lubancat:~/rk3588_linux_aarch64/rknn_yolov8_demo$ ./rknn_yolov8_demo model/yolov8.rknn model/bus.jpg
运行一张的速度太慢了,批量去运行写一个batch_process.sh文件
cat@lubancat:~/rk3588_linux_aarch64/rknn_yolov8_demo$ vim batch_process.sh
export LD_LIBRARY_PATH=./lib
#!/bin/bash
# 指定图片文件夹路径
IMAGE_DIR="./image_file"
# 遍历图片文件夹中的所有图片文件
for image_file in "$IMAGE_DIR"/*.jpg; do
if [ -f "$image_file" ]; then
echo "Processing $image_file"
./rknn_yolov8_demo ./model/yolov8.rknn "$image_file"
fi
done
终端设备执行操作全记录
cat@lubancat:~$ ls
Desktop Downloads Pictures Python-3.10.14 Videos librga rknn_mobilenet_demo_Linux rknn_model_zoo
Documents Music Public Templates cmake-3.29.0-linux-aarch64 rk3588_linux_aarch64 rknn_model
cat@lubancat:~$ cd rk3588_linux_aarch64/
cat@lubancat:~/rk3588_linux_aarch64$ ls
rknn_yolov8_demo
cat@lubancat:~/rk3588_linux_aarch64$ cd rknn_yolov8_demo/
cat@lubancat:~/rk3588_linux_aarch64/rknn_yolov8_demo$ ls
lib model rknn_yolov8_demo
cat@lubancat:~/rk3588_linux_aarch64/rknn_yolov8_demo$ ll
total 984
drwxrwxr-x 4 cat cat 4096 May 13 15:06 ./
drwxrwxrwx 3 cat cat 4096 May 13 15:06 ../
drwxrwxr-x 2 cat cat 4096 May 13 15:06 lib/
drwxrwxr-x 2 cat cat 4096 May 13 15:06 model/
-rw-rw-r-- 1 cat cat 991216 May 13 15:06 rknn_yolov8_demo
cat@lubancat:~/rk3588_linux_aarch64/rknn_yolov8_demo$ chmod 777 *
cat@lubancat:~/rk3588_linux_aarch64/rknn_yolov8_demo$ export LD_LIBRARY_PATH=./lib
cat@lubancat:~/rk3588_linux_aarch64/rknn_yolov8_demo$ ./rknn_yolov8_demo model/yolov8.rknn model/bus.jpg
load lable ./model/coco_80_labels_list.txt
model input num: 1, output num: 9
input tensors:
index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
output tensors:
index=0, name=318, n_dims=4, dims=[1, 64, 80, 80], n_elems=409600, size=409600, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-58, scale=0.117659
index=1, name=onnx::ReduceSum_326, n_dims=4, dims=[1, 80, 80, 80], n_elems=512000, size=512000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003104
index=2, name=331, n_dims=4, dims=[1, 1, 80, 80], n_elems=6400, size=6400, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003173
index=3, name=338, n_dims=4, dims=[1, 64, 40, 40], n_elems=102400, size=102400, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-45, scale=0.093747
index=4, name=onnx::ReduceSum_346, n_dims=4, dims=[1, 80, 40, 40], n_elems=128000, size=128000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003594
index=5, name=350, n_dims=4, dims=[1, 1, 40, 40], n_elems=1600, size=1600, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003627
index=6, name=357, n_dims=4, dims=[1, 64, 20, 20], n_elems=25600, size=25600, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-34, scale=0.083036
index=7, name=onnx::ReduceSum_365, n_dims=4, dims=[1, 80, 20, 20], n_elems=32000, size=32000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003874
index=8, name=369, n_dims=4, dims=[1, 1, 20, 20], n_elems=400, size=400, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
model is NHWC input fmt
model input height=640, width=640, channel=3
origin size=640x640 crop size=640x640
input image: 640 x 640, subsampling: 4:2:0, colorspace: YCbCr, orientation: 1
scale=1.000000 dst_box=(0 0 639 639) allow_slight_change=1 _left_offset=0 _top_offset=0 padding_w=0 padding_h=0
src width=640 height=640 fmt=0x1 virAddr=0x0x1d996ea0 fd=0
dst width=640 height=640 fmt=0x1 virAddr=0x0x1dac2eb0 fd=0
src_box=(0 0 639 639)
dst_box=(0 0 639 639)
color=0x72
rga_api version 1.10.1_[0]
rknn_run
person @ (211 241 282 506) 0.864
bus @ (96 136 549 449) 0.864
person @ (109 235 225 535) 0.860
person @ (477 226 560 522) 0.848
person @ (79 327 116 513) 0.306
write_image path: out.png width=640 height=640 channel=3 data=0x1d996ea0
cat@lubancat:~/rk3588_linux_aarch64/rknn_yolov8_demo$ ls
lib model out.png rknn_yolov8_demo
cat@lubancat:~/rk3588_linux_aarch64/rknn_yolov8_demo$
rknpu2 examples 文件
使用 rknpu2 编译构建输出文件(install)推送到开发板
在rknpu2 / examples / rknn_yolov5_demo目录下生成 install
推送到aarch64设备开发板 install / rknn_yolov5_demo_Linux
export LD_LIBRARY_PATH=./lib
#./rknn_yolov5_demo model/<TARGET_PLATFORM>/yolov5s-640-640.rknn model/bus.jpg
chmod 777 *
./rknn_yolov5_demo model/RK3588/yolov5s-640-640.rknn model/bus.jpg
注意:如果在 lib 文件夹中找不到 librga.so
,请尝试搜索 librga.so
的位置并将其添加到LD_LIBRARY_PATH。使用以下命令添加到LD_LIBRARY_PATH。
export LD_LIBRARY_PATH=./lib:<LOCATION_LIBRGA.SO>
GITHUB的airockchip/librga地址
效果:
视频演示指南:
添加 test.mp4 视频测试
H264 H264型
通过 ffmpeg 转换为 h264 :
#ffmpeg -i xxx.mp4 -vcodec h264 xxx.h264
ffmpeg -i model/test.mp4 -vcodec h264 model/test.h264
#./rknn_yolov5_video_demo model/<TARGET_PLATFORM>/yolov5s-640-640.rknn xxx.h264 264
./rknn_yolov5_video_demo model/RK3588/yolov5s-640-640.rknn model/test.h264 264
通过 ffmpeg 转换为 h265 :
#ffmpeg -i xxx.mp4 -vcodec hevc xxx.hevc
ffmpeg -i model/test.mp4 -vcodec hevc model/test.hevc
H265
#./rknn_yolov5_video_demo model/<TARGET_PLATFORM>/yolov5s-640-640.rknn xxx.hevc 265
./rknn_yolov5_video_demo model/RK3588/yolov5s-640-640.rknn model/test.hevc 265
RTSP
#./rknn_yolov5_video_demo model/<TARGET_PLATFORM>/yolov5s-640-640.rknn <RTSP_URL> 265
./rknn_yolov5_video_demo model/RK3588/yolov5s-640-640.rknn <RTSP_URL> 265
效果:
终端设备执行操作全记录
cat@lubancat:~$ ls
Desktop Music Python-3.10.14 cmake-3.29.0-linux-aarch64 rknn_yolov5_demo_Linux
Documents Pictures Templates librga
Downloads Public Videos rk3588_linux_aarch64
cat@lubancat:~$ cd rknn_yolov5_demo_Linux/
cat@lubancat:~/rknn_yolov5_demo_Linux$ ls
lib model out.h264 out.jpg rknn_yolov5_demo rknn_yolov5_video_demo
cat@lubancat:~/rknn_yolov5_demo_Linux$ rm -rf out.h264
cat@lubancat:~/rknn_yolov5_demo_Linux$ ls
lib model out.jpg rknn_yolov5_demo rknn_yolov5_video_demo
cat@lubancat:~/rknn_yolov5_demo_Linux$ ./rknn_yolov5_video_demo model/RK3588/yolov5s-640-640.rknn model/test.hevc 265
Loading mode...
省略....
time_gap=-10found last frame reset decoder
waiting finish
cat@lubancat:~/rknn_yolov5_demo_Linux$ ls
lib model out.h264 out.jpg rknn_yolov5_demo rknn_yolov5_video_demo