前言

        香橙派内置了6T算力的NPU，想着可以跑一下yolov5，看看香橙派的速度如何。

        在开始部署之前，需要具备一定的linux技能——vim、linux常见指令、conda等等。如果没有这些技能的话，做下去会有一定的难度，可以先看几遍了解一下流程；也可以边学边部署。

香橙派环境

Orangepi5pro 1.0.4 ubuntu jammy desktop xfce linux6.1.43.7z

python=3.10.0

conda:Miniconda3-py310_23.11.0-2-Linux-x86_64.sh

参考文章香橙派5 RK3588 yolov5模型转换rknn及部署踩坑全记录 orangepi 5-CSDN博客

第一步——在PC端生成文件

 这一步是在PC端（笔记本或台式电脑）上实现的。

(1)、在电脑上使用WSL安装Ubuntu22.04系统并配置conda环境

可以在应用商城搜索WSL，点击安装

具体可以看这个视频：【WSL2】在你的win10/11电脑上安装Linux子系统+Ubuntu+图形化界面_哔哩哔哩_bilibili

conda环境可以看我的上一篇博客香橙派5pro（orangepi5pro）安装miniconda3_orange pi 5 pro install conda-CSDN博客

虽然是在PC端,但都是linux环境,上传.sh文件安装就行(上传过程自行百度)

(2)下载rknn-toolkit2源码

rockchip-linux/rknn-toolkit2 at v1.5.2 (github.com)https://github.com/rockchip-linux/rknn-toolkit2/tree/v1.5.2

(3)一、在windows上

①下载yolov5源码以及yolov5s.pt

Release v6.0 - YOLOv5n 'Nano' models, Roboflow integration, TensorFlow export, OpenCV DNN support · ultralytics/yolov5 (github.com)https://github.com/ultralytics/yolov5/releases/tag/v6.0

②将yolov5.pt转换成yolov5.onnx

按照官方文档操作,将class Detect(nn.Moudle)类的子函数forword更改

文件名字

 改为:

def forward(self, x):
        z = []  # inference output
        for i in range(self.nl):
            x[i] = self.m[i](x[i])  # conv

        return x

③然后，在yolov5目录下，打开终端输入命令

python export.py --weights yolov5n.pt --data data/coco128.yaml --include onnx --opset 12 --batch-size 1

注意注意！opset一定要为12，不然后面onnx转rknn会报错。weights自己选你训练完成的best.pt，data选你自己设置的，这里会生成一个.onnx文件

  二、在windows中的linux上

④onnx转化为rknn格式

进入用conda创建的虚拟环境中，假设叫yolo_py310，python=3.10.0

conda activate yolo_py310

假设你刚刚将rknn-toolkit2源码和yolov5源码上传到/root/Download/下面

这个rknn是rknn-toolkit2-1.5.2文件的重命名

在rknn/doc/，运行这个指令

pip install -r requirements_cp310-1.5.2.txt -i https://mirror.baidu.com/pypi/simple

进入/rknn/packages/中

pip install rknn_toolkit2-1.5.2+b642f30c-cp310-cp310-linux_x86_64.whl

中间会出现这个错误

具体看这个博客：Ubuntu 20.04安装RKNN-Toolkit2出现tf-estimator-nightly==2.8.0.dev2021122109找不到的问题_no matching distribution found for tf-estimator-ni-CSDN博客

这个上面这个博客的命令时需要科学上网！！！

        完成后，输入命令 python，输入以下代码如果不报错说明安装成功，使用键盘Ctrl+Z退出此模式。

from rknn.api import RKNN

        将（2）③中获得的.onnx文件放到rknn/examples/onnx/yolov5文件夹下，终端里进入该文件夹。用你喜欢的编辑器修改 test.py里面的一些内容

根据你的板子情况添加芯片型号

情况像下面的差不多就时成功了

此时会出现一个yolo5s.rknn的文件

⑤准备文件

1.创建一个名为deploy.py的文件

#deploy.py
import numpy as np
import cv2
from rknnlite.api import RKNNLite

RKNN_MODEL = 'yolov5n.rknn'

QUANTIZE_ON = True

OBJ_THRESH = 0.25
NMS_THRESH = 0.45
IMG_SIZE = 640

CLASSES = ("person", "bicycle", "car", "motorbike ", "aeroplane ", "bus ", "train", "truck ", "boat", "traffic light",
           "fire hydrant", "stop sign ", "parking meter", "bench", "bird", "cat", "dog ", "horse ", "sheep", "cow", "elephant",
           "bear", "zebra ", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite",
           "baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife ",
           "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza ", "donut", "cake", "chair", "sofa",
           "pottedplant", "bed", "diningtable", "toilet ", "tvmonitor", "laptop	", "mouse	", "remote ", "keyboard ", "cell phone", "microwave ",
           "oven ", "toaster", "sink", "refrigerator ", "book", "clock", "vase", "scissors ", "teddy bear ", "hair drier", "toothbrush ")


def sigmoid(x):
    return 1 / (1 + np.exp(-x))


def xywh2xyxy(x):
    # Convert [x, y, w, h] to [x1, y1, x2, y2]
    y = np.copy(x)
    y[:, 0] = x[:, 0] - x[:, 2] / 2  # top left x
    y[:, 1] = x[:, 1] - x[:, 3] / 2  # top left y
    y[:, 2] = x[:, 0] + x[:, 2] / 2  # bottom right x
    y[:, 3] = x[:, 1] + x[:, 3] / 2  # bottom right y
    return y


def process(input, mask, anchors):

    anchors = [anchors[i] for i in mask]
    grid_h, grid_w = map(int, input.shape[0:2])

    box_confidence = sigmoid(input[..., 4])
    box_confidence = np.expand_dims(box_confidence, axis=-1)

    box_class_probs = sigmoid(input[..., 5:])

    box_xy = sigmoid(input[..., :2])*2 - 0.5

    col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)
    row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)
    col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)
    row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)
    grid = np.concatenate((col, row), axis=-1)
    box_xy += grid
    box_xy *= int(IMG_SIZE/grid_h)

    box_wh = pow(sigmoid(input[..., 2:4])*2, 2)
    box_wh = box_wh * anchors

    box = np.concatenate((box_xy, box_wh), axis=-1)

    return box, box_confidence, box_class_probs


def filter_boxes(boxes, box_confidences, box_class_probs):
    """Filter boxes with box threshold. It's a bit different with origin yolov5 post process!

    # Arguments
        boxes: ndarray, boxes of objects.
        box_confidences: ndarray, confidences of objects.
        box_class_probs: ndarray, class_probs of objects.

    # Returns
        boxes: ndarray, filtered boxes.
        classes: ndarray, classes for boxes.
        scores: ndarray, scores for boxes.
    """
    boxes = boxes.reshape(-1, 4)
    box_confidences = box_confidences.reshape(-1)
    box_class_probs = box_class_probs.reshape(-1, box_class_probs.shape[-1])

    _box_pos = np.where(box_confidences >= OBJ_THRESH)
    boxes = boxes[_box_pos]
    box_confidences = box_confidences[_box_pos]
    box_class_probs = box_class_probs[_box_pos]

    class_max_score = np.max(box_class_probs, axis=-1)
    classes = np.argmax(box_class_probs, axis=-1)
    _class_pos = np.where(class_max_score >= OBJ_THRESH)

    boxes = boxes[_class_pos]
    classes = classes[_class_pos]
    scores = (class_max_score* box_confidences)[_class_pos]

    return boxes, classes, scores


def nms_boxes(boxes, scores):
    """Suppress non-maximal boxes.

    # Arguments
        boxes: ndarray, boxes of objects.
        scores: ndarray, scores of objects.

    # Returns
        keep: ndarray, index of effective boxes.
    """
    x = boxes[:, 0]
    y = boxes[:, 1]
    w = boxes[:, 2] - boxes[:, 0]
    h = boxes[:, 3] - boxes[:, 1]

    areas = w * h
    order = scores.argsort()[::-1]

    keep = []
    while order.size > 0:
        i = order[0]
        keep.append(i)

        xx1 = np.maximum(x[i], x[order[1:]])
        yy1 = np.maximum(y[i], y[order[1:]])
        xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])
        yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])

        w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)
        h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)
        inter = w1 * h1

        ovr = inter / (areas[i] + areas[order[1:]] - inter)
        inds = np.where(ovr <= NMS_THRESH)[0]
        order = order[inds + 1]
    keep = np.array(keep)
    return keep


def yolov5_post_process(input_data):
    masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]
    anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45],
               [59, 119], [116, 90], [156, 198], [373, 326]]

    boxes, classes, scores = [], [], []
    for input, mask in zip(input_data, masks):
        b, c, s = process(input, mask, anchors)
        b, c, s = filter_boxes(b, c, s)
        boxes.append(b)
        classes.append(c)
        scores.append(s)

    boxes = np.concatenate(boxes)
    boxes = xywh2xyxy(boxes)
    classes = np.concatenate(classes)
    scores = np.concatenate(scores)

    nboxes, nclasses, nscores = [], [], []
    for c in set(classes):
        inds = np.where(classes == c)
        b = boxes[inds]
        c = classes[inds]
        s = scores[inds]

        keep = nms_boxes(b, s)

        nboxes.append(b[keep])
        nclasses.append(c[keep])
        nscores.append(s[keep])

    if not nclasses and not nscores:
        return None, None, None

    boxes = np.concatenate(nboxes)
    classes = np.concatenate(nclasses)
    scores = np.concatenate(nscores)

    return boxes, classes, scores


def draw(image, boxes, scores, classes):
    """Draw the boxes on the image.

    # Argument:
        image: original image.
        boxes: ndarray, boxes of objects.
        classes: ndarray, classes of objects.
        scores: ndarray, scores of objects.
        all_classes: all classes name.
    """
    for box, score, cl in zip(boxes, scores, classes):
        top, left, right, bottom = box
        print('class: {}, score: {}'.format(CLASSES[cl], score))
        print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom))
        top = int(top)
        left = int(left)
        right = int(right)
        bottom = int(bottom)

        cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2)
        cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score),
                    (top, left - 6),
                    cv2.FONT_HERSHEY_SIMPLEX,
                    0.6, (0, 0, 255), 2)


def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):
    # Resize and pad image while meeting stride-multiple constraints
    shape = im.shape[:2]  # current shape [height, width]
    if isinstance(new_shape, int):
        new_shape = (new_shape, new_shape)

    # Scale ratio (new / old)
    r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])

    # Compute padding
    ratio = r, r  # width, height ratios
    new_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))
    dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1]  # wh padding

    dw /= 2  # divide padding into 2 sides
    dh /= 2

    if shape[::-1] != new_unpad:  # resize
        im = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)
    top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))
    left, right = int(round(dw - 0.1)), int(round(dw + 0.1))
    im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color)  # add border
    return im, ratio, (dw, dh)


if __name__ == '__main__':

    # Create RKNN object
    rknn = RKNNLite()

    # load RKNN model
    print('--> Load RKNN model')
    ret = rknn.load_rknn(RKNN_MODEL)


    # Init runtime environment
    print('--> Init runtime environment')
    ret = rknn.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2)  #使用0 1 2三个NPU核心
    # ret = rknn.init_runtime('rk3566')
    if ret != 0:
        print('Init runtime environment failed!')
        exit(ret)
    print('done')

    # Set inputs
    img = cv2.imread(IMG_PATH)
    # img, ratio, (dw, dh) = letterbox(img, new_shape=(IMG_SIZE, IMG_SIZE))
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))

    # Inference
    outputs = rknn.inference(inputs=[img])


    # post process
    input0_data = outputs[0]
    input1_data = outputs[1]
    input2_data = outputs[2]

    input0_data = input0_data.reshape([3, -1]+list(input0_data.shape[-2:]))
    input1_data = input1_data.reshape([3, -1]+list(input1_data.shape[-2:]))
    input2_data = input2_data.reshape([3, -1]+list(input2_data.shape[-2:]))

    input_data = list()
    input_data.append(np.transpose(input0_data, (2, 3, 0, 1)))
    input_data.append(np.transpose(input1_data, (2, 3, 0, 1)))
    input_data.append(np.transpose(input2_data, (2, 3, 0, 1)))

    boxes, classes, scores = yolov5_post_process(input_data)

    img_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
    if boxes is not None:
        draw(img_1, boxes, scores, classes)
    # show output
    cv2.imshow("post process result", img_1)
    cv2.waitKey(0)
    cv2.destroyAllWindows()

    rknn.release()

这是基于原来test.py做的一些修改，删除了导出模型的部分，只保留了加载模型和推理部分。接下来把这几个文件放在同一个文件夹，传到香橙派的 /home/orangepi/Desktop/yolov5 目录下

第二步——在香橙派上部署

（1）将rknn-toolkit2-1.5.2.zip上传到香橙派上并解压，上传到Download目录下面

在香橙派上创建python=3.10.0的虚拟环境（yolo_py310），进入虚拟环境

将①版本升级为1.5.2，结合下面这个两个网站进行升级

rknn3588如何升级驱动_rk2588 rknpu驱动-CSDN博客

RK3588S rknnlite版本1.4升级到1.5.2，yolov5推理速度变慢 · Issue #242 · rockchip-linux/rknn-toolkit2 (github.com) 

将②版本升级到0.9.6，这个博客的做法可以升级，升级后要重启

香橙派5B-RK3588-NPU内核更新至0.9.6_rk3588更新npu驱动-CSDN博客

（2）下载rknn-toolkit2-1.5.2依赖

将下载源设置为清华源

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

pip install rknn/rknn-toolkit2-lite/packages/rknn_toolkit_lite2-1.5.2-cp310-cp310-linux_aarch64.whl

安装剩下的依赖包

pip install opencv-python=4.5.5.64

运行deploy.py

python deploy.py

运行之后看到这个图像就是部署成功了