52、训练paddleSeg模型，部署自己的模型到OAK相机上

news2024/10/6 16:18:41

基本思想：简单记录一下训练过程，数据集在coco基础上进行，进行筛选出杯子的数据集，然后进行训练，比较简单， 从coco数据集中筛选出杯子的数据集，然后在labelme数据集的基础上，转成paddleseg数据集，然后训练即可，生成的标签在代码中添加相应的数据格式，贴到txt即可

实验模型链接: https://pan.baidu.com/s/1w50vkX1kLfEhj2labK1xuQ?pwd=79qk 提取码: 79qk

一、数据集准备45、实例分割的labelme数据集转coco数据集以及coco数据集转labelme数据集、转paddleSeg数据集_sxj731533730的博客-CSDN博客_实例分割labelme

得到杯子数据集：链接: https://pan.baidu.com/s/1DWf7d1xWAscAKmIvNYJ9Rw?pwd=n2vs 提取码: n2vs

二、配置文件使用修改pp_liteseg_stdc1_camvid_960x720_10k.yml

ubuntu@ubuntu:~/PaddleSeg/configs/pp_liteseg$ cp pp_liteseg_stdc1_camvid_960x720_10k.yml pp_liteseg_stdc1_camvid_300x300_10k.yml

文件内容

batch_size: 6  # total: 4*6
iters: 100000
 
train_dataset:
  type: Dataset
  dataset_root: /home/ubuntu/PaddleSeg/paddleSegCup/datasets/train
  num_classes: 2 #backgroud+cup
  mode: train
  train_path: /home/ubuntu/PaddleSeg/paddleSegCup/datasets/train/train.txt
  transforms:
    - type: ResizeStepScaling
      min_scale_factor: 0.5
      max_scale_factor: 2.5
      scale_step_size: 0.25
    - type: RandomPaddingCrop
      crop_size: [300, 300]
    - type: RandomHorizontalFlip
    - type: RandomDistort
      brightness_range: 0.5
      contrast_range: 0.5
      saturation_range: 0.5
    - type: Normalize
 
val_dataset:
  type: Dataset
  dataset_root: /home/ubuntu/PaddleSeg/paddleSegCup/datasets/val
  num_classes: 2
  mode: val
  val_path: /home/ubuntu/PaddleSeg/paddleSegCup/datasets/val/val.txt
  transforms:
    - type: Normalize
 
optimizer:
  type: sgd
  momentum: 0.9
  weight_decay: 5.0e-4
 
lr_scheduler:
  type: PolynomialDecay
  learning_rate: 0.01
  end_lr: 0
  power: 0.9
  warmup_iters: 200
  warmup_start_lr: 1.0e-5
 
loss:
  types:
    - type: OhemCrossEntropyLoss
      min_kept: 250000   # batch_size * 300 * 300 // 16
    - type: OhemCrossEntropyLoss
      min_kept: 250000
    - type: OhemCrossEntropyLoss
      min_kept: 250000
  coef: [1, 1, 1]
 
model:
  type: PPLiteSeg
  backbone:
    type: STDC1
    pretrained: https://bj.bcebos.com/paddleseg/dygraph/PP_STDCNet1.tar.gz
  arm_out_chs: [32, 64, 128]
  seg_head_inter_chs: [32, 64, 64]

训练起来了

ubuntu@ubuntu:~/PaddleSeg$ ubuntu@ubuntu:~/PaddleSeg$ python3 train.py --config configs/pp_liteseg/pp_liteseg_stdc1_camvid_300x300_10k.yml --do_eval
2022-11-25 16:46:23 [INFO]	
------------Environment Information-------------
platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
Python: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
Paddle compiled with cuda: True
NVCC: Build cuda_11.1.TC455_06.29069683_0
cudnn: 8.2
GPUs used: 1
CUDA_VISIBLE_DEVICES: None
GPU: ['GPU 0: NVIDIA GeForce']
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
PaddleSeg: 2.6.0
PaddlePaddle: 2.3.2
OpenCV: 4.6.0
------------------------------------------------
2022-11-25 16:46:23 [INFO]	
---------------Config Information---------------
batch_size: 6
iters: 10000
loss:
  coef:
  - 1
  - 1
  - 1
  types:
2022-11-25 16:54:19 [INFO]	[TRAIN] epoch: 1, iter: 10/10000, loss: 2.7239, lr: 0.000460, batch_cost: 0.2893, reader_cost: 0.01094, ips: 20.7363 samples/sec | ETA 00:48:10
2022-11-25 16:54:19 [INFO]	[TRAIN] epoch: 1, iter: 20/10000, loss: 2.3742, lr: 0.000959, batch_cost: 0.0511, reader_cost: 0.00009, ips: 117.4557 samples/sec | ETA 00:08:29
2022-11-25 16:54:20 [INFO]	[TRAIN] epoch: 1, iter: 30/10000, loss: 1.9726, lr: 0.001459, batch_cost: 0.0536, reader_cost: 0.00026, ips: 111.8903 samples/sec | ETA 00:08:54
2022-11-25 16:54:20 [INFO]	[TRAIN] epoch: 2, iter: 40/10000, loss: 1.7898, lr: 0.001958, batch_cost: 0.0576, reader_cost: 0.00709, ips: 104.1587 samples/sec | ETA 00:09:33
2022-11-25 16:54:21 [INFO]	[TRAIN] epoch: 2, iter: 50/10000, loss: 2.6318, lr: 0.002458, batch_cost: 0.0550, reader_cost: 0.00426, ips: 109.1434 samples/sec | ETA 00:09:06
2022-11-25 16:54:21 [INFO]	[TRAIN] epoch: 2, iter: 60/10000, loss: 2.1906, lr: 0.002957, batch_cost: 0.0566, reader_cost: 0.00435, ips: 106.0024 samples/sec | ETA 00:09:22
2022-11-25 16:54:22 [INFO]	[TRAIN] epoch: 2, iter: 70/10000, loss: 1.9887, lr: 0.003457, batch_cost: 0.0567, reader_cost: 0.00542, ips: 105.8548 samples/sec | ETA 00:09:22
2022-11-25 16:54:23 [INFO]	[TRAIN] epoch: 3, iter: 80/10000, loss: 2.3479, lr: 0.003956, batch_cost: 0.0611, reader_cost: 0.01129, ips: 98.2484 samples/sec | ETA 00:10:05
2022-11-25 16:54:23 [INFO]	[TRAIN] epoch: 3, iter: 90/10000, loss: 2.0537, lr: 0.004456, batch_cost: 0.0551, reader_cost: 0.00373, ips: 108.8724 samples/sec | ETA 00:09:06
2022-11-25 16:54:24 [INFO]	[TRAIN] epoch: 3, iter: 100/10000, loss: 2.0187, lr: 0.004955, batch_cost: 0.0539, reader_cost: 0.00411, ips: 111.2684 samples/sec | ETA 00:08:53
2022-11-25 16:54:24 [INFO]	[TRAIN] epoch: 3, iter: 110/10000, loss: 2.1657, lr: 0.005455, batch_cost: 0.0508, reader_cost: 0.00069, ips: 118.2217 samples/sec | ETA 00:08:21

训练完成和测试

ubuntu@ubuntu:~/PaddleSeg/output$ ls
iter_10000  iter_6000  iter_7000  iter_8000  iter_9000

三、测试

ubuntu@ubuntu:~/PaddleSeg$ python3 predict.py --config /home/ubuntu/PaddleSeg/configs/pp_liteseg/pp_liteseg_stdc1_camvid_300x300_10k.yml --model_path /home/ubuntu/PaddleSeg/output/best_model/model.pdparams --image_path /home/ubuntu/PaddleSeg/paddleSegCup/datasets/val/JPEGImages/000000002157.jpg

测试结果

三、转模型，从modelparam到onnx，然后到openvino,最后到blob

1)onnx转换，

model = SavedSegmentationNet(model)  # add argmax to the last layer

后续错误不用在意,这里测试以427 640 图片为例子，还是建议统一图片尺寸在训练，因为voc数据集大小不统一，所以，我只生成了一个427 640 的数据集

ubuntu@ubuntu:~/PaddleSeg$ python3 deploy/python/infer_onnx_trt.py --config /home/ubuntu/PaddleSeg/configs/pp_liteseg/pp_liteseg_stdc1_camvid_300x300_10k.yml --model_path /home/ubuntu/PaddleSeg/output/best_model/model.pdparams --save_dir ./saved --width 640 --height 427
W1126 10:34:49.439234 19118 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 11.7, Runtime API Version: 11.1
W1126 10:34:49.441439 19118 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
2022-11-26 10:34:50 [INFO]	Loading pretrained model from https://bj.bcebos.com/paddleseg/dygraph/PP_STDCNet1.tar.gz
2022-11-26 10:34:50 [INFO]	There are 145/145 variables loaded into STDCNet.
2022-11-26 10:34:50 [INFO]	Loading pretrained model from /home/ubuntu/PaddleSeg/output/best_model/model.pdparams
2022-11-26 10:34:50 [INFO]	There are 247/247 variables loaded into PPLiteSeg.
2022-11-26 10:34:50 [INFO]	Loaded trained params of model successfully
input shape: [1, 3, 427, 640]
out shape: (1, 1, 427, 640)


2022-11-26 09:15:33 [INFO]	Static PaddlePaddle model saved in ./saved/paddle_model_static_onnx_temp_dir.
[Paddle2ONNX] Start to parse PaddlePaddle model...
[Paddle2ONNX] Model file path: ./saved/paddle_model_static_onnx_temp_dir/model.pdmodel
[Paddle2ONNX] Paramters file path: ./saved/paddle_model_static_onnx_temp_dir/model.pdiparams
[Paddle2ONNX] Start to parsing Paddle model...
[Paddle2ONNX] Use opset_version = 11 for ONNX export.
[Paddle2ONNX] PaddlePaddle model is exported as ONNX format now.
2022-11-26 09:15:33 [INFO]	ONNX model saved in ./saved/pp_liteseg_stdc1_camvid_300x300_10k_model.onnx.
Completed export onnx model.

2）转openvino

ubuntu@ubuntu:~/PaddleSeg$ python3 /opt/intel/openvino_2021/deployment_tools/model_optimizer/mo.py --input_model /home/ubuntu/PaddleSeg/saved/pp_liteseg_stdc1_camvid_300x300_10k_model.onnx --output_dir /home/ubuntu/PaddleSeg/saved/FP16 --input_shape [1,3,427,640] --data_type FP16 --scale_values [127.5,127.5,127.5] --mean_values [127.5,127.5,127.5

cmakelist.txt

cmake_minimum_required(VERSION 3.4.1)
set(CMAKE_CXX_STANDARD 14)


project(nanodet_demo)

find_package(OpenCV REQUIRED)
find_package(ngraph REQUIRED)
find_package(InferenceEngine REQUIRED)

include_directories(
        ${OpenCV_INCLUDE_DIRS}
        ${CMAKE_CURRENT_SOURCE_DIR}
        ${CMAKE_CURRENT_BINARY_DIR}
)

add_executable(nanodet_demo main.cpp )

target_link_libraries(
        nanodet_demo
        ${InferenceEngine_LIBRARIES}
        ${NGRAPH_LIBRARIES}
        ${OpenCV_LIBS}
)

main.cpp

#include <inference_engine.hpp>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <iostream>
#include <vector>
#include <chrono>
#include <iomanip> // Header file needed to use setprecision

using namespace std;
using namespace cv;

void preprocess(cv::Mat image, InferenceEngine::Blob::Ptr &blob) {
    int img_w = image.cols;
    int img_h = image.rows;
    int channels = 3;

    InferenceEngine::MemoryBlob::Ptr mblob = InferenceEngine::as<InferenceEngine::MemoryBlob>(blob);
    if (!mblob) {
        THROW_IE_EXCEPTION << "We expect blob to be inherited from MemoryBlob in matU8ToBlob, "
                           << "but by fact we were not able to cast inputBlob to MemoryBlob";
    }
    // locked memory holder should be alive all time while access to its buffer happens
    auto mblobHolder = mblob->wmap();

    float *blob_data = mblobHolder.as<float *>();


    for (size_t c = 0; c < channels; c++) {
        for (size_t h = 0; h < img_h; h++) {
            for (size_t w = 0; w < img_w; w++) {
                blob_data[c * img_w * img_h + h * img_w + w] =
                        (float) image.at<cv::Vec3b>(h, w)[c];
            }
        }
    }
}

int main(int argc, char **argv) {
    cv::Mat bgr = cv::imread("/home/ubuntu/PaddleSeg/paddleSegCup/datasets/val/JPEGImages/000000002157.jpg");
    int orignal_width = bgr.cols;
    int orignal_height = bgr.rows;
    int target_width = 640;
    int target_height = 427;
    cv::Mat resize_img;
    cv::resize(bgr, resize_img, cv::Size(target_width, target_height));

    cv::Mat rgb;
    cv::cvtColor(resize_img, rgb, cv::COLOR_BGR2RGB);


    // resize_img.convertTo(resize_img, CV_32FC1, 1.0 / 255, 0);
    //resize_img = (resize_img - 0.5) / 0.5;
    auto start = chrono::high_resolution_clock::now();    //开始时间

    std::string input_name_ = "x";
    std::string output_name_ = "argmax_0.tmp_0";
    std::string model_path = "/home/ubuntu/PaddleSeg/saved/FP16/pp_liteseg_stdc1_camvid_300x300_10k_model.xml";
    InferenceEngine::Core ie;
    InferenceEngine::CNNNetwork model = ie.ReadNetwork(model_path);
    // prepare input settings
    InferenceEngine::InputsDataMap inputs_map(model.getInputsInfo());
    input_name_ = inputs_map.begin()->first;
    InferenceEngine::InputInfo::Ptr input_info = inputs_map.begin()->second;
    //input_info->setPrecision(InferenceEngine::Precision::FP32);
    //input_info->setLayout(InferenceEngine::Layout::NCHW);



    //prepare output settings
    InferenceEngine::OutputsDataMap outputs_map(model.getOutputsInfo());
    for (auto &output_info : outputs_map) {
        std::cout << "Output:" << output_info.first << std::endl;
        output_info.second->setPrecision(InferenceEngine::Precision::FP32);
    }

    //get network
    InferenceEngine::ExecutableNetwork network_ = ie.LoadNetwork(model, "CPU");
    InferenceEngine::InferRequest infer_request_ = network_.CreateInferRequest();
    InferenceEngine::Blob::Ptr input_blob = infer_request_.GetBlob(input_name_);
    preprocess(rgb, input_blob);

    // do inference
    infer_request_.Infer();
    const InferenceEngine::Blob::Ptr pred_blob = infer_request_.GetBlob(output_name_);

    auto m_pred = InferenceEngine::as<InferenceEngine::MemoryBlob>(pred_blob);
    auto m_pred_holder = m_pred->rmap();
    const float *pred = m_pred_holder.as<const float *>();
    auto end = chrono::high_resolution_clock::now();    //结束时间
    auto duration = (end - start).count();
    cout << "程序运行时间：" << std::setprecision(10) << duration / 1000000000.0 << "s"
         << "；  " << duration / 1000000.0 << "ms"
         << "；  " << duration / 1000.0 << "us"
         << endl;

    int w = target_height;
    int h = target_width;
    std::vector<int> vec_host_scores;
    for (int i = 0; i < w * h; i++) {
        vec_host_scores.emplace_back(pred[i]);
    }

    int num_class = 1;
    vector<int> color_map(num_class * 3);
    for (int i = 0; i < num_class; i++) {
        int j = 0;
        int lab = i;
        while (lab) {
            color_map[i * 3] |= ((lab >> 0 & 1) << (7 - j));
            color_map[i * 3 + 1] |= (((lab >> 1) & 1) << (7 - j));
            color_map[i * 3 + 2] |= (((lab >> 2) & 1) << (7 - j));
            j += 1;
            lab >>= 3;
        }
    }

    cv::Mat pseudo_img(w, h, CV_8UC3, cv::Scalar(0, 0, 0));
    for (int r = 0; r < w; r++) {
        for (int c = 0; c < h; c++) {
            int idx = vec_host_scores[r * h + c];
            pseudo_img.at<Vec3b>(r, c)[0] = color_map[idx * 3];
            pseudo_img.at<Vec3b>(r, c)[1] = color_map[idx * 3 + 1];
            pseudo_img.at<Vec3b>(r, c)[2] = color_map[idx * 3 + 2];
        }
    }
    cv::Mat result;
    cv::addWeighted(resize_img, 0.4, pseudo_img, 0.6, 0, result, 0);

    cv::imshow("pseudo_img", pseudo_img);
    cv::imwrite("pseudo_img.jpg", pseudo_img);
    cv::imshow("bgr", bgr);
    cv::imwrite("resize_img.jpg", resize_img);
    cv::imshow("result", result);
    cv::imwrite("result.jpg", result);
    cv::waitKey(0);
    return 0;

}

测试结果

 3)转OAK模型

ubuntu@ubuntu:/opt/intel/openvino_2021/deployment_tools/tools$ sudo chmod 777 compile_tool/
ubuntu@ubuntu:/opt/intel/openvino_2021/deployment_tools/tools$ cd compile_tool/
ubuntu@ubuntu:/opt/intel/openvino_2021/deployment_tools/tools/compile_tool$ ./compile_tool -m /home/ubuntu/PaddleSeg/saved/FP16/pp_liteseg_stdc1_camvid_300x300_10k_model.xml -ip U8 -d MYRIAD -VPU_NUMBER_OF_SHAVES 4 -VPU_NUMBER_OF_CMX_SLICES 4
Inference Engine: 
	IE version ......... 2021.4.1
	Build ........... 2021.4.1-3926-14e67d86634-releases/2021/4

Network inputs:
    x : U8 / NCHW
Network outputs:
    bilinear_interp_v2_13.tmp_0 : FP16 / NCHW
[Warning][VPU][Config] Deprecated option was used : VPU_MYRIAD_PLATFORM
Done. LoadNetwork time elapsed: 5132 ms
ubuntu@ubuntu:/opt/intel/openvino_2021/deployment_tools/tools/compile_tool$ cp pp_liteseg_stdc1_camvid_300x300_10k_model.blob /home/ubuntu/PaddleSeg/saved/FP16

cmakelist.txt 测试图片的

cmake_minimum_required(VERSION 3.16)
project(untitled15)
set(CMAKE_CXX_STANDARD 11)
find_package(OpenCV REQUIRED)
#message(STATUS ${OpenCV_INCLUDE_DIRS})
#添加头文件
include_directories(${OpenCV_INCLUDE_DIRS})
include_directories(${CMAKE_SOURCE_DIR}/include)
include_directories(${CMAKE_SOURCE_DIR}/include/utility)
#链接Opencv库
find_package(depthai CONFIG REQUIRED)
add_executable(untitled15 main.cpp include/utility/utility.cpp)
target_link_libraries(untitled15 ${OpenCV_LIBS}  depthai::opencv )

main.cpp


#include <stdio.h>
#include <string>
#include <iostream>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>

#include "utility.hpp"
#include <vector>
#include "depthai/depthai.hpp"
using namespace std;
using namespace std::chrono;
using namespace cv;
int post_process(std::vector<int> vec_host_scores,cv::Mat resize_img,cv::Mat &result, vector<int> color_map,int w,int h){



    cv::Mat pseudo_img(w, h, CV_8UC3, cv::Scalar(0, 0, 0));
    for (int r = 0; r < w; r++) {
        for (int c = 0; c < h; c++) {
            int idx = vec_host_scores[r*h  + c];
            pseudo_img.at<Vec3b>(r, c)[0] = color_map[idx * 3];
            pseudo_img.at<Vec3b>(r, c)[1] = color_map[idx * 3 + 1];
            pseudo_img.at<Vec3b>(r, c)[2] = color_map[idx * 3 + 2];
        }
    }

    cv::addWeighted(resize_img, 0.4, pseudo_img, 0.6, 0, result, 0);
    //cv::imshow("pseudo_img", pseudo_img);
    cv::imwrite(".pseudo_img.jpg", pseudo_img);
    // cv::imshow("bgr", resize_img);
    cv::imwrite("resize_img.jpg", resize_img);
    //cv::imshow("result", result);
    cv::imwrite("result.jpg", result);
    //cv::waitKey(0);
    return 0;
}


int main(int argc, char **argv) {
    int num_class = 256;
    vector<int> color_map(num_class * 3);
    for (int i = 0; i < num_class; i++) {
        int j = 0;
        int lab = i;
        while (lab) {
            color_map[i * 3] |= ((lab >> 0 & 1) << (7 - j));
            color_map[i * 3 + 1] |= (((lab >> 1) & 1) << (7 - j));
            color_map[i * 3 + 2] |= (((lab >> 2) & 1) << (7 - j));
            j += 1;
            lab >>= 3;
        }
    }
    int target_width=427;
    int target_height=640;
    dai::Pipeline pipeline;
    //定义
    auto cam = pipeline.create<dai::node::XLinkIn>();
    cam->setStreamName("inFrame");
    auto net = pipeline.create<dai::node::NeuralNetwork>();
    dai::OpenVINO::Blob blob("/opt/intel/openvino_2021.4.689/deployment_tools/tools/compile_tool/pp_liteseg_stdc1_camvid_300x300_10k_model.blob");
    net->setBlob(blob);
    net->input.setBlocking(false);

    //基本熟练明白oak的函数使用了　
    cam->out.link(net->input);



    //定义输出
    auto xlinkParserOut = pipeline.create<dai::node::XLinkOut>();
    xlinkParserOut->setStreamName("parseOut");
    auto xlinkoutOut = pipeline.create<dai::node::XLinkOut>();
    xlinkoutOut->setStreamName("out");

    auto xlinkoutpassthroughOut = pipeline.create<dai::node::XLinkOut>();

    xlinkoutpassthroughOut->setStreamName("passthrough");


    net->out.link(xlinkParserOut->input);

    net->passthrough.link(xlinkoutpassthroughOut->input);

    //结构推送相机
    dai::Device device(pipeline);
    //取帧显示
    auto inqueue = device.getInputQueue("inFrame");//maxsize 代表缓冲数据
    auto detqueue = device.getOutputQueue("parseOut", 8, false);//maxsize 代表缓冲数据

    bool printOutputLayersOnce=true;

    cv::Mat frame=cv::imread("/home/ubuntu/PaddleSeg/paddleSegCup/datasets/val/JPEGImages/000000002157.jpg");
    while(true) {

        if(frame.empty()) break;

        auto img = std::make_shared<dai::ImgFrame>();
        frame = resizeKeepAspectRatio(frame, cv::Size(target_height, target_width), cv::Scalar(0));
        toPlanar(frame, img->getData());
        img->setTimestamp(steady_clock::now());
        img->setWidth(target_height);
        img->setHeight(target_width);
        inqueue->send(img);

        auto inNN = detqueue->get<dai::NNData>();
        if( printOutputLayersOnce&&inNN) {
            std::cout << "Output layer names: ";
            for(const auto& ten : inNN->getAllLayerNames()) {
                std::cout << ten << ", ";
            }
            std::cout << std::endl;
            printOutputLayersOnce = false;
        }
        cv::Mat result;
        auto pred=inNN->getLayerInt32(inNN->getAllLayerNames()[0]);

        post_process(pred,frame,result,color_map,target_width,target_height);
        cv::imshow("demo", frame);
        cv::imshow("result", result);
        cv::imwrite("result.jpg",result);
        int key = cv::waitKey(1);
        if(key == 'q' || key == 'Q') return 0;
    }

//    while (true) {
//
//
//        auto ImgFrame = outqueue->get<dai::ImgFrame>();
//        auto frame = ImgFrame->getCvFrame();
//
//        auto inNN = detqueue->get<dai::NNData>();
//        if( printOutputLayersOnce&&inNN) {
//            std::cout << "Output layer names: ";
//            for(const auto& ten : inNN->getAllLayerNames()) {
//                std::cout << ten << ", ";
//            }
//            std::cout << std::endl;
//            printOutputLayersOnce = false;
//        }
//        cv::Mat result;
//        auto pred=inNN->getLayerInt32(inNN->getAllLayerNames()[0]);
//
//        post_process(pred,frame,result,color_map,target_width,target_height);
//        cv::imshow("demo", frame);
//        cv::imshow("result", result);
//        cv::imwrite("result.jpg",result);
//        cv::waitKey(1);
//
//
//    }


    return 0;
}

测试结果

实际测试视频472 640 的帧率在16fps左右


#include <stdio.h>
#include <string>
#include <iostream>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>

#include "utility.hpp"
#include <vector>
#include "depthai/depthai.hpp"

using namespace std;
using namespace std::chrono;
using namespace cv;

int post_process(std::vector<int> vec_host_scores, cv::Mat resize_img, cv::Mat &result, vector<int> color_map, int w,
                 int h) {


    cv::Mat pseudo_img(w, h, CV_8UC3, cv::Scalar(0, 0, 0));
    for (int r = 0; r < w; r++) {
        for (int c = 0; c < h; c++) {
            int idx = vec_host_scores[r * h + c];
            pseudo_img.at<Vec3b>(r, c)[0] = color_map[idx * 3];
            pseudo_img.at<Vec3b>(r, c)[1] = color_map[idx * 3 + 1];
            pseudo_img.at<Vec3b>(r, c)[2] = color_map[idx * 3 + 2];
        }
    }

    cv::addWeighted(resize_img, 0.4, pseudo_img, 0.6, 0, result, 0);
    //cv::imshow("pseudo_img", pseudo_img);
    cv::imwrite(".pseudo_img.jpg", pseudo_img);
    // cv::imshow("bgr", resize_img);
    cv::imwrite("resize_img.jpg", resize_img);
    //cv::imshow("result", result);
    cv::imwrite("result.jpg", result);
    //cv::waitKey(0);
    return 0;
}


int main(int argc, char **argv) {
    int num_class = 256;
    vector<int> color_map(num_class * 3);
    for (int i = 0; i < num_class; i++) {
        int j = 0;
        int lab = i;
        while (lab) {
            color_map[i * 3] |= ((lab >> 0 & 1) << (7 - j));
            color_map[i * 3 + 1] |= (((lab >> 1) & 1) << (7 - j));
            color_map[i * 3 + 2] |= (((lab >> 2) & 1) << (7 - j));
            j += 1;
            lab >>= 3;
        }
    }
    int target_width = 427;
    int target_height = 640;
    dai::Pipeline pipeline;
    //定义
    auto cam = pipeline.create<dai::node::ColorCamera>();
    cam->setBoardSocket(dai::CameraBoardSocket::RGB);
    cam->setResolution(dai::ColorCameraProperties::SensorResolution::THE_1080_P);
    cam->setPreviewSize(target_height, target_width);  // NN input
    cam->setInterleaved(false);

    auto net = pipeline.create<dai::node::NeuralNetwork>();
    dai::OpenVINO::Blob blob("/home/ubuntu/PaddleSeg/saved/FP16/pp_liteseg_stdc1_camvid_300x300_10k_model.blob");
    net->setBlob(blob);
    net->input.setBlocking(false);

    //基本熟练明白oak的函数使用了　
    cam->preview.link(net->input);



    //定义输出
    auto xlinkParserOut = pipeline.create<dai::node::XLinkOut>();
    xlinkParserOut->setStreamName("parseOut");
    auto xlinkoutOut = pipeline.create<dai::node::XLinkOut>();
    xlinkoutOut->setStreamName("out");

    auto xlinkoutpassthroughOut = pipeline.create<dai::node::XLinkOut>();

    xlinkoutpassthroughOut->setStreamName("passthrough");


    net->out.link(xlinkParserOut->input);

    net->passthrough.link(xlinkoutpassthroughOut->input);

    //结构推送相机
    dai::Device device(pipeline);
    //取帧显示
    auto outqueue = device.getOutputQueue("passthrough", 8, false);//maxsize 代表缓冲数据
    auto detqueue = device.getOutputQueue("parseOut", 8, false);//maxsize 代表缓冲数据

    bool printOutputLayersOnce = true;
    auto startTime = steady_clock::now();
    int counter = 0;
    float fps = 0;
    while (true) {




    auto ImgFrame = outqueue->get<dai::ImgFrame>();
    auto frame = ImgFrame->getCvFrame();

    auto inNN = detqueue->get<dai::NNData>();
    if (printOutputLayersOnce && inNN) {
        std::cout << "Output layer names: ";
        for (const auto &ten : inNN->getAllLayerNames()) {
            std::cout << ten << ", ";
        }
        std::cout << std::endl;
        printOutputLayersOnce = false;
    }
    cv::Mat result;
    auto pred = inNN->getLayerInt32(inNN->getAllLayerNames()[0]);

    post_process(pred, frame, result, color_map, target_width, target_height);

    counter++;
    auto currentTime = steady_clock::now();
    auto elapsed = duration_cast<duration<float>>(currentTime - startTime);
    if (elapsed > seconds(1)) {
        fps = counter / elapsed.count();
        counter = 0;
        startTime = currentTime;
    }
        std::stringstream fpsStr;
        fpsStr << "NN fps: " << std::fixed << std::setprecision(2) << fps;
        cv::putText(result, fpsStr.str(), cv::Point(2, result.rows - 4), cv::FONT_HERSHEY_TRIPLEX, 0.4, cv::Scalar(0,255,0));
        //cv::imshow("demo", frame);
        cv::imshow("result", result);
        //cv::imwrite("result.jpg", result);
        cv::waitKey(1);


    }


    return 0;
}

测试数据

/home/ubuntu/CLionProjects/untitled5/cmake-build-debug/untitled15
[19443010C130FF1200] [1.5] [1.155] [NeuralNetwork(1)] [warning] Network compiled for 4 shaves, maximum available 13, compiling for 6 shaves likely will yield in better performance
Output layer names: argmax_0.tmp_0, 
NN fps: 0.00
NN fps: 0.00
NN fps: 0.00
NN fps: 0.00
NN fps: 0.00
NN fps: 0.00
NN fps: 0.00
NN fps: 0.00
NN fps: 0.00
NN fps: 0.00
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 10.58
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.26
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 15.65
NN fps: 14.99
NN fps: 14.99
NN fps: 14.99
NN fps: 14.99
NN fps: 14.99
NN fps: 14.99
NN fps: 14.99

测试图片

有时候识别效果还是差，数据集可能太少了。。毕竟才150张。。。

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/37303.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！

相关文章

学生个人网页设计作品：基于HTML+CSS+JavaScript实现摄影艺术网站 DIV布局简单的摄影主题网站

学生个人网页设计作品：基于HTML+CSS+JavaScript实现摄影艺术网站 DIV布局简单的摄影主题网站

&#x1f389;精彩专栏推荐 &#x1f4ad;文末获取联系 ✍️ 作者简介: 一个热爱把逻辑思维转变为代码的技术博主 &#x1f482; 作者主页: 【主页——&#x1f680;获取更多优质源码】 &#x1f393; web前端期末大作业&#xff1a; 【&#x1f4da;毕设项目精品实战案例 (10…
阅读更多...
NATAPP 访问vue-cli启动的项目

NATAPP 访问vue-cli启动的项目

由于疫情原因&#xff0c;最近又静默居家办公了&#xff0c;由于项目需要&#xff0c;不得不再使用一下natapp了&#xff0c;这个东西两年前就用过了&#xff0c;不过以前访问的不是vue-cli项目&#xff0c;特此记录一下&#xff0c;因为这里有个坑&#xff0c;上午搞了半天没搞…
阅读更多...
APP逆向案例之（二）对加固APP进行分析和破解

APP逆向案例之（二）对加固APP进行分析和破解

说明&#xff1a;对加固APP进行分析和破解&#xff0c;对发现新版本提示关掉 1、先对APP窗口类行进HOOK&#xff0c;确定窗口提示用的是那个类。 android hooking watch class android.app.AlertDialog 2、发现一个非常明显的函数 setCancelable objection -g com.hello.qq…
阅读更多...
50个html+css+js项目小练习（二：动画的倒计时效果）

50个html+css+js项目小练习（二：动画的倒计时效果）

2.animated-navigation 实现效果&#xff1a; 倒计时321后显示go数字依次从x轴负方向&#xff0c;顺时针倒下去 xy第一个数&#xff08;0,y&#xff09;—>&#xff08;-x,0&#xff09; 第一个数字倒下去的同时&#xff0c;第二个数从x轴正方向升起 第二个数&#xff08;x…
阅读更多...
1. 开篇：既简单又复杂的基础框架

1. 开篇：既简单又复杂的基础框架

同样的基础但不简单 之前在写 《从 0 开始深入学习 Spring》 小册时&#xff0c;阿熊提到过一件事&#xff1a;学习 JavaEE 开发的第一个框架&#xff0c;大多数是推荐 MyBatis 的&#xff0c;因为它相对简单&#xff0c;学习起来也相对轻松。不过不要因为 MyBatis 入门简单&a…
阅读更多...
【电商】管理后台篇之安全、菜单、通知管理

【电商】管理后台篇之安全、菜单、通知管理

系统管理第一篇我们介绍了账号管理相关的业务&#xff0c;这一篇我们介绍下其他几个常见的业务&#xff1a;安全管理、菜单管理和通知管理。 本篇介绍的几个常见业务如下&#xff1a; 安全管理&#xff1a;监控每个账号的登录情况和操作情况通知管理&#xff1a;给使用后台系统…
阅读更多...
[附源码]计算机毕业设计JAVA企业信息网站

[附源码]计算机毕业设计JAVA企业信息网站

[附源码]计算机毕业设计JAVA企业信息网站 项目运行 环境配置&#xff1a; Jdk1.8 Tomcat7.0 Mysql HBuilderX&#xff08;Webstorm也行&#xff09; Eclispe&#xff08;IntelliJ IDEA,Eclispe,MyEclispe,Sts都支持&#xff09;。 项目技术&#xff1a; SSM mybatis M…
阅读更多...
自定义qtquick 插件模块，支持qmldesigner

自定义qtquick 插件模块，支持qmldesigner

自定义qtquick 插件模块&#xff0c;支持qmldesigner wmxModule.pro #------------------------------------------------- # # Project created by wmx # #-------------------------------------------------QT core qml quickTARGET wmxModule TEMPLATE lib CONF…
阅读更多...
图像处理QPixmap、Picture、QBitmap

图像处理QPixmap、Picture、QBitmap

QPixmap 类是一种可用作绘制设备的屏幕外图像表示形式。 QPixmap可以使用QLabel或QAbstractButton的子类之一可以轻松显示在屏幕上 QPixmap 对象可以按值传递&#xff0c;因为 QPixmap 类使用隐式数据共享&#xff0c;也可以流式传输。 QPixmap可以和QImage之间进行转换&…
阅读更多...
【关于lombok框架一文秒懂】

【关于lombok框架一文秒懂】

目录 1. Lombok框架 2. 安装lombok插件 1. Lombok框架 Lombok框架是用于在编译期自动生成相关代码的&#xff0c;例如&#xff1a;Setters & Getters、hashCode()、equals()、toString()、无参数构造方法、全参数构造方法等。 相关的注解有&#xff1a; Data&#xff1…
阅读更多...
python中的编码格式

python中的编码格式

1- 字符集和编码 1&#xff09; 初代编码 – ASCII 1- 物理机与电脑的交互&#xff1a;物理机中有个组件叫二极管&#xff0c; 可以通过电流通过二极管显示的高电平和低电平来记录信号。 2- 二极管的高电平和低电平则被计算机解读为0 和 1 3- 多个0和1拼接起来成为了二进制数据…
阅读更多...
自动化测试框架设计原理详解

自动化测试框架设计原理详解

一、自动化测试场景 1、测试四象限 并不是所有测试类型都适合自动化。哪些测试更适合自动化？哪些更适合手工测试？ 根据Brain Marick提出的测试四象限，我们可以对测试进行归类，将其划分到4个象限中，以解答这些问题，如下图所示。 第一象限：面向技术和指导开发，该象限中…
阅读更多...
ActivityPub 笔记

ActivityPub 笔记

ActivityPub 协议是一个去中心化的社交网络协议&#xff0c;其基于 ActivityStreams 2.0 数据格式。 在 ActivityPub 协议里&#xff0c;一个用户在服务器上的角色为“参与者&#xff08;actor&#xff09;”。用户在不同的服务器上的角色为不同的“参与者”。每一名“参与者”…
阅读更多...
LSTM和GRU

LSTM和GRU

LSTM和GRU RNN的优缺点 优点&#xff1a; 可以捕获序列之间的关系&#xff1b;共享参数。 缺点&#xff1a; 存在梯度消失和梯度爆炸问题&#xff1b;RNN的训练是一个比较困难的任务&#xff1b;无法处理长序列问题。 LSTM LSTM可以处理长序列问题&#xff0c;同样在之前…
阅读更多...
openssl做文件处理（base64，MD5，sha256等）

openssl做文件处理（base64，MD5，sha256等）

这次使用openssl&#xff0c;发现openssl不仅可以用来做加密和解密&#xff0c;实际上也可以用来做文件的处理&#xff0c;比如base64转码、解码&#xff0c;文件md5的计算等。实现这些&#xff0c;即可以从命令行去做&#xff0c;也可以通过代码去实现。 1. 命令行操作 1. ba…
阅读更多...
【ML特征工程】第 5 章 ：分类变量：机器鸡时代的鸡蛋计数

【ML特征工程】第 5 章 ：分类变量：机器鸡时代的鸡蛋计数

&#x1f50e;大家好&#xff0c;我是Sonhhxg_柒&#xff0c;希望你看完之后&#xff0c;能对你有所帮助&#xff0c;不足请指正&#xff01;共同学习交流&#x1f50e; &#x1f4dd;个人主页&#xff0d;Sonhhxg_柒的博客_CSDN博客 &#x1f4c3; &#x1f381;欢迎各位→点赞…
阅读更多...
想做副业没有方向，这三条告诉你什么是副业思维

想做副业没有方向，这三条告诉你什么是副业思维

怎样副业赚钱&#xff1f;教你几招&#xff0c;掌控自己的固有思维 你了解钱是怎么来的吗&#xff1f;你如果弄不懂这一点&#xff0c;你也是很难赚到钱的。 钱不是苦的&#xff0c;辛苦努力挣的基本都是一点钱。 假如将你的工作时长换为钱&#xff0c;你可以清晰地赚多少钱…
阅读更多...
Java并发-线程上下文切换与死锁

Java并发-线程上下文切换与死锁

理解线程的上下文切换 概述&#xff1a;在多线程编程中&#xff0c;线程个数一般都大于 CPU 个数&#xff0c;而每个 CPU 同一时&#xff0d;刻只能被一个线程使用&#xff0c;为了让用户感觉多个线程是在同时执行的&#xff0c; CPU 资源的分配采用了时间片轮转的策略&#x…
阅读更多...
javacc之路0--- 安装与使用

javacc之路0--- 安装与使用

官网&#xff1a; https://javacc.github.io/javacc/ https://github.com/javacc/javacc#getting-started 安装 下载解压 执行&#xff1a; mvn package 将 javacc-7.0.10.jar 重命名为 javacc.jar 并将目录下的scripts文件夹加入到环境变量中。 执行javacc命令验证是否成功…
阅读更多...
【Qt 按钮】QPushButton所有函数和样式

【Qt 按钮】QPushButton所有函数和样式

【Qt 按钮】QPushButton所有函数和样式一、QSS语句 &#xff08;界面样式大全&#xff09;二、 构造函数三、Geometry &#xff08;获取属性&#xff09;四、 QFont五、setFont六、text七、setText八、move九、resize十、adjustSize[按钮自动适应文本大小]十一、setFocus十二、…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023