目录
1 调试jetson-mpeg视频解码模块
1.1 修改config.json
1.2 Picture size 0x0 is invalid
1.3 Process(): Send package failed. Maximum number of attempts reached
1.4 Picture size 2239821608x65535 is invalid
1.5 保存h264文件解码之后的测试图片
1.6 保存RTSP视频解码之后的测试图片
2 调试cv-cuda图片处理模块
2.1 NVCV_ERROR_NOT_IMPLEMENTED: Batch image format must not have subsampled planes,
2.2 保存NVConvertFormat之后的图片
参考文献:
记录下将CNStream流处理多路并发Pipeline框架适配到NVIDIA Jetson AGX Orin的过程,以及过程中遇到的问题,我的jetson盒子是用jetpack5.1.3重新刷机之后的,这是系列博客的第四篇,前三篇链接如下:
完整指南:CNStream流处理多路并发框架适配到NVIDIA Jetson Orin (一) 依赖库编译、第三方库编译安装-CSDN博客
完整指南:CNStream流处理多路并发框架适配到NVIDIA Jetson Orin (二) 源码架构流程梳理、代码编写-CSDN博客
完整指南:CNStream流处理多路并发框架适配到NVIDIA Jetson Orin (三) 代码编译、各种问题解决、代码修改-CSDN博客
这篇博客记录下程序运行过程中遇到的错误、以及调试解决各种错误的过程。
1 调试jetson-mpeg视频解码模块
1.1 修改config.json
先把config.json修改成最简单的方式,只保留解码模块
{
"profiler_config" : {
"enable_profiling" : false,
"enable_tracing" : false
},
"subgraph:decode" : {
"config_path" : "configs/decode_config.json"
},
"subgraph:business" : {
"config_path" : "configs/business.json"
}
}
然后开始运行、调试,
1.2 Picture size 0x0 is invalid
I0909 17:30:40.658268 943351 data_handler_file.cpp:304] [TACLStream SOURCE INFO] [FileHandlerImpl] OnParserInfo(): [5g3223522224262]: Got video info.
I0909 17:30:40.658645 943351 decode_impl_nv.cpp:45] [InferServer] [DecodeFFmpeg] Create(): Use codec type: h264_nvmpi
[h264_nvmpi @ 0xffff100035f0] [IMGUTILS @ 0xffff34fc4770] Picture size 0x0 is invalid
[h264_nvmpi @ 0xffff100035f0] video_get_buffer: image parameters invalid
[h264_nvmpi @ 0xffff100035f0] get_buffer() failed
[h264_nvmpi @ 0xffff100035f0] [IMGUTILS @ 0xffff34fc4770] Picture size 0x0 is invalid
[h264_nvmpi @ 0xffff100035f0] video_get_buffer: image parameters invalid
[h264_nvmpi @ 0xffff100035f0] get_buffer() failed
E0909 17:30:40.659263 943351 decode_impl_nv.cpp:67] [InferServer] [DecodeFFmpeg] Failed to open codec
E0909 17:30:40.660058 943351 decode.cpp:64] [InferServer] [DecodeService] Create(): Create decoder failed
E0909 17:30:40.660167 943351 video_decoder.cpp:88] [TACLStream SOURCE ERROR] [5g3223522224262]: Create decoder failed
E0909 17:30:40.660254 943351 data_handler_file.cpp:315] [TACLStream SOURCE ERROR] [FileHandlerImpl] OnParserInfo(): Create decoder failed, ret = 0
E0909 17:30:40.660997 943351 data_handler_file.cpp:220] [TACLStream SOURCE ERROR] [FileHandlerImpl] Loop(): [5g3223522224262]: PrepareResources failed.
W0909 17:30:40.661371 941831 task_session_mgr.cpp:855] [TACLStream SESSION_MGR WARN] [TNVPipeline] received stream error from stream: 5g3223522224262, remove it from pipeline.
在ffmpeg的源码中搜了下这个报错,
看了下代码,应该是宽高没设置的问题,但是我找了半天这个ff_get_buffer是怎么被avcodec_open2函数调用的,没找到,于是我直接吧nvmpi和ffmepg这两个库都编译成debug版本然后调试。
首先把jetson-ffmpeg编译成debug版本,修改其中的cmake那一行
git clone https://github.com/Keylost/jetson-ffmpeg.git
cd jetson-ffmpeg
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Debug ..
make
sudo make install
sudo ldconfig
然后把ffmpeg编译一个debug版本,修改其中的config那一行
./configure --enable-static --enable-shared --enable-nvmpi --enable-debug=3 --disable-optimizations
git clone git://source.ffmpeg.org/ffmpeg.git -b release/4.4 --depth=1
cd ffmpeg
cp /data/chw/jetson-ffmpeg/ffmpeg_patches/ffmpeg4.4_nvmpi.patch /data/chw/ffmpeg/ffmpeg_nvmpi.patch
git apply ffmpeg_nvmpi.patch
./configure --enable-static --enable-shared --enable-nvmpi
make -j8
make install
ldconfig
然后调试看,找到了调用关系
然后报错是因为这里的宽高没给他赋值,判断是0就会报错。
另外,在查找这个bug的过程中发现,ffmpeg里面还有个问题就是数据格式如果传NV12那么里面也会报错提示不支持,所以这里还需要修改ffmpeg内部的源码,重新编译库,
另外,我的解码代码加上这几行
codec_context_->pix_fmt = AV_PIX_FMT_NV12;
codec_context_->width = create_params_.max_width;
codec_context_->height = create_params_.max_height;
1.3 Process(): Send package failed. Maximum number of attempts reached
这个错误我检查了代码也没发现错误呀,调试也没调试出来啥,然后我看了下我的视频文件是用的一个mp4文件,我换成一个.h264视频文件就没有这个错误了。
1.4 Picture size 2239821608x65535 is invalid
[h264_nvmpi @ 0xffff00000e30] [IMGUTILS @ 0xffff26affa60] Picture size 2239821608x65535 is invalid
[h264_nvmpi @ 0xffff00000e30] [IMGUTILS @ 0xffff26affa10] Picture size 0x0 is invalid
[h264_nvmpi @ 0xffff00000e30] video_get_buffer: image parameters invalid
[h264_nvmpi @ 0xffff00000e30] get_buffer() failed
E0910 17:38:43.887912 1224545 decode_impl_nv.cpp:70] [InferServer] [DecodeFFmpeg] Failed to open codec
E0910 17:38:43.890803 1224545 decode.cpp:64] [InferServer] [DecodeService] Create(): Create decoder failed
E0910 17:38:43.891697 1224545 video_decoder.cpp:88] [TACLStream SOURCE ERROR] [5g3223522224262]: Create decoder failed
E0910 17:38:43.891896 1224545 data_handler_rtsp.cpp:515] [TACLStream SOURCE ERROR] [RtspHandlerImpl] DecodeLoop(): Create decoder failed.
我把mp4文件换成RTSP,报错
我调试进去看了下是因为宽高没被赋值。然后我调试到处理RTSP流的地方,发现
其实这里是有获取这两个值的代码的,并且我调试看到也有值,只不过这两行代码之前被注释掉了,放开注释。
然后重新编译、运行,刚才的报错消失。
1.5 保存h264文件解码之后的测试图片
在这里增加测试代码,直接用opencv保存图片
void DecodeFFmpeg::OnFrame(AVFrame *av_frame, uint32_t frame_id) {
static int debug_count = 0;
cv::Mat yuv_mat(av_frame_->height * 3 / 2, av_frame_->width, CV_8UC1, av_frame_->data[0]);//调试代码,记得修改。
cv::Mat bgr_mat;
cv::cvtColor(yuv_mat, bgr_mat, cv::COLOR_YUV2BGR_NV12); //调试代码,记得修改,这是原代码
debug_count = debug_count + 1;
char decode_name[20] = {};
if((debug_count < 3000) && (debug_count % 2 == 0))
{
sprintf(decode_name, "OnDecodeFrame_%d.jpg", debug_count);
cv::imwrite(decode_name, bgr_mat);
}
BufSurface *surf{};
if (create_params_.GetBufSurf(&surf, av_frame_->width, av_frame->height, BUF_COLOR_FORMAT_BGR,
create_params_.surf_timeout_ms, create_params_.userdata) < 0) {
LOG(ERROR) << "[InferServer] [DecoderAcl] OnFrame(): Get BufSurface failed";
OnError(-1);
return;}
if (surf->mem_type != BUF_MEMORY_MANAGED) {
LOG(ERROR) << "[InferServer] [DecoderAcl] OnFrame(): BufSurface memory type must be BUF_MEMORY_MANAGED";
return;
}
发现图片保存不正常,
好吧,
我写代码直接保存yuv文件
void DecodeFFmpeg::SaveYUVFrame(AVFrame *av_frame, const char *filename) {
FILE *file = fopen(filename, "wb");
if (!file) {
fprintf(stderr, "Could not open %s for writing\n", filename);
return;
}
// 写入 Y 分量
for (int i = 0; i < av_frame->height; i++) {
fwrite(av_frame->data[0] + i * av_frame->linesize[0], 1, av_frame->width, file);
}
// 写入交错的 UV 分量
for (int i = 0; i < av_frame->height / 2; i++) {
fwrite(av_frame->data[1] + i * av_frame->linesize[1], 1, av_frame->width, file);
}
// // 写入 U 分量
// for (int i = 0; i < av_frame->height / 2; i++) {
// fwrite(av_frame->data[1] + i * av_frame->linesize[1], 1, av_frame->width / 2, file);
// }
// // 写入 V 分量
// for (int i = 0; i < av_frame->height / 2; i++) {
// fwrite(av_frame->data[1] + (av_frame->height / 2)*av_frame->linesize[1] + i * av_frame->linesize[1], 1, av_frame->width / 2, file);
// }
fclose(file);
}
然后
void DecodeFFmpeg::OnFrame(AVFrame *av_frame, uint32_t frame_id) {
static int debug_count = 0;
cv::Mat yuv_mat(av_frame_->height * 3 / 2, av_frame_->width, CV_8UC1, av_frame_->data[0]);//调试代码,记得修改。
cv::Mat bgr_mat;
cv::cvtColor(yuv_mat, bgr_mat, cv::COLOR_YUV2BGR_NV12); //调试代码,记得修改,这是原代码
debug_count = debug_count + 1;
char decode_name[20] = {};
// 保存 YUV 数据
if (debug_count < 3000 && debug_count % 2 == 0) {
char yuv_filename[20];
sprintf(yuv_filename, "frame_%d.yuv", debug_count);
SaveYUVFrame(av_frame, yuv_filename);
}
if((debug_count < 3000) && (debug_count % 2 == 0))
{
sprintf(decode_name, "OnDecodeFrame_%d.jpg", debug_count);
cv::imwrite(decode_name, bgr_mat);
}
yuv文件打开正常。
那么就是我的yuv转BGR的代码有问题了,
我把代码改成下面这种
void DecodeFFmpeg::OnFrame(AVFrame *av_frame, uint32_t frame_id) {
static int debug_count = 0;
//cv::Mat yuv_mat(av_frame_->height * 3 / 2, av_frame_->width, CV_8UC1, av_frame_->data[0]);//调试代码,记得修改。
cv::Mat yuv_mat(av_frame_->height * 3 / 2, av_frame_->width, CV_8UC1);//调试代码,记得修改。
// 拷贝 Y 分量数据
memcpy(yuv_mat.data, av_frame->data[0], av_frame->height * av_frame->linesize[0]);
// 拷贝 UV 分量数据
memcpy(yuv_mat.data + av_frame->height * av_frame->linesize[0], av_frame->data[1], av_frame->height / 2 * av_frame->linesize[1]);
cv::Mat bgr_mat;
cv::cvtColor(yuv_mat, bgr_mat, cv::COLOR_YUV2BGR_NV12); //调试代码,记得修改,这是原代码
debug_count = debug_count + 1;
char decode_name[20] = {};
// 保存 YUV 数据
if (debug_count < 3000 && debug_count % 2 == 0) {
char yuv_filename[20];
sprintf(yuv_filename, "frame_%d.yuv", debug_count);
SaveYUVFrame(av_frame, yuv_filename);
}
if((debug_count < 3000) && (debug_count % 2 == 0))
{
sprintf(decode_name, "OnDecodeFrame_%d.jpg", debug_count);
cv::imwrite(decode_name, bgr_mat);
}
跟之前的区别在与我先创建一个mat,然后我memcpy分别从data[0]和data[1]拷贝两次,这样保存的图片正常,那么问题出在哪里。我觉得在于av_frame_->data[0]是Y分量,av_frame_->data[1]是UV分量,但是av_frame_->data[0]和av_frame_->data[1]之间并不是连续的,我用计算器验证一下,这个是av_frame_->data[0]和av_frame_->data[1]的差。
这个是1920*1080的值。
1.6 保存RTSP视频解码之后的测试图片
解码RTSP视频流,保存图片正常。
2 调试cv-cuda图片处理模块
2.1 NVCV_ERROR_NOT_IMPLEMENTED: Batch image format must not have subsampled planes,
运行报错
terminate called after throwing an instance of 'nvcv::Exception'
what(): NVCV_ERROR_NOT_IMPLEMENTED: Batch image format must not have subsampled planes, but it is: NVCV_IMAGE_FORMAT_NV12
Aborted (core dumped)
直接问下必应
修改代码,将
nvcv::Tensor::Requirements in_reqs = nvcv::Tensor::CalcRequirements(1, { buf_surf.width_stride, buf_surf.height }, CastColorFmt(buf_surf.color_format));
修改成下面的代码,也就是按照FMT_U8给yuv申请tensor。
if (buf_surf.color_format == BUF_COLOR_FORMAT_NV12 || buf_surf.color_format == BUF_COLOR_FORMAT_NV21) {
in_reqs = nvcv::Tensor::CalcRequirements(1, { buf_surf.width, buf_surf.height * 3 / 2 }, nvcv::FMT_U8);
}
else {
in_reqs = nvcv::Tensor::CalcRequirements(1, { buf_surf.width, buf_surf.height }, CastColorFmt(buf_surf.color_format));
}
2.2 保存NVConvertFormat之后的图片
我在这个函数最后面增加测试代码,保存测试图片
int TransformerNV::NVConvertFormat(BufSurface *src, BufSurface *dst, TransformParams *transform_params) {
auto& src_surf = src->surface_list[0];
auto& dst_surf = dst->surface_list[0];
auto src_tensor = GetTensorFromBufSurf(src_surf);
auto dst_tensor = GetTensorFromBufSurf(dst_surf);
NVCVColorConversionCode cvt_code{ NVCV_COLOR_YUV2BGR_NV12 };
switch (src_surf.color_format) {
case BUF_COLOR_FORMAT_NV12:
{
if (dst_surf.color_format == BUF_COLOR_FORMAT_BGR) {
cvt_code = NVCV_COLOR_YUV2BGR_NV12;
}
}break;
case BUF_COLOR_FORMAT_BGR:
{
if (dst_surf.color_format == BUF_COLOR_FORMAT_NV12) {
cvt_code = NVCV_COLOR_BGR2YUV_NV12;
}
}break;
case BUF_COLOR_FORMAT_RGB:
{
if (dst_surf.color_format == BUF_COLOR_FORMAT_NV12) {
cvt_code = NVCV_COLOR_RGB2YUV_NV12;
}
}break;
default:
cvt_code = NVCV_COLOR_YUV2BGR_NV12;
}
if ((src_surf.color_format == BUF_COLOR_FORMAT_NV12 || src_surf.color_format == BUF_COLOR_FORMAT_NV21)
&& (dst_surf.color_format == BUF_COLOR_FORMAT_BGR || dst_surf.color_format == BUF_COLOR_FORMAT_RGB)) {
cvt_code = NVCV_COLOR_YUV2BGR_NV12;
}
else if ((src_surf.color_format == BUF_COLOR_FORMAT_NV12 || src_surf.color_format == BUF_COLOR_FORMAT_NV21)
&& (dst_surf.color_format == BUF_COLOR_FORMAT_BGR || dst_surf.color_format == BUF_COLOR_FORMAT_RGB)) {
cvt_code = NVCV_COLOR_YUV2BGR_NV12;
}
(*cvtcolor_op_)(reinterpret_cast<cudaStream_t>(cu_stream_), src_tensor, dst_tensor, cvt_code);
CUDA_SAFECALL(cuStreamSynchronize(cu_stream_)
, "[InferServer] [TransformerNV] NVConvertFormat failed.", -1);
// Assuming 'dst_tensor' holds the converted BGR data.
cv::Mat bgr_image(dst_surf.height, dst_surf.width, CV_8UC3, dst->surface_list[0].data_ptr);
cv::imwrite("output_image.jpg", bgr_image);
return 0;
}
结果如下,
我知道是什么原因,这是因为前面解码的时候,解码结束之后赋值内存的时候用的是av_frame_->data,问题原因就和上面保存解码测试图片一样的。
参考文献:
在NVIDIA Jetson AGX Orin中使用jetson-ffmpeg调用硬件编解码加速处理-CSDN博客
NVIDIA Jetson AGX Orin源码编译安装CV-CUDA-CSDN博客
GitHub - Cambricon/CNStream: CNStream is a streaming framework for building Cambricon machine learning pipelines http://forum.cambricon.com https://gitee.com/SolutionSDK/CNStream
easydk/samples/simple_demo/common/video_decoder.cpp at master · Cambricon/easydk · GitHub
aclStream流处理多路并发Pipeline框架中 视频解码 代码调用流程整理、类的层次关系整理、回调函数赋值和调用流程整理-CSDN博客
aclStream流处理多路并发Pipeline框架中VEncode Module代码调用流程整理、类的层次关系整理、回调函数赋值和调用流程整理-CSDN博客
FFmpeg/doc/examples at master · FFmpeg/FFmpeg · GitHub
GitHub - CVCUDA/CV-CUDA: CV-CUDA™ is an open-source, GPU accelerated library for cloud-scale image processing and computer vision.
如何使用FFmpeg的解码器—FFmpeg API教程 · FFmpeg原理
C++ API — CV-CUDA Beta documentation (cvcuda.github.io)
CV-CUDA/tests/cvcuda/system at main · CVCUDA/CV-CUDA · GitHub
Resize — CV-CUDA Beta documentation
CUDA Runtime API :: CUDA Toolkit Documentation
CUDA Toolkit Documentation 12.6 Update 1
完整指南:CNStream流处理多路并发框架适配到NVIDIA Jetson Orin (一) 依赖库编译、第三方库编译安装-CSDN博客
完整指南:CNStream流处理多路并发框架适配到NVIDIA Jetson Orin (二) 源码架构流程梳理、代码编写-CSDN博客