femu使用记录
- 环境搭建
- libnvme：libnvme.so.1: cannot open shared object file: No such file or directory
- 调试方法
Briefly speaking, FEMU is a fast, accurate, scalable, and extensible NVMe SSD Emulator. Based upon QEMU/KVM, FEMU is exposed to Guest OS (Linux) as an NVMe block device (e.g. /dev/nvme0nX). It supports emulating different types of SSDs
这篇博客用于记录femu搭建与使用过程中遇到的问题，持续更新
环境搭建
首先是搭建环境，刚开始按照博客安装FEMU，并使用FEMU模拟SSD黑盒、OCSSD、NoSSD。，自己制作镜像，但失败了，故选择第一种方法，下载femu提供的镜像，下载完成后直接运行run_zns.sh脚本，不需要第二种方法的安装操作系统，修改grub等操作
# 邮件内容
Hi,
Thank you again for your interest in FEMU!
The tarball size is 1.4GB. After decompression, the VM image file size is ~14GB.
Downloading and decompression might take a while depending on your Internet
connection speed and CPU processing capability. Please wait patiently!
Below are the steps to download, decompress, and verify the FEMU VM image.
mkdir -p ~/images
cd ~/images
wget http://people.cs.uchicago.edu/~huaicheng/femu/femu-vm.tar.xz
tar xJvf femu-vm.tar.xz
After these steps, you will get two files: "u20s.qcow2" and "u20s.md5sum".
You can verify the integrity of the VM image file by doing:
md5sum u20s.qcow2 > tmp.md5sum
diff tmp.md5sum u20s.md5sum
If diff complains that the above two files differ, then the VM image file is
corrupted. Please redo the above steps.
The user account and guest OS of the VM:
- username: femu
- passwd : femu
- Guest OS: Ubuntu 20.04.1 server, with kernel 5.4
If you think FEMU is useful to your project, we appreciate you could consider
starring the FEMU github repo to support us, thanks!
Best,
Huaicheng
以root权限运行nvme list能看到zns ssd，则表示搭建环境成功
root@fvm /h/f/libnvme (master)# nvme list
Node SN Model Namespace Usage Format FW Rev
--------------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1 vZNSSD0 FEMU ZNS-SSD Controller 1 4.29 GB / 4.29 GB 512 B + 0 B 1.0
可使用ssh连接虚拟机，端口号即为hostfwd定义的端口号（8080），vscode连接更方便
自己制作镜像经验
源码编译qemu
sudo apt-get install git fakeroot build-essential ncurses-dev xz-utils libssl-dev bc flex libelf-dev bison meson
wget https://download.qemu.org/qemu-8.0.0.tar.xz
tar xvJf qemu-8.0.0.tar.xz
cd qemu-8.0.0
./configure --enable-kvm --enable-debug --enable-vnc --enable-werror --target-list="x86_64-softmmu" --enable-sdl
make
make install
qemu运行虚拟机无反应，只输出一行提示信息:VNC server running on 127.0.0.1:5900
libnvme：libnvme.so.1: cannot open shared object file: No such file or directory
libnvme
This is the libnvme development C library. libnvme provides type definitions for NVMe specification structures, enumerations, and bit fields, helper functions to construct, dispatch, and decode commands and payloads, and utilities to connect, scan, and manage nvme devices on a Linux system.
使用以下命令编译源文件
gcc zns_rw.c -o zns_rw -lnvme
执行
./zns_rw
./zns_rw: error while loading shared libraries: libnvme.so.1: cannot open shared object file: No such file or directory
使用ldd列出文件依赖的动态库
ldd zns_rw
linux-vdso.so.1 (0x00007ffe90b75000)
libnvme.so.1 => not found
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f396cb49000)
/lib64/ld-linux-x86-64.so.2 (0x00007f396cd49000)
也就是说链接器不知道去哪找到libnvme.so.1，查看libnvme库install的输出
meson install -C .build
ninja: Entering directory `.build'
ninja: no work to do.
Installing src/libnvme.so.1.4.0 to /usr/local/lib64
Installing src/libnvme-mi.so.1.4.0 to /usr/local/lib64
Installing /home/femu/libnvme/src/libnvme.h to /usr/local/include
Installing /home/femu/libnvme/src/libnvme-mi.h to /usr/local/include
Installing /home/femu/libnvme/src/nvme/api-types.h to /usr/local/include/nvme
Installing /home/femu/libnvme/src/nvme/fabrics.h to /usr/local/include/nvme
Installing /home/femu/libnvme/src/nvme/filters.h to /usr/local/include/nvme
Installing /home/femu/libnvme/src/nvme/ioctl.h to /usr/local/include/nvme
Installing /home/femu/libnvme/src/nvme/linux.h to /usr/local/include/nvme
Installing /home/femu/libnvme/src/nvme/log.h to /usr/local/include/nvme
Installing /home/femu/libnvme/src/nvme/nbft.h to /usr/local/include/nvme
Installing /home/femu/libnvme/src/nvme/tree.h to /usr/local/include/nvme
Installing /home/femu/libnvme/src/nvme/types.h to /usr/local/include/nvme
Installing /home/femu/libnvme/src/nvme/util.h to /usr/local/include/nvme
Installing /home/femu/libnvme/src/nvme/mi.h to /usr/local/include/nvme
Installing /home/femu/libnvme/.build/meson-private/libnvme.pc to /usr/local/lib64/pkgconfig
Installing /home/femu/libnvme/.build/meson-private/libnvme-mi.pc to /usr/local/lib64/pkgconfig
故设置动态库路径为/usr/local/lib64重新编译
gcc zns_rw.c -o zns_rw -lnvme -L /usr/local/lib64/ -Wl,-rpath=/usr/local/lib64/
ldd zns_rw
linux-vdso.so.1 (0x00007ffe233b1000)
libnvme.so.1 => /usr/local/lib64/libnvme.so.1 (0x00007fc367ee5000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc367cec000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc367f1d000)
-L选项告诉链接器在链接时寻找动态库的路径
-Wl,-rpath选项告诉链接器在运行时寻找动态库的路径
man gcc | grep "Wl"
-Wl,option
Pass option as an option to the linker. If option contains commas, it is split into multiple options at the commas. You can use this syntax to pass an argument to the option. For example, -Wl,-Map,output.map passes -Map output.map to the
linker. When using the GNU linker, you can also get the same effect with -Wl,-Map=output.map.
ld --help | grep "rpath"
Just link symbols (if directory, same as --rpath)
-rpath PATH Set runtime shared library search path
-rpath-link PATH Set link time shared library search path
相关博客
I don’t understand -Wl,-rpath -Wl,
gcc编译过程、gcc命令参数、静态库和动态库搜索路径
《c专家编程》读书笔记：第五章 对链接的思考
调试方法
一：重新编译femu，在femu-compile.sh configure后加上–enable-debug选项，保证优化等级为-O0
# 编译输出
Compilation
host CPU : x86_64
host endianness : little
C compiler : cc -m64 -mcx16
Host C compiler : cc -m64 -mcx16
C++ compiler : c++ -m64 -mcx16
CFLAGS : -g -O0
CXXFLAGS : -g -O0
二：gdb运行程序
gdb --args x86_64-softmmu/qemu-system-x86_64 \
-name "FEMU-ZNSSD-VM" \
-enable-kvm \
-cpu host \
-smp 4 \
-m 4G \
-device virtio-scsi-pci,id=scsi0 \
-device scsi-hd,drive=hd0 \
-drive file=/root/images/u20s.qcow2,if=none,aio=native,cache=none,format=qcow2,id=hd0 \
-device femu,devsz_mb=4096,femu_mode=3 \
-net user,hostfwd=tcp::8080-:22 \
-net nic,model=virtio \
-nographic \
-qmp unix:./qmp-sock,server,nowait 2>&1 | tee log
(gdb) handle SIGUSR1 nostop
Signal Stop Print Pass to program Description
SIGUSR1 No Yes Yes User defined signal 1
三： 调试演示
而后就是常规的gdb操作了，拿一个错误的程序来调试
(gdb) b zns.c:1110
Breakpoint 1 at 0x489f0b: file ../hw/femu/zns/zns.c, line 1110.
(gdb) r
# 开启虚拟机
错误的读写测试代码
// SPDX-License-Identifier: LGPL-2.1-or-later
/**
* This file is part of libnvme.
* Copyright (c) 2020 Western Digital Corporation or its affiliates.
*
* Authors: Keith Busch <keith.busch@wdc.com>
*/
/**
* Search out for ZNS type namespaces, and if found, report their properties.
*/
#include <stdio.h>
#include <string.h>
#include <stdbool.h>
#include <stdlib.h>
#include <libnvme.h>
#include <inttypes.h>
#include <errno.h>
#define TEST_SIZE (4096)
#define TEST_BLOCK (8)
static void rw_test(nvme_ns_t n) {
char write_data[TEST_SIZE];
char read_data[TEST_SIZE];
memset(write_data, 0x12, TEST_SIZE);
memset(read_data, 0x34, TEST_SIZE);
int fd = nvme_ns_get_fd(n);
int nsid = nvme_ns_get_nsid(n);
uint64_t req_res;
struct nvme_zns_append_args arg;
arg.data = write_data;
arg.data_len = TEST_SIZE;
arg.fd = fd;
arg.nsid = nsid;
arg.nlb = TEST_BLOCK;
arg.zslba = 0;
arg.result = &req_res;
arg.args_size = sizeof(arg);
arg.timeout = 1000;
int ret = nvme_zns_append(&arg);
printf("ret:%d req res:%lx erron:%d\n", ret, req_res, errno);
}
int main() {
nvme_subsystem_t s;
nvme_root_t r;
nvme_host_t h;
nvme_ctrl_t c;
nvme_ns_t n;
r = nvme_scan(NULL);
if (!r) return -1;
nvme_for_each_host(r, h) {
nvme_for_each_subsystem(h, s) {
nvme_subsystem_for_each_ctrl(s, c) {
nvme_ctrl_for_each_ns(c, n) {
if (nvme_ns_get_csi(n) == NVME_CSI_ZNS) {
rw_test(n);
}
}
}
}
}
nvme_free_tree(r);
}
在虚拟机中执行该程序（用vscode连接8080端口）
gdb显示
Ubuntu 20.04.1 LTS fvm ttyS0
fvm login: [Switching to Thread 0x7ffff5c2d700 (LWP 423223)]
Thread 9 "qemu-system-x86" hit Breakpoint 1, zns_do_write (n=0x555557cae9d0, req=0x7ffedc36a3a0, append=true, wrz=false) at ../hw/femu/zns/zns.c:1110
1110 printf("****************Append Failed***************\n");
(gdb) p/x status
$1 = 0x4002
(gdb)
也可以在执行测试程序前再打断点
类似于以下操作步骤
Thread 1 "qemu-system-x86" received signal SIGINT, Interrupt.
0x00007ffff75c2a96 in __ppoll (fds=0x555556f074a0, nfds=20, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:44
44 ../sysdeps/unix/sysv/linux/ppoll.c: 没有那个文件或目录.
(gdb) b zns_do_write
Breakpoint 1 at 0x5555559ddcab: file ../hw/femu/zns/zns.c, line 1051.
(gdb) c
Continuing.
[Switching to Thread 0x7ffff5c2d700 (LWP 426245)]
Thread 9 "qemu-system-x86" hit Breakpoint 1, zns_do_write (n=0x1, req=0x7ffedc0eb000, append=false, wrz=true) at ../hw/femu/zns/zns.c:1051
1051 {
通过debug
四：解决问题的经历
1 刚开始的时候想要调试femu，就想使用6.081 qemu调试内核的方式，但也不知道符号表在哪，并且femu虚拟出来的是一个设备，总觉得和调试内核不一样，后面看到博客qemu侧 块设备调试记录（一），具体内容没怎么看，只是发现原来可以直接将qemu当作可执行程序来调试，故用同样的方法试了试femu，发现可以。
2 gdb运行的时候时不时有输出 “Program received signal SIGUSR1, User defined signal 1.” 打断运行，故执行“handle SIGUSR1 nostop”，使得用户定义中断不停止运行。
3 错误击中断点时用bt发现各个变量都优化掉了“optimized out”，故需要修改编译选项
看到执行femu-compile.sh的输出中有如下信息
Compilation
host CPU : x86_64
host endianness : little
C compiler : cc -m64 -mcx16
Host C compiler : cc -m64 -mcx16
C++ compiler : c++ -m64 -mcx16
CFLAGS : -g -O2
CXXFLAGS : -g -O2
全局搜索Compilation,-O,optimization等相关信息，然后一步一步倒退终于看到了相应的选项 --enable-debug
// meson.build
option_cflags = (get_option('debug') ? ['-g'] : [])
if get_option('optimization') != 'plain'
option_cflags += ['-O' + get_option('optimization')]
endif
// configure
--enable-debug)
# Enable debugging options that aren't excessively noisy
debug_tcg="yes"
meson_option_parse --enable-debug-mutex ""
meson_option_add -Doptimization=0
fortify_source="no"
加上 --enable-debug选项后
Compilation
host CPU : x86_64
host endianness : little
C compiler : cc -m64 -mcx16
Host C compiler : cc -m64 -mcx16
C++ compiler : c++ -m64 -mcx16
CFLAGS : -g -O0
CXXFLAGS : -g -O0
