编译安装qemu-devel @FreeBSD

缘起

使用cbsd创建riscv jail的时候提示：

you have no qemu-user, please install qemu-devle with BSD_USER and STATIC ops (emulators/qemu-devel)

使用pkg安装之后，创建的riscv jail启动报错：

Starting jail: fbriscv, parallel timeout=5
chroot: /bin/sh: Exec format error
* Copying emulator to /bin
emulator error: /usr/sbin/chroot /usr/jails/basejail/base_riscv_riscv64_14.1 /bin/qemu-riscv64-static /bin/sh -c "echo ping": Assertion failed: (p_rcu_reader->depth != 0), function rcu_read_unlock, file /wrkdirs/usr/ports/emulators/qemu-user-static-devel/work/qemu-bsd-user-64ff0f053df/include/qemu/rcu.h, line 101.

于是准备重新编译安装qemu-devel

开工

首先安装prots系统：

git clone --depth 1 https://git.FreeBSD.org/ports.git /usr/ports

编译安装qemu-devel

进入/usr/ports/emulators/qemu-devel

执行make config

make config

选中STATIC_LINK 回车

注意选了STATIC_LINK就不能选X11 support 和GTK3 GTK 3 GUI toolkit support

经实验，选中STATIC_LINK无法编译成功，要去掉才行。

另外编译的时候需要用

make

而不能用make -j 8 ，带了-j参数会编译失败。

编译安装后提示：

FreeBSD host notes
==================

- Needs to set net.link.tap.user_open sysctl in order to use /dev/tap*
  networking as non-root.  Don't forget to adjust device node
  permissions in /etc/devfs.rules.

- slirp (usermode networking) is fixed now, you still have to manually
  do: echo nameserver 10.0.2.3 >/etc/resolv.conf but that is normal. And
  you have to wait a bit for dhclient to do its thing; traffic to
  address 10.0.2.2 is routed to 127.1 on the host.

- The -smb option (smb-export local dir to guest using the default slirp
  networking) needs the samba port/package installed in addition to
  qemu. (SAMBA knob.)

- If you want to use usb devices connected to the host in the guest
  you can use usbredir over the network (see below); also unless you are
  running qemu as root you then need to fix permissions for /dev/ugen*
  device nodes: put a rule in /etc/devfs.rules, activate it in
  /etc/rc.conf and run /etc/rc.d/devfs restart.  Example devfs.rules:

	[ugen_ruleset=20]
	add path 'ugen*' mode 660 group operator

  corresponding rc.conf line:

	devfs_system_ruleset="ugen_ruleset"

- If you want to test the new usb network redirection (USBREDIR option)
  see this thread by Hans de Goede <hdegoede <at> redhat.com>:

	http://thread.gmane.org/gmane.comp.emulators.qemu/110176/focus=110183

  Quote:

  Example usage:

  1) Start usbredirserver for a usb device:
  sudo usbredirserver 045e:0772
  2) Start qemu with usb2 support + a chardev talking to usbredirserver +
     a usb-redir device using this chardev:
  qemu -usb \
    -readconfig docs/ich9-ehci-uhci.cfg \
    -chardev socket,id=usbredirchardev,host=localhost,port=4000 \
    -device usb-redir,chardev=usbredirchardev,id=usbredirdev ...

  [you would replace docs/ich9-ehci-uhci.cfg with e.g.
  /usr/local/share/doc/qemu/docs/ich9-ehci-uhci.cfg, but turns out
  ehci was broken for me here with FreeBSD guests and the previous
  qemu version at least, I got:

	FETCHENTRY: entry at 22C5484 is of type 2 which is not supported yet
processing error - resetting ehci HC
	Assertion failed: (0), function ehci_advance_state, file /data/ports/emulators/qemu-devel/work/qemu-0.15.0/hw/usb-ehci.c, line 2045.

  The new qemu version works better tho.]

- Still usb: since the hub is no longer attached to the uchi controller and
  the wakeup mechanism, resume interrupt is not implemented yet linux guests
  will suspend the bus, i.e. they wont see devices usb_add'ed after its
  (linux') uhci module got loaded.  Workaround: either add devices before
  linux loads the module or rmmod and modprobe it afterwards.  [Not sure
  if this still applies to the new libusb host code used on recent
  10-current.]

- If you get repeated `atapi_poll called!' console messages with FreeBSD
  guests or other weird cdrom problems then thats probably because the guest
  has atapicam loaded, which for reasons still to be determined has problems
  with qemu's now by default enabled cdrom dma.  You can build the port with
  CDROM_DMA disabled to disable it.  [Looks like this is fixed in recent
  FreeBSD guest versions.]

- If you build qemu wihout SDL and then get crashes running it try passing it
  -nographic.

- qemu's network boot roms (-boot n) have a bug when bootfiles sizes are a
  multiple of blksize, if this affects you (like with FreeBSD's /boot/pxeboot)
  you can do like

	cp /boot/pxeboot pxeboot-qemu && chmod +w pxeboot-qemu && echo >>pxeboot-qemu

  and then use pxeboot-qemu. Actually you need recent btx code because
  of the real mode boot problem, so use at least pxeboot from there.

- If you use slirp (usernet, the default) and want to mount nfs into the
  guest and you are not running qemu as root, then mountd(8) on the
  exporting box needs to be run with -n in order to accept requests from
  ports >= 1024.

- (not FreeBSD-specific:) There have been reports of qcow2 corruption with (at
  least) win2k guests on recent kvm (which uses similar qcow2 code than qemu
  now, see this thread:

	http://lists.gnu.org/archive/html/qemu-devel/2009-02/msg00713.html -

  the consensus on that thread seems to be that qcow(2) code has always been
  experimental and you should use raw images if you want reliability; raw is
  also usually faster.)  You should be able to migrate existing images to raw
  using qemu-img(1)'s convert function; raw doesn't support advanced features
  like snapshots tho.  [a few important qcow2 bugfixed have been committed in
  the meantime so this _might_ be less of an issue now; and meanwhile there
  also is the new qed format - I don't know how stable that one is.]

- (also not FreeBSD-specific:)  It is recommended to pass raw images using the
  new -drive syntax, specifying format=raw explicitly in order to avoid
  malicious guests being able to exploit the format autodetection thats
  otherwise getting used.  (Not that you should run malicious guests anyway,
  but this eleminates at least a known attack vector.)

- qemu now has improved physical cdrom support, but still there is at
  least one known problem: you need to have the guest eject the disc if you
  want to change it/take it out, or otherwise the guest may continue using
  state (like size) of the old disc.  (You can also do like `change ide1-cd0
  /dev/acd0' in the monitor after taking out the disc if a guest cannot eject
  it itself.)

- The default configuration location (qemu-ifup script etc.) has been changed
  from /etc to PREFIX/etc (usually /usr/local/etc).  Move your files
  accordingly.

- The pcap code (-net nic... -net pcap,ifname=...) should work properly now,
  with only one exception:  Advanced features like TSO used on the host
  interface can cause oversize packets which now do get truncated to avoid
  confusing/panicing guests but of course still will cause retransmissions.
  So if you see slow throughput and `pcap_send: packet size > ..., truncating'
  messages on qemu's tty try disabling TSO etc on the host interface at least
  while using pcap.

- kqemu is no longer supported in qemu upstream after the 0.11 branch
  was created, which means also not in this version.  (Linux has moved
  on to kvm now for qemu(-like) virtualization needs, so if you want qemu
  to go faster and don't want to switch to virtualbox or stick to the older
  emulators/qemu port which is at 0.11.1 atm and as such still supports
  kqemu you should help getting the FreeBSD kvm port updated and
  completed:

	http://wiki.freebsd.org/FabioChecconi/PortingLinuxKVMToFreeBSD

  )

===> SECURITY REPORT: 
      This port has installed the following files which may act as network
      servers and may therefore pose a remote security risk to the system.
/usr/local/bin/qemu-img
/usr/local/bin/qemu-system-nios2
/usr/local/bin/qemu-system-riscv64
/usr/local/bin/qemu-system-mips64
/usr/local/bin/qemu-system-mips64el
/usr/local/bin/qemu-system-loongarch64
/usr/local/bin/qemu-system-microblaze
/usr/local/bin/qemu-system-sparc
/usr/local/bin/qemu-system-riscv32
/usr/local/bin/qemu-nbd
/usr/local/bin/qemu-system-ppc64
/usr/local/bin/qemu-system-avr
/usr/local/bin/qemu-system-sh4
/usr/local/bin/qemu-system-or1k
/usr/local/bin/qemu-system-cris
/usr/local/bin/qemu-x86_64
/usr/local/bin/qemu-system-mipsel
/usr/local/bin/qemu-system-alpha
/usr/local/bin/qemu-edid
/usr/local/bin/qemu-system-sparc64
/usr/local/bin/qemu-arm
/usr/local/bin/qemu-storage-daemon
/usr/local/bin/qemu-system-ppc
/usr/local/bin/qemu-system-microblazeel
/usr/local/bin/qemu-system-xtensa
/usr/local/bin/qemu-system-aarch64
/usr/local/bin/qemu-system-xtensaeb
/usr/local/bin/qemu-io
/usr/local/bin/qemu-i386
/usr/local/bin/qemu-system-sh4eb
/usr/local/bin/qemu-system-arm
/usr/local/bin/qemu-system-m68k
/usr/local/bin/qemu-system-i386
/usr/local/bin/qemu-system-x86_64
/usr/local/bin/qemu-system-s390x
/usr/local/bin/qemu-system-mips
/usr/local/bin/qemu-system-rx
/usr/local/bin/qemu-system-hppa
/usr/local/bin/qemu-system-tricore

      If there are vulnerabilities in these programs there may be a security
      risk to the system. FreeBSD makes no guarantee about the security of
      ports included in the Ports Collection. Please type 'make deinstall'
      to deinstall the port if this is a concern.

      For more information, and contact details about the security
      status of this software, see the following webpage: 
https://www.qemu.org/

编译安装的时候有很多波折，见后面调试部分

pkg 安装qemu

本来已经pkg 安装qemu-devel，再把qemu装上，发现还是新装了几个软件

pkg install qemu

装好后提示：

Message from qemu-8.2.2_1:

--
FreeBSD host notes
==================

- Needs to set net.link.tap.user_open sysctl in order to use /dev/tap*
networking as non-root. Don't forget to adjust device node permissions in
/etc/devfs.rules.

- Expect timer problems when guest kernel HZ is > hosts. The linux 2.6
kernel uses 1000 by default btw. (changed to 250 later, and recent linux
kernels now no longer have a fixed HZ, aka `tickless kernel'...)
Enabling /dev/rtc doesn't seem to help either (not included since it needs
a patch to emulators/rtc.)

- The -smb option (smb-export local dir to guest using the default
slirp networking) needs the samba port/package installed
in addition to qemu. (SAMBA knob.)

- If you want to use usb devices connected to the host in the guest
yot can use usbredir over the network (see below); also unless you are
running qemu as root you then need to fix permissions for /dev/ugen*
device nodes: if you are on 5.x or later (devfs) put a rule in
/etc/devfs.rules, activate it in /etc/rc.conf and run /etc/rc.d/devfs
restart. Example devfs.rules:

[ugen_ruleset=20]
add path 'ugen*' mode 660 group operator

corresponding rc.conf line:

devfs_system_ruleset="ugen_ruleset"

- Still usb: since the hub is no longer attached to the uchi controller and
the wakeup mechanism, resume interrupt is not implemented yet linux guests
will suspend the bus, i.e. they wont see devices usb_add'ed after its
(linux') uhci module got loaded. Workaround: either add devices before
linux loads the module or rmmod and modprobe it afterwards. [Not sure
if this still applies to the new libusb host code used on recent
10-current.]

- If you get repeated `atapi_poll called!' console messages with FreeBSD
guests or other weird cdrom problems then thats probably because the guest
has atapicam loaded, which for reasons still to be determined has problems
with qemu's now by default enabled cdrom dma. You can build the port with
CDROM_DMA disabled to disable it. [Looks like this is fixed in recent
FreeBSD guest versions.]

- If you build qemu wihout SDL and then get crashes running it try passing it
-nographic. This should probably be default in that case...

- qemu's network boot roms (-boot n) have a bug when bootfiles sizes are a
multiple of blksize, if this affects you (like with FreeBSD's /boot/pxeboot)
you can do like

cp /boot/pxeboot pxeboot-qemu && chmod +w pxeboot-qemu && echo >>pxeboot-qemu

and then use pxeboot-qemu. Actually you need recent btx code
(from after 7.0 was released) because of the real mode boot
problem, so use at least pxeboot from there. And I just did that
for the pxeboot extracted out of

ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/200805/7.0-STABLE-200805-i386-bootonly.iso

and placed it here:

http://people.freebsd.org/~nox/qemu/pxeboot-qemu

- If you use slirp (usernet, the default) and want to mount nfs into the guest
and you are not running qemu as root, then mountd(8) on the exporting box
needs to be run with -n in order to accept requests from ports >= 1024.
root@fbhost:/usr/ports/emulators/qemu-devel #

后来又重新安装了qemu-devel

pkg install qemu-devel

当时之所以编译安装，就是为了STATIC_LINK这个参数。但是编译安装的时候用了这个参数也编译不过去，所以不如直接pkg按钻个方便了。

调试

报错pkg-static: open(/usr/ports/textproc/py-sphinx/work-py311/.metadir.py311-sphinx): No such file or directory

--- security-check ---
--- fake-pkg.py311-sphinx ---
===> Registering installation for py311-sphinx-5.3.0_1,1 as automatic
pkg-static: open(/usr/ports/textproc/py-sphinx/work-py311/.metadir.py311-sphinx): No such file or directory
*** [fake-pkg.py311-sphinx] Error code 1

make[2]: stopped in /usr/ports/textproc/py-sphinx

尝试 pkg install textproc/py-sphinx

不行。尝试pkg install textproc/py-sphinx_rtd_theme

还是不行。尝试pkg install python311 ，还是不行

删除work-py311目录，不行

期间发现安装的python311无法执行，原来是老终端里没有更新path，重新登录即可执行python3.11

执行python3.11 -m ensurepip ,这样就安装好了Successfully installed pip-24.0 setuptools-65.5.0

还是不行。在python里面安装sphinx_rtd_theme

python3.11 -m pip install sphinx_rtd_theme

还是不行，把Makefile修改一下，去掉sphinx的依赖

# BUILD_DEPENDS=        sphinx-build:textproc/py-sphinx \
                ${PYTHON_PKGNAMEPREFIX}sphinx_rtd_theme>0:textproc/py-sphinx_rtd_theme@${PY_FLAV
OR} \

出现新的报错

如何提高pip安装软件的速度

python3.11 -m pip config set global.index-url  https://mirror.baidu.com/pypi/simple

报错pkg-static: open(/usr/ports/devel/bison/work/.metadir.bison): No such file or directory

===> Registering installation for bison-3.8.2_2,1 as automatic
pkg-static: open(/usr/ports/devel/bison/work/.metadir.bison): No such file or directory
*** [fake-pkg.bison] Error code 1

pkg install bison-3.8.2_1,1
搞定

报错没有cmake

pkg install gmake 搞定

报错没有ninja

发现Makefile还有几个需求，一起装了

pkg install ninja perl5 pkgconfig

结果不行，还是一个一个装吧：pkg install ninja

安装pkg install print/texinfo

报错../meson.build:2420:15: ERROR: C prefer_static library 'inotify' not found

Header "sys/inotify.h" has symbol "inotify_init" : YES
Header "sys/inotify.h" has symbol "inotify_init1" : YES

../meson.build:2420:15: ERROR: C prefer_static library 'inotify' not found

A full log can be found at /usr/ports/emulators/qemu-devel/work/qemu-ff202817dc2b0b3b42992fa7f1c
e503f081068fe/build/meson-logs/meson-log.txt
ERROR: meson setup failed
===> Script "configure" failed unexpectedly.
Please report the problem to bofh@FreeBSD.org [maintainer] and attach the
"/usr/ports/emulators/qemu-devel/work/qemu-ff202817dc2b0b3b42992fa7f1ce503f081068fe/config.log"
including the output of the failure of your make command. Also, it might be
a good idea to provide an overview of all packages installed on your system
怀疑是把Makefile里面的sphinx注释掉的原因。把注释去掉，还是报错

BUILD_DEPENDS=  sphinx-build:textproc/py-sphinx

到目录/usr/ports/textproc/py-sphinx_rtd_theme

然后make install

再回到目录/usr/ports/emulators/qemu-devel

make install ,依旧。

针对这个报错，重新编译meson

cd /usr/ports/devel/meson && make install

依旧。

但是把STATIC_LINK选项去掉，能编译下去了。

最后编译完了之后报错...

编译结束之后报错几个文件找不到

把make -j 8 改成make ，就又能继续下去了。

但是最终make install的时候报错：

===> Registering installation for qemu-devel-8.3.0.20240229
pkg-static: Unable to access file /usr/ports/emulators/qemu-devel/work/stage/usr/local/include/fdt.h:No such file or directory
pkg-static: Unable to access file /usr/ports/emulators/qemu-devel/work/stage/usr/local/include/libfdt.h:No such file or directory
pkg-static: Unable to access file /usr/ports/emulators/qemu-devel/work/stage/usr/local/include/libfdt_env.h:No such file or directory
pkg-static: Unable to access file /usr/ports/emulators/qemu-devel/work/stage/usr/local/lib/libfdt.a:No such file or directory
pkg-static: Unable to access file /usr/ports/emulators/qemu-devel/work/stage/usr/local/libdata/pkgconfig/libfdt.pc:No such file or directory
*** Error code 1
可能是前面pkg install qemu导致的，执行pkg install qemu-devel

然后再来make install，提示需要先uninstall reinstall ，结果还是报错文件找不到。

最后是重新删除build目录，创建build目录，重新make config ，make install，终于编译成功了。