接前一篇文章:
本文内容参考:
《趣谈Linux操作系统》 —— 刘超,极客时间
《QEMU/KVM》源码解析与应用 —— 李强,机械工业出版社
QEMU内存管理模型
特此致谢!
QEMU内存初始化
1. 基本结构
在开始介绍内存初始化的时候,首先需要对QEMU中几个与内存相关的数据结构进行介绍。
(1)AddressSpace
首先是AddressSpace结构。AddressSpace结构用来表示一个虚拟机或者虚拟CPU能够访问的所有物理地址。注意,这里的“访问”和“能够访问”并不是一回事。与进程的地址空间一样,一个进程的虚拟地址空间为4GB(32位),但这并不是说操作系统需要为进程分配这么大的空间。同样,QEMU中的AddressSpace表示的是一段地址空间,整个系统可以有一个全局的地址空间,CPU可以有自己的地址空间视角,设备也可以有自己的地址空间视角。
AddressSpace的意义从名字就可以看出,是针对整个地址空间,最顶层的内存管理结构。对于x86架构来说,其实是有两种寻址方式的,一种是memory一种是IO。相应的QEMU会维护两个AddressSpace结构,address_space_memory和address_space_io。
AddressSpace的定义在include/qemu/typedefs.h中,如下:
typedef struct AddressSpace AddressSpace;
而struct AddressSpace的定义在include/exec/memory.h中,代码如下:
/**
* struct AddressSpace: describes a mapping of addresses to #MemoryRegion objects
*/
struct AddressSpace {
/* private: */
struct rcu_head rcu;
char *name;
MemoryRegion *root;
/* Accessed via RCU. */
struct FlatView *current_map;
int ioeventfd_nb;
struct MemoryRegionIoeventfd *ioeventfds;
QTAILQ_HEAD(, MemoryListener) listeners;
QTAILQ_ENTRY(AddressSpace) address_spaces_link;
};
根据struct AddressSpace的注释,此结构描述了一个到MemoryRegion对象的地址映射。
其中:
- MemoryRegion *root成员表示AddressSpace对应的一个根MemoryRegion。
- struct FlatView *current_map成员表示该地址空间是一个平坦模式下的视图。
- QEMU的其它子系统可以注册地址变更的事件,所有注册的信息都通过QTAILQ_HEAD(, MemoryListener) listeners成员连接起来。
- 所有的AddressSpace通过address_spaces_link这个node连接起来(QTAILQ_ENTRY(AddressSpace) address_spaces_link),链表头是address_spaces。
AddressSpace的定义其实比较简单,有两个QEMU内存管理框架的核心结构,就是MemoryRegion和FlatView,两者配合使用,MemoryRegion是基础(MemoryRegion),用于正向管理;Flatview用于反向查找。
对于一个地址空间(AddressSpace),会有多个内存区域(MemoryRegion)组成树型结构。这其中,root是这棵树的根。另外,还有一个MemoryListener链表,当内存区域发生变化的时候,需要做一些动作,使得用户态和内核态能够协同,就是由这些MemoryListener来完成的。
在QEMU monitor中输入“info qtree”,可以看到所有的AddressSpace,如下所示:
$ qemu-system-x86_64 -vnc :1 -monitor stdio
QEMU 6.2.0 monitor - type 'help' for more information
(qemu) info mtree
address-space: memory
0000000000000000-ffffffffffffffff (prio 0, i/o): system
0000000000000000-0000000007ffffff (prio 0, ram): alias ram-below-4g @pc.ram 0000000000000000-0000000007ffffff
0000000000000000-ffffffffffffffff (prio -1, i/o): pci
00000000000a0000-00000000000bffff (prio 1, i/o): vga-lowmem
00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
00000000000e0000-00000000000fffff (prio 1, rom): alias isa-bios @pc.bios 0000000000020000-000000000003ffff
00000000fd000000-00000000fdffffff (prio 1, ram): vga.vram
00000000feb80000-00000000feb9ffff (prio 1, i/o): e1000-mmio
00000000febb0000-00000000febb0fff (prio 1, i/o): vga.mmio
00000000febb0000-00000000febb017f (prio 0, i/o): edid
00000000febb0400-00000000febb041f (prio 0, i/o): vga ioports remapped
00000000febb0500-00000000febb0515 (prio 0, i/o): bochs dispi interface
00000000febb0600-00000000febb0607 (prio 0, i/o): qemu extended regs
00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 00000000000a0000-00000000000bffff
00000000000c0000-00000000000c3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c0000-00000000000c3fff
00000000000c4000-00000000000c7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c4000-00000000000c7fff
00000000000c8000-00000000000cbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000c8000-00000000000cbfff
00000000000cb000-00000000000cdfff (prio 1000, ram): alias kvmvapic-rom @pc.ram 00000000000cb000-00000000000cdfff
00000000000cc000-00000000000cffff (prio 1, ram): alias pam-rom @pc.ram 00000000000cc000-00000000000cffff
00000000000d0000-00000000000d3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d0000-00000000000d3fff
00000000000d4000-00000000000d7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d4000-00000000000d7fff
00000000000d8000-00000000000dbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000d8000-00000000000dbfff
00000000000dc000-00000000000dffff (prio 1, ram): alias pam-rom @pc.ram 00000000000dc000-00000000000dffff
00000000000e0000-00000000000e3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e0000-00000000000e3fff
00000000000e4000-00000000000e7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e4000-00000000000e7fff
00000000000e8000-00000000000ebfff (prio 1, ram): alias pam-ram @pc.ram 00000000000e8000-00000000000ebfff
00000000000ec000-00000000000effff (prio 1, ram): alias pam-ram @pc.ram 00000000000ec000-00000000000effff
00000000000f0000-00000000000fffff (prio 1, ram): alias pam-rom @pc.ram 00000000000f0000-00000000000fffff
00000000fec00000-00000000fec00fff (prio 0, i/o): ioapic
00000000fed00000-00000000fed003ff (prio 0, i/o): hpet
00000000fee00000-00000000feefffff (prio 4096, i/o): apic-msi
address-space: I/O
0000000000000000-000000000000ffff (prio 0, i/o): io
0000000000000000-0000000000000007 (prio 0, i/o): dma-chan
0000000000000008-000000000000000f (prio 0, i/o): dma-cont
0000000000000020-0000000000000021 (prio 0, i/o): pic
0000000000000040-0000000000000043 (prio 0, i/o): pit
0000000000000060-0000000000000060 (prio 0, i/o): i8042-data
0000000000000061-0000000000000061 (prio 0, i/o): pcspk
0000000000000064-0000000000000064 (prio 0, i/o): i8042-cmd
0000000000000070-0000000000000071 (prio 0, i/o): rtc
0000000000000070-0000000000000070 (prio 0, i/o): rtc-index
000000000000007e-000000000000007f (prio 0, i/o): kvmvapic
0000000000000080-0000000000000080 (prio 0, i/o): ioport80
0000000000000081-0000000000000083 (prio 0, i/o): dma-page
0000000000000087-0000000000000087 (prio 0, i/o): dma-page
0000000000000089-000000000000008b (prio 0, i/o): dma-page
000000000000008f-000000000000008f (prio 0, i/o): dma-page
0000000000000092-0000000000000092 (prio 0, i/o): port92
00000000000000a0-00000000000000a1 (prio 0, i/o): pic
00000000000000b2-00000000000000b3 (prio 0, i/o): apm-io
00000000000000c0-00000000000000cf (prio 0, i/o): dma-chan
00000000000000d0-00000000000000df (prio 0, i/o): dma-cont
00000000000000f0-00000000000000f0 (prio 0, i/o): ioportF0
0000000000000170-0000000000000177 (prio 0, i/o): ide
00000000000001ce-00000000000001d1 (prio 0, i/o): vbe
00000000000001f0-00000000000001f7 (prio 0, i/o): ide
0000000000000376-0000000000000376 (prio 0, i/o): ide
0000000000000378-000000000000037f (prio 0, i/o): parallel
00000000000003b4-00000000000003b5 (prio 0, i/o): vga
00000000000003ba-00000000000003ba (prio 0, i/o): vga
00000000000003c0-00000000000003cf (prio 0, i/o): vga
00000000000003d4-00000000000003d5 (prio 0, i/o): vga
00000000000003da-00000000000003da (prio 0, i/o): vga
00000000000003f1-00000000000003f5 (prio 0, i/o): fdc
00000000000003f6-00000000000003f6 (prio 0, i/o): ide
00000000000003f7-00000000000003f7 (prio 0, i/o): fdc
00000000000003f8-00000000000003ff (prio 0, i/o): serial
00000000000004d0-00000000000004d0 (prio 0, i/o): elcr
00000000000004d1-00000000000004d1 (prio 0, i/o): elcr
0000000000000510-0000000000000511 (prio 0, i/o): fwcfg
0000000000000514-000000000000051b (prio 0, i/o): fwcfg.dma
0000000000000600-000000000000063f (prio 0, i/o): piix4-pm
0000000000000600-0000000000000603 (prio 0, i/o): acpi-evt
0000000000000604-0000000000000605 (prio 0, i/o): acpi-cnt
0000000000000608-000000000000060b (prio 0, i/o): acpi-tmr
0000000000000700-000000000000073f (prio 0, i/o): pm-smbus
0000000000000cf8-0000000000000cfb (prio 0, i/o): pci-conf-idx
0000000000000cf9-0000000000000cf9 (prio 1, i/o): piix3-reset-control
0000000000000cfc-0000000000000cff (prio 0, i/o): pci-conf-data
0000000000005658-0000000000005658 (prio 0, i/o): vmport
000000000000ae00-000000000000ae17 (prio 0, i/o): acpi-pci-hotplug
000000000000af00-000000000000af1f (prio 0, i/o): acpi-cpu-hotplug
000000000000afe0-000000000000afe3 (prio 0, i/o): acpi-gpe0
000000000000c000-000000000000c03f (prio 1, i/o): e1000-io
000000000000c040-000000000000c04f (prio 1, i/o): piix-bmdma-container
000000000000c040-000000000000c043 (prio 0, i/o): piix-bmdma
000000000000c044-000000000000c047 (prio 0, i/o): bmdma
000000000000c048-000000000000c04b (prio 0, i/o): piix-bmdma
000000000000c04c-000000000000c04f (prio 0, i/o): bmdma
address-space: cpu-memory-0
0000000000000000-ffffffffffffffff (prio 0, i/o): system
0000000000000000-0000000007ffffff (prio 0, ram): alias ram-below-4g @pc.ram 0000000000000000-0000000007ffffff
0000000000000000-ffffffffffffffff (prio -1, i/o): pci
00000000000a0000-00000000000bffff (prio 1, i/o): vga-lowmem
00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
00000000000e0000-00000000000fffff (prio 1, rom): alias isa-bios @pc.bios 0000000000020000-000000000003ffff
00000000fd000000-00000000fdffffff (prio 1, ram): vga.vram
00000000feb80000-00000000feb9ffff (prio 1, i/o): e1000-mmio
00000000febb0000-00000000febb0fff (prio 1, i/o): vga.mmio
00000000febb0000-00000000febb017f (prio 0, i/o): edid
00000000febb0400-00000000febb041f (prio 0, i/o): vga ioports remapped
00000000febb0500-00000000febb0515 (prio 0, i/o): bochs dispi interface
00000000febb0600-00000000febb0607 (prio 0, i/o): qemu extended regs
00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 00000000000a0000-00000000000bffff
00000000000c0000-00000000000c3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c0000-00000000000c3fff
00000000000c4000-00000000000c7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c4000-00000000000c7fff
00000000000c8000-00000000000cbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000c8000-00000000000cbfff
00000000000cb000-00000000000cdfff (prio 1000, ram): alias kvmvapic-rom @pc.ram 00000000000cb000-00000000000cdfff
00000000000cc000-00000000000cffff (prio 1, ram): alias pam-rom @pc.ram 00000000000cc000-00000000000cffff
00000000000d0000-00000000000d3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d0000-00000000000d3fff
00000000000d4000-00000000000d7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d4000-00000000000d7fff
00000000000d8000-00000000000dbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000d8000-00000000000dbfff
00000000000dc000-00000000000dffff (prio 1, ram): alias pam-rom @pc.ram 00000000000dc000-00000000000dffff
00000000000e0000-00000000000e3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e0000-00000000000e3fff
00000000000e4000-00000000000e7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e4000-00000000000e7fff
00000000000e8000-00000000000ebfff (prio 1, ram): alias pam-ram @pc.ram 00000000000e8000-00000000000ebfff
00000000000ec000-00000000000effff (prio 1, ram): alias pam-ram @pc.ram 00000000000ec000-00000000000effff
00000000000f0000-00000000000fffff (prio 1, ram): alias pam-rom @pc.ram 00000000000f0000-00000000000fffff
00000000fec00000-00000000fec00fff (prio 0, i/o): ioapic
00000000fed00000-00000000fed003ff (prio 0, i/o): hpet
00000000fee00000-00000000feefffff (prio 4096, i/o): apic-msi
address-space: cpu-smm-0
0000000000000000-ffffffffffffffff (prio 0, i/o): memory
0000000000000000-00000000ffffffff (prio 1, i/o): alias smram @smram 0000000000000000-00000000ffffffff
0000000000000000-ffffffffffffffff (prio 0, i/o): alias memory @system 0000000000000000-ffffffffffffffff
address-space: i440FX
0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
address-space: PIIX3
0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
address-space: VGA
0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
address-space: e1000
0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
0000000000000000-ffffffffffffffff (prio 0, i/o): alias bus master @system 0000000000000000-ffffffffffffffff
address-space: piix3-ide
0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
address-space: PIIX4_PM
0000000000000000-ffffffffffffffff (prio 0, i/o): bus master container
memory-region: pc.ram
0000000000000000-0000000007ffffff (prio 0, ram): pc.ram
memory-region: pc.bios
00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
memory-region: pci
0000000000000000-ffffffffffffffff (prio -1, i/o): pci
00000000000a0000-00000000000bffff (prio 1, i/o): vga-lowmem
00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
00000000000e0000-00000000000fffff (prio 1, rom): alias isa-bios @pc.bios 0000000000020000-000000000003ffff
00000000fd000000-00000000fdffffff (prio 1, ram): vga.vram
00000000feb80000-00000000feb9ffff (prio 1, i/o): e1000-mmio
00000000febb0000-00000000febb0fff (prio 1, i/o): vga.mmio
00000000febb0000-00000000febb017f (prio 0, i/o): edid
00000000febb0400-00000000febb041f (prio 0, i/o): vga ioports remapped
00000000febb0500-00000000febb0515 (prio 0, i/o): bochs dispi interface
00000000febb0600-00000000febb0607 (prio 0, i/o): qemu extended regs
00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
memory-region: smram
0000000000000000-00000000ffffffff (prio 0, i/o): smram
00000000000a0000-00000000000bffff (prio 0, ram): alias smram-low @pc.ram 00000000000a0000-00000000000bffff
memory-region: system
0000000000000000-ffffffffffffffff (prio 0, i/o): system
0000000000000000-0000000007ffffff (prio 0, ram): alias ram-below-4g @pc.ram 0000000000000000-0000000007ffffff
0000000000000000-ffffffffffffffff (prio -1, i/o): pci
00000000000a0000-00000000000bffff (prio 1, i/o): vga-lowmem
00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
00000000000e0000-00000000000fffff (prio 1, rom): alias isa-bios @pc.bios 0000000000020000-000000000003ffff
00000000fd000000-00000000fdffffff (prio 1, ram): vga.vram
00000000feb80000-00000000feb9ffff (prio 1, i/o): e1000-mmio
00000000febb0000-00000000febb0fff (prio 1, i/o): vga.mmio
00000000febb0000-00000000febb017f (prio 0, i/o): edid
00000000febb0400-00000000febb041f (prio 0, i/o): vga ioports remapped
00000000febb0500-00000000febb0515 (prio 0, i/o): bochs dispi interface
00000000febb0600-00000000febb0607 (prio 0, i/o): qemu extended regs
00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 00000000000a0000-00000000000bffff
00000000000c0000-00000000000c3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c0000-00000000000c3fff
00000000000c4000-00000000000c7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000c4000-00000000000c7fff
00000000000c8000-00000000000cbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000c8000-00000000000cbfff
00000000000cb000-00000000000cdfff (prio 1000, ram): alias kvmvapic-rom @pc.ram 00000000000cb000-00000000000cdfff
00000000000cc000-00000000000cffff (prio 1, ram): alias pam-rom @pc.ram 00000000000cc000-00000000000cffff
00000000000d0000-00000000000d3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d0000-00000000000d3fff
00000000000d4000-00000000000d7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000d4000-00000000000d7fff
00000000000d8000-00000000000dbfff (prio 1, ram): alias pam-rom @pc.ram 00000000000d8000-00000000000dbfff
00000000000dc000-00000000000dffff (prio 1, ram): alias pam-rom @pc.ram 00000000000dc000-00000000000dffff
00000000000e0000-00000000000e3fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e0000-00000000000e3fff
00000000000e4000-00000000000e7fff (prio 1, ram): alias pam-rom @pc.ram 00000000000e4000-00000000000e7fff
00000000000e8000-00000000000ebfff (prio 1, ram): alias pam-ram @pc.ram 00000000000e8000-00000000000ebfff
00000000000ec000-00000000000effff (prio 1, ram): alias pam-ram @pc.ram 00000000000ec000-00000000000effff
00000000000f0000-00000000000fffff (prio 1, ram): alias pam-rom @pc.ram 00000000000f0000-00000000000fffff
00000000fec00000-00000000fec00fff (prio 0, i/o): ioapic
00000000fed00000-00000000fed003ff (prio 0, i/o): hpet
00000000fee00000-00000000feefffff (prio 4096, i/o): apic-msi
- address-space: memory
第一个是系统全局的AddressSpace,表示虚拟机能够访问的所有地址。
- address-space: I/O
I/O表示x86系统下I/O端口的地址空间。
- address-space: cpu-memory-0
cpu-memory-x表示CPU视角下的地址空间。
- i440FX和PIIX3
i440FX和PIIX3是设备视角下的地址空间,虽然大部分情况下很多都是相同的,但是逻辑意义并不一样。
至此,QEMU中与内存相关的第一个数据结构AddressSpace就讲解完了。
(2)MemoryRegion
第二个是基本数据结构是MemoryRegion。MemoryRegion结构的定义在include/exec/memory.h中,如下:
#define TYPE_MEMORY_REGION "memory-region"
DECLARE_INSTANCE_CHECKER(MemoryRegion, MEMORY_REGION,
TYPE_MEMORY_REGION)
在此不详细展开宏,而直接给出struct MemoryRegion定义,也在include/exec/memory.h中,代码如下:
/** MemoryRegion:
*
* A struct representing a memory region.
*/
struct MemoryRegion {
Object parent_obj;
/* private: */
/* The following fields should fit in a cache line */
bool romd_mode;
bool ram;
bool subpage;
bool readonly; /* For RAM regions */
bool nonvolatile;
bool rom_device;
bool flush_coalesced_mmio;
uint8_t dirty_log_mask;
bool is_iommu;
RAMBlock *ram_block;
Object *owner;
/* owner as TYPE_DEVICE. Used for re-entrancy checks in MR access hotpath */
DeviceState *dev;
const MemoryRegionOps *ops;
void *opaque;
MemoryRegion *container;
int mapped_via_alias; /* Mapped via an alias, container might be NULL */
Int128 size;
hwaddr addr;
void (*destructor)(MemoryRegion *mr);
uint64_t align;
bool terminates;
bool ram_device;
bool enabled;
bool warning_printed; /* For reservations */
uint8_t vga_logging_count;
MemoryRegion *alias;
hwaddr alias_offset;
int32_t priority;
QTAILQ_HEAD(, MemoryRegion) subregions;
QTAILQ_ENTRY(MemoryRegion) subregions_link;
QTAILQ_HEAD(, CoalescedMemoryRange) coalesced;
const char *name;
unsigned ioeventfd_nb;
MemoryRegionIoeventfd *ioeventfds;
RamDiscardManager *rdm; /* Only for RAM */
/* For devices designed to perform re-entrant IO into their own IO MRs */
bool disable_reentrancy_guard;
};
MemoryRegion表示的是虚拟机中的一段内存区域(对比:AddressSpace结构用来表示一个虚拟机或者虚拟CPU能够访问的所有物理地址,其描述了一个到MemoryRegion对象的地址映射)。MemoryRegion是内存模拟中的核心结构,整个内存的模拟都是通过MemoryRegion构成的无环图完成的。图的叶子节点是实际分配给虚拟机的物理内存或者MMIO,中间节点则表示内存总线,内存控制是其它MemoryRegion的别名。AddressSpace与MemoryRegion的关系如下图所示:
对于MemoryRegion各成员的详细讲解,请看下回。