sqlite mmap

https://www.sqlite.org/mmap.html

1. 内存映射 I/O 的基本原理

默认机制（传统 I/O）
SQLite 默认通过 xRead() 和 xWrite() 方法（对应 read()/write() 系统调用）访问数据库文件。这些方法需要将数据从内核缓冲区复制到用户空间，反之亦然。
内存映射 I/O
从 SQLite 3.7.17 开始，新增了 xFetch() 和 xUnfetch() 方法。内存映射允许 SQLite 直接将数据库文件的部分或全部内容映射到进程的虚拟内存空间中，通过指针直接访问数据，避免复制操作。

2. 内存映射 I/O 的优缺点

优点

性能提升
- 减少数据在内核缓冲区和用户空间之间的复制次数，特别适合频繁随机读取的场景（如大型数据库查询）。
- 对 I/O 密集型操作（如复杂查询）有显著优化。
减少内存占用
- SQLite 共享操作系统的页缓存（Page Cache），无需单独维护数据副本。

缺点

I/O 错误处理问题
- 内存映射文件的 I/O 错误（如磁盘故障）会触发信号（Signal）。若应用程序未捕获这些信号，可能导致程序崩溃。
操作系统依赖
- 要求操作系统支持统一缓冲区缓存（Unified Buffer Cache），否则可能引发数据损坏（尤其在多进程混合使用内存映射与传统 I/O 时）。
- 某些操作系统的统一缓冲区缓存实现存在 Bug。
性能不确定性
- 并非所有场景都能提升性能。极端情况下（如小文件频繁访问），内存映射可能比传统 I/O 更慢。
Windows 的局限性
- Windows 无法截断（Truncate）内存映射文件。当执行 VACUUM 或 auto_vacuum 缩减数据库时，文件末尾的未使用空间不会被释放，但后续操作可复用该空间。
- 旧版 SQLite（< 3.7.0）可能误报此类文件为损坏。

3. 内存映射 I/O 的工作流程

读取数据

传统方式（xRead()）
- SQLite 分配堆内存 → 调用 xRead() → 从内核缓冲区复制数据到用户空间。
内存映射方式（xFetch()）
- SQLite 调用 xFetch() 获取指向内存页的指针。
- 若成功，直接通过指针访问数据，无需复制。
- 若失败（返回 NULL），回退到 xRead()。

写入数据

无论是否使用内存映射，SQLite 始终在修改数据前将页面复制到堆内存：
1. 确保其他进程看不到未提交的更改（事务隔离）。
2. 防止应用程序误操作指针导致数据库损坏。
修改完成后，通过 xWrite() 将数据写回磁盘。

结论：内存映射主要优化读取性能，对写入性能影响有限。

4. 配置内存映射 I/O

核心参数：`PRAGMA mmap_size`

设置内存映射大小

PRAGMA mmap_size = 268435456; -- 启用内存映射（例如 256MB）
PRAGMA mmap_size = 0;         -- 禁用内存映射

仅映射文件的前 N 字节，超出部分仍用传统 xRead()。
若文件小于 N，映射整个文件。

默认值与上限

默认值
- 默认 mmap_size = 0（禁用）。可通过以下方式修改：
  - 编译时：SQLITE_DEFAULT_MMAP_SIZE 宏。
  - 启动时：sqlite3_config(SQLITE_CONFIG_MMAP_SIZE, X, Y)。
硬性上限
- 编译时上限：SQLITE_MAX_MMAP_SIZE 宏（默认 1GB）。若设为 0，禁用内存映射。
- 运行时上限：可通过 sqlite3_config(SQLITE_CONFIG_MMAP_SIZE, X, Y) 降低或归零，但不能超过编译时上限。

平台限制

OpenBSD 等系统：因缺乏统一缓冲区缓存，强制设置硬性上限为 0（禁用内存映射）。

5. 使用建议

适用场景
- 读多写少、大型数据库、高并发查询。
- 确保操作系统支持且稳定（如 Linux、macOS）。
注意事项
- 测试性能提升效果，避免盲目启用。
- 处理 Windows 的未释放空间问题（需 SQLite ≥3.7.0）。
- 监控内存占用，防止虚拟内存耗尽。

总结

SQLite 的内存映射 I/O 通过直接操作虚拟内存提升读取性能，但需权衡兼容性、内存占用和平台限制。合理配置 mmap_size 并结合业务场景测试是关键。写入操作仍依赖传统 I/O 机制，因此内存映射主要优化查询性能。

The default mechanism by which SQLite accesses and updates database disk files is the xRead() and xWrite() methods of the sqlite3_io_methods VFS object. These methods are typically implemented as “read()” and “write()” system calls which cause the operating system to copy disk content between the kernel buffer cache and user space.

Beginning with version 3.7.17 (2013-05-20), SQLite has the option of accessing disk content directly using memory-mapped I/O and the new xFetch() and xUnfetch() methods on sqlite3_io_methods.

There are advantages and disadvantages to using memory-mapped I/O. Advantages include:

Many operations, especially I/O intensive operations, can be faster since content need not be copied between kernel space and user space.

The SQLite library may need less RAM since it shares pages with the operating-system page cache and does not always need its own copy of working pages.

But there are also disadvantages:

An I/O error on a memory-mapped file cannot be caught and dealt with by SQLite. Instead, the I/O error causes a signal which, if not caught by the application, results in a program crash.

The operating system must have a unified buffer cache in order for the memory-mapped I/O extension to work correctly, especially in situations where two processes are accessing the same database file and one process is using memory-mapped I/O while the other is not. Not all operating systems have a unified buffer cache. In some operating systems that claim to have a unified buffer cache, the implementation is buggy and can lead to corrupt databases.

Performance does not always increase with memory-mapped I/O. In fact, it is possible to construct test cases where performance is reduced by the use of memory-mapped I/O.

Windows is unable to truncate a memory-mapped file. Hence, on Windows, if an operation such as VACUUM or auto_vacuum tries to reduce the size of a memory-mapped database file, the size reduction attempt will silently fail, leaving unused space at the end of the database file. No data is lost due to this problem, and the unused space will be reused again the next time the database grows. However if a version of SQLite prior to 3.7.0 runs PRAGMA integrity_check on such a database, it will (incorrectly) report database corruption due to the unused space at the end. Or if a version of SQLite prior to 3.7.0 writes to the database while it still has unused space at the end, it may make that unused space inaccessible and unavailable for reuse until after the next VACUUM.

Because of the potential disadvantages, memory-mapped I/O is disabled by default. To activate memory-mapped I/O, use the mmap_size pragma and set the mmap_size to some large number, usually 256MB or larger, depending on how much address space your application can spare. The rest is automatic. The PRAGMA mmap_size statement will be a silent no-op on systems that do not support memory-mapped I/O.

How Memory-Mapped I/O Works

To read a page of database content using the legacy xRead() method, SQLite first allocates a page-size chunk of heap memory then invokes the xRead() method which causes the database page content to be copied into the newly allocated heap memory. This involves (at a minimum) a copy of the entire page.

But if SQLite wants to access a page of the database file and memory mapped I/O is enabled, it first calls the xFetch() method. The xFetch() method asks the operating system to return a pointer to the requested page, if possible. If the requested page has been or can be mapped into the application address space, then xFetch returns a pointer to that page for SQLite to use without having to copy anything. Skipping the copy step is what makes memory mapped I/O faster.

SQLite does not assume that the xFetch() method will work. If a call to xFetch() returns a NULL pointer (indicating that the requested page is not currently mapped into the applications address space) then SQLite silently falls back to using xRead(). An error is only reported if xRead() also fails.

When updating the database file, SQLite always makes a copy of the page content into heap memory before modifying the page. This is necessary for two reasons. First, changes to the database are not supposed to be visible to other processes until after the transaction commits and so the changes must occur in private memory. Second, SQLite uses a read-only memory map to prevent stray pointers in the application from overwriting and corrupting the database file.

After all needed changes are completed, xWrite() is used to move the content back into the database file. Hence the use of memory mapped I/O does not significantly change the performance of database changes. Memory mapped I/O is mostly a benefit for queries.

Configuring Memory-Mapped I/O

The “mmap_size” is the maximum number of bytes of the database file that SQLite will try to map into the process address space at one time. The mmap_size applies separately to each database file, so the total amount of process address space that could potentially be used is the mmap_size times the number of open database files.

To activate memory-mapped I/O, an application can set the mmap_size to some large value. For example:

PRAGMA mmap_size=268435456;

To disable memory-mapped I/O, simply set the mmap_size to zero:

PRAGMA mmap_size=0;

If mmap_size is set to N then all current implementations map the first N bytes of the database file and use legacy xRead() calls for any content beyond N bytes. If the database file is smaller than N bytes, then the entire file is mapped. In the future, new OS interfaces could, in theory, map regions of the file other than the first N bytes, but no such implementation currently exists.

The mmap_size is set separately for each database file using the “PRAGMA mmap_size” statement. The usual default mmap_size is zero, meaning that memory mapped I/O is disabled by default. However, the default mmap_size can be increased either at compile-time using the SQLITE_DEFAULT_MMAP_SIZE macro or at start-time using the sqlite3_config(SQLITE_CONFIG_MMAP_SIZE,…) interface.

SQLite also maintains a hard upper bound on the mmap_size. Attempts to increase the mmap_size above this hard upper bound (using PRAGMA mmap_size) will automatically cap the mmap_size at the hard upper bound. If the hard upper bound is zero, then memory mapped I/O is impossible. The hard upper bound can be set at compile-time using the SQLITE_MAX_MMAP_SIZE macro. If SQLITE_MAX_MMAP_SIZE is set to zero, then the code used to implement memory mapped I/O is omitted from the build. The hard upper bound is automatically set to zero on certain platforms (ex: OpenBSD) where memory mapped I/O does not work due to the lack of a unified buffer cache.

If the hard upper bound on mmap_size is non-zero at compilation time, it may still be reduced or zeroed at start-time using the sqlite3_config(SQLITE_CONFIG_MMAP_SIZE,X,Y) interface. The X and Y parameters must both be 64-bit signed integers. The X parameter is the default mmap_size of the process and the Y is the new hard upper bound. The hard upper bound cannot be increased above its compile-time setting using SQLITE_CONFIG_MMAP_SIZE but it can be reduced or zeroed.