LuaJIT 学习(5)—— string.buffer 库

news2025/3/19 1:46:29

文章目录

    • Using the String Buffer Library
      • Buffer Objects
      • Buffer Method Overview
    • Buffer Creation and Management
      • `local buf = buffer.new([size [,options]]) local buf = buffer.new([options])`
      • `buf = buf:reset()`
      • `buf = buf:free()`
    • Buffer Writers
      • `buf = buf:put([str|num|obj] [,…])`
      • `buf = buf:putf(format, …)`
      • `buf = buf:putcdata(cdata, len)`FFI
      • `buf = buf:set(str) `
      • `buf = buf:set(cdata, len)`FFI
      • `ptr, len = buf:reserve(size)`FFI
      • `buf = buf:commit(used)`FFI
    • Buffer Readers
      • `len = #buf`
      • `res = str|num|buf .. str|num|buf […]`
      • `buf = buf:skip(len)`
      • `str, … = buf:get([len|nil] [,…])`
      • `str = buf:tostring() `
      • `str = tostring(buf)`
      • `ptr, len = buf:ref()`FFI
    • Serialization of Lua Objects
        • 例子:序列化 Lua 对象
    • Error handling
    • FFI caveats
        • 例子说明:

The string buffer library allows high-performance manipulation of string-like data.

Unlike Lua strings, which are constants, string buffers are mutable sequences of 8-bit (binary-transparent) characters. Data can be stored, formatted and encoded into a string buffer and later converted, extracted or decoded.

The convenient string buffer API simplifies common string manipulation tasks, that would otherwise require creating many intermediate strings. String buffers improve performance by eliminating redundant memory copies, object creation, string interning and garbage collection overhead. In conjunction with the FFI library, they allow zero-copy operations.

The string buffer library also includes a high-performance serializer for Lua objects.

Using the String Buffer Library

The string buffer library is built into LuaJIT by default, but it’s not loaded by default. Add this to the start of every Lua file that needs one of its functions:

local buffer = require("string.buffer")

The convention for the syntax shown on this page is that buffer refers to the buffer library and buf refers to an individual buffer object.

Please note the difference between a Lua function call, e.g. buffer.new() (with a dot) and a Lua method call, e.g. buf:reset() (with a colon).

Buffer Objects

A buffer object is a garbage-collected Lua object. After creation with buffer.new(), it can (and should) be reused for many operations. When the last reference to a buffer object is gone, it will eventually be freed by the garbage collector, along with the allocated buffer space.

Buffers operate like a FIFO (first-in first-out) data structure. Data can be appended (written) to the end of the buffer and consumed (read) from the front of the buffer. These operations may be freely mixed.

The buffer space that holds the characters is managed automatically — it grows as needed and already consumed space is recycled. Use buffer.new(size) and buf:free(), if you need more control.

The maximum size of a single buffer is the same as the maximum size of a Lua string, which is slightly below two gigabytes. For huge data sizes, neither strings nor buffers are the right data structure — use the FFI library to directly map memory or files up to the virtual memory limit of your OS.

Buffer Method Overview

  • The buf:put*()-like methods append (write) characters to the end of the buffer.
  • The buf:get*()-like methods consume (read) characters from the front of the buffer.
  • Other methods, like buf:tostring() only read the buffer contents, but don’t change the buffer.
  • The buf:set() method allows zero-copy consumption of a string or an FFI cdata object as a buffer.
  • The FFI-specific methods allow zero-copy read/write-style operations or modifying the buffer contents in-place. Please check the FFI caveats below, too.
  • Methods that don’t need to return anything specific, return the buffer object itself as a convenience. This allows method chaining, e.g.: buf:reset():encode(obj) or buf:skip(len):get()

Buffer Creation and Management

local buf = buffer.new([size [,options]]) local buf = buffer.new([options])

Creates a new buffer object.

The optional size argument ensures a minimum initial buffer size. This is strictly an optimization when the required buffer size is known beforehand. The buffer space will grow as needed, in any case.

The optional table options sets various serialization options.

buf = buf:reset()

Reset (empty) the buffer. The allocated buffer space is not freed and may be reused.

buf = buf:free()

The buffer space of the buffer object is freed. The object itself remains intact, empty and may be reused.

Note: you normally don’t need to use this method. The garbage collector automatically frees the buffer space, when the buffer object is collected. Use this method, if you need to free the associated memory immediately.

Buffer Writers

buf = buf:put([str|num|obj] [,…])

Appends a string str, a number num or any object obj with a __tostring metamethod to the buffer. Multiple arguments are appended in the given order.

Appending a buffer to a buffer is possible and short-circuited internally. But it still involves a copy. Better combine the buffer writes to use a single buffer.

buf = buf:putf(format, …)

Appends the formatted arguments to the buffer. The format string supports the same options as string.format().

buf = buf:putcdata(cdata, len)FFI

Appends the given len number of bytes from the memory pointed to by the FFI cdata object to the buffer. The object needs to be convertible to a (constant) pointer.

buf = buf:set(str)

buf = buf:set(cdata, len)FFI

This method allows zero-copy consumption of a string or an FFI cdata object as a buffer. It stores a reference to the passed string str or the FFI cdata object in the buffer. Any buffer space originally allocated is freed. This is not an append operation, unlike the buf:put*() methods.

After calling this method, the buffer behaves as if buf:free():put(str) or buf:free():put(cdata, len) had been called. However, the data is only referenced and not copied, as long as the buffer is only consumed.

In case the buffer is written to later on, the referenced data is copied and the object reference is removed (copy-on-write semantics).

The stored reference is an anchor for the garbage collector and keeps the originally passed string or FFI cdata object alive.

ptr, len = buf:reserve(size)FFI

buf = buf:commit(used)FFI

The reserve method reserves at least size bytes of write space in the buffer. It returns an uint8_t * FFI cdata pointer ptr that points to this space.

The available length in bytes is returned in len. This is at least size bytes, but may be more to facilitate efficient buffer growth. You can either make use of the additional space or ignore len and only use size bytes.

The commit method appends the used bytes of the previously returned write space to the buffer data.

This pair of methods allows zero-copy use of C read-style APIs:

local MIN_SIZE = 65536
repeat
  local ptr, len = buf:reserve(MIN_SIZE)
  local n = C.read(fd, ptr, len)
  if n == 0 then break end -- EOF.
  if n < 0 then error("read error") end
  buf:commit(n)
until false

The reserved write space is not initialized. At least the used bytes must be written to before calling the commit method. There’s no need to call the commit method, if nothing is added to the buffer (e.g. on error).

Buffer Readers

len = #buf

Returns the current length of the buffer data in bytes.

res = str|num|buf .. str|num|buf […]

The Lua concatenation operator .. also accepts buffers, just like strings or numbers. It always returns a string and not a buffer.

Note that although this is supported for convenience, this thwarts one of the main reasons to use buffers, which is to avoid string allocations. Rewrite it with buf:put() and buf:get().

Mixing this with unrelated objects that have a __concat metamethod may not work, since these probably only expect strings.

buf = buf:skip(len)

Skips (consumes) len bytes from the buffer up to the current length of the buffer data.

str, … = buf:get([len|nil] [,…])

Consumes the buffer data and returns one or more strings. If called without arguments, the whole buffer data is consumed. If called with a number, up to len bytes are consumed. A nil argument consumes the remaining buffer space (this only makes sense as the last argument). Multiple arguments consume the buffer data in the given order.

Note: a zero length or no remaining buffer data returns an empty string and not nil.

str = buf:tostring()

str = tostring(buf)

Creates a string from the buffer data, but doesn’t consume it. The buffer remains unchanged.

Buffer objects also define a __tostring metamethod. This means buffers can be passed to the global tostring() function and many other functions that accept this in place of strings. The important internal uses in functions like io.write() are short-circuited to avoid the creation of an intermediate string object.

ptr, len = buf:ref()FFI

Returns an uint8_t * FFI cdata pointer ptr that points to the buffer data. The length of the buffer data in bytes is returned in len.

The returned pointer can be directly passed to C functions that expect a buffer and a length. You can also do bytewise reads (local x = ptr[i]) or writes (ptr[i] = 0x40) of the buffer data.

In conjunction with the skip method, this allows zero-copy use of C write-style APIs:

repeat
  local ptr, len = buf:ref()
  if len == 0 then break end
  local n = C.write(fd, ptr, len)
  if n < 0 then error("write error") end
  buf:skip(n)
until n >= len

Unlike Lua strings, buffer data is not implicitly zero-terminated. It’s not safe to pass ptr to C functions that expect zero-terminated strings. If you’re not using len, then you’re doing something wrong.

Serialization of Lua Objects

略过

例子:序列化 Lua 对象
local buffer = require("string.buffer")

-- 创建一个元表
local mt1 = { __index = function(t, k) return "default" end }
local mt2 = { __index = function(t, k) return "another default" end }

-- 创建需要序列化的表
local t1 = setmetatable({ key1 = "value1", key2 = "value2" }, mt1)
local t2 = setmetatable({ key1 = "value3", key2 = "value4" }, mt2)

-- 定义字典和元表的数组
local dict = {"key1", "key2"}
local metatable = {mt1, mt2}

-- 使用 buffer.new() 进行序列化
local buffer_obj = buffer.new({
    dict = dict,
    metatable = metatable
})

-- 假设序列化后的数据为序列化函数 `encode()`
local serialized_data = buffer_obj:encode({t1, t2})

-- 反序列化
local decoded_data = buffer_obj:decode(serialized_data)

-- 访问解码后的数据
for _, tbl in ipairs(decoded_data) do
    print(tbl.key1, tbl.key2)
end

Error handling

Many of the buffer methods can throw an error. Out-of-memory or usage errors are best caught with an outer wrapper for larger parts of code. There’s not much one can do after that, anyway.

OTOH, you may want to catch some errors individually. Buffer methods need to receive the buffer object as the first argument. The Lua colon-syntax obj:method() does that implicitly. But to wrap a method with pcall(), the arguments need to be passed like this:

local ok, err = pcall(buf.encode, buf, obj)
if not ok then
  -- Handle error in err.
end

FFI caveats

The string buffer library has been designed to work well together with the FFI library. But due to the low-level nature of the FFI library, some care needs to be taken:

First, please remember that FFI pointers are zero-indexed. The space returned by buf:reserve() and buf:ref() starts at the returned pointer and ends before len bytes after that.

I.e. the first valid index is ptr[0] and the last valid index is ptr[len-1]. If the returned length is zero, there’s no valid index at all. The returned pointer may even be NULL.

The space pointed to by the returned pointer is only valid as long as the buffer is not modified in any way (neither append, nor consume, nor reset, etc.). The pointer is also not a GC anchor for the buffer object itself.

Buffer data is only guaranteed to be byte-aligned. Casting the returned pointer to a data type with higher alignment may cause unaligned accesses. It depends on the CPU architecture whether this is allowed or not (it’s always OK on x86/x64 and mostly OK on other modern architectures).

FFI pointers or references do not count as GC anchors for an underlying object. E.g. an array allocated with ffi.new() is anchored by buf:set(array, len), but not by buf:set(array+offset, len). The addition of the offset creates a new pointer, even when the offset is zero. In this case, you need to make sure there’s still a reference to the original array as long as its contents are in use by the buffer.

例子说明:
  1. 正常的引用:当你使用 buf:set(array, len) 时,这个 array 是一个通过 FFI 创建的数组,它会被作为 buf 的参数传递进去。在这种情况下,array 被引用,并且只要 buf 依然存在并持有这个引用,array 不会被垃圾回收器回收。这里 array 是一个“垃圾回收锚点”(GC anchor),即它会被垃圾回收器追踪。
  2. 添加偏移量后的情况:当你通过 array + offset 创建一个新的指针时(即通过加偏移量来引用 array 中的某个元素),这时创建的是一个新的指针对象。即使 offset 为零,array + offset 仍然会被视为一个新的指针。这个新的指针不会自动被垃圾回收器追踪,因为它并没有直接引用 array
    • 问题:这意味着,如果你只使用 array + offset(即偏移后的指针),垃圾回收器可能会认为原始的 array 对象不再被使用,最终回收掉 array,即使 buf 仍然依赖于它的内容。这会导致访问已回收的内存,造成未定义行为或崩溃。

Even though each LuaJIT VM instance is single-threaded (but you can create multiple VMs), FFI data structures can be accessed concurrently. Be careful when reading/writing FFI cdata from/to buffers to avoid concurrent accesses or modifications. In particular, the memory referenced by buf:set(cdata, len) must not be modified while buffer readers are working on it. Shared, but read-only memory mappings of files are OK, but only if the file does not change.

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.coloradmin.cn/o/2317512.html

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈,一经查实,立即删除!

相关文章

qt介绍图表 charts 一

qt chartsj基于Q的Graphics View框架&#xff0c;其核心组件是QChartView和QChart.QChartView是一个显示图表的独立部件&#xff0c;基类为QGraphicsView.QChar类管理图表的序列&#xff0c;图例和轴示意图。 绘制一个cos和sin曲线图&#xff0c;效果如下 实现代码 #include…

Transformer:GPT背后的造脑工程全解析(含手搓过程)

Transformer&#xff1a;GPT背后的"造脑工程"全解析&#xff08;含手搓过程&#xff09; Transformer 是人工智能领域的革命性架构&#xff0c;通过自注意力机制让模型像人类一样"全局理解"上下文关系。它摒弃传统循环结构&#xff0c;采用并行计算实现高…

S32K144入门笔记(十):TRGMUX的初始化

目录 1. 概述 2. 代码配置 1. 概述 书接上回&#xff0c;TRGMUX本质上是一个多路选择开关&#xff0c;根据用户手册中的描述&#xff0c;它可以实现多个输入的选择输出&#xff0c;本篇文章将验证如何通过配置工具来生成初始化配置代码。 2. 代码配置 笔者通过配置TRGMUX实现…

有了大模型为何还需要Agent智能体

一、什么是Agent&#xff1f; Agent&#xff08;智能体&#xff09; 是一种能感知环境、自主决策、执行动作的智能实体&#xff0c;当它与大语言模型&#xff08;如通义千问QWen、GPT&#xff09;结合时&#xff0c;形成一种**“增强型AI系统”**。其核心架构如下&#xff1a;…

DNS主从服务器

1.1环境准备 作用系统IP主机名web 服务器redhat9.5192.168.33.8webDNS 主服务器redhat9.5192.168.33.18dns1DNS 从服务器redhat9.5192.168.33.28dns2客户端redhat9.5192.168.33.7client 1.2修改主机名和IP地址 web服务器 [rootweb-8 ~]# hostnamectl hostname web [rootweb-8…

Flume详解——介绍、部署与使用

1. Flume 简介 Apache Flume 是一个专门用于高效地 收集、聚合、传输 大量日志数据的 分布式、可靠 的系统。它特别擅长将数据从各种数据源&#xff08;如日志文件、消息队列等&#xff09;传输到 HDFS、HBase、Kafka 等大数据存储系统。 特点&#xff1a; 可扩展&#xff1…

【Linux系列】文件压缩

&#x1f49d;&#x1f49d;&#x1f49d;欢迎来到我的博客&#xff0c;很高兴能够在这里和您见面&#xff01;希望您在这里可以感受到一份轻松愉快的氛围&#xff0c;不仅可以获得有趣的内容和知识&#xff0c;也可以畅所欲言、分享您的想法和见解。 推荐:kwan 的首页,持续学…

微服务架构中10个常用的设计模式

​在当今的微服务架构中&#xff0c;常见的十种设计模式&#xff0c;分别是服务发现模式、API网关模式、断路器模式、边车模式、负载均衡模式、Saga事务模式、CQRS模式、分片模式、分布式日志跟踪模式、熔断与降级模式 。其中&#xff0c;服务发现模式十分关键&#xff0c;通过…

Vue3组件+leaflet,实现重叠marker的Popup切换显示

一、前言 GIS开发过程中&#xff0c;经常需要绘制marker&#xff0c;这些marker很大概率会有坐标相同导致的叠加问题&#xff0c;这种情况下会降低使用体验感。所以我们可以将叠加的marker的popup做一个分页效果&#xff0c;可以切换显示的marker。 二、技术要点 我们以leaf…

机器学习之距离度量方法

常见的距离度量方法及相关函数、图示如下: 1. 欧几里得距离(Euclidean Distance) 函数公式:对于两个 ( n ) 维向量 ( x = ( x 1 , x 2 , ⋯   ,

3.1 在VisionPro脚本中添加CogGraphicLabel

本案例需要实现如下功能&#xff1a; 1.加载toolBlock 2.加载图片&#xff0c; 3.运行Block 4.VisionPro中添加脚本显示数值。 见下图&#xff1a;详细代码&#xff08;C#以及visionPro&#xff09;见下面链接&#xff1a; https://download.csdn.net/download/qq_340474…

AI:Machine Learning Data Science

机器学习与数据科学 左侧 机器学习 Machine Learning 机器学习是一门多领域交叉学科&#xff0c;涉及概率论、统计学、逼近论、凸分析、算法复杂度理论等多门学科。专门研究计算机怎样模拟或实现人类的学习行为&#xff0c;以获取新的知识或技能&#xff0c;重新组织已有的知…

软件需求分类、需求获取(高软46)

系列文章目录 软件需求分类&#xff0c;需求获取 文章目录 系列文章目录前言一、软件需求二、获取需求三、真题总结 前言 本节讲明软件需求分类、需求获取的相关知识。 一、软件需求 二、获取需求 三、真题 总结 就是高软笔记&#xff0c;大佬请略过&#xff01;

嵌入式Linux | 什么是 BootLoader、Linux 内核(kernel)、和文件系统?

01 什么是 BootLoader 呢&#xff1f; 它是个引导程序&#xff0c;也就是硬件复位以后第一个要执行的程序&#xff0c;它主要工作就是初始化操作系统运行的环境&#xff0c;比如说内存、定时器、缓冲器等&#xff0c;当这个工作做完以后&#xff0c;再把操作系统的代码加载…

函数(函数的概念、库函数、自定义函数、形参和实参、return语句、数组做函数参数、嵌套调用和链式访问、函数的声明和定义、static和extern)

一、函数的概念 •C语⾔中的函数&#xff1a;⼀个完成某项特定的任务的⼀⼩段代码 •函数又被翻译为子函数&#xff08;更准确&#xff09; •在C语⾔中我们⼀般会⻅到两类函数&#xff1a;库函数 ⾃定义函数 二、库函数 1 .标准库和头文件 •C语⾔的国际标准ANSIC规定了⼀…

ImGui 学习笔记(五) —— 字体文件加载问题

ImGui 加载字体文件的函数似乎存在编码问题&#xff0c;这一点可能跟源文件的编码也有关系&#xff0c;我目前源文件编码是 UTF-16。 当参数中包含中文字符时&#xff0c;ImGui 内部将字符转换为宽字符字符集时候&#xff0c;采用的 MultiByteToWideChar API 参数不太对&#…

OpenCV计算摄影学(20)非真实感渲染之增强图像的细节函数detailEnhance()

操作系统&#xff1a;ubuntu22.04 OpenCV版本&#xff1a;OpenCV4.9 IDE:Visual Studio Code 编程语言&#xff1a;C11 算法描述 此滤波器增强特定图像的细节。 cv::detailEnhance用于增强图像的细节&#xff0c;通过结合空间域和频率域的处理&#xff0c;提升图像中特定细节…

Android PC 要来了?Android 16 Beta3 出现 Enable desktop experience features 选项

在之前的 《Android 桌面窗口新功能推进》 我们就聊过&#xff0c;Google 就一直在努力改进 Android 的内置桌面模式&#xff0c;例如添加了适当的窗口标题、捕捉窗口的能力、悬停选项、窗口大小调整、最小化支持、app-to-web 等。 比如在搭载 Android 15 QPR 1 Beta 2 的 Pix…

Git常用操作之GitLab

Git常用操作之GitLab 小薛博客官网&#xff1a;小薛博客Git常用操作之GitLab官方地址 1、GitLab安装 https://gitlab.cn/install/ 1、Docker安装GitLab https://docs.gitlab.cn/jh/install/docker.html 1、设置卷位置 在设置其他所有内容之前&#xff0c;请配置一个新的…

Netty基础—NIO的使用简介

1.Buffer缓冲区 (1)Buffer缓冲区的作用 在NIO中&#xff0c;所有的数据都是通过使用Buffer缓冲区来处理的。如果要通过NIO&#xff0c;将数据写到文件和网络或从文件和网络中读取数据&#xff0c;那么就需要使用Buffer缓冲区来进行处理。 (2)Buffer缓冲区的4个核心概念 Buffer缓…