原创不易，转载请注明出处：
https://blog.csdn.net/q258523454/article/details/129988906
前言
我们项目中会用到各种序列化工具，到底哪一种是最适合我们的？ 需要从序列化、反序列化、存储大小各个方法面来衡量，我之前有实测过jackson、fastjson、kryo、protostuff序列化的数据，大概情况如下：
CPU: i7 2.80GHz
内存: 16GB
系统: 64位
备注：每次插入1个 StudentObject, 每个 StudentObject中含有 100000 个 Student 对象
600次,共6000万个对象,实测数据.
| 方式 | 序列化 | 反序列化 | 存储大小 |
| fastjson | 70 ms | 43 ms | 4.3 M |
| jackson | 68 ms | 135 ms | 7.6 M |
| kryo | 19 ms | 20 ms | 1.7 M |
| protoStuff | 14 ms | 30 ms | 2.1 M |
首选: kryo，其次 protoStuff
代码实现
下面的代码将会实现，一个项目中，Redisson同时使用多个序列化工具，让Redisson用不同的序列化来作客户端。文末也有总结。
maven包
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<!-- ——————————————————mybatis 数据库 BEGIN —————————————————————————— -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.13</version>
</dependency>
<dependency>
<groupId>com.github.pagehelper</groupId>
<artifactId>pagehelper-spring-boot-starter</artifactId>
<version>1.2.13</version>
</dependency>
<dependency>
<groupId>com.baomidou</groupId>
<artifactId>mybatis-plus-boot-starter</artifactId>
<version>3.4.0</version>
<exclusions>
<exclusion>
<artifactId>mybatis</artifactId>
<groupId>org.mybatis</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
<version>3.5.6</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.46</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId>
</dependency>
<!-- redisson -->
<dependency>
<groupId>org.redisson</groupId>
<artifactId>redisson-spring-boot-starter</artifactId>
<version>3.17.1</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.11</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.24</version>
</dependency>
<dependency>
<groupId>com.dyuproject.protostuff</groupId>
<artifactId>protostuff-core</artifactId>
<version>1.1.3</version>
</dependency>
<dependency>
<groupId>com.dyuproject.protostuff</groupId>
<artifactId>protostuff-runtime</artifactId>
<version>1.1.3</version>
</dependency>
<dependency>
<groupId>com.esotericsoftware</groupId>
<artifactId>kryo</artifactId>
<version>5.4.0</version>
</dependency>
</dependencies>Redisson抽象类
为了让各个序列化工具复用代码，我们先定义 RedissonClientInterface
public interface RedissonClientInterface {
RedissonClient getRedissonClient();
}将公共能力抽象到 AbstractRedissonService 类
public abstract class AbstractRedissonService implements RedissonClientInterface {
/**
* 默认保存时间
*/
private static final long DEFAULT_EXPIRE_TIME_SECONDS = 3600L;
public RLock getLock(String key) {
return getRedissonClient().getLock(key);
}
/**
* 设置key-value
*/
public void set(String key, Object value, long seconds) {
if (Objects.isNull(value)) {
return;
}
if (seconds <= 0) {
seconds = DEFAULT_EXPIRE_TIME_SECONDS;
}
getRedissonClient().getBucket(key).set(value, seconds, TimeUnit.SECONDS);
}
/**
* 批量设置key-value
*/
public <T> void setBatch(Map<String, T> map, long seconds) {
if (seconds <= 0) {
seconds = DEFAULT_EXPIRE_TIME_SECONDS;
}
RBatch batch = getRedissonClient().createBatch();
long finalSeconds = seconds;
map.forEach((k, v) -> {
batch.getBucket(k).setAsync(v, finalSeconds, TimeUnit.SECONDS);
});
batch.execute();
}
/**
* 获取value
* 没有则返回 null
*/
public Object get(String key) {
return getRedissonClient().getBucket(key).get();
}
/**
* 批量获取value
*/
public List<?> getBatch(List<String> keys) {
RBatch batch = getRedissonClient().createBatch();
keys.forEach(key -> batch.getBucket(key).getAsync());
BatchResult<?> execute = batch.execute();
return execute.getResponses();
}
/**
* 设置key-value,同时返回旧值
* 旧值不存在则返回 null
*/
public Object getAndSet(String key, Object value, long seconds) {
if (seconds <= 0) {
seconds = DEFAULT_EXPIRE_TIME_SECONDS;
}
return getRedissonClient().getBucket(key).getAndSet(value, seconds, TimeUnit.SECONDS);
}
/**
* 获取所有的指定前缀 keys
* 例如: "test:*"
*/
public Set<String> getKeys(String prefix) {
Iterable<String> keysByPattern = getRedissonClient().getKeys().getKeysByPattern(prefix);
Set<String> keys = new HashSet<>();
for (String key : keysByPattern) {
keys.add(key);
}
return keys;
}
/**
* 删除key
*/
public boolean removeKey(String key) {
return getRedissonClient().getBucket(key).delete();
}
/**
* 删除key
*/
public void removeKeyAsync(String key) {
getRedissonClient().getBucket(key).deleteAsync();
}
/**
* 批量删除
*/
public void removeKeyBatch(Collection<String> keys) {
RBatch batch = getRedissonClient().createBatch();
keys.forEach(key -> batch.getBucket(key).deleteAsync());
batch.execute();
}
}定义FastJson序列化编码
继承实现Redisson的BaseCodec，自定义fastjson编码器
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.parser.ParserConfig;
import com.alibaba.fastjson.serializer.SerializerFeature;
import com.dyuproject.protostuff.Schema;
import com.dyuproject.protostuff.runtime.RuntimeSchema;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
import io.netty.buffer.ByteBufInputStream;
import io.netty.buffer.ByteBufOutputStream;
import org.redisson.client.codec.BaseCodec;
import org.redisson.client.handler.State;
import org.redisson.client.protocol.Decoder;
import org.redisson.client.protocol.Encoder;
import java.io.IOException;
public class FastJsonCodec extends BaseCodec {
private static final Schema SCHEMA = RuntimeSchema.getSchema(ProtoStuffDataWrapper.class);
@Override
public Decoder<Object> getValueDecoder() {
return decoder;
}
@Override
public Encoder getValueEncoder() {
return encoder;
}
private final Encoder encoder = new Encoder() {
@Override
public ByteBuf encode(Object in) throws IOException {
ByteBuf out = ByteBufAllocator.DEFAULT.buffer();
try {
ByteBufOutputStream os = new ByteBufOutputStream(out);
JSON.writeJSONString(os, in, SerializerFeature.WriteClassName);
return os.buffer();
} catch (IOException e) {
// ByteBuf 只有在出现异常的时候才需要主动释放,因为否则重复释放会报错: IllegalReferenceCountException: refCnt: 0
out.release();
throw e;
} catch (Exception e) {
out.release();
throw new IOException(e);
}
}
};
private final Decoder<Object> decoder = new Decoder<Object>() {
public Object decode(ByteBuf buf, State state) throws IOException {
// 添加待序列化对象的扫描包，如果不添加扫描包，无法实现序列化和反序列化
// 另外待序列化的对象必须有一个默认的构造器，否则也无法实现序列化和反序列化。
ParserConfig.getGlobalInstance().addAccept("redisson.entity");
return JSON.parseObject(new ByteBufInputStream(buf), Object.class);
}
};
}
定义ProtoStuff序列化编码
注意，我们要先定义Protostuff 包装类，Protostuff 是基于POJO进行序列化和反序列化操作, 有时候对非定义POJO要用这个类辅助.
**
* Protostuff 包装类
* Protostuff 是基于POJO进行序列化和反序列化操作, 有时候对非定义POJO要用这个类辅助.
* 作用:
* 1.主要用于对 Map、List、String、Enum 等进行序列化/反序列化
* 2.对未知的 Object.class 可以用这个通用包装类
*/
class ProtoStuffDataWrapper<T> {
private T data;
public ProtoStuffDataWrapper() {
}
public ProtoStuffDataWrapper(T data) {
super();
this.data = data;
}
public T getData() {
return data;
}
public void setData(T data) {
this.data = data;
}
}
实现Protostuff序列化
import com.dyuproject.protostuff.LinkedBuffer;
import com.dyuproject.protostuff.ProtostuffIOUtil;
import com.dyuproject.protostuff.Schema;
import com.dyuproject.protostuff.runtime.RuntimeSchema;
import io.netty.buffer.ByteBuf;
import io.netty.buffer.ByteBufAllocator;
import io.netty.buffer.ByteBufInputStream;
import io.netty.buffer.ByteBufOutputStream;
import lombok.extern.slf4j.Slf4j;
import org.redisson.client.codec.BaseCodec;
import org.redisson.client.handler.State;
import org.redisson.client.protocol.Decoder;
import org.redisson.client.protocol.Encoder;
import java.io.IOException;
@Slf4j
public class ProtoStuffCodec extends BaseCodec {
/**
* 继承 Redisson Codec 实现后, 因为没有Object.Class信息,因此必须用一个封装类来定义 Schema, 否则无法 prostuff 序列化
*/
private static final Schema SCHEMA = RuntimeSchema.getSchema(ProtoStuffDataWrapper.class);
@Override
public Decoder<Object> getValueDecoder() {
return decoder;
}
@Override
public Encoder getValueEncoder() {
return encoder;
}
@SuppressWarnings({"unchecked", "rawtypes"})
private final Encoder encoder = new Encoder() {
@Override
public ByteBuf encode(Object in) throws IOException {
if (null == in) {
throw new NullPointerException();
}
LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE);
ByteBuf out = ByteBufAllocator.DEFAULT.buffer();
ByteBufOutputStream os = new ByteBufOutputStream(out);
try {
ProtostuffIOUtil.writeTo(os, new ProtoStuffDataWrapper(in), SCHEMA, buffer);
return os.buffer();
} catch (Exception e) {
out.release();
throw e;
} finally {
// 注意:finally 不要调用 release() 释放 ByteBuf, 因为重复释放会报错: io.netty.util.IllegalReferenceCountException: refCnt: 0
// ProtostuffIOUtil.writeTo() 会调用 ByteBufOutputStream.write() 已经进行'软'置空
buffer.clear();
}
}
};
@SuppressWarnings({"unchecked", "rawtypes"})
private final Decoder<Object> decoder = new Decoder<Object>() {
@Override
public Object decode(ByteBuf buf, State state) throws IOException {
ProtoStuffDataWrapper ProtoStuffDataWrapper = new ProtoStuffDataWrapper<>();
ProtostuffIOUtil.mergeFrom(new ByteBufInputStream(buf), ProtoStuffDataWrapper, SCHEMA);
return ProtoStuffDataWrapper.getData();
}
};
}定义Jackson序列化编码
无需实现，Redisson自带：org.redisson.codec.JsonJacksonCodec
定义Kryo序列化编码
无需实现，Redisson自带：org.redisson.codec.Kryo5Codec
注入客户端 RedissonClient
每个序列化单独用一个客户端
import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import redisson.codec.FastJsonCodec;
import redisson.codec.ProtoStuffCodec;
import org.redisson.Redisson;
import org.redisson.api.RedissonClient;
import org.redisson.client.codec.Codec;
import org.redisson.codec.JsonJacksonCodec;
import org.redisson.codec.Kryo5Codec;
import org.redisson.config.ClusterServersConfig;
import org.redisson.config.Config;
import org.redisson.config.SingleServerConfig;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.util.CollectionUtils;
import java.util.List;
@Slf4j
@Data
@Configuration
@ConfigurationProperties(prefix = "spring.redis")
public class RedissonConfig {
private Cluster cluster;
private String host;
private String port;
private String password;
@Data
public static class Cluster {
private List<String> nodes;
/**
* Maximum number of redirects to follow when executing commands across the cluster.
*/
private Integer maxRedirects;
}
/**
* RedissonClient bean
*/
@Primary
@Bean(name = "redissonClient", destroyMethod = "shutdown")
public RedissonClient redissonClient() {
return getRedissonClient(new JsonJacksonCodec());
}
@Bean(name = "kryoRedisson", destroyMethod = "shutdown")
public RedissonClient kryoRedisson() {
return getRedissonClient(new Kryo5Codec());
}
@Bean(name = "protoStuffRedisson", destroyMethod = "shutdown")
public RedissonClient protoStuffRedisson() {
return getRedissonClient(new ProtoStuffCodec());
}
@Bean(name = "fastJsonRedisson", destroyMethod = "shutdown")
public RedissonClient fastJsonRedisson() {
return getRedissonClient(new FastJsonCodec());
}
/**
* 根据 codec 创建 RedissonClient
*
* @param codec codec
* @return RedissonClient
*/
public RedissonClient getRedissonClient(Codec codec) {
Config redissonConfig = new Config();
redissonConfig.setCodec(codec);
// 设置'看门狗'续命时间, 默认30秒
redissonConfig.setLockWatchdogTimeout(30 * 1000);
if (null != cluster && !CollectionUtils.isEmpty(cluster.getNodes())) {
List<String> nodes = cluster.getNodes();
String[] clusterNodes = new String[nodes.size()];
for (int i = 0; i < nodes.size(); i++) {
clusterNodes[i] = "redis://" + nodes.get(i);
}
// 集群模式
ClusterServersConfig clusterServersConfig = redissonConfig.useClusterServers();
clusterServersConfig.addNodeAddress(clusterNodes)
.setPassword(password)
.setSlaveConnectionMinimumIdleSize(24)
.setMasterConnectionMinimumIdleSize(24);
} else {
// 单点模式
SingleServerConfig singleServerConfig = redissonConfig.useSingleServer();
singleServerConfig.setAddress("redis://" + host + ":" + port)
.setPassword(password);
}
return Redisson.create(redissonConfig);
}
}
注入客户端Service
以 kryoRedisson 为例，其他的一样，把Qualifier的名字改一下即可（redissonClient、 kryoRedisson、 protoStuffRedisson、 fastJsonRedisson）
import redisson.service.base.AbstractRedissonService;
import org.redisson.api.RedissonClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.stereotype.Component;
@Component
public class RedissonKryoService extends AbstractRedissonService {
@Autowired
@Qualifier("kryoRedisson")
private RedissonClient redissonClient;
@Override
public RedissonClient getRedissonClient() {
return redissonClient;
}
}
结果测试
测试代码如下，运行即可，其他的序列化工具测试写法一样，替换service即可。
import lombok.extern.slf4j.Slf4j;
import redisson.entity.Student;
import redisson.entity.StudentListObject;
import redisson.service.RedissonKryoService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.util.StopWatch;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import java.util.Set;
import java.util.UUID;
@Slf4j
@RestController
public class KryoController {
private static final long EXPIRE_TIME_SECONDS = 600;
public static final String PREFIX = "test:kryo:studentList:";
@Autowired
private RedissonKryoService redissonKryoService;
@GetMapping(value = "/redisson/kryo/set")
public String studentSet() {
Student student = new Student();
student.setId(1);
student.setAge(19);
student.setName("zhangsan");
redissonKryoService.set("kryo_student", student, EXPIRE_TIME_SECONDS);
return "ok";
}
@GetMapping(value = "/redisson/kryo/get")
public String studentGet() {
return "" + redissonKryoService.get("kryo_student");
}
/**
* 测试平均速度:
* 序列化效率 比 JackSon、FastJson 好
* 100次插入,总耗时:2184,平均耗时:21
* 100次插入,总耗时:1714,平均耗时:17
* 100次插入,总耗时:1627,平均耗时:16
* 100次插入,总耗时:1641,平均耗时:16
* 100次插入,总耗时:2451,平均耗时:24
* 100次插入,总耗时:1995,平均耗时:19
*/
@GetMapping(value = "/redisson/kryo/set/list")
public String studentSetList() {
// 为了测试准确, 每次插入前, 先清空旧数据
Set<String> keys = redissonKryoService.getKeys(PREFIX + "*");
redissonKryoService.removeKeyBatch(keys);
log.info("清空 " + PREFIX + keys.size() + "个");
// 创建插入对象(每个对象List有n个)
int n = 100000;
StudentListObject studentListObject = new StudentListObject();
studentListObject.setName("studentListObject");
studentListObject.setStudentList(StudentUtil.getRandomStudentList(n));
// 循环插入100次,测试效率
int count = 100;
StopWatch stopWatch = new StopWatch();
for (int i = 0; i < count; i++) {
String uuid = UUID.randomUUID().toString().replaceAll("-", "").substring(0, 10);
stopWatch.start("" + i);
redissonKryoService.set(PREFIX + uuid, studentListObject, EXPIRE_TIME_SECONDS);
stopWatch.stop();
}
// log.info(stopWatch.prettyPrint());
log.info(count + "次插入,总耗时:{},平均耗时:{}", stopWatch.getTotalTimeMillis(), stopWatch.getTotalTimeMillis() / stopWatch.getTaskCount());
return stopWatch.getTotalTimeMillis() + "";
}
/**
* 测试平均速度:
* 反序列化效率 比 JackSon、FastJson 好
* 100次查询,总耗时:2657,平均耗时:26
* 100次查询,总耗时:1962,平均耗时:19
* 100次查询,总耗时:2055,平均耗时:20
* 100次查询,总耗时:1980,平均耗时:19
* 100次查询,总耗时:1910,平均耗时:19
* 100次查询,总耗时:1968,平均耗时:19
*/
@GetMapping(value = "/redisson/kryo/get/list")
public String studentGetList() {
// 获取所有key
Set<String> keys = redissonKryoService.getKeys(PREFIX + "*");
Object list = null;
StopWatch stopWatch = new StopWatch();
// 循环依次查询
int i = 0;
for (String key : keys) {
stopWatch.start("" + i);
list = redissonKryoService.get(key);
stopWatch.stop();
i++;
}
// log.info(stopWatch.prettyPrint());
log.info(keys.size() + "次查询,总耗时:{},平均耗时:{}", stopWatch.getTotalTimeMillis(), stopWatch.getTotalTimeMillis() / stopWatch.getTaskCount());
return list + "";
}
}总结
各个序列化效率总结，可以看我这篇文章：
https://blog.csdn.net/q258523454/article/details/129953047
jackson、fastjson、kryo、protobuf等序列化效率对比【全】_kryo jackson-CSDN博客
我们通过同时使用多个序列化工具，以及实际数据测试的效果来看。
综合性能推荐使用：kryo 和 protoStuff
原创不易，转载请注明出处：
https://blog.csdn.net/q258523454/article/details/129988906
