缘起

今年(2023年) 2月的时候做了个适配Amazon S3对象存储接口的需求，由于4月份自学考试临近，一直在备考就拖着没总结记录下，开发联调过程中也出现过一些奇葩的问题，最近人刚从考试缓过来顺手记录一下。

S3对象存储的基本概念

S3是什么？

Amazon S3(Simple Storage Service)对象存储出现得比较早且使用简单的RESTful API，于是成为了对象存储服务(Object Storage Service，OSS)业内的标准接口规范。

S3的逻辑模型

如下图，我们可以把S3的存储空间想象成无限的，想存储一个任意格式的文件到S3服务中，只需要知道要把它放到哪个桶(Bucket)中，它的名字（Object Id）应该是什么。

按图中的模型，可简单理解为S3是由若干个桶（Bucket）组成，每个桶中包含若干个不同标识的对象（Object），还有就是统一的访问入口(RESTful API)，这样基本就足够了。

Minio客户端方式操作S3

详细API文档：Java Client API Reference — MinIO Object Storage for Linux

以下代码异常处理做了简化，真实使用时请注意捕获异常做处理。

引入依赖

Maven:

	<dependency>
	<groupId>io.minio</groupId>
	<artifactId>minio</artifactId>
	<version>8.5.2</version>
	</dependency>

Gradle:

	dependencies {
	implementation("io.minio:minio:8.5.2")
	}

初始化客户端

	private static final String HTTP_PROTOCOL = "http";
	private MinioClient minioClient;
	private String endpoint = "http://192.168.0.8:9200";
	private String accessKey = "testKey";
	private String secretKey = "testSecretKey";
	public void init() throws MalformedURLException {
	URL endpointUrl = new URL(endpoint);
	try {
	// url上无端口号时，识别http为80端口，https为443端口
	int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort();
	boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true;
	//@formatter:off
	this.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security)
	.credentials(accessKey, secretKey).build();
	//@formatter:on
	// 忽略证书校验，防止自签名证书校验失败导致无法建立连接
	this.minioClient.ignoreCertCheck();
	} catch (Exception e) {
	e.printStackTrace();
	}
	}

建桶

	public boolean createBucket(String bucket) {
	try {
	minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());
	} catch (Exception e) {
	e.printStackTrace();
	return false;
	}
	return true;
	}

删桶

	public boolean deleteBucket(String bucket) {
	try {
	minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build());
	logger.info("删除桶[{}]成功", bucket);
	} catch (Exception e) {
	e.printStackTrace();
	return false;
	}
	return true;
	}

判断桶是否存在

	public boolean bucketExists(String bucket) {
	try {
	return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());
	} catch (Exception e) {
	e.printStackTrace();
	return false;
	}
	}

上传对象

	public void upload(String bucket, String objectId, InputStream input) {
	try {
	//@formatter:off
	minioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId)
	.stream(input, input.available(), -1)
	.build());
	//@formatter:on
	} catch (Exception e) {
	e.printStackTrace();
	}
	}

下载对象

提供两个下载方法，一个将输入流返回，另一个用参数输出流写出

	public InputStream download(String bucket, String objectId) {
	try {
	return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build());
	} catch (Exception e) {
	e.printStackTrace();
	}
	return null;
	}
	public void download(String bucket, String objectId, OutputStream output) {
	//@formatter:off
	try (InputStream input = minioClient.getObject(
	GetObjectArgs.builder().bucket(bucket).object(objectId).build())) {
	IOUtils.copyLarge(input, output);
	} catch (Exception e) {
	e.printStackTrace();
	}
	//@formatter:on
	}

删除对象

	public boolean deleteObject(String bucket, String objectId) {
	//@formatter:off
	try {
	minioClient.removeObject(RemoveObjectArgs.builder()
	.bucket(bucket).object(objectId).build());
	} catch (Exception e) {
	e.printStackTrace();
	}
	//@formatter:on
	return true;
	}

判断对象是否存在

	public boolean objectExists(String bucket, String key) {
	//@formatter:off
	try {
	// minio客户端未提供判断对象是否存在的方法，此方法中调用出现异常时说明对象不存在
	minioClient.statObject(StatObjectArgs.builder()
	.bucket(bucket).object(key).build());
	} catch (Exception e) {
	return false;
	}
	//@formatter:on
	return true;
	}

完整代码

	import java.io.InputStream;
	import java.io.OutputStream;
	import java.net.MalformedURLException;
	import java.net.URL;
	import org.apache.tomcat.util.http.fileupload.IOUtils;
	import org.slf4j.Logger;
	import org.slf4j.LoggerFactory;
	import io.minio.BucketExistsArgs;
	import io.minio.GetObjectArgs;
	import io.minio.MakeBucketArgs;
	import io.minio.MinioClient;
	import io.minio.PutObjectArgs;
	import io.minio.RemoveBucketArgs;
	import io.minio.RemoveObjectArgs;
	import io.minio.StatObjectArgs;
	public class S3MinioClientDemo {
	private static final Logger logger = LoggerFactory.getLogger(S3MinioClientDemo.class);
	private static final String HTTP_PROTOCOL = "http";
	private MinioClient minioClient;
	private String endpoint = "http://192.168.0.8:9200";
	private String accessKey = "testKey";
	private String secretKey = "testSecretKey";
	public void init() throws MalformedURLException {
	URL endpointUrl = new URL(endpoint);
	try {
	// url上无端口号时，识别http为80端口，https为443端口
	int port = endpointUrl.getPort() != -1 ? endpointUrl.getPort() : endpointUrl.getDefaultPort();
	boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol()) ? false : true;
	//@formatter:off
	this.minioClient = MinioClient.builder().endpoint(endpointUrl.getHost(), port, security)
	.credentials(accessKey, secretKey).build();
	//@formatter:on
	// 忽略证书校验，防止自签名证书校验失败导致无法建立连接
	this.minioClient.ignoreCertCheck();
	} catch (Exception e) {
	e.printStackTrace();
	}
	}
	public boolean createBucket(String bucket) {
	try {
	boolean found = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());
	if (found) {
	logger.info("桶名[{}]已存在", bucket);
	return false;
	}
	minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucket).build());
	} catch (Exception e) {
	e.printStackTrace();
	}
	return true;
	}
	public boolean deleteBucket(String bucket) {
	try {
	minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucket).build());
	logger.info("删除桶[{}]成功", bucket);
	} catch (Exception e) {
	e.printStackTrace();
	return false;
	}
	return true;
	}
	public boolean bucketExists(String bucket) {
	try {
	return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());
	} catch (Exception e) {
	e.printStackTrace();
	return false;
	}
	}
	public void upload(String bucket, String objectId, InputStream input) {
	try {
	//@formatter:off
	minioClient.putObject(PutObjectArgs.builder().bucket(bucket).object(objectId)
	.stream(input, input.available(), -1)
	.build());
	//@formatter:on
	} catch (Exception e) {
	e.printStackTrace();
	}
	}
	public InputStream download(String bucket, String objectId) {
	try {
	return minioClient.getObject(GetObjectArgs.builder().bucket(bucket).object(objectId).build());
	} catch (Exception e) {
	e.printStackTrace();
	}
	return null;
	}
	public void download(String bucket, String objectId, OutputStream output) {
	//@formatter:off
	try (InputStream input = minioClient.getObject(
	GetObjectArgs.builder().bucket(bucket).object(objectId).build())) {
	IOUtils.copyLarge(input, output);
	} catch (Exception e) {
	e.printStackTrace();
	}
	//@formatter:on
	}
	public boolean objectExists(String bucket, String objectId) {
	//@formatter:off
	try {
	// minio客户端未提供判断对象是否存在的方法，此方法中调用出现异常时说明对象不存在
	minioClient.statObject(StatObjectArgs.builder()
	.bucket(bucket).object(objectId).build());
	} catch (Exception e) {
	return false;
	}
	//@formatter:on
	return true;
	}
	public boolean deleteObject(String bucket, String objectId) {
	//@formatter:off
	try {
	minioClient.removeObject(RemoveObjectArgs.builder()
	.bucket(bucket).object(objectId).build());
	} catch (Exception e) {
	e.printStackTrace();
	}
	//@formatter:on
	return true;
	}
	public void close() {
	minioClient = null;
	}
	}

Amazon S3 SDK方式操作S3

官方API文档：What is Amazon S3? - Amazon Simple Storage Service

这里由于项目上提供的SDK和文档都是1.x的，这里就暂时只提供1.x的代码

引入依赖

Maven:

	<dependency>
	<groupId>com.amazonaws</groupId>
	<artifactId>aws-java-sdk-s3</artifactId>
	<version>1.11.300</version>
	</dependency>

Gradle:

	dependencies {
	implementation 'com.amazonaws:aws-java-sdk-s3:1.11.300'
	}

初始化客户端

	private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class);
	private AmazonS3 s3client;
	private String endpoint = "http://192.168.0.8:9200";
	private String accessKey = "testKey";
	private String secretKey = "testSecretKey";
	public void init() throws MalformedURLException {
	URL endpointUrl = new URL(endpoint);
	String protocol = endpointUrl.getProtocol();
	int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort();
	ClientConfiguration clientConfig = new ClientConfiguration();
	clientConfig.setSignerOverride("S3SignerType");
	clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase()));
	// 禁用证书检查，避免https自签证书校验失败
	System.setProperty("com.amazonaws.sdk.disableCertChecking", "true");
	// 屏蔽 AWS 的 MD5 校验，避免校验导致的下载抛出异常问题
	System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true");
	AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey);
	// 创建 S3Client 实例
	AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig);
	s3client.setEndpoint(endpointUrl.getHost() + ":" + port);
	s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());
	this.s3client = s3client;
	}

建桶

	public boolean createBucket(String bucket) {
	String bucketName = parseBucketName(bucket);
	try {
	if (s3client.doesBucketExist(bucketName)) {
	logger.warn("bucket[{}]已存在", bucketName);
	return false;
	}
	s3client.createBucket(bucketName);
	} catch (Exception e) {
	e.printStackTrace();
	}
	return true;
	}

删桶

	public boolean deleteBucket(String bucket) {
	try {
	s3client.deleteBucket(bucket);
	logger.info("删除bucket[{}]成功", bucket);
	} catch (Exception e) {
	e.printStackTrace();
	return false;
	}
	return true;
	}

判断桶是否存在

	public boolean bucketExists(String bucket) {
	try {
	return s3client.doesBucketExist(bucket);
	} catch (Exception e) {
	e.printStackTrace();
	}
	return false;
	}

上传对象

	public void upload(String bucket, String objectId, InputStream input) {
	try {
	// 创建文件上传的元数据
	ObjectMetadata meta = new ObjectMetadata();
	// 设置文件上传长度
	meta.setContentLength(input.available());
	// 上传
	s3client.putObject(bucket, objectId, input, meta);
	} catch (Exception e) {
	e.printStackTrace();
	}
	}

下载对象

	public InputStream download(String bucket, String objectId) {
	try {
	S3Object o = s3client.getObject(bucket, objectId);
	return o.getObjectContent();
	} catch (Exception e) {
	e.printStackTrace();
	}
	return null;
	}
	public void download(String bucket, String objectId, OutputStream out) {
	S3Object o = s3client.getObject(bucket, objectId);
	try (InputStream in = o.getObjectContent()) {
	IOUtils.copyLarge(in, out);
	} catch (Exception e) {
	e.printStackTrace();
	}
	}

删除对象

	public boolean deleteObject(String bucket, String objectId) {
	try {
	s3client.deleteObject(bucket, objectId);
	} catch (Exception e) {
	e.printStackTrace();
	return false;
	}
	return true;
	}

判断对象是否存在

	public boolean existObject(String bucket, String objectId) {
	try {
	return s3client.doesObjectExist(bucket, objectId);
	} catch (Exception e) {
	e.printStackTrace();
	return false;
	}
	}

完整代码

	import java.io.InputStream;
	import java.io.OutputStream;
	import java.net.MalformedURLException;
	import java.net.URL;
	import org.apache.tomcat.util.http.fileupload.IOUtils;
	import org.slf4j.Logger;
	import org.slf4j.LoggerFactory;
	import com.amazonaws.ClientConfiguration;
	import com.amazonaws.Protocol;
	import com.amazonaws.auth.AWSCredentials;
	import com.amazonaws.auth.BasicAWSCredentials;
	import com.amazonaws.services.s3.AmazonS3;
	import com.amazonaws.services.s3.AmazonS3Client;
	import com.amazonaws.services.s3.S3ClientOptions;
	import com.amazonaws.services.s3.model.ObjectMetadata;
	import com.amazonaws.services.s3.model.S3Object;
	/**
	* S3对象存储官方SDK实现
	*
	* @author ZhangChenguang
	* @date 2023年2月2日
	*/
	@SuppressWarnings("deprecation")
	public class S3SdkDemo {
	private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class);
	private AmazonS3 s3client;
	private String endpoint = "http://192.168.0.8:9200";
	private String accessKey = "testKey";
	private String secretKey = "testSecretKey";
	public void init() throws MalformedURLException {
	URL endpointUrl = new URL(endpoint);
	String protocol = endpointUrl.getProtocol();
	int port = endpointUrl.getPort() == -1 ? endpointUrl.getDefaultPort() : endpointUrl.getPort();
	ClientConfiguration clientConfig = new ClientConfiguration();
	clientConfig.setSignerOverride("S3SignerType");
	clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase()));
	// 禁用证书检查，避免https自签证书校验失败
	System.setProperty("com.amazonaws.sdk.disableCertChecking", "true");
	// 屏蔽 AWS 的 MD5 校验，避免校验导致的下载抛出异常问题
	System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true");
	AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey);
	// 创建 S3Client 实例
	AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig);
	s3client.setEndpoint(endpointUrl.getHost() + ":" + port);
	s3client.setS3ClientOptions(S3ClientOptions.builder().setPathStyleAccess(true).build());
	this.s3client = s3client;
	}
	public boolean createBucket(String bucket) {
	try {
	s3client.createBucket(bucket);
	} catch (Exception e) {
	e.printStackTrace();
	}
	return true;
	}
	public boolean deleteBucket(String bucket) {
	try {
	s3client.deleteBucket(bucket);
	logger.info("删除bucket[{}]成功", bucket);
	} catch (Exception e) {
	e.printStackTrace();
	return false;
	}
	return true;
	}
	public boolean bucketExists(String bucket) {
	try {
	return s3client.doesBucketExist(bucket);
	} catch (Exception e) {
	e.printStackTrace();
	}
	return false;
	}
	public void upload(String bucket, String objectId, InputStream input) {
	try {
	// 创建文件上传的元数据
	ObjectMetadata meta = new ObjectMetadata();
	// 设置文件上传长度
	meta.setContentLength(input.available());
	// 上传
	s3client.putObject(bucket, objectId, input, meta);
	} catch (Exception e) {
	e.printStackTrace();
	}
	}
	public InputStream download(String bucket, String objectId) {
	try {
	S3Object o = s3client.getObject(bucket, objectId);
	return o.getObjectContent();
	} catch (Exception e) {
	e.printStackTrace();
	}
	return null;
	}
	public void download(String bucket, String objectId, OutputStream out) {
	S3Object o = s3client.getObject(bucket, objectId);
	try (InputStream in = o.getObjectContent()) {
	IOUtils.copyLarge(in, out);
	} catch (Exception e) {
	e.printStackTrace();
	}
	}
	public boolean existObject(String bucket, String objectId) {
	try {
	return s3client.doesObjectExist(bucket, objectId);
	} catch (Exception e) {
	e.printStackTrace();
	return false;
	}
	}
	public boolean deleteObject(String bucket, String objectId) {
	try {
	s3client.deleteObject(bucket, objectId);
	} catch (Exception e) {
	e.printStackTrace();
	return false;
	}
	return true;
	}
	public void close() {
	s3client = null;
	}
	}

遇到的问题

1、bucket名称必须是小写，不支持下划线

• 处理方式：写方法转换下bucket名称，将大写转小写，将下划线替换为中划线。

2、minio客户端下载非官方S3存储的文件时，如果响应头的Content-Length与实际文件大小不符，会导致minio客户端包装的okhttp3报错

报错信息：

	Caused by: java.net.ProtocolException: unexpected end of stream
	at okhttp3.internal.http1.Http1ExchangeCodec$FixedLengthSource.read(Http1ExchangeCodec.java:430) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286) ~[okhttp-3.14.9.jar:?]
	at okio.RealBufferedSource$1.read(RealBufferedSource.java:447) ~[okio-1.17.2.jar:?]
	at com.jiuqi.nr.file.utils.FileUtils.writeInput2Output(FileUtils.java:83) ~[nr.file-2.5.7.jar:?]
	at com.jiuqi.nr.file.impl.FileAreaServiceImpl.download(FileAreaServiceImpl.java:395) ~[nr.file-2.5.7.jar:?]
	... 122 more

抓包发现问题的图：

最终换成了S3官方SDK可用了。

PS：客户现场部署的S3是浪潮公司提供的，如果现场遇到这个情况，就不要固执去找对方对线了，完全没用。。

总结

S3存储的基本操作就记录到这里了，由于没有S3存储就没尝试官方SDK的V2版本，由于这些代码是总结时从业务代码里抽取出来的，可能会有点问题，但大体思路已经有了。

希望对读者有所用处，觉得写得不错和有帮到你，欢迎点个赞，您的支持就是我的鼓励！