aws s3 参与s3game寻找宝藏游戏挑战学习s3对象存储

参考资料

Pirates S3game workshop
http://s3game-level1.s3-website.us-east-2.amazonaws.com/level1.html
https://blog.benclmnt.com/notes/s3-game/
https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/index.html

强烈推荐这种寓教于乐的方式学习知识，作者的巧思实在太赞，有趣的灵魂~

之前见过类似的前端过关挑战，偶然发现了一个aws s3 game的挑战游戏，还是蛮有意思的，一共11关，涉及到s3的方方面面。

Level1 - Public access
Level2 - Get specific object by its ID
Level3 - Metadata
Level4 - Tags
Level5 - Versioning
Level6 - S3 Select
Level7 - Presigned URL
Level8 - CloudFront
Level9 - Referer
Level10 - Storage Classes
Level11 - Encryption

入口如下：

http://s3game-level1.s3-website.us-east-2.amazonaws.com/level1.html

在这里插入图片描述

注意，请不要查看本文章的答案，努力享受寓教于乐的乐趣吧~

Level1

第一关介绍了s3 托管静态网站的注意事项，以及website endpoint和restapi endpoint的区别。

通关目标是查看桶内的文件

$ aws s3 ls s3://s3game-level1/ --region us-east-2
An error occurred (InvalidToken) when calling the ListObjectsV2 operation: The provided token is malformed or otherwise invalid.

提示说桶策略为，允许所有人访问

Bucket policy is:
  "Effect": "Allow",
  "Principal": "*",
  "Action": "s3:ListBucket",
  "Resource": "arn:aws:s3:::s3game-level1"

在中国区无法访问海外的存储桶

$ aws s3 ls s3://s3game-level1/ --no-sign-request
An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist

我本地配了凭证，反而无法访问桶内的对象，可见这个桶只允许匿名访问

$ aws s3 ls s3://s3game-level1/ --no-sign-request --region us-east-2
2020-06-01 08:58:46      24361 S3game.png
2020-04-20 06:39:19       1568 level1-hint2.html
2020-04-20 06:39:19       1484 level1-hint3.html
2020-04-20 06:39:19       1566 level1-hint4.html
2020-04-20 12:28:16       2240 level1.html
2020-06-01 09:09:01       1936 s3game.html
2020-05-02 08:08:53        115 treasure1

提示说需要找到名为treasure的文件，然后通过链接进入下一关

$ aws s3 cp s3://s3game-level1/treasure1 --no-sign-request --region us-east-2 .
download: s3://s3game-level1/treasure1 to ./treasure1
[ec2-user@ip-172-31-18-4 s3game]$ cat treasure1
The secret code is 748l6b6xwzl6
Go to https://s3game-level2.s3.us-east-2.amazonaws.com/level2-748l6b6xwzl6.html

Level2

主要介绍了s3访问路径的区别，列举如下

virtual-hosted style URL

https://george-work.s3.cn-north-1.amazonaws.com/thefile.gif

path style URL

https://s3.Region.amazonaws.com/bucket-name/key-name

s3访问点

https://AccessPointName-AccountId.s3-accesspoint.region.amazonaws.com.

s3前缀访问
```
S3://bucket-name/key-name
```

其中path style URL2022年9月30日之后已经不在支持了

我们需要获取第二个treasure文件，这一关比较简单

$ aws s3 cp s3://s3game-level2/treasure2 --no-sign-request --region us-east-2 .
download: s3://s3game-level2/treasure2 to ./treasure2
$ cat treasure2
The secret is 76qp7mlpzyg1
Strange signs are scratched on the lid of the treasure chest:
AKIAZBIEGK7G3GDBOOIQ  nJkVYeENVtcSuqZ2ueuXWVkHrZ5zfKxY8Z0E/msp
Go to https://s3game-level3.s3.us-east-2.amazonaws.com/level3-76qp7mlpzyg1.html

文件中出现了疑似aksk的东西，哈哈哈估计是下一关的提示

Level3

第三关开始就提到了aksk认证，保证机密性和不泄露，这提示我们无法匿名访问这个桶，配置上一关的aksk

$ aws configure --profile s3game
AWS Access Key ID [None]: AKIAZBIEGK7G3GDBOOIQ
AWS Secret Access Key [None]: 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
Default region name [None]: us-east-2
Default output format [None]:

我们需要通过获取对象元数据获取随机字符串并通过以下链接进入下一关

https://s3game-level4-<THE CODE>.s3.us-east-2.amazonaws.com/level4.html

我们通过以往的管理已经知道第三关的桶名为s3game-level3，果然匿名访问直接denied

$ aws s3 ls s3://s3game-level3/ --region us-east-2 --no-sign-request
An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
$ aws s3 ls s3://s3game-level3/ --profile s3game
2020-04-20 06:44:08       1721 level3-76qp7mlpzyg1-hint2.html
2020-04-20 06:44:08       1787 level3-76qp7mlpzyg1-hint3.html
2020-04-20 11:22:48       1873 level3-76qp7mlpzyg1-hint4.html
2020-04-20 12:27:42       1990 level3-76qp7mlpzyg1.html
2020-05-02 08:12:11        234 treasure3_has_no_secret_code

查看treasure3_has_no_secret_code的内容

$ aws s3 cp s3://s3game-level3/treasure3_has_no_secret_code --profile s3game .
download: s3://s3game-level3/treasure3_has_no_secret_code to ./treasure3_has_no_secret_code
$ cat treasure3_has_no_secret_code
Hm... the chest is empty :(
Let's look around, may be secret code is somewhere else...
Think about where else the code can be hidden?

Find the code and go to https://s3game-level4-<THE CODE>.s3.us-east-2.amazonaws.com/level4.html

接下来提示我们查看对象元数据获取更多信息，这里我们需要使用s3cli工具的低等级api head-object，这会检索对象元数据，必须有对象的read权限才行

$ aws s3api head-object --bucket s3game-level3 --key treasure3_has_no_secret_code --profile s3game
{
    "AcceptRanges": "bytes",
    "LastModified": "2020-05-02T08:12:11+00:00",
    "ContentLength": 234,
    "ETag": "\"6244574e387025dec9b9b589773d4d29\"",
    "ContentType": "binary/octet-stream",
    "Metadata": {
        "x-amz-meta-secret-code": "k73045aztqln"
    }
}

就这样我们得到了下一关的密钥，作者巧思

https://s3game-level4-k73045aztqln.s3.us-east-2.amazonaws.com/level4.html

Level4

开局让我们仔细检查这一关的桶名？果然这个桶不存在，回头仔细看桶全称为s3game-level4-k73045aztqln。。。

$ aws s3 ls s3://s3game-level4 --profile s3game
An error occurred (NoSuchBucket) when calling the ListObjectsV2 operation: The specified bucket does not exist
$ aws s3 ls s3://s3game-level4-k73045aztqln --profile s3game
2020-04-20 06:45:01       2040 level4-hint2.html
2020-05-24 19:45:57       1462 level4.html
2020-05-23 17:24:41        270 treasure4_also_has_no_secret_code

提示来了，s3中的对象是不可变的，编辑对象的元数据等于是创建新的对象。而tag和subresources是可变的（不会修改对象）。很明显提示我们获取对象的tag信息。

$ aws s3 cp s3://s3game-level4-k73045aztqln/treasure4_also_has_no_secret_code --profile s3game .
download: s3://s3game-level4-k73045aztqln/treasure4_also_has_no_secret_code to ./treasure4_also_has_no_secret_code
$ cat treasure4_also_has_no_secret_code
Oh... this chest is empty as well :(
Let's look around, may be secret code is somewhere else...
You already know about metadata. Think where else the code can be hidden?
Find the code and go to https://s3game-level5-<THE CODE>.s3.us-east-2.amazonaws.com/level5.html
$ aws s3api get-object-tagging --bucket s3game-level4-k73045aztqln --key treasure4_also_has_no_secret_code --profile s3game
{
    "TagSet": [
        {
            "Key": "secret_code",
            "Value": "8v95e5rv7z4i"
        }
    ]
}

进入下一关，注意桶名还是带密钥的全称

https://s3game-level5-8v95e5rv7z4i.s3.us-east-2.amazonaws.com/level5.html

Level5

提示，有时候一些对象存在但又不存在，我猜可能要问版本控制和删除标记。果然没有发现treasure文件。

$ aws s3 ls s3://s3game-level5-8v95e5rv7z4i --profile s3game
2020-05-02 08:35:32       2005 level5-hint2.html
2020-05-02 08:35:32       2802 level5-hint3.html
2020-04-20 11:08:44       1698 level5-hint4.html
2020-04-20 11:04:58       1648 level5.html

开启版本控制的存储桶，会对每个对象添加一个唯一id（没开之前的对象id为null），因此同一个桶中可能存在两个同样key的对象，但是版本id不同。默认s3的request api只会获取最近版本的对象。

By default, the GET action returns the current version of an object. To return a different version, use the versionId subresource

If you supply a versionId , you need the s3:GetObjectVersion permission to access a specific version of an object. If you request a specific version, you do not need to have the s3:GetObject permission.
If the current version of the object is a delete marker, Amazon S3 behaves as if the object was deleted and includes x-amz-delete-marker: true in the response.

版本控制下，删除对象不会真的删除，而是增加一个删除标记。推测treasure文件被删除标记覆盖了。那么接下来就是找到这个文件。

果然开启了版本控制

$ aws s3api get-bucket-versioning --bucket s3game-level5-8v95e5rv7z4i --profile s3game
{
    "Status": "Enabled"
}

列出版本对象

$ aws s3api list-object-versions --bucket s3game-level5-8v95e5rv7z4i --profile s3game
{
    "Versions": [
        {
            "ETag": "\"9693f24274d7ec88a8563d662c34be99\"",
            "Size": 126,
            "StorageClass": "STANDARD",
            "Key": "treasure5_is_deleted",
            "VersionId": "344PQOyFqocF0TI66MbLynNNdQqHfBz3",
            "IsLatest": false,
            "LastModified": "2020-05-02T08:20:06+00:00"
        }
    ],
    "DeleteMarkers": [
        {
            "Key": "treasure5_is_deleted",
            "VersionId": "EtCsdZVg383Pj4qEB9XzjPKwMWMMd_d8",
            "IsLatest": true,
            "LastModified": "2020-05-02T08:20:38+00:00"
        }
    ]
}

通过版本id获取这个对象到本地即可。此外无法下载delete marker

$ aws s3api get-object --bucket s3game-level5-8v95e5rv7z4i --key treasure5_is_deleted --version-id 344PQOyFqocF0TI66MbLynNNdQqHfBz3 --profile s3game treasure5
$ aws s3api get-object --bucket s3game-level5-8v95e5rv7z4i --key treasure5_is_deleted --version-id EtCsdZVg383Pj4qEB9XzjPKwMWMMd_d8 --profile s3game deletemarker
An error occurred (MethodNotAllowed) when calling the GetObject operation: The specified method is not allowed against this resource.
$ cat treasure5
The code is vjv45x1gux81
Find the code and go to https://s3game-level6-vjv45x1gux81.s3.us-east-2.amazonaws.com/level6.html

进入下一关

Level6

提示使用s3 select api

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/select-object-content.html

用法如下

aws s3api select-object-content \
    --bucket my-bucket \
    --key my-data-file.csv \
    --expression "select * from s3object limit 100" \
    --expression-type 'SQL' \
    --input-serialization '{"CSV": {}, "CompressionType": "NONE"}' \
    --output-serialization '{"CSV": {}}' "output.csv"

查看桶对象，有一个s3select.csv.gz对象

$  aws s3 ls s3://s3game-level6-vjv45x1gux81 --profile s3game
2020-04-20 14:39:42       1785 level6-hint2.html
2020-07-01 14:58:43       2404 level6.html
2020-05-02 08:52:25   12884420 s3select.csv.gz

其实可以下载下来解压查看内容，这一关就是为了用这个api

aws s3api select-object-content --profile s3game \
	--bucket s3game-level6-vjv45x1gux81 \
	--key s3select.csv.gz \
	--expression "SELECT s.Answer FROM s3object s WHERE Category = 'TREASURE'" \
	--expression-type 'SQL' \
	--input-serialization '{"CSV": {"FileHeaderInfo": "USE", "FieldDelimiter": ";"}, "CompressionType": "GZIP"}' \
	--output-serialization '{"CSV": {}}' \
	treasure6

进入下一关

$ cat treasure6
Go to https://s3game-level7-zhovpo4j8588.s3.us-east-2.amazonaws.com/level7.html

Level7

这一关考验权限的内容。默认s3对象都是私有的，只有owner有权限访问。但是owner可以通过创建presigner url暂时授予拥有url的人访问对象的权，例如

$ aws s3 presign s3://<BUCKET>/<OBJECT> --expires-in 3600

直接访问是没有权限的

$ aws s3 ls s3://s3game-level7-zhovpo4j8588 --profile s3game
An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

玩了个tricky，这个其实就是文件名

在这里插入图片描述

获取预签名

$ aws s3 cp s3://s3game-level7-zhovpo4j8588/somethingstrange  --profile s3game .
download: s3://s3game-level7-zhovpo4j8588/somethingstrange to ./somethingstrange
$ cat somethingstrange
https://s3game-level7-zhovpo4j8588.s3.amazonaws.com/treasure7_is_presigned?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAZBIEGK7GW6B6MKWQ%2F20230115%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20230115T084502Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEKn%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMiJGMEQCIBT12LVX%2BLG%2F8izT9jS0obpHK01AypvENhEYiJqjPaolAiBpYzCWYlgcPySni%2FXY0RE5cBmAOZbDSNhl5mFBF%2ByQTSrNAwgSEAIaDDYyMTE2ODQ0OTQ4NSIMFSwfP5UxhESHqGVUKqoD64vhzId7f5IF2ZWQpRgH3wj3Q9baR%2BlUk%2Bi0CYYZfgzQIrDT6ivxvPvpDT9Ab2YB94MI9Tejp491EiCUAtM1YnO1XRcoMqtvopZTRRpmKBq3cokr1eWQ5owqyyaK0fSlHJE4jPSKGznCe%2Bc%2BM51TctQbZPrE87zS5PdQ7ZWGhLidsUVimToaaX2SWMgLM6H6l2XwZNr463HFI%2FOPOlf4jnuIQ15yZ7f4%2FgGGtjBazz3d7LMX4SlspIz1DRQlG2UFx7Ja4OfjbxoUmfcUcV0an2c8XmIvLxrtfBhPtl4v7PZ4P1jF18lk%2FXFhTI%2FIHBwxmtM8ueUE8GnrUfs%2FX033UTPJh7WTW6QKK96LBg7%2BYIdb4%2Fw0EhYdLOCHlfxk2p2cQtt%2BaKRjQ%2BlyeAGvcC%2BaVnYnAoUBxq14sZ4AYCvJIVF%2BVD2xBCDvopLfI6TVmvO4oDC%2FUyJ%2F4Bgwb%2BSzI1IsdfJQ3DnJNQ71bwAjZdw7m%2BxEOqSWZvlBnGISBRibdEK3nJE1qeJ%2FrqN%2F5R3tvdjy1A6r24FhqbTe9BtEpGiwjsZWeFtO7HL26yFnMI76jp4GOp8BCY72ytFtfqsYgaPBIQ%2Bq8PnIBG7HOEGLqtn5v%2B9xnS5Jy%2Fwx7QbCX8NALpDXMuW2Vg5roioom0N9xlkVqXLEu53Pm8%2BImKdWv%2BqDTqT%2FXoxvH9%2FYOgY9AZdfSpPXDyppkvCiR539QY4r6m4wUpyPOEvkytHYYvUH0VUcFmppSSB4kA0fvdvng7AUcpr01OF%2BETHhemNiphOAs%2FLQ9jFT&X-Amz-Signature=1bd9732edcbfb6649493c2c38d4d4aaabb6a33a2887329e9506b99b1dec57e4f

用浏览器打开上面的链接，进入下一关

The code is v6g8tp7ra2ld
Find the code and go to https://s3game-level8-v6g8tp7ra2ld.s3.us-east-2.amazonaws.com/level8.html

Level8

涉及到了cloudfront，如果s3桶是cdn的源，可以通过控制，cloudfront控制对象访问

直接无法访问对象

$ aws s3 ls s3://s3game-level8-v6g8tp7ra2ld --profile s3game
2020-04-24 09:17:55       2265 level8.html
2020-05-02 09:37:19        126 treasure8_CDN
$ aws s3 cp s3://s3game-level8-v6g8tp7ra2ld/treasure8_CDN --profile s3game .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden

提示桶策略为

{
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity xxx"
    },
    "Action": "s3:GetObject",
    "Resource": "arn:aws:s3:::bucket/*"
}

提示You can get the object using d2suiw06vujwz3.cloudfront.net CloudFront distribution URL，通过cdn访问对象即可

$ curl -O https://d2suiw06vujwz3.cloudfront.net/treasure8_CDN
$ cat treasure8_CDN
The code is 781xtls2quvy
Find the code and go to https://s3game-level9-781xtls2quvy.s3.us-east-2.amazonaws.com/level9.html

进入下一关

Level9

http请求头中的refer能够识别访问的来源。查看对象

$ aws s3 ls s3://s3game-level9-781xtls2quvy --profile s3game
2020-04-21 08:00:25       1540 level9-hint2.html
2020-05-02 09:47:05       2246 level9.html
2020-05-02 09:50:22        128 treasure9_referer

桶策略如下

{
  "Effect": "Allow",
  "Principal": "*",
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::s3game-level9-781xtls2quvy/treasure9_referer",
  "Condition": {
      "StringLike": {
          "aws:Referer": [
              "http://s3game.treasure"
          ]
      }
  }

通过这个条件键过滤了外部访问，主体是*那我们可以直接curl

$ curl --referer http://s3game.treasure https://s3game-level9-781xtls2quvy.s3.us-east-2.amazonaws.com/treasure9_referer

The code is gac6tf83erp6
Find the code and go to https://s3game-level10-gac6tf83erp6.s3.us-east-2.amazonaws.com/level10.html

进入下一关

Level10

s3提供了不同的存储类，可以设置声明周期策略自动进行存储类的转换

Just find the object in Standard-IA storage class

直接查看文件发现太多了

$ aws s3 ls s3://s3game-level10-gac6tf83erp6 --profile s3game
2020-05-02 10:00:31        128 03kufblhnwiw
2020-05-02 09:57:44        128 0f6e843c99to
2020-05-02 09:59:59        128 0jx774dmh047
2020-05-02 09:59:48        128 0rtmc3bs4kb6
2020-05-23 17:00:26        128 0sg0snw2oktd
2020-05-02 10:01:58        128 13iqnird613i
2020-05-02 09:57:56        128 13itwr8uz0iw
2020-05-02 09:59:47        128 183e62jsoa09
2020-05-23 17:01:11        128 1a43gkanmyp6
2020-05-02 09:59:04        128 1b8k99grgj7w
2020-05-02 09:59:28        128 1d5iso2985uv
2020-05-23 17:01:14        128 1khbex6o9kpa
2020-05-23 17:01:09        128 1oj1j3nbtjon
2020-05-23 17:00:37        128 1wv7zieqe08w
2020-05-02 10:02:40        128 1zn06q85m31a
2020-05-23 17:01:03        128 227jh4msboxj
2020-05-23 17:00:20        128 25ffnknd3rtl

使用低级api获取对象，存储类是对象的一个属性，我们可以进行筛选

$ aws s3api list-objects --bucket s3game-level10-gac6tf83erp6 --profile s3game
{
    "Contents": [
        {
            "Key": "03kufblhnwiw",
            "LastModified": "2020-05-02T10:00:31+00:00",
            "ETag": "\"4730dca4ccc205769b149c0c84abf15c\"",
            "Size": 128,
            "StorageClass": "STANDARD"
        },
        {
            "Key": "0f6e843c99to",
            "LastModified": "2020-05-02T09:57:44+00:00",
            "ETag": "\"13a308d161965ab89137b2cba7072938\"",
            "Size": 128,
            "StorageClass": "STANDARD"
        },
...
}

查看standard-ia的对象

$ aws s3api list-objects --bucket s3game-level10-gac6tf83erp6 --query 'Contents[?StorageClass == `STANDARD_IA`]' --profile s3game
[
    {
        "Key": "djq30a807iyq",
        "LastModified": "2020-05-23T16:56:46+00:00",
        "ETag": "\"b29bd78cd969aec7862407ff045d8aeb\"",
        "Size": 128,
        "StorageClass": "STANDARD_IA"
    }
]

进入下一关

$ aws s3 cp s3://s3game-level10-gac6tf83erp6/djq30a807iyq --profile s3game .
download: s3://s3game-level10-gac6tf83erp6/djq30a807iyq to ./djq30a807iyq
[ec2-user@ip-172-31-18-4 s3game]$ cat djq30a807iyq
The code is djq30a807iyq
Find the code and go to https://s3game-level11-djq30a807iyq.s3.us-east-2.amazonaws.com/level11.html

Level11

这一关涉及到对象加密。s3加密方式有以下几种

服务端加密
- sse-s3
- sse-kms
- sse-c
客户端加密

注意：服务器端加密只加密对象数据，而不加密对象元数据

提示加密方式为sse-c客户自定义密钥，加密密钥为UkXp2s5v8y/B?E(H+MbPeShVmYq3t6w9

$ aws s3 ls s3://s3game-level11-djq30a807iyq --profile s3game
2020-05-02 10:11:55       2485 level11.html
2020-05-02 10:06:33        128 treasure11_encryption

直接获取对象即可

$ aws s3 cp s3://s3game-level11-djq30a807iyq/treasure11_encryption . --sse-c-key "UkXp2s5v8y/B?E(H+MbPeShVmYq3t6w9" --sse-c --profile s3game
$ cat treasure11_encryption
The code is bk0m5ax5n92o
Find the code and go to https://s3game-level12-bk0m5ax5n92o.s3.us-east-2.amazonaws.com/level12.html