Secret 的定义
Kubernetes(k8s)中的 Secret 是一种用于存储敏感信息的 Kubernetes 资源对象,如密码、API 密钥、证书等。Secret 被设计为用于安全地存储和管理敏感数据,并且可以通过 Volume 或环境变量的方式将这些数据提供给 Pod 中的容器。
简单来讲, K8S 的secret 和 configmap 的作用都是存放configuration 配置数据
但是 configmap 不适合存放证书, 密码等敏感数据。
官方推荐把敏感配置 存放在 secret 中, 会更加安全, 虽然个人认为也不怎么安全
Secret 创建
跟configmap 类似
screct 也可以用如下kubeclt命令来查看创建的例子
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl create secret -h
Create a secret with specified type.
A docker-registry type secret is for accessing a container registry.
A generic type secret indicate an Opaque secret type.
A tls type secret holds TLS certificate and its associated key.
Available Commands:
docker-registry Create a secret for use with a Docker registry
generic Create a secret from a local file, directory, or literal value
tls Create a TLS secret
Usage:
kubectl create secret (docker-registry | generic | tls) [options]
Use "kubectl create secret <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).
Secret 的3个子类
由上面看出 secret 也分3个子类
- docker-repository 这个类别是存放docker 私服的验证token, 会保存为json 格式, 假如k8s deployment拉取的镜像是from docker 私服, 则很可能需要provide 这个 docker-repository 类别的secret
- tls 用于存放ssl 证书 和 passwork 等信息
- generic 正常的text , 密码等配置
本文主要详细介绍 generic 类型, 不会cover 其他两种
generic 类型 secret的创建
1. by kubectl command line
还是先看一下官方例子
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl create secret generic -h
Create a secret based on a file, directory, or specified literal value.
A single secret may package one or more key/value pairs.
When creating a secret based on a file, the key will default to the basename of the file, and the value will default to
the file content. If the basename is an invalid key or you wish to chose your own, you may specify an alternate key.
When creating a secret based on a directory, each file whose basename is a valid key in the directory will be packaged
into the secret. Any directory entries except regular files are ignored (e.g. subdirectories, symlinks, devices, pipes,
etc).
Examples:
# Create a new secret named my-secret with keys for each file in folder bar
kubectl create secret generic my-secret --from-file=path/to/bar
# Create a new secret named my-secret with specified keys instead of names on disk
kubectl create secret generic my-secret --from-file=ssh-privatekey=path/to/id_rsa
--from-file=ssh-publickey=path/to/id_rsa.pub
# Create a new secret named my-secret with key1=supersecret and key2=topsecret
kubectl create secret generic my-secret --from-literal=key1=supersecret --from-literal=key2=topsecret
# Create a new secret named my-secret using a combination of a file and a literal
kubectl create secret generic my-secret --from-file=ssh-privatekey=path/to/id_rsa --from-literal=passphrase=topsecret
# Create a new secret named my-secret from env files
kubectl create secret generic my-secret --from-env-file=path/to/foo.env --from-env-file=path/to/bar.env
...
这里就不一一作例子了, 选最常用那个
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl create secret generic test-sec1 --from-literal=username=u123 --from-literal=password=p123456
secret/test-sec1 created
这里的 test-sec1 是secret item的名字, 在这个item 其实里面创建了2个 secret 对象
分别是
username:u123
password:p12345
注意, 如果value 包含特殊字符, 则需要用引号包住, 否则base64 encode 会有gap
2. by yaml
格式
apiVersion: v1
kind: Secret
metadata:
name: my-secret
type: Opaque
data:
username: <base64-encoded-username>
password: <base64-encoded-password>
注意的是, secret里的值都需要 base64 encoded之后才写入yaml文件, 但是base64 encoded 的text 是十分容易decoded, 所以建议密码还是要先加密再encoded 写入
Secret 的查看
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl get secret
NAME TYPE DATA AGE
default-token-7khb9 kubernetes.io/service-account-token 3 175d
test-sec1 Opaque 2 25m
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$
注意这里的Opaque 就是 generic的意思, Data 列2 是有连个子项(username , password)
用 Kubectl describe secret 只能看到size 信息
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl describe secret test-sec1
Name: test-sec1
Namespace: default
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
password: 7 bytes
username: 4 bytes
用 kubectl get secret xxx -o yaml 可以查看encoded 后的值
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl get secret test-sec1 -o yaml
apiVersion: v1
data:
password: cDEyMzQ1Ng==
username: dTEyMw==
kind: Secret
metadata:
creationTimestamp: "2024-08-17T13:47:38Z"
name: test-sec1
namespace: default
resourceVersion: "6132863"
uid: 2a027099-090b-478b-b460-e42991a50c62
type: Opaque
很方便地 decoded 出来真正的值
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ echo cDEyMzQ1Ng== | base64 --decode
p123456
所以说也不是很安全
Secret 的使用
例子介绍
这里我们用springboot 访问mysql的用户名和密码作例子
原本 cloud order service 的配置文件是这样的
application-prod.yml
...
spring:
datasource:
url: jdbc:mysql://192.168.0.42:3306/demo_cloud_order?useUnicode=true&characterEncoding=utf-8&useSSL=false&allowPublicKeyRetrieval=true
username: cloud_order
password: ENC(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)
driver-class-name: com.mysql.cj.jdbc.Driver
...
我们增加1个yaml 文件作为例子
application-k8sprod
...
spring:
datasource:
url: jdbc:mysql://192.168.0.42:3306/demo_cloud_order?useUnicode=true&characterEncoding=utf-8&useSSL=false&allowPublicKeyRetrieval=true
driver-class-name: com.mysql.cj.jdbc.Driver
...
吧user name 和 密码去掉了
那么怎么启动这个springboot service呢, 当然是在启动参数中传入
方法1, 吧secret map 到 环境变量
创建 secret 包含db 的username 和 password (加密后的)
这里用yaml方式
cloud-order-db-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: cloud-order-db-secret
type: Opaque
data:
# echo cloud-order | base64
username: Y2xvdWQtb3JkZXIK
password: RU5DKE9KaDRYVGFXejFixxxxxxxxxxxxx
执行
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl apply -f cloud-order-db-secret.yaml
secret/cloud-order-db-secret created
修改deployment yaml
deployment-cloud-order-with-secret-command-notwork.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels: # label of this deployment
app: cloud-order # custom defined
author: nvd11
name: deployment-cloud-order # name of this deployment
namespace: default
spec:
replicas: 3 # desired replica count, Please note that the replica Pods in a Deployment are typically distributed across multiple nodes.
revisionHistoryLimit: 10 # The number of old ReplicaSets to retain to allow rollback
selector: # label of the Pod that the Deployment is managing,, it's mandatory, without it , we will get this error
# error: error validating data: ValidationError(Deployment.spec.selector): missing required field "matchLabels" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector ..
matchLabels:
app: cloud-order
strategy: # Strategy of upodate
type: RollingUpdate # RollingUpdate or Recreate
rollingUpdate:
maxSurge: 25% # The maximum number of Pods that can be created over the desired number of Pods during the update
maxUnavailable: 25% # The maximum number of Pods that can be unavailable during the update
template: # Pod template
metadata:
labels:
app: cloud-order # label of the Pod that the Deployment is managing. must match the selector, otherwise, will get the error Invalid value: map[string]string{"app":"bq-api-xxx"}: `selector` does not match template `labels`
spec: # specification of the Pod
containers:
- image: europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/cloud-order:1.0.2 # image of the container
imagePullPolicy: Always
name: container-cloud-order
command: ["bash"]
# does not work, as the password format is ENV(xxxxx) , and the java -jar command could not handle (), after I added double quotes "ENV(xxxx)", service failed to start due to db password error
# unless I set the plain text password to secret , but I don't think it's secure enough
args:
- "-c"
- |
java -jar -Dserver.port=8080 app.jar --spring.profiles.active=$APP_ENVIRONMENT --spring.datasource.username=$DB_USER --spring.datasource.password=$DB_PASSWORD
env: # set env varaibles
- name: APP_ENVIRONMENT # name of the environment variable
value: k8sprod # value of the environment variable
- name: APP_TAG
valueFrom: # value from config map
configMapKeyRef:
name: cloud-order-app-tag-config # name of the configMap item
key: APP_TAG # key from the config map item
- name: DB_USER
valueFrom: # value from secret
secretKeyRef:
name: cloud-order-db-secret
key: username
- name: DB_PASSWORD
valueFrom: # value from secret
secretKeyRef:
name: cloud-order-db-secret
key: password
restartPolicy: Always # Restart policy for all containers within the Pod
terminationGracePeriodSeconds: 10 # The period of time in seconds given to the Pod to terminate gracefully
但是实际上这个方法对这个case 不work
因为java -jar 无法处理 ENV(XXX)
例如
gateman@MoreFine-S500:~/tmp$ java -jar -Dserver.port=8085 app.jar --spring.profiles.active=k8sprod --spring.datasource.username=cloud-order --spring.datasource.password=ENC(OJh4XTaWz1beLRY/1cMxxxx)
bash: syntax error near unexpected token `('
不能正确地转义括号
当我加上"" 双引号包住ENV()
则启动失败, 报错db 密码错误
我暂时无法解决这个问题, 除非我把明文密码放入secret , 不使用ENV() , 但是上面说过了, 把明文密码放入secret 我认为不够安全, 很容易decoded 出来
方法2, 吧secret 保存到文件
前提是其实
java 启动spring boot 命令有1个特性, 可以额外指定包含另1个配置文件
java -jar -Dserver.port=8085 -Dspring.config.additional-location=db-config.yaml app.jar --spring.profiles.active=k8sprod
注意这里的 spring.config.additional-location 必须用 -D 作为VM 参数传入, 而不是 --spring.boot 参数传入
所以我们的方向是吧username 和 password 的配置写入到1个卷中
创建 secret 包含db 的username 和 password (加密后的)
cloud-order-db-secret-file.yaml
apiVersion: v1
kind: Secret
metadata:
name: cloud-order-db-secret-file
type: Opaque
data:
# atasource:
# username: cloud_order
# password: ENC(OJh4XTaWz1beLRY/1cM4Xaxxx)
dbconfig: c3ByaW5nOgogIGRhdGFzb3VyY2U6Cxxxx
注意这里的encoded 码是 原本的格式 , 必须符合spring 的配置
修改 deployment yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels: # label of this deployment
app: cloud-order # custom defined
author: nvd11
name: deployment-cloud-order # name of this deployment
namespace: default
spec:
replicas: 3 # desired replica count, Please note that the replica Pods in a Deployment are typically distributed across multiple nodes.
revisionHistoryLimit: 10 # The number of old ReplicaSets to retain to allow rollback
selector: # label of the Pod that the Deployment is managing,, it's mandatory, without it , we will get this error
# error: error validating data: ValidationError(Deployment.spec.selector): missing required field "matchLabels" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector ..
matchLabels:
app: cloud-order
strategy: # Strategy of upodate
type: RollingUpdate # RollingUpdate or Recreate
rollingUpdate:
maxSurge: 25% # The maximum number of Pods that can be created over the desired number of Pods during the update
maxUnavailable: 25% # The maximum number of Pods that can be unavailable during the update
template: # Pod template
metadata:
labels:
app: cloud-order # label of the Pod that the Deployment is managing. must match the selector, otherwise, will get the error Invalid value: map[string]string{"app":"bq-api-xxx"}: `selector` does not match template `labels`
spec: # specification of the Pod
containers:
- image: europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/cloud-order:1.0.2 # image of the container
imagePullPolicy: Always
name: container-cloud-order
command: ["bash"]
args:
- "-c"
- |
java -jar -Dserver.port=8080 -Dspring.config.additional-location=/app/config/db-config.yaml app.jar --spring.profiles.active=$APP_ENVIRONMENT
env: # set env varaibles
- name: APP_ENVIRONMENT # name of the environment variable
value: k8sprod # value of the environment variable
- name: APP_TAG
valueFrom: # value from config map
configMapKeyRef:
name: cloud-order-app-tag-config # name of the configMap item
key: APP_TAG # key from the config map item
- name: DB_USER
valueFrom: # value from secret
secretKeyRef:
name: cloud-order-db-secret
key: username
- name: DB_PASSWORD
valueFrom: # value from secret
secretKeyRef:
name: cloud-order-db-secret
key: password
volumeMounts: # volume mount
- name: db-config
mountPath: /app/config
volumes:
- name: db-config
secret:
secretName: cloud-order-db-secret-file
items:
- key: dbconfig
path: db-config.yaml
restartPolicy: Always # Restart policy for all containers within the Pod
terminationGracePeriodSeconds: 10 # The period of time in seconds given to the Pod to terminate gracefully
注意上面有几点
- command 容器启动命令被改成 java -jar -Dserver.port=8080 -Dspring.config.additional-location=/app/config/db-config.yaml app.jar --spring.profiles.active=$APP_ENVIRONMENT
- 把 secret的内容写入 到/app/config/db-config
检查和测试
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl apply -f deployment-cloud-order-with-secret.yaml
deployment.apps/deployment-cloud-order created
首先部署
再进入容器, 查看/app/config/db-config.yaml 是否被生成
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl exec -it deployment-cloud-order-588c9444dc-7cphs /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
bash-4.4# cat /app/config/db-config.yaml
spring:
datasource:
username: cloud_order
password: ENC(OJh4XTaWz1beLRY/1cM4Xa6IP/mrxxxxxx)
正确生成
测试接口:
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/info | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1922 0 1922 0 0 3616 0 --:--:-- --:--:-- --:--:-- 3612
{
"app": "Cloud Order Service",
"appEnvProfile": "k8sprod",
"version": "1.0.2",
"hostname": "deployment-cloud-order-588c9444dc-cjtkv",
"dbUrl": "jdbc:mysql://192.168.0.42:3306/demo_cloud_order?useUnicode=true&characterEncoding=utf-8&useSSL=false&allowPublicKeyRetrieval=true",
"description": "This is a simple Spring Boot application to for cloud order...",
"SystemVariables": {
"PATH": "/usr/java/openjdk-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"CLUSTERIP_CLOUD_ORDER_SERVICE_HOST": "10.98.117.97",
"CLUSTERIP_BQ_API_SERVICE_SERVICE_PORT": "8080",
"KUBERNETES_PORT": "tcp://10.96.0.1:443",
"DB_USER": "cloud-order\n",
"JAVA_HOME": "/usr/java/openjdk-17",
"KUBERNETES_SERVICE_HOST": "10.96.0.1",
"LANG": "C.UTF-8",
"CLUSTERIP_CLOUD_ORDER_PORT": "tcp://10.98.117.97:8080",
"CLUSTERIP_BQ_API_SERVICE_PORT_8080_TCP": "tcp://10.100.68.154:8080",
"CLUSTERIP_CLOUD_ORDER_PORT_8080_TCP": "tcp://10.98.117.97:8080",
"CLUSTERIP_BQ_API_SERVICE_PORT_8080_TCP_PROTO": "tcp",
"PWD": "/app",
"JAVA_VERSION": "17.0.2",
"_": "/usr/java/openjdk-17/bin/java",
"CLUSTERIP_CLOUD_ORDER_PORT_8080_TCP_ADDR": "10.98.117.97",
"KUBERNETES_PORT_443_TCP": "tcp://10.96.0.1:443",
"KUBERNETES_PORT_443_TCP_ADDR": "10.96.0.1",
"CLUSTERIP_CLOUD_ORDER_PORT_8080_TCP_PORT": "8080",
"CLUSTERIP_BQ_API_SERVICE_PORT": "tcp://10.100.68.154:8080",
"CLUSTERIP_BQ_API_SERVICE_PORT_8080_TCP_PORT": "8080",
"KUBERNETES_PORT_443_TCP_PROTO": "tcp",
"APP_ENVIRONMENT": "k8sprod",
"KUBERNETES_SERVICE_PORT": "443",
"CLUSTERIP_CLOUD_ORDER_SERVICE_PORT": "8080",
"CLUSTERIP_BQ_API_SERVICE_PORT_8080_TCP_ADDR": "10.100.68.154",
"APP_TAG": "cloud-order-app-tag",
"HOSTNAME": "deployment-cloud-order-588c9444dc-cjtkv",
"CLUSTERIP_CLOUD_ORDER_PORT_8080_TCP_PROTO": "tcp",
"CLUSTERIP_BQ_API_SERVICE_SERVICE_HOST": "10.100.68.154",
"KUBERNETES_PORT_443_TCP_PORT": "443",
"KUBERNETES_SERVICE_PORT_HTTPS": "443",
"SHLVL": "1",
"HOME": "/root",
"DB_PASSWORD": "ENC(OJh4XTaWz1beLRY/1cM4Xa6IP/mrKgx2CdVRysJl+BZ/+7eHswcEzQWFF1PR/Hhl)"
}
}
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/health | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 364 0 364 0 0 735 0 --:--:-- --:--:-- --:--:-- 736
{
"status": "UP",
"components": {
"db": {
"status": "UP",
"details": {
"database": "MySQL",
"validationQuery": "isValid()"
}
},
"diskSpace": {
"status": "UP",
"details": {
"total": 20891439104,
"free": 12675047424,
"threshold": 10485760,
"path": "/app/.",
"exists": true
}
},
"livenessState": {
"status": "UP"
},
"ping": {
"status": "UP"
},
"readinessState": {
"status": "UP"
}
},
"groups": [
"liveness",
"readiness"
]
}
测试通过!