写在前面
- 生产环境中的 k8s 集群安全不可忽略，即使是内网环境
- 容器化的应用部署虽然本质上没有变化，始终是机器上的一个进程
- 但是提高了安全问题的处理的复杂性
- 分享一个开源的 k8s 集群安全合规检查/漏洞扫描 工具
kubescape - 博文内容涉及：
kubescape简介介绍kubescape命令行工具安装，扫描运行的集群kubescape在集群下安装，通过 kubescape Clound 可视化查看扫描信息
- 理解不足小伙伴帮忙指正
- 需要有科学上网环境
对每个人而言，真正的职责只有一个：找到自我。然后在心中坚守其一生，全心全意，永不停息。所有其它的路都是不完整的，是人的逃避方式，是对大众理想的懦弱回归，是随波逐流，是对内心的恐惧 ——赫尔曼·黑塞《德米安》
简单介绍
k8s 安全问题不可忽略
镜像的不可变性使我们能够方便地部署、测试和发布镜像到其他环境，这是一个很大的优势，但也带来了潜在的风险：镜像及其依赖在过时或者被弃用时，无法自动更新或打新的安全补丁，尤其一些基础镜像，重新做镜像，需要依赖一些厂商，或者开源项目 team 来维护。
RedHat 在 2022 年 Kubernetes 安全报告中对 300 多名 DevOps、工程和安全专业人士进行了调查，发现对 容器安全威胁的担忧以及对容器安全投资的缺乏 是 31% 的受访者对容器策略最普遍的担忧。
支持这些担忧的是 93% 的受访者在过去 12 个月内在其 Kubernetes 环境中经历过至少一次安全事件，这些事件有时会导致收入或客户流失。超过一半的受访者 (55%) 在过去一年中还因为安全问题而不得不推迟应用程序的推出。
尽管媒体广泛关注网络攻击，但该报告强调，实际上是错误配置让 IT 专业人员彻夜难眠。Kubernetes 是高度可定制的，具有可以影响应用程序安全状况的各种配置选项。因此，受访者最担心的是容器和 Kubernetes 环境中的错误配置导致的风险暴露 (46%) —— 几乎是对攻击的担忧程度 (16%) 的三倍。尽可能自动化配置管理有助于缓解这些问题，因此安全工具 —— 而不是人类 —— 提供了帮助开发人员和 DevOps 团队更安全地配置容器和 Kubernetes 的护栏。
以上内容来自：Redhat Blog(The State of Kubernetes Security in 2022)
Kubescape 简单介绍
Kubescape 是一个开源的 Kubernetes 安全平台。它的功能包括 风险分析、安全合规性 和 错误配置扫描。针对 DevSecOps 从业者或平台工程师，提供易于使用的 CLI 界面、灵活的输出格式和自动扫描功能。同时对于小集群提供了免费的 在线 面板工具，它为 Kubernetes 用户和管理员节省了宝贵的时间、精力和资源。
Kubescape 可以扫描运行的集群、静态 YAML 文件和 本地 Helm Charts。它根据多个框架（包括 NSA-CISA、MITRE ATT&CK®和CIS Benchmark）检测错误配置。
Kubescape 由 ARMO 创建，是一个 Cloud Native Computing Foundation (CNCF) 沙盒项目。
Kubescape 如果小伙伴觉得重的话， kubectl 有一个类似的插件，个人十分推荐，叫做 kube-score ,很轻量，也可以做一些简单的合规性扫描。但是只有扫描合规性的提示，没有规范出处。
Kubescape 安装
当前的集群信息
┌──[root@vms100.liruilongs.github.io]-[~]
└─$kubectl get nodes
NAME STATUS ROLES AGE VERSION
vms100.liruilongs.github.io Ready control-plane 8d v1.25.1
vms101.liruilongs.github.io Ready control-plane 8d v1.25.1
vms102.liruilongs.github.io Ready control-plane 8d v1.25.1
vms103.liruilongs.github.io Ready <none> 8d v1.25.1
vms105.liruilongs.github.io Ready <none> 8d v1.25.1
vms106.liruilongs.github.io Ready <none> 8d v1.25.1
vms107.liruilongs.github.io Ready <none> 8d v1.25.1
vms108.liruilongs.github.io Ready <none> 8d v1.25.1
┌──[root@vms100.liruilongs.github.io]-[~]
└─$
集群节点信息
┌──[root@vms100.liruilongs.github.io]-[~]
└─$hostnamectl
Static hostname: vms100.liruilongs.github.io
Icon name: computer-vm
Chassis: vm
Machine ID: e93ae3f6cb354f3ba509eeb73568087e
Boot ID: a1150b6d97dc4afbb81dae58f131a487
Virtualization: vmware
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 5.4.230-1.el7.elrepo.x86_64
Architecture: x86-64
┌──[root@vms100.liruilongs.github.io]-[~]
└─$
Kubescape 命令行工具安装
Kubescape CLI 安装
通过下面的方式自动安装
curl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash
如果没有科学上网，找一台可以访问的集群，下载 install.sh 文件，按照下面方式修改，获取 curl 命令，自行下载。
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$vim install.sh
.......
# curl --progress-bar -L $DOWNLOAD_URL -o $OUTPUT
echo "curl --progress-bar -L $DOWNLOAD_URL -o $OUTPUT"
exit 1
...
运行 shell，获取下载命令，找有网的机器下载
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$sh install.sh
Installing Kubescape...
curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescape
上传到指定位置
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$ls
install.sh kubescape-ubuntu-latest
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$mv kubescape-ubuntu-latest /root/.kubescape/kubescape
mv：是否覆盖"/root/.kubescape/kubescape"？ y
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$
修改脚本，再次运行。
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$vim install.sh
# exit 1
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$sh install.sh
Installing Kubescape...
curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescape
Finished Installation.
Your current version is: v2.0.183 [git enabled in build: true]
Usage: $ kubescape scan --enable-host-scan
到这里 ，kubescape 命令行工具即安装成功，扫描当前运行的集群可以运行如下命令。
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubescape scan --enable-host-scan --format html --output results.html --verbose
[info] Kubescape scanner starting
[info] Installing host scanner
[info] Downloading/Loading policy definitions
[success] Downloaded/Loaded policy
[info] Accessing Kubernetes objects
[success] Accessed to Kubernetes objects
[info] Requesting images vulnerabilities results
[success] Requested images vulnerabilities results
[info] Requesting Host scanner data
[info] Host scanner version : v1.0.39
◑[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-xshr6; error: the server could not find the requested resource (get pods http:host-scanner-xshr6:7888)
◒[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-4tgnp; error: the server could not find the requested resource (get pods http:host-scanner-4tgnp:7888)
.......
[success] Done scanning. Cluster: kubernetes-admin-kubernetes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
################################################################################
ApiVersion: hostdata.kubescape.cloud/v1beta0
Kind: KubeletInfo
Name: vms105.liruilongs.github.io
Controls: 21 (Failed: 6, Excluded: 0)
+----------+--------------------------------+------------------------------------+------------------------+
| SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION |
+----------+--------------------------------+------------------------------------+------------------------+
| High | CIS-4.1.7 Ensure that the | https://hub.armosec.io/docs/c-0168 | |
| | certificate authorities file | | |
| | permissions are set to 600 or | | |
| | more restrictive | | |
+ +--------------------------------+------------------------------------+------------------------+
| | CIS-4.1.9 If the kubelet | https://hub.armosec.io/docs/c-0170 | |
| | config.yaml configuration | | |
| | file is being used validate | | |
| | permissions set to 600 or more | | |
| | restrictive | | |
+----------+--------------------------------+------------------------------------+------------------------+
........
+----------+--------------------------------+------------------------------------+------------------------+
| Low | CIS-4.2.6 Ensure that the | https://hub.armosec.io/docs/c-0177 | protectKernelDefaults |
| | --protect-kernel-defaults | | |
| | argument is set to true | | |
+ +--------------------------------+------------------------------------+------------------------+
| | CIS-4.2.7 Ensure that the | https://hub.armosec.io/docs/c-0178 | makeIPTablesUtilChains |
| | --make-iptables-util-chains | | |
| | argument is set to true | | |
+----------+--------------------------------+------------------------------------+------------------------+
################################################################################
ApiVersion: v1
Kind: Namespace
Name: kubescape
Controls: 18 (Failed: 17, Excluded: 0)
+----------+--------------------------------+------------------------------------+----------------------------------------------------------------+
| SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION |
+----------+--------------------------------+------------------------------------+----------------------------------------------------------------+
| High | CIS-5.2.11 Minimize the | https://hub.armosec.io/docs/c-0202 | metadata.labels[pod-security.kubernetes.io/enforce]=baseline |
| | admission of Windows | | |
| | HostProcess Containers | | |
+ +--------------------------------+------------------------------------+ +
| | CIS-5.2.2 Minimize the | https://hub.armosec.io/docs/c-0193 | |
| | admission of privileged | | |
| | containers | | |
+----------+--------------------------------+------------------------------------+----------------------------------------------------------------+
| Medium | CIS-5.2.1 Ensure that | https://hub.armosec.io/docs/c-0192 | metadata.labels[pod-security.kubernetes.io/enforce]=YOUR_VALUE |
| | the cluster has at least | | |
| | one active policy control | | |
| | mechanism in place | | |
+ +--------------------------------+------------------------------------+----------------------------------------------------------------+
.......
..........
################################################################################
ApiVersion: hostdata.kubescape.cloud/v1beta0
Kind: ControlPlaneInfo
Name: vms102.liruilongs.github.io
Controls: 25 (Failed: 1, Excluded: 0)
+----------+--------------------------------+------------------------------------+-----------------------+
| SEVERITY | CONTROL NAME | DOCS | ASSISTANT REMEDIATION |
+----------+--------------------------------+------------------------------------+-----------------------+
| High | CIS-1.1.20 Ensure that the | https://hub.armosec.io/docs/c-0111 | |
| | Kubernetes PKI certificate | | |
| | file permissions are set to | | |
| | 600 or more restrictive | | |
+----------+--------------------------------+------------------------------------+-----------------------+
################################################################################
ApiVersion: apps/v1
Kind: Deployment
Name: local-path-provisioner
Namespace: local-path-storage
Controls: 35 (Failed: 18, Excluded: 0)
...
........
......
输出的合规信息和漏洞信息。
异常问题
我在 kubescape 多次次扫描中，集群因部分节点因为端口问题，无法发生调度，节点上的 kubescape 对于的 pod 无法自行删除，如果上一次扫描的 pod 或则 ns 没有删除，那么下一次的扫描无法进行，之前创建的 pod 和 ns 状态一直为 Terminating, 解决办法需要 对命名空间进行彻底删除。
这里执行完命令会进入阻塞状态。
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubevirt]
└─$kubectl delete ns kubescape-host-scanner
namespace "kubescape-host-scanner" deleted
运行脚本删除
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s_shell_secript]
└─$cat delete_namespace.sh
#!/bin/bash
coproc kubectl proxy --port=30990 &
if [ $# -eq 0 ] ; then
echo "后面加上你所要删除的ns."
exit 1
fi
kubectl get namespace $1 -o json > logging.json
sed -i '/"finalizers"/{n;d}' logging.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @logging.json http://127.0.0.1:30990/api/v1/namespaces/${1}/finalize
kill %1
┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s_shell_secript]
└─$sh delete_namespace.sh kubescape-host-scanner
也可以离线运行Kubescape，时间网络原因，这里不做分享，有需要的小伙伴可以到 github 上的项目地址查看详细信息。
一些其他的用法
扫描正在运行的 Kubernetes 集群：
kubescape scan --enable-host-scan --verbose
使用替代的 kubeconfig 文件：
kubescape scan --kubeconfig cluster.conf
扫描特定的命名空间：
kubescape scan --include-namespaces development,staging,production
排除某些命名空间：
kubescape scan --exclude-namespaces kube-system,kube-public
部署前扫描本地 YAML/JSON 文件：
kubescape scan *.yaml
┌──[root@vms100.liruilongs.github.io]-[~/ansible/helm]
└─$kubescape scan --enable-host-scan kube-prometheus-stack.yaml --format html --output resout.html
[info] Kubescape scanner starting
[warning] in setContextMetadata. case: git-url; error: repository host 'gitee.com' not supported
[info] Downloading/Loading policy definitions
[success] Downloaded/Loaded policy
[info] Accessing local objects
[success] Done accessing local objects
[info] Scanning GitLocal
[success] Done scanning GitLocal
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Controls: 55 (Failed: 31, Excluded: 0, Skipped: 0)
Failed Resources by Severity: Critical — 0, High — 34, Medium — 151, Low — 24
+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
| SEVERITY | CONTROL NAME | FAILED RESOURCES | EXCLUDED RESOURCES | ALL RESOURCES | % RISK-SCORE |
+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
| High | Resources memory limit and request | 7 | 0 | 7 | 100% |
| High | Resource limits | 7 | 0 | 7 | 100% |
| High | List Kubernetes secrets | 4 | 0 | 7 | 57% |
。。。。。。。。。。。。。。。。。。。。。
| Medium | Allow privilege escalation | 6 | 0 | 7 | 86% |
| Medium | Ingress and Egress blocked | 7 | 0 | 7 | 100% |
。。。。。。。。。。。。。。。。。。。。。
| Medium | CIS-5.4.1 Prefer using secrets as files over secrets as environment variables | 1 | 0 | 7 | 14% |
| Medium | CIS-5.7.2 Ensure that the seccomp profile is set to docker/default in your pod definitions | 7 | 0 | 7 | 100% |
| Medium | CIS-5.7.4 The default namespace should not be used | 56 | 0 | 61 | 92% |
| Low | Immutable container filesystem | 6 | 0 | 7 | 86% |
| Low | Configured readiness probe | 5 | 0 | 7 | 71% |
| Low | Malicious admission controller (validating) | 1 | 0 | 1 | 100% |
| Low | Pods in default namespace | 7 | 0 | 7 | 100% |
| Low | Naked PODs | 1 | 0 | 1 | 100% |
| Low | Label usage for resources | 3 | 0 | 7 | 43% |
| Low | K8s common labels usage | 1 | 0 | 7 | 14% |
+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
| | RESOURCE SUMMARY | 66 | 0 | 72 | 43.26% |
+----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+
FRAMEWORKS: ArmoBest (risk: 33.51), cis-v1.23-t1.0.1 (risk: 62.75), DevOpsBest (risk: 61.72), AllControls (risk: 36.46), MITRE (risk: 11.79), NSA (risk: 33.74)
[success] Scan results saved. filename: resout.html
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Scan results have not been submitted: run kubescape with the '--account' flag
For more details: https://hub.armosec.io/docs/installing-kubescape?utm_campaign=Submit&utm_medium=CLI&utm_source=GitHub
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Run with '--verbose'/'-v' flag for detailed resources view
┌──[root@vms100.liruilongs.github.io]-[~/ansible/helm]
└─$l
从 Git 存储库扫描 Kubernetes 清单文件：
kubescape scan https://github.com/kubescape/kubescape
扫描 Helm 图表,kubescape 将加载默认的 VALUES 文件。
kubescape scan </path/to/directory>
扫描 Kustomize 目录,Kubescape 将使用 kustomize 文件生成 Kubernetes YAML 对象并扫描它们以确保安全。
kubescape scan </path/to/directory>
使用 NSA 框架扫描正在运行的 Kubernetes 集群：
kubescape scan framework nsa
使用 MITRE ATT&CK® 框架扫描正在运行的 Kubernetes 集群：
kubescape scan framework mitre
使用控件名称或控件 ID 扫描特定控件。请参阅控件列表。
kubescape scan control "Privileged container"
指定报告输出格式
JSON：
kubescape scan --format json --format-version v2 --output results.json
XML：
kubescape scan --format junit --output results.xml
PDF:
kubescape scan --format pdf --output results.pdf
普罗米修斯指标：
kubescape scan --format prometheus
HTML
kubescape scan --format html --output results.html
显示所有扫描到的资源（包括通过的资源）：
kubescape scan --verbose
Kubescape 在集群下安装
Kubescape 也可以在集群下安装，通过集群安装，可以在 cloud 里的 UI 界面查看具体扫描信息，集群中安装 Kubescape，先决条件
- 确保您拥有 Kubescape Cloud 帐户——如果没有，请在此处注册
- 您需要拥有对集群的安装权限（您应该能够创建 Deployments、CronJobs、ConfigMaps 和 Secrets）
- 你必须有 Kubectl 和 Helm
集群要求
- Kubescape 运算符组件至少需要 400Mib RAM 和 400m CPU
具体的安装可以参考下的教程： 需要注册 Kubescape Cloud，并且 只有的工作节点小于 10 个的时候才免费。
https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster
注册登录
这里登录完会弹出一个安装部署，安装安装部署安装即可
添加 Helm 源，并更新
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$helm repo add kubescape https://kubescape.github.io/helm-charts/
"kubescape" has been added to your repositories
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "rancher-stable" chart repository
...Successfully got an update from the "botkube" chart repository
...Successfully got an update from the "awx-operator" chart repository
...Successfully got an update from the "kubescape" chart repository
Update Complete. ⎈Happy Helming!⎈
运行 charts
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$helm upgrade --install kubescape kubescape/kubescape-cloud-operator -n kubescape --create-namespace --set account=97f09924-0c06-42e4-bdad-5b333321af77 --set clusterName=`kubectl config current-context`
Release "kubescape" does not exist. Installing it now.
.........
......................
NAME: kubescape
LAST DEPLOYED: Sat Feb 4 10:03:36 2023
NAMESPACE: kubescape
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing kubescape-cloud-operator version 1.9.5.
In a few minutes your scan results will be available in the following link:
https://cloud.armosec.io/config-scanning/kubernetes-admin-kubernetes
You can see and change the values of your's recurring configurations daily scan in the following link:
https://cloud.armosec.io/settings/assets/clusters/scheduled-scans?cluster=kubernetes-admin-kubernetes
> kubectl -n kubescape get cj kubescape-scheduler -o=jsonpath='{.metadata.name}{"\t"}{.spec.schedule}{"\n"}'
You can see and change the values of your's recurring images daily scan in the following link:
https://cloud.armosec.io/settings/assets/images
> kubectl -n kubescape get cj kubevuln-scheduler -o=jsonpath='{.metadata.name}{"\t"}{.spec.schedule}{"\n"}'
See you!!!
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$
运行完成，安装提示操作，点击刚才注册生成的页面。
验证 Kubescape 在集群中运行状态
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$kubectl -n kubescape get deployments.apps
NAME READY UP-TO-DATE AVAILABLE AGE
gateway 1/1 1 1 19m
kubescape 1/1 1 1 19m
kubevuln 1/1 1 1 19m
operator 1/1 1 1 19m
kubescape 会定期扫描
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$kubectl get cronjobs.batch -n kubescape
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
kubescape-scheduler 21 3 * * * False 0 <none> 20m
kubevuln-scheduler 13 22 * * * False 0 <none> 20m
┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape]
└─$
在登录的主页中可以看到集群和第一次扫描结果
安全合规性扫描信息
可以通过不同的维度来看
合规性的维度查看
集群 Pod/Deploy 的维度查看
具体的 合规配置信息查看
漏洞扫描
关于 kubescape 和小伙伴分享到这里，时间关系，没有深入太多。偶尔听大佬谈到，所以研究一下，有需要的小伙伴快去尝试吧。
博文部分内容参考
文中涉及参考链接内容版权归原作者所有，如有侵权请告知
https://github.com/kubescape/kubescape
https://www.redhat.com/en/blog/state-kubernetes-security-2022-1
https://betterprogramming.pub/image-vulnerability-scanning-for-optimal-kubernetes-security-c3ba933190ef
https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster
© 2018-2023 liruilonger@gmail.com，All rights reserved. 保持署名-非商用-自由转载-相同方式共享(创意共享 3.0 许可证)
