在Kubernetes集群中,Pod状态为“被驱逐(evicted)”表示Pod无法在当前节点上继续运行,已被集群从节点上移除。
问题分析:
节点磁盘空间不足 ,使用df -h查看磁盘使用情况
可以看到根目录 / 已100%满,导致pod被驱逐
查看占用空间最大的目录,可以看到/var占用了20G
[root@k8s-master apply]# du -sh /* 2>/dev/null | sort -hr
20G /var
4.2G /usr
3.9G /root
491M /opt
295M /home
193M /boot
172M /etc
150M /run
124M /kubectl-argo-rollouts-linux-amd64
28K /monitor
16K /lost+found
4.0K /srv
4.0K /mnt
4.0K /media
4.0K /id_rsa.pub
4.0K /afs
0 /tmp
0 /sys
0 /sbin
0 /proc
0 /lib64
0 /lib
0 /dev
0 /bin
清理/var 目录
进入/var逐层分析,检测var下的子目录大小
[root@k8s-master apply]# cd /var
[root@k8s-master var]# du -sh * 2>/dev/null | sort -hr
19G lib
676M log
332M cache
36K spool
20K tmp
12K kerberos
12K db
8.0K empty
4.0K yp
4.0K preserve
4.0K opt
4.0K nis
4.0K local-path-provisioner
4.0K local
4.0K games
4.0K ftp
4.0K crash
4.0K adm
0 run
0 mail
0 lock
清理Docker数据
删除所有未使用的容器、镜像、卷和缓存
[root@k8s-master var]# docker system prune -a -f
[root@k8s-master var]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
07f99d0b452f hub.17usoft.com/gsairgroup/tool-portal:b684c29d "docker-entrypoint.s…" 17 hours ago Up 17 hours 3000/tcp, 8080/tcp nervous_euler
1107187c9e61 apolloconfig/apollo-adminservice "/apollo-adminservic…" 25 hours ago Up 25 hours 0.0.0.0:8090->8090/tcp apollo-adminservice
f75ec993e063 apolloconfig/apollo-portal "/apollo-portal/scri…" 25 hours ago Up 25 hours 0.0.0.0:8070->8070/tcp apollo-portal
a3cc9a54bc6d apolloconfig/apollo-configservice "/apollo-configservi…" 25 hours ago Up 25 hours 0.0.0.0:8080->8080/tcp apollo-configservice
feca1915a4c7 mysql:8.0 "docker-entrypoint.s…" 25 hours ago Up 25 hours 0.0.0.0:3306->3306/tcp, 33060/tcp apollo-mysql
07a829c9ae88 hub.17usoft.com/gsairgroup/timeline-service:master-fe94ee5 "/bin/sh" 34 hours ago Up 34 hours 8080/tcp, 9090/tcp wonderful_lichterman
00483430d7d3 5e785d005ccc "/usr/bin/kube-contr…" 39 hours ago Up 39 hours k8s_calico-kube-controllers_calico-kube-controllers-697d846cf4-wplzw_kube-system_7b3c76be-3dc8-466c-b2b9-21b5f6bf3c50_0
9f883bb71d98 8d147537fb7d "/coredns -conf /etc…" 39 hours ago Up 39 hours k8s_coredns_coredns-7d89d9b6b8-585rc_kube-system_9ced5570-6a86-4397-8f1a-ae080e21c38f_0
fa3084652e29 8d147537fb7d "/coredns -conf /etc…" 39 hours ago Up 39 hours k8s_coredns_coredns-7d89d9b6b8-clvgn_kube-system_8a1a76b2-7ce1-4e8b-8345-1c874b8bc198_0
2863213c1950 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 "/pause" 39 hours ago Up 39 hours k8s_POD_coredns-7d89d9b6b8-clvgn_kube-system_8a1a76b2-7ce1-4e8b-8345-1c874b8bc198_0
0c0faa9863bb registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 "/pause" 39 hours ago Up 39 hours k8s_POD_coredns-7d89d9b6b8-585rc_kube-system_9ced5570-6a86-4397-8f1a-ae080e21c38f_0
19088b072072 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 "/pause" 39 hours ago Up 39 hours k8s_POD_calico-kube-controllers-697d846cf4-wplzw_kube-system_7b3c76be-3dc8-466c-b2b9-21b5f6bf3c50_026c4f07db245 08616d26b8e7 "start_runit" 39 hours ago Up 39 hours k8s_calico-node_calico-node-gq2wj_kube-system_1a026e1d-9424-4884-93d3-b74a8bedb74b_0
3350890701b5 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 "/pause" 39 hours ago Up 39 hours k8s_POD_calico-node-gq2wj_kube-system_1a026e1d-9424-4884-93d3-b74a8bedb74b_0
1ee7c444baf0 77c8bfac1781 "/usr/local/bin/kube…" 40 hours ago Up 40 hours k8s_kube-proxy_kube-proxy-dtqdn_kube-system_94f28854-13c6-4483-8f16-26651d0ad9f7_0
d7cf20062e40 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 "/pause" 40 hours ago Up 40 hours k8s_POD_kube-proxy-dtqdn_kube-system_94f28854-13c6-4483-8f16-26651d0ad9f7_0
ad961ccfa202 fce326961ae2 "etcd --advertise-cl…" 40 hours ago Up 40 hours k8s_etcd_etcd-k8s-master_kube-system_5709e96ef17ce9e30e67cf45d0377549_2
95247db9a58e registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 "/pause" 40 hours ago Up 40 hours k8s_POD_etcd-k8s-master_kube-system_5709e96ef17ce9e30e67cf45d0377549_0
04d1716d923b d4893b67e97f "kube-scheduler --au…" 40 hours ago Up 40 hours k8s_kube-scheduler_kube-scheduler-k8s-master_kube-system_56e5f657c0f5842192c9009d22729ac6_22
414292bdca76 c7ab721dfdae "kube-controller-man…" 40 hours ago Up 40 hours k8s_kube-controller-manager_kube-controller-manager-k8s-master_kube-system_7319ae284ccecb49c0164089eaf03446_22
953c9b59080a 2b5e9c96248f "kube-apiserver --ad…" 40 hours ago Up 40 hours k8s_kube-apiserver_kube-apiserver-k8s-master_kube-system_1156d3f2edec396a386c715bc506a05e_3
92df1a9b0841 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 "/pause" 40 hours ago Up 40 hours k8s_POD_kube-scheduler-k8s-master_kube-system_56e5f657c0f5842192c9009d22729ac6_0
91d3e9d0bc25 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 "/pause" 40 hours ago Up 40 hours k8s_POD_kube-apiserver-k8s-master_kube-system_1156d3f2edec396a386c715bc506a05e_0
9e56d1c18ce0 registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 "/pause" 40 hours ago Up 40 hours k8s_POD_kube-controller-manager-k8s-master_kube-system_7319ae284ccecb49c0164089eaf03446_0
d051c56fa19d 2eaf94c71682 "/usr/bin/docker-ent…" 2 weeks ago Up 40 hours 0.0.0.0:9000->9000/tcp minio
0cfd3f86b76e moby/buildkit:buildx-stable-1 "buildkitd" 3 weeks ago Up 40 hours buildx_buildkit_default
[root@k8s-master var]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hub.17usoft.com/gsairgroup/tool-portal b684c29d c48fc7daaf38 17 hours ago 815MB
mysql 8.0 c6f7ec307380 3 days ago 772MB
moby/buildkit buildx-stable-1 3c2d38015344 3 weeks ago 209MB
quay.io/minio/minio latest 2eaf94c71682 5 weeks ago 182MB
apolloconfig/apollo-portal latest cc502c758c3b 2 months ago 291MB
apolloconfig/apollo-adminservice latest 0fc2047c9a04 2 months ago 307MB
apolloconfig/apollo-configservice latest 185f23e7f2d6 2 months ago 311MB
swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/calico/kube-controllers v3.25.0 5e785d005ccc 2 years ago 71.6MB
swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/calico/node v3.25.0 08616d26b8e7 2 years ago 245MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver v1.22.17 2b5e9c96248f 2 years ago 128MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager v1.22.17 c7ab721dfdae 2 years ago 122MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler v1.22.17 d4893b67e97f 2 years ago 52.7MB
registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy v1.22.17 77c8bfac1781 2 years ago 104MB
registry.cn-hangzhou.aliyuncs.com/google_containers/etcd 3.5.6-0 fce326961ae2 2 years ago 299MB
hub.17usoft.com/gsairgroup/timeline-service master-fe94ee5 db53ed5a2383 2 years ago 48.8MB
registry.cn-hangzhou.aliyuncs.com/google_containers/coredns v1.8.4 8d147537fb7d 3 years ago 47.6MB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause 3.5 ed210e3e4a5b 4 years ago 683kB
可以看到 根目录的使用降下来了
容忍 DiskPressure 污点
在此添加容忍配置
应用新配置可以看到pod正常运行