概述
Pod的生命周期包含从Pod创建事件的触发到Pod被停止的整个流程。了解Pod的生命周期方便日常排障,并能帮助较深入了解K8s。
在Pod生命周期中有两个重要的标识:Pod Condition 和 Pod Phase。Pod Phase提供了一个Pod当前状况的概览,可以帮助我们快速定位Pod当前在生命周期的哪个阶段,但是Pod Phase无法提供关于Pod状态更详细的信息。在Debug过程中,我们需要更多的准确反应Pod状态的指标,此时Pod Condition可以提供帮助。Pod Condition反应Pod是否已经到达了一个特定的阶段,并且为什么。不同于Pod Phase,Pod Condition有多个同时存在的标识,这些标识反映了Pod不同纬度的状态。
Pod Phase
Pod Phase 包含 Pending, Running, Succeeded, Failed, Unknow 5个阶段,我们重点关注从启动到运行的两个阶段: Pending & Running。
Pending指一个Pod已经被K8s确认接受,容器镜像被拉取到节点机成功,到所有Pod内容器成功启动之前的阶段。每个新启动的Pod一定会经历这个阶段,Pod处于Pending阶段是很正常的现象。但是一个或多个Pod长时间处于Pending状态,无法转换到Running状态表明系统是有问题的。几个可能导致Pod长时间处于Pending状态的原因:
1)Pod scheduler 工作异常。
2)集群内可分配资源不足。
3)镜像拉取异常。
上述三个原因可能有多个子原因,产生问题时需俱体分析。
Running指一个Pod中所有的Container都已启动,在此之后会一直维持Running。Running后Pod会被kubelet开始进行就绪探测(readiness probe)。
Pod Conditions
Pod Conditions 包含PodScheduled, Ready,Initialized,ContainersReady,DisruptionTarget,我们重点关注PodScheduled, Ready, ContainersReady这三个状态。与Pod Phase不同的是,多个Pod Conditions会同时存在,通过True & False 标识该Pod是否满足了某个Condition。
PodScheduled表示该Pod已经被成功调度,被分配了可用节点。PodScheduled为False的原因和调度相关,包含:
1)Pod scheduler 工作异常
2)集群内可分配资源不足
ContainersReady表示容器中的所有containers都已经通过了readiness probe的探测,准备好开始服务。ContainersReady为False的原因主要包含:
1)Container中1号进程启动失败
2)Container的Readiness Probe探测失败
Ready表示除ContainersReady外,Pod对于存储,网络的需求已经满足,Init Container的工作也执行完毕。当Pod处于Ready的状态时,表明该Pod已经完全准备好接收请求提供服务了。此时Pod的IP会被加入到对应Endpoint资源的Pod对象列表中,endpoint的信息会被同步到kube-proxy,此时Pod就真正要接收请求了。
值得注意的是,ContainersReady和Ready这两个状态通常会同时达到。Ready为False的原因多出了:
1)存储,网络需求未满足
2)init container未完成工作
实验
Pending + PodScheduled:False
由于没有合适的宿主节点导致Pod无法被调度
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-deployment
labels:
app: redis
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:latest
ports:
- containerPort: 6379
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
nodeSelector:
disktype: ssd
Pending + PodScheduled:True&ContainersReady:False
由于镜像服务不可用导致Pod虽然被调度但是无法下载镜像
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-deployment
labels:
app: redis
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: myprivateregistry.com/redis:latest
ports:
- containerPort: 6379
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
Running + PodScheduled:True&ContainersReady:False
Pod内容器启动,由于错误配置就绪探针导致ContainersReady一直错误。注意此处我们故意配置了错误的就绪探针,在真实场景中
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-deployment
labels:
app: redis
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:latest
ports:
- containerPort: 6379
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 150
periodSeconds: 10
Running + PodScheduled:True&ContainersReady:True&Ready:True
以下是一个能够正常启动并就绪的Pod yaml描述文件
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-deployment
labels:
app: redis
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:latest
ports:
- containerPort: 6379
readinessProbe:
tcpSocket:
port: 6379
initialDelaySeconds: 10
periodSeconds: 10
总结
在 Kubernetes 中,理解 Pod Phase 和 Pod Conditions 对于有效管理和监控 Pod 的生命周期至关重要。Pod Phase 描述了 Pod 的主要生命周期阶段,而 Pod Conditions 提供了更具体的健康状况信息,帮助我们诊断和解决问题。这些状态信息不仅对于日常的故障排查非常有用,例如帮助我们快速定位为什么大量 Pod 处于 Pending 状态,而且对于开发相关的 Kubernetes 操作也非常关键,如自定义的负载均衡器需要根据 Pod 的就绪状态来更新其配置。此外,理解包括那些较不常见的如 Evicted 在内的状态,可以让我们更全面地掌握和应对各种集群状况。这种知识的深入不仅能提升我们对系统的掌控,还能优化我们的应用性能和可靠性。
Appendix
Pod Phases 在K8s代码中的定义
const (
// PodPending means the pod has been accepted by the system, but one or more of the containers
// has not been started. This includes time before being bound to a node, as well as time spent
// pulling images onto the host.
PodPending PodPhase = "Pending"
// PodRunning means the pod has been bound to a node and all of the containers have been started.
// At least one container is still running or is in the process of being restarted.
PodRunning PodPhase = "Running"
// PodSucceeded means that all containers in the pod have voluntarily terminated
// with a container exit code of 0, and the system is not going to restart any of these containers.
PodSucceeded PodPhase = "Succeeded"
// PodFailed means that all containers in the pod have terminated, and at least one container has
// terminated in a failure (exited with a non-zero exit code or was stopped by the system).
PodFailed PodPhase = "Failed"
// PodUnknown means that for some reason the state of the pod could not be obtained, typically due
// to an error in communicating with the host of the pod.
// Deprecated in v1.21: It isn't being set since 2015 (74da3b14b0c0f658b3bb8d2def5094686d0e9095)
PodUnknown PodPhase = "Unknown"
)
Pod Conditions 在K8s代码中的定义
// These are valid conditions of pod.
const (
// PodScheduled represents status of the scheduling process for this pod.
PodScheduled PodConditionType = "PodScheduled"
// PodReady means the pod is able to service requests and should be added to the
// load balancing pools of all matching services.
PodReady PodConditionType = "Ready"
// PodInitialized means that all init containers in the pod have started successfully.
PodInitialized PodConditionType = "Initialized"
// ContainersReady indicates whether all containers in the pod are ready.
ContainersReady PodConditionType = "ContainersReady"
// DisruptionTarget indicates the pod is about to be terminated due to a
// disruption (such as preemption, eviction API or garbage-collection).
DisruptionTarget PodConditionType = "DisruptionTarget"
)