k8s通过nginx-ingress实现集群外网访问功能
一：ingress概述
1.1 ingress 工作原理
step1：ingress contronler通过与k8s的api进行交互，动态的去感知k8s集群中ingress服务规则的变化，然后读取它，并按照定义的ingress规则，转发到k8s集群中对应的service。
step2：而这个ingress规则写明了哪个域名对应k8s集群中的哪个service，然后再根据ingress-controller中的nginx配置模板，生成一段对应的nginx配置。
step3：然后再把该配置动态的写到ingress-controller的pod里，该ingress-controller的pod里面运行着一个nginx服务，控制器会把生成的nginx配置写入到nginx的配置文件中，然后reload一下，使其配置生效,以此来达到域名分配置及动态更新的效果。
1.2 ingress可以解决的问题
1）动态配置服务
如果按照传统方式, 当新增加一个服务时, 我们可能需要在流量入口加一个反向代理指向我们新的k8s服务. 而如果用了Ingress, 只需要配置好这个服务, 当服务启动时, 会自动注册到Ingress的中, 不需要而外的操作。
2）减少不必要的端口暴露
配置过k8s的都清楚, 第一步是要关闭防火墙的, 主要原因是k8s的很多服务会以NodePort方式映射出去, 这样就相当于给宿主机打了很多孔, 既不安全也不优雅. 而Ingress可以避免这个问题, 除了Ingress自身服务可能需要映射出去, 其他服务都不要用NodePort方式。
二、部署nginx-ingress
2.0 2.2 相关部署组件说明
| PodName | namespace | ServiceName | ports | 部署方式 | 作用 |
|---|---|---|---|---|---|
| ingress-nginx-controller | test | ingress-nginx-controller | NodePort :80(http)、443（https） | DaemonSet | 实现基于灵活的 ingress 策略定义的服务路由功能 |
| ingress-nginx-admission-create | test | ingress-nginx-controller-admission | LoadBalancer: 443 | Job | 是用来创建证书的，需要指定证书的名称、域名、ns等信息 |
| ingress-nginx-admission-patch | test | ingress-nginx-controller-admission | LoadBalancer: 443 | Job | 将前面创建的证书中的ca提取出来，写入到指定的admission webhook配置中 |
| ingress实例 | test | ingress | nginx转发具体配置文件 |
2.1 编写nginx-ingreess相关资源文件
[root@master1 ingress]# cat > nginx-ingress.yaml << EOF
apiVersion: v1
kind: Namespace
metadata:
name: test
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx
namespace: test
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-admission
namespace: test
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx
namespace: test
rules:
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- endpoints
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resourceNames:
- ingress-controller-leader
resources:
- configmaps
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- coordination.k8s.io
resourceNames:
- ingress-controller-leader
resources:
- leases
verbs:
- get
- update
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- create
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-admission
namespace: test
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- create
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
- namespaces
verbs:
- list
- watch
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- networking.k8s.io
resources:
- ingresses/status
verbs:
- update
- apiGroups:
- networking.k8s.io
resources:
- ingressclasses
verbs:
- get
- list
- watch
- apiGroups:
- discovery.k8s.io
resources:
- endpointslices
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-admission
rules:
- apiGroups:
- admissionregistration.k8s.io
resources:
- validatingwebhookconfigurations
verbs:
- get
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx
namespace: test
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: test
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-admission
namespace: test
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: test
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx
subjects:
- kind: ServiceAccount
name: ingress-nginx
namespace: test
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-admission
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ingress-nginx-admission
subjects:
- kind: ServiceAccount
name: ingress-nginx-admission
namespace: test
---
apiVersion: v1
data:
allow-snippet-annotations: "true"
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-controller
namespace: test
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-controller
namespace: test
spec:
externalTrafficPolicy: Local
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- appProtocol: http
name: http
port: 80
protocol: TCP
targetPort: http
- appProtocol: https
name: https
port: 443
protocol: TCP
targetPort: https
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: NodePort
---
apiVersion: v1
kind: Service
metadata:
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-controller-admission
namespace: test
spec:
ports:
- appProtocol: https
name: https-webhook
port: 443
targetPort: webhook
selector:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
type: LoadBalancer
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-controller
namespace: test
spec:
minReadySeconds: 0
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
spec:
hostNetwork: true
containers:
- args:
- /nginx-ingress-controller
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-controller-leader
- --controller-class=k8s.io/ingress-nginx
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: LD_PRELOAD
value: /usr/local/lib/libmimalloc.so
image: anjia0532/google-containers.ingress-nginx.controller:v1.4.0
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
livenessProbe:
failureThreshold: 5
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: controller
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
- containerPort: 8443
name: webhook
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
cpu: 100m
memory: 90Mi
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
runAsUser: 101
volumeMounts:
- mountPath: /usr/local/certificates/
name: webhook-cert
readOnly: true
dnsPolicy: ClusterFirst
nodeSelector:
kubernetes.io/os: linux
serviceAccountName: ingress-nginx
terminationGracePeriodSeconds: 300
volumes:
- name: webhook-cert
secret:
secretName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-admission-create
namespace: test
spec:
template:
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-admission-create
spec:
containers:
- args:
- create
- --host=ingress-nginx-controller-admission,ingress-nginx-controller-admission.$(POD_NAMESPACE).svc
- --namespace=$(POD_NAMESPACE)
- --secret-name=ingress-nginx-admission
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: anjia0532/google-containers.ingress-nginx.kube-webhook-certgen:v20220916-gd32f8c343
imagePullPolicy: IfNotPresent
name: create
securityContext:
allowPrivilegeEscalation: false
nodeSelector:
kubernetes.io/os: linux
restartPolicy: OnFailure
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccountName: ingress-nginx-admission
---
apiVersion: batch/v1
kind: Job
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-admission-patch
namespace: test
spec:
template:
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-admission-patch
spec:
containers:
- args:
- patch
- --webhook-name=ingress-nginx-admission
- --namespace=$(POD_NAMESPACE)
- --patch-mutating=false
- --secret-name=ingress-nginx-admission
- --patch-failure-policy=Fail
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
image: anjia0532/google-containers.ingress-nginx.kube-webhook-certgen:v20220916-gd32f8c343
imagePullPolicy: IfNotPresent
name: patch
securityContext:
allowPrivilegeEscalation: false
nodeSelector:
kubernetes.io/os: linux
restartPolicy: OnFailure
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccountName: ingress-nginx-admission
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: nginx
spec:
controller: k8s.io/ingress-nginx
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
app.kubernetes.io/component: admission-webhook
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.4.0
name: ingress-nginx-admission
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
service:
name: ingress-nginx-controller-admission
namespace: test
path: /networking/v1/ingresses
failurePolicy: Fail
matchPolicy: Equivalent
name: validate.nginx.ingress.kubernetes.io
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
sideEffects: None
EOF
应用生效
[root@master1 ingress]# kubectl apply -f nginx-ingress.yaml
serviceaccount/ingress-nginx created
serviceaccount/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
configmap/ingress-nginx-controller created
service/ingress-nginx-controller created
service/ingress-nginx-controller-admission created
daemonset.apps/ingress-nginx-controller created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
验证
[root@master1 ingress]# kubectl get pods -n test
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-z2xgq 0/1 Completed 0 3m8s
ingress-nginx-admission-patch-qpnh7 0/1 Completed 2 3m8s
ingress-nginx-controller-kc7km 1/1 Running 0 3m8s
ingress-nginx-controller-knjm6 0/1 CrashLoopBackOff 3 (19s ago) 3m8s
ingress-nginx-controller-mzqjn 1/1 Running 0 3m8s
ingress-nginx-controller-xcxsd 1/1 Running 0 3m8s
nfs-client-provisioner-fb55999fb-pcrqt 1/1 Running 0 4h11m
web-0 1/1 Running 0 4h5m
web-1 1/1 Running 0 4h5m
[root@master1 ingress]# kubectl logs -n test ingress-nginx-controller-knjm6
-------------------------------------------------------------------------------
NGINX Ingress controller
Release: v1.4.0
Build: 50be2bf95fd1ef480420e2aa1d6c5c7c138c95ea
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.19.10
-------------------------------------------------------------------------------
F0524 06:17:54.168788 6 main.go:67] port 80 is already in use. Please check the flag --http-port
报错解决：
由POD报错日志可知，80端口被占用了，解决后重启pod即可。
[root@master1 ingress]# docker ps|grep rancher
56e840839dc1 rancher/rancher:v2.7.0-rc12 "entrypoint.sh" 7 days ago Up 7 days 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp rancher
[root@master1 ingress]# docker stop rancher
rancher
[root@master1 ingress]# kubectl get pods -n test
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-z2xgq 0/1 Completed 0 8m46s
ingress-nginx-admission-patch-qpnh7 0/1 Completed 2 8m46s
ingress-nginx-controller-kc7km 1/1 Running 0 8m46s
ingress-nginx-controller-knjm6 0/1 CrashLoopBackOff 6 (53s ago) 8m46s
ingress-nginx-controller-mzqjn 1/1 Running 0 8m46s
ingress-nginx-controller-xcxsd 1/1 Running 0 8m46s
nfs-client-provisioner-fb55999fb-pcrqt 1/1 Running 0 4h17m
web-0 1/1 Running 0 4h11m
web-1 1/1 Running 0 4h11m
[root@master1 ingress]# kubectl delete pods -n test ingress-nginx-controller-knjm6
pod "ingress-nginx-controller-knjm6" deleted
[root@master1 ingress]# kubectl get pods -n test
NAME READY STATUS RESTARTS AGE
ingress-nginx-admission-create-z2xgq 0/1 Completed 0 9m13s
ingress-nginx-admission-patch-qpnh7 0/1 Completed 2 9m13s
ingress-nginx-controller-kc7km 1/1 Running 0 9m13s
ingress-nginx-controller-mzqjn 1/1 Running 0 9m13s
ingress-nginx-controller-r7knt 1/1 Running 0 12s
ingress-nginx-controller-xcxsd 1/1 Running 0 9m13s
nfs-client-provisioner-fb55999fb-pcrqt 1/1 Running 0 4h17m
web-0 1/1 Running 0 4h11m
web-1 1/1 Running 0 4h11m
创建一个ingree，测试外网访问
查看当前svc
[root@master1 ingress]# kubectl get svc -n test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.96.117.202 <none> 80:32210/TCP,443:32008/TCP 11m
ingress-nginx-controller-admission LoadBalancer 10.96.131.36 <pending> 443:32639/TCP 11m
nginx ClusterIP None <none> 80/TCP 4h14m
You have new mail in /var/spool/mail/root
[root@master1 ingress]# kubectl get svc -n test nginx -oyaml
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"nginx"},"name":"nginx","namespace":"test"},"spec":{"clusterIP":"None","ports":[{"name":"web","port":80}],"selector":{"app":"nginx"}}}
creationTimestamp: "2023-05-24T02:11:37Z"
labels:
app: nginx
name: nginx
namespace: test
resourceVersion: "2499378"
uid: a4584c4d-51ea-4bf0-b711-880090ad1dae
spec:
clusterIP: None
clusterIPs:
- None
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: web
port: 80
protocol: TCP
targetPort: 80
selector:
app: nginx
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
创建ingress
cat > web-ing.yaml << EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: web-ing
namespace: test
spec:
rules:
- host: web.example.com
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: Prefix
EOF
应用
[root@master1 ingress]# kubectl apply -f web-ing.yaml
ingress.networking.k8s.io/web-ing created
验证
# 确认部署
- Ingress-nginx-controller:
为了高可用，部署方式采用DaemonSet，所以集群内可调度的节点都会部署一个（master节点不可调度），并确认处于Runnning状态。
- Ingress-nginx-controller服务
使用NodePort方式，确认PORTS后面出现30000以上的本地端口
[root@master1 nfs-provisioner]# kubectl get pod,svc -n test
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-admission-create-z2xgq 0/1 Completed 0 67m
pod/ingress-nginx-admission-patch-qpnh7 0/1 Completed 2 67m
pod/ingress-nginx-controller-kc7km 1/1 Running 0 67m
pod/ingress-nginx-controller-mzqjn 1/1 Running 0 67m
pod/ingress-nginx-controller-r7knt 1/1 Running 0 58m
pod/ingress-nginx-controller-xcxsd 1/1 Running 0 67m
pod/nfs-client-provisioner-fb55999fb-pcrqt 1/1 Running 0 5h15m
pod/web-0 1/1 Running 0 5h9m
pod/web-1 1/1 Running 0 5h9m
pod/web-7849c945f4-k9xzz 1/1 Running 0 21m
pod/web-7849c945f4-x246j 1/1 Running 0 21m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller NodePort 10.96.117.202 <none> 80:32210/TCP,443:32008/TCP 67m
service/ingress-nginx-controller-admission LoadBalancer 10.96.131.36 <pending> 443:32639/TCP 67m
service/nginx ClusterIP 10.96.6.151 <none> 80/TCP 21m
[root@master1 nfs-provisioner]# telnet 10.140.20.142 32210
Trying 10.140.20.142...
Connected to 10.140.20.142.
Escape character is '^]'.
^CConnection closed by foreign host.
访问验证
[root@master1 nfs-provisioner]# kubectl exec -n test web-7849c945f4-k9xzz -it bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@web-7849c945f4-k9xzz:/# ls /usr/share/nginx/html/
root@web-7849c945f4-k9xzz:/#
root@web-7849c945f4-k9xzz:/# ls /usr/share/nginx/html/
root@web-7849c945f4-k9xzz:/# echo 1 > /usr/share/nginx/html/index.html
root@web-7849c945f4-k9xzz:/# curl http://localhost/
1
root@web-7849c945f4-k9xzz:/# exit
[root@master1 nfs-provisioner]# kubectl get svc -n test
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller NodePort 10.96.117.202 <none> 80:32210/TCP,443:32008/TCP 51m
ingress-nginx-controller-admission LoadBalancer 10.96.131.36 <pending> 443:32639/TCP 51m
nginx ClusterIP 10.96.6.151 <none> 80/TCP 5m40s
[root@master1 nfs-provisioner]# kubectl get ing -n test
NAME CLASS HOSTS ADDRESS PORTS AGE
web-ing nginx web.example.com 10.96.117.202 80 26m
#登录k8s集群外的一台服务器做完hosts解析后进行验证
root@k8s-master1:~# tail -n 1 /etc/hosts
10.140.20.141 web.example.com
root@k8s-master1:~# curl http://web.example.com
1
