11--kubernetes的Ingress应用

前言：本章主要记录ingress暴露服务方式，会详细解释其原理及两种网络模式应用实操。

1、Ingress概念详解

Kubernetes 暴露服务的方式目前只有三种：LoadBlancer Service、NodePort Service、Ingress，Service属于4层负载均衡，而本章的ingress属于7层负载均衡。

要理解 Ingress，需要区分两个概念：Ingress（类似老板角色）和 Ingress-Controller（类似职工角色）。

Ingress 对象： 这是 Kubernetes 中的一个 API 对象/资源对象，用 YAML文件配置。它定义了请求如何转发到 Service 的规则，可以理解为配置模板。

Ingress-Controller： 这是具体实现反向代理和负载均衡的程序。它解析 Ingress 定义的规则，并根据配置转发请求。

简而言之，Ingress-Controller 负责实际的请求转发，并通过多种方式暴露在集群入口。外部请求首先到达 Ingress-Controller，而 Ingress 对象则告知 Ingress-Controller 如何转发请求，如哪些域名和路径转发到哪些服务。

Service 在第四层传输层（TCP/IP 层）以 IP:PORT 形式存在，对于不同的 URL 地址，可能需要不同的后端服务或虚拟服务器。这种应用层的转发机制通过 Kubernetes 的 Service 机制无法实现，此时可以使用 Ingress 策略和具体的 Ingress-Controller。Ingress 提供七层负载均衡能力（当然也无法脱离4层存在），如外部可访问的 URL、负载均衡、SSL、基于名称的虚拟主机等。在集群流量接入层，Ingress 的高可用性至关重要。

Kubernetes 并没有自带 Ingress Controller，实际上ingress-controller只是一个统称，具体实现有多种，需要自己单独安装，目前，由k8s维护的ingress-controller只有google云的GCE与ingress-nginx两个，常用的是 Ingress-nginx Controller（基于nginx七层反向代理来实现）。

Ingress 一般由三个组件组成：

Nginx 反向代理负载均衡器（干活用的工具）：负责实际的请求转发和负载均衡。
Ingress Controller（员工）：作为控制器，与 Kubernetes API 交互，实时获取后端 Service 和 Pod 的变化（如新增或删除）。它结合 Ingress 定义的规则生成配置，动态更新 Nginx 负载均衡器，并使配置生效，实现服务自动发现。
Ingress（老板）：定义请求转发规则。通过 YAML 文件配置，指定某个域名的请求如何转发到集群中的指定 Service。可以为一个或多个 Service 定义一个或多个 Ingress 规则。

2、Ingress部署实战

2.1、部署 Ingress-Controller（Nginx）

下载Ingress控制器：

网盘链接: https://pan.baidu.com/s/1Y3eB-4OXeoOQp1AVoaXdYw?pwd=tfdm 提取码: tfdm

github地址：https://github.com/kubernetes/ingress-nginx/archive/nginx-0.30.0.tar.gz

[root@k8s-master1 ~]# wget https://github.com/kubernetes/ingress-nginx/archive/nginx-0.30.0.tar.gz
[root@k8s-master1 ~]# tar -zxvf ingress-nginx-nginx-0.30.0.tar.gz -C /mnt/
[root@k8s-master1 ~]# cd /mnt/ingress-nginx-nginx-0.30.0/deploy/static/
[root@k8s-master1 static]# ls
configmap.yaml  mandatory.yaml  namespace.yaml  provider  rbac.yaml  with-rbac.yaml
[root@k8s-master1 static]# mkdir /root/ingress
[root@k8s-master1 static]# cp mandatory.yaml /root/ingress
[root@k8s-master1 ~]# cd /root/ingress
[root@k8s-master1 ~]# cd /root/ingress
[root@k8s-master1 ingress]# vim mandatory.yaml 
修改详情如下（修改后的文件网盘连接内有）

1、修改pod控制器为daemonset
apiVersion: apps/v1
kind: DaemonSet    #修改
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  #replicas: 1    #注释掉
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx

2、宿主机网络共享，注释掉node标签选择
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      hostNetwork: true #添加宿主机网络共享
      serviceAccountName: nginx-ingress-serviceaccount
      #nodeSelector:
        #kubernetes.io/os: linux    #这两行注释掉

这里不注释的话需要使用
kubectl label nodes k8s-node1 kubernetes.io/os: linux=true为node节点添加标签

3、修改镜像为阿里镜像（直接拉取也可以，但网络不稳定，成功拉取后记得上传到本地仓库）
#image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
image: registry.cn-chengdu.aliyuncs.com/k8s_module_images/kubernetes-ingress-controller-nginx-ingress-controller:0.30.0

[root@k8s-master1 ingress]# kubectl apply -f mandatory.yaml
此处一般会有警告提示修改api版本，忽略的话不影响后续使用
[root@k8s-master1 ingress]# kubectl get pod -n ingress-nginx
NAME                             READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-f6brt   1/1     Running   0          2m30s
nginx-ingress-controller-jbst2   1/1     Running   0          2m30s

此时Ingress-Controller部署完毕

2.2、部署web应用

这里选用http和nginx两个作为两种不同的web服务器，这里使用hostnetwork网络方式，所以部署的svc是默认的clusterip

[root@k8s-master1 ingress]# vim my-apache.yaml 
[root@k8s-master1 ingress]# cat my-apache.yaml 
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-apache
spec:
 selector:
   matchLabels:
     run: my-apache
 replicas: 2
 template:
  metadata:
   labels:
    run: my-apache
  spec:
   containers:
   - name: my-apache
     image: registry.cn-chengdu.aliyuncs.com/liumuquan_app/httpd:2.4-alpine3.17
     ports:
     - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
 name: my-apache
 labels:
  run: my-apache
spec:
 #type: NodePort
 ports:
 - port: 80
   targetPort: 80
   #nodePort: 30001
 selector:
  run: my-apache

[root@k8s-master1 ingress]# vim my-nginx.yaml
[root@k8s-master1 ingress]# cat my-nginx.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-nginx
spec:
 selector:
   matchLabels:
     run: my-nginx
 replicas: 2
 template:
  metadata:
   labels:
    run: my-nginx
  spec:
   containers:
   - name: my-nginx
     image: registry.cn-chengdu.aliyuncs.com/liumuquan_app/nginx:1.12.0-alpine
     ports:
     - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
 name: my-nginx
 labels:
  run: my-nginx
spec:
 #type: NodePort
 ports:
 - port: 80
   targetPort: 80
   #nodePort: 30002
 selector:
  run: my-nginx

创建两个web服务

[root@k8s-master1 ingress]# kubectl apply -f my-apache.yaml 
deployment.apps/my-apache created
service/my-apache created
[root@k8s-master1 ingress]# kubectl apply -f my-nginx.yaml 
deployment.apps/my-nginx created
service/my-nginx created
[root@k8s-master1 ingress]# kubectl get pod -o wide
NAME                         READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
my-apache-764b8f49c5-g6hl6   1/1     Running   0          23s   10.244.1.60   k8s-node1   <none>           <none>
my-apache-764b8f49c5-t5ndt   1/1     Running   0          23s   10.244.2.45   k8s-node2   <none>           <none>
my-nginx-5c49645686-rm6h8    1/1     Running   0          19s   10.244.1.61   k8s-node1   <none>           <none>
my-nginx-5c49645686-z7g82    1/1     Running   0          19s   10.244.2.46   k8s-node2   <none>           <none>
[root@k8s-master1 ingress]# kubectl get endpoints
NAME                                             ENDPOINTS                       AGE
cluster.local-nfs-redis-nfs-client-provisioner   <none>                          2d23h
kubernetes                                       192.168.188.101:6443            23d
my-apache                                        10.244.1.60:80,10.244.2.45:80   34s
my-nginx                                         10.244.1.61:80,10.244.2.46:80   30s
[root@k8s-master1 ingress]# kubectl get svc
NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP   23d
my-apache    ClusterIP   10.96.232.117   <none>        80/TCP    39s
my-nginx     ClusterIP   10.99.230.106   <none>        80/TCP    35s

web服务创建完成

2.3、配置ingress规则

1）宿主机网络模式 hostNetwork: true

[root@k8s-master1 ingress]# vim ingress-test.yaml
[root@k8s-master1 ingress]# cat ingress-test.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: test-ingress
 namespace: default
 annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
 rules:  #定义转发规则
 - host: www.apache111.com  #指定域名方式
   http:
    paths:
    - path: /  #指定访问的路径
      pathType: Prefix  #定义路径的类型
      backend:   #定义转发后端的服务
       service:  #定义转发的service
         name: my-apache
         port:
          number: 80 #由于Ingress控制器开启了hostNetwork: true。且clusterip的配置这里必须设置80
 - host: www.nginx222.com
   http:
    paths:
    - path: /
      pathType: Prefix
      backend:
       service:
         name: my-nginx
         port:
          number: 80

[root@k8s-master1 ingress]# kubectl apply -f ingress-test.yaml 
ingress.networking.k8s.io/test-ingress created
[root@k8s-master1 ingress]# kubectl get ingress
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME           CLASS    HOSTS                                ADDRESS   PORTS   AGE
test-ingress   <none>   www.apache111.com,www.nginx222.com             80      8s

配置物理机解析

windows电脑：

C:\Windows\System32\drivers\etc\hosts内添加

192.168.188.102 k8s-node1 www.apache111.com www.nginx222.com
192.168.188.103 k8s-node2 www.apache111.com www.nginx222.com

使用物理机浏览器访问

2）修改网络模式为nodeport

[root@k8s-master1 ingress]# kubectl delete -f ingress-test.yaml 
[root@k8s-master1 ingress]# kubectl delete -f my-apache.yaml 
[root@k8s-master1 ingress]# kubectl delete -f my-nginx.yaml 
[root@k8s-master1 ingress]# kubectl delete -f mandatory.yaml

ingress-controller配置如下

[root@k8s-master1 ingress]# vim ingress-controller.yaml
[root@k8s-master1 ingress]# cat ingress-controller.yaml
---
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
#kind: DaemonSet
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
#      hostNetwork: true   # 关闭此网络模式
      serviceAccountName: nginx-ingress-serviceaccount
#      nodeSelector:
#        custom/ingress-controller-ready: "true"
#        kubernetes.io/os: linux
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-chengdu.aliyuncs.com/k8s_module_images/kubernetes-ingress-controller-nginx-ingress-controller:0.30.0 
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

---

apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - min:
      memory: 90Mi
      cpu: 100m
    type: Container

创建ingress-controller

[root@k8s-master1 ingress]# kubectl apply -f ingress-controller.yaml
[root@k8s-master1 ingress]# kubectl get pod -n ingress-nginx
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-7b5d6f74c7-sxm7l   1/1     Running   0          85s
nginx-ingress-controller-7b5d6f74c7-tqpgl   1/1     Running   0          85s

修改web-pod文件svc为nodeport方式

[root@k8s-master1 ingress]# vim my-apache.yaml 
[root@k8s-master1 ingress]# cat my-apache.yaml 
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-apache
spec:
 selector:
   matchLabels:
     run: my-apache
 replicas: 2
 template:
  metadata:
   labels:
    run: my-apache
  spec:
   containers:
   - name: my-apache
     image: registry.cn-chengdu.aliyuncs.com/liumuquan_app/httpd:2.4-alpine3.17
     ports:
     - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
 name: my-apache
 labels:
  run: my-apache
spec:
 type: NodePort
 ports:
 - port: 80
   targetPort: 80
   nodePort: 30001
 selector:
  run: my-apache

[root@k8s-master1 ingress]# vim my-nginx.yaml 
[root@k8s-master1 ingress]# cat my-nginx.yaml 
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: my-nginx
spec:
 selector:
   matchLabels:
     run: my-nginx
 replicas: 2
 template:
  metadata:
   labels:
    run: my-nginx
  spec:
   containers:
   - name: my-nginx
     image: registry.cn-chengdu.aliyuncs.com/liumuquan_app/nginx:1.12.0-alpine
     ports:
     - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
 name: my-nginx
 labels:
  run: my-nginx
spec:
 type: NodePort
 ports:
 - port: 80
   targetPort: 80
   nodePort: 30002
 selector:
  run: my-nginx

创建webpod和svc

[root@k8s-master1 ingress]# kubectl apply -f my-apache.yaml 
deployment.apps/my-apache created
service/my-apache created
[root@k8s-master1 ingress]# kubectl apply -f my-nginx.yaml 
deployment.apps/my-nginx created
service/my-nginx created
[root@k8s-master1 ingress]# kubectl get pod
NAME                         READY   STATUS    RESTARTS   AGE
my-apache-764b8f49c5-5bgdb   1/1     Running   0          19s
my-apache-764b8f49c5-ck4vl   1/1     Running   0          19s
my-nginx-5c49645686-7csv8    1/1     Running   0          15s
my-nginx-5c49645686-ptxhv    1/1     Running   0          15s
[root@k8s-master1 ingress]# kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        23d
my-apache    NodePort    10.105.128.125   <none>        80:30001/TCP   22s
my-nginx     NodePort    10.99.184.224    <none>        80:30002/TCP   18s

ingress无需修改

[root@k8s-master1 ingress]# kubectl apply -f ingress-test.yaml 
ingress.networking.k8s.io/test-ingress created

此时物理机访问需要添加nodeport暴露的端口号才能正常访问