Kubernetes 部署 nfs-subdir-external-provisioner

news2024/10/5 21:18:26

概述

官方GitHub及参考文档:GitHub - kubernetes-sigs/nfs-subdir-external-provisioner: Dynamic sub-dir volume provisioner on a remote NFS server.

部署nfs-subdir-external-provisioner提供StorageClass服务

步骤

nfs 服务器准备

/etc/exports

# cat /etc/exports
/nfsshare *(rw,no_root_squash,sync)

install nfs-utils

yum -y install nfs-utils
systemctl enable --now nfs

namespace

apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: nfs-server
  name: nfs-server
spec:
  finalizers:
    - kubernetes

class

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-client
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
  archiveOnDelete: "true"
  # 定义路径格式
  pathPattern: "${.PVC.namespace}/${.PVC.annotations.nfs.io/storage-path}"

不同的呈现方式

rbac

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-server
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-client-provisioner-runner
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: run-nfs-client-provisioner
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs-server
roleRef:
  kind: ClusterRole
  name: nfs-client-provisioner-runner
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-server
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: leader-locking-nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-server
subjects:
  - kind: ServiceAccount
    name: nfs-client-provisioner
    # replace with namespace where provisioner is deployed
    namespace: nfs-server
roleRef:
  kind: Role
  name: leader-locking-nfs-client-provisioner
  apiGroup: rbac.authorization.k8s.io

deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-client-provisioner
  labels:
    app: nfs-client-provisioner
  # replace with namespace where provisioner is deployed
  namespace: nfs-server
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-client-provisioner
  template:
    metadata:
      labels:
        app: nfs-client-provisioner
    spec:
      serviceAccountName: nfs-client-provisioner
      containers:
        - name: nfs-client-provisioner
          # image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
          image: registry.cn-beijing.aliyuncs.com/xngczl/nfs-subdir-external-provisione:v4.0.0
          volumeMounts:
            - name: nfs-client-root
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME
              value: k8s-sigs.io/nfs-subdir-external-provisioner
              # 设置高可用允许选举
            - name: ENABLE_LEADER_ELECTION
              value: "True"
            - name: NFS_SERVER
              value: 192.168.164.16
            - name: NFS_PATH
              value: /nfsshare
      volumes:
        - name: nfs-client-root
          nfs:
            server: 192.168.164.16
            path: /nfsshare

验证命令

kubectl logs -f --tail=20 nfs-client-provisioner-5856c5fc68-2r4k2  -n nfs-server

kubectl get sc

测试

pvc

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: "nfs-client"
spec:
  storageClassName: nfs-client
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi

pod

kind: Pod
apiVersion: v1
metadata:
  name: test-pod
spec:
  containers:
  - name: test-pod
    image: busybox:stable
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && exit 0 || exit 1"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: test-claim

Error

No such file or directory

Warning FailedMount 15s (x8 over 79s) kubelet MountVolume.SetUp failed for volume "nfs-client-r oot" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs 192.168.164.16:/nfsshare /etc/cni/net.d/pods/67154eef-96d4-46cc-af01-653496a4f1a7/vo lumes/kubernetes.io~nfs/nfs-client-root
Output: mount.nfs: mounting 192.168.164.16:/nfsshare failed, reason given by server: No such file or directory

报错原因: 配置文件编写错误：

修改正确的配置文件/etc/exports后，需要重启nfs服务生效。

mount挂载文件，有坏超级块

Mounting arguments: --description=Kubernetes transient mount for /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root --scope -- mount -t nfs 172.16.41.7:/data/nfs_storage /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root

Output: Running scope as unit: run-rdcc7cfa6560845969628fc551606e69d.scope

mount: /data/kubernetes/kubelet/pods/2ca70aa9-433c-4d10-8f87-154ec9569504/volumes/kubernetes.io~nfs/nfs-client-root: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

Warning FailedMount 10s kubelet, node1.ayunw.cn MountVolume.SetUp failed for volume "nfs-client-root" : mount failed: exit status 32

Mounting command: systemd-run

报错原因: 各个Kubernetes节点需要安装nfs-utils

yum -y install nfs-utils 执行安装即可

waiting for a volume to be created

Normal ExternalProvisioning 8s (x17 over 3m42s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "k8s-sigs.io/nfs-subdir-external-provisioner" or manually created by system administrator