云原生之容器编排实践-在K8S集群中使用Registry2搭建私有镜像仓库

背景

基于前面搭建的3节点 Kubernetes 集群，今天我们使用 Registry2 搭建私有镜像仓库，这在镜像安全性以及离线环境下运维等方面具有重要意义。

Note: 由于是测试环境，以下创建了一个 local-storage 的 StorageClass ，并使用本地磁盘的方式创建使用 PV ，实际建议使用 NFS 。

虚机资源

共用到了三台虚机，1台作为 Master 节点，2台 Worker 节点。

主机名	IP	说明
k8s-master	172.16.201.25	主节点
k8s-node1	172.16.201.26	工作节点
k8s-node2	172.16.201.27	工作节点

[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES                  AGE   VERSION
k8s-master   Ready    control-plane,master   37h   v1.20.9
k8s-node1    Ready    <none>                 35h   v1.20.9
k8s-node2    Ready    <none>                 35h   v1.20.9

系统环境

[root@k8s-master ~]# uname -a
Linux k8s-master 3.10.0-1160.71.1.el7.x86_64 #1 SMP Tue Jun 28 15:37:28 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@k8s-master ~]# cat /proc/version 
Linux version 3.10.0-1160.71.1.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Tue Jun 28 15:37:28 UTC 2022
[root@k8s-master ~]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)

使用Registry2搭建私有镜像仓库

采用 YAML 文件的方式，搭建私有 Docker Registry 的整体流程：

创建一个Docker Registry的命名空间
创建一个持久卷来存储Docker Registry的数据
创建一个Secret用于存储Docker Registry的凭证
创建一个Deployment来部署Docker Registry
创建一个Service来暴露Docker Registry
登录并推送镜像到私有镜像仓库

创建一个Docker Registry的命名空间

apiVersion: v1
kind: Namespace
metadata:
  name: docker-registry

创建一个持久卷来存储Docker Registry的数据

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
  namespace: docker-registry
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain

---

apiVersion: v1
kind: PersistentVolume
metadata:
  name: docker-registry-pv
  labels:
    pv: docker-registry-pv
spec:
  capacity: 
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/docker
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-node1

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: docker-registry-pvc
  namespace: docker-registry
spec:
  resources:
    requests:
      storage: 5Gi
  accessModes:
    - ReadWriteMany
  storageClassName: local-storage
  selector:
    matchLabels:
      pv: docker-registry-pv

Note：

没有必要将namespace字段添加到kind: persistantVolume因为PersistentVolumes绑定(bind)是排他的；
出于安全考虑Pod不会被调度到Master Node上，也就是说默认情况下Master节点（taint：污点）不参与工作负载。如果没有配置nodeAffinity，在创建PV时会报错：

0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn’t tolerate, 2 node(s) didn’t find available persistent volumes to bind.

解决方法：添加 nodeAffinity ，这里将 PV 创建到 k8s-node1 节点；记得先在 Worker 节点上创建 PV 目录：/data/docker

创建一个Secret用于存储Docker Registry的凭证

mkdir auth

docker run --entrypoint htpasswd registry:2 -Bbn username password > auth/htpasswd

报错：unable to start container process: exec: “htpasswd”: executable file not found in $PATH: unknown.
原因： registry 镜像在 2.x 相关版本中删除了 /usr/bin/htpasswd 文件，导致创建容器时找不到可执行文件，参考：https://github.com/docker/distribution-library-image/issues/106

使用旧版本的 registry 或者在服务器本地安装 httpd ，再执行生成密码的命令。

yum install httpd
htpasswd -Bbn username password > auth/htpasswd
kubectl create secret generic docker-registry-secret --from-file=auth/htpasswd -n docker-registry

# 创建了一个Opaque类型的Secret
[root@k8s-master ~]# kubectl get secret -n docker-registry
NAME                     TYPE                                  DATA   AGE
default-token-prfsz      kubernetes.io/service-account-token   3      11m
docker-registry-secret   Opaque                                1      20s

创建 Docker 私有镜像仓库的 secret ，这里是在 docker-registry 命名空间，如果新增了 namespace ，那么这个 namespace 就需要单独添加一次 secret ，而且这个 namespace 下拉取镜像时也需要添加 imagePullSecrets 。

kubectl create secret docker-registry ruoyi-registry-secret --docker-server=10.96.198.223:5000 --docker-username=username --docker-password=password  -n docker-registry

[root@k8s-master ~]# kubectl get secret -n docker-registry
NAME                     TYPE                                  DATA   AGE
default-token-prfsz      kubernetes.io/service-account-token   3      2d23h
docker-registry-secret   Opaque                                1      2d23h
ruoyi-registry-secret     kubernetes.io/dockerconfigjson        1      3s

Note:

docker-registry-secret: 用于配置Registry的账号与域名，并在控制台通过username与password登录、推送镜像到私有镜像仓库；
ruoyi-registry-secret: 用于通过在不同的Worker节点上拉取私有镜像。

将上述创建 NameSpace , StorageClass , PV 以及 PVC 的 YAML 合并为一个 docker-registry-ns-sc-pv-pvc.yaml 并执行。

kubectl apply -f docker-registry-ns-sc-pv-pvc.yaml

# 查看sc,pv,pvc状态
[root@k8s-master registry]# kubectl get sc,pv,pvc -n docker-registry
NAME                                        PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
storageclass.storage.k8s.io/local-storage   kubernetes.io/no-provisioner   Retain          WaitForFirstConsumer   false                  4m45s

NAME                                  CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS    REASON   AGE
persistentvolume/docker-registry-pv   5Gi        RWX            Retain           Available           local-storage            4m45s

NAME                                        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
persistentvolumeclaim/docker-registry-pvc   Pending                                      local-storage   4m45s

看到 PVC 一直是 Pending 状态。

# 查看PVC详细信息
[root@k8s-master registry]# kubectl describe pvc -n docker-registry
Name:          docker-registry-pvc
Namespace:     docker-registry
StorageClass:  local-storage
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age                  From                         Message
  ----    ------                ----                 ----                         -------
  Normal  WaitForFirstConsumer  9s (x26 over 6m13s)  persistentvolume-controller  waiting for first consumer to be created before binding

提示需要有个消费者进行关联，后面应用 Deployment 后， PVC 状态会变为 Binding 。

创建一个Deployment来部署Docker Registry

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: docker-registry
  name: docker-registry
  namespace: docker-registry
spec:
  replicas: 1
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      securityContext:
        runAsUser: 0
      containers:
        - name: docker-registry 
          image: registry:2 
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 5000
              name: web
              protocol: TCP
          resources:
            requests:
              memory: 200Mi
              cpu: "0.1"
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          env:
            - name: REGISTRY_AUTH
              value: htpasswd
            - name: REGISTRY_AUTH_HTPASSWD_REALM
              value: Registry Realm
            - name: REGISTRY_AUTH_HTPASSWD_PATH
              value: /auth/htpasswd
          volumeMounts:
            - name: docker-registry-data
              mountPath: /var/lib/registry/
            - name: docker-registry-auth
              mountPath: /auth
              readOnly: true
      volumes:
        - name: docker-registry-data
          persistentVolumeClaim:
            claimName: docker-registry-pvc
        - name: docker-registry-auth
          secret:
            secretName: docker-registry-secret

创建一个Service来暴露Docker Registry

apiVersion: v1
kind: Service
metadata:
  name: docker-registry-service
  namespace: docker-registry 
spec:
  ports:
    - name: port-name
      port: 5000 
      protocol: TCP
      targetPort: 5000
  selector:
    app: docker-registry
  type: NodePort

部署 Deployment 与 Service 。

kubectl apply -f docker-registry-deploy-svc.yaml

[root@k8s-master registry]# kubectl get deploy -n docker-registry
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
docker-registry   1/1     1            1           21s

[root@k8s-master registry]# kubectl get svc -n docker-registry
NAME                      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
docker-registry-service   NodePort   10.96.198.223   <none>        5000:30858/TCP   15s

登录私有镜像仓库

# 直接查看下镜像仓库的内容，提示未认证
[root@k8s-master ~]# curl http://10.96.198.223:5000/v2/_catalog
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"registry","Class":"","Name":"catalog","Action":"*"}]}]}

# 使用用户名、密码登录，报错：http: server gave HTTP response to HTTPS client
[root@k8s-master ~]# docker login -uusername -ppassword 10.96.198.223:5000
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://10.96.198.223:5000/v2/: http: server gave HTTP response to HTTPS client

# 修改配置文件，添加insecure-registries
[root@k8s-master ~]# vi /etc/docker/daemon.json
"insecure-registries": ["10.96.198.223:5000"]

# 重启Docker服务
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl restart docker

# 再次使用用户名、密码登录，成功
[root@k8s-master registry]# docker login -uusername -ppassword 10.96.198.223:5000
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

# 会自动生成~/.docker/config.json文件
[root@k8s-master ~]# cat ~/.docker/config.json
{
        "auths": {
                "10.96.198.223:5000": {
                        "auth": "dXNlcm5hbWU6cGFzc3dvcmQ="
                }
        }
}

# 附带用户名、密码，获取仓库内镜像，为空
[root@k8s-master ~]# curl -u username:password http://10.96.198.223:5000/v2/_catalog
{"repositories":[]}

推送镜像到私有仓库

# 以hello-world为例测试推送镜像到私有仓库功能
[root@k8s-master ~]# docker run hello-world
[root@k8s-master ~]# docker images
REPOSITORY                                                                 TAG        IMAGE ID       CREATED         SIZE
calico/node                                                                v3.20.6    daeec7e26e1f   17 months ago   156MB
calico/pod2daemon-flexvol                                                  v3.20.6    39b166f3f936   17 months ago   18.6MB
calico/cni                                                                 v3.20.6    13b6f63a50d6   17 months ago   138MB
calico/kube-controllers                                                    v3.20.6    4dc6e7685020   17 months ago   60.2MB
registry                                                                   2          b8604a3fe854   2 years ago     26.2MB
hello-world                                                                latest     feb5d9fea6a5   2 years ago     13.3kB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-proxy                v1.20.9    8dbf9a6aa186   2 years ago     99.7MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-scheduler            v1.20.9    295014c114b3   2 years ago     47.3MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-controller-manager   v1.20.9    eb07fd4ad3b4   2 years ago     116MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-apiserver            v1.20.9    0d0d57e4f64c   2 years ago     122MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/etcd                      3.4.13-0   0369cf4303ff   3 years ago     253MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/coredns                   1.7.0      bfe3a36ebd25   3 years ago     45.2MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/pause                     3.2        80d28bedfe5d   3 years ago     683kB

# 对hello-world镜像重新打标签
[root@k8s-master ~]# docker tag hello-world 10.96.198.223:5000/hello-world:1
[root@k8s-master ~]# docker images
REPOSITORY                                                                 TAG        IMAGE ID       CREATED         SIZE
calico/node                                                                v3.20.6    daeec7e26e1f   17 months ago   156MB
calico/pod2daemon-flexvol                                                  v3.20.6    39b166f3f936   17 months ago   18.6MB
calico/cni                                                                 v3.20.6    13b6f63a50d6   17 months ago   138MB
calico/kube-controllers                                                    v3.20.6    4dc6e7685020   17 months ago   60.2MB
registry                                                                   2          b8604a3fe854   2 years ago     26.2MB
10.96.198.223:5000/hello-world                                             1          feb5d9fea6a5   2 years ago     13.3kB
hello-world                                                                latest     feb5d9fea6a5   2 years ago     13.3kB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-proxy                v1.20.9    8dbf9a6aa186   2 years ago     99.7MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-scheduler            v1.20.9    295014c114b3   2 years ago     47.3MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-controller-manager   v1.20.9    eb07fd4ad3b4   2 years ago     116MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/kube-apiserver            v1.20.9    0d0d57e4f64c   2 years ago     122MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/etcd                      3.4.13-0   0369cf4303ff   3 years ago     253MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/coredns                   1.7.0      bfe3a36ebd25   3 years ago     45.2MB
registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/pause                     3.2        80d28bedfe5d   3 years ago     683kB

# 推送到私有镜像仓库，成功
[root@k8s-master ~]# docker push 10.96.198.223:5000/hello-world:1
The push refers to repository [10.96.198.223:5000/hello-world]
e07ee1baac5f: Pushed 
1: digest: sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4 size: 525

# 附带用户名、密码，获取仓库内镜像
[root@k8s-master ~]# curl -u username:password http://10.96.198.223:5000/v2/_catalog
{"repositories":["hello-world"]}

# 退出操作
[root@k8s-master ~]# docker logout 10.96.198.223:5000

安装Registry前端

直接通过 Docker 安装 Registry 前端，通过用户名与密码登录后，可以网页展示已推送的私有镜像。

docker run -d -e ENV_DOCKER_REGISTRY_HOST=10.96.198.223 -e ENV_DOCKER_REGISTRY_PORT=5000 -p 5001:80 konradkleine/docker-registry-frontend:v2

认证

私有镜像列表

小总结

作为测试环境，本次使用 Registry2 搭建了私有镜像仓库，实际生产环境建议使用 Harbor 。 Registry2 和 Harbor 都是用于 Docker 镜像存储和分发的解决方案，但它们各自具有不同的特点：

Registry2

基本功能：Registry2（通常称为Docker Registry）是Docker官方的开源镜像仓库，提供了存储和分发Docker镜像的基本功能。
简单轻量：它比较轻量级，适合需要快速部署的场景。
自主部署：可以自主部署在私有环境中，满足私有化需求。
缺少高级功能：相比于Harbor，Registry2缺少一些高级功能，如图形用户界面、角色基础的访问控制、镜像扫描和签名等。

Harbor

企业级特性：Harbor是一个开源的企业级Docker Registry解决方案，提供了Registry2的所有基本功能，并增加了许多企业级特性。
图形用户界面：Harbor提供了用户友好的图形界面，便于管理和浏览镜像。
访问控制：支持基于角色的访问控制，可以细粒度地管理用户权限。
安全性：提供了镜像扫描和签名功能，增强了镜像的安全性。
高可用性：支持高可用部署，适合企业级应用。

总结来说，如果你需要一个简单的、轻量级的 Docker 镜像仓库， Registry2 可能是一个不错的选择。而如果你需要更多的企业级特性，如图形界面、细粒度的访问控制、镜像安全扫描等， Harbor 会是更好的选择。

If you have any questions or any bugs are found, please feel free to contact me.

Your comments and suggestions are welcome!