参考 文章：Helm部署Harbor_helm harbor_风向决定发型丶的博客-CSDN博客
安装好后使用 nerd
containerd对接harbor_containerd 容器 insecure-registries 配置_柠是柠檬的檬的博客-CSDN博客
推送镜像 Containerd 对接私有镜像仓库 Harbor - 知乎
接下来我们来测试下如何在 containerd 中使用 Harbor 镜像仓库。
首先我们需要将私有镜像仓库配置到 containerd 中去，修改 containerd 的配置文件 /etc/containerd/config.toml：
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."core.harbor.domain".tls]
insecure_skip_verify = true
[plugins."io.containerd.grpc.v1.cri".registry.configs."core.harbor.domain".auth]
username = "admin"
password = "Harbor12345"
[plugins."io.containerd.grpc.v1.cri".registry.headers]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."core.harbor.domain"]
endpoint = ["https://core.harbor.domain"]
在 plugins."io.containerd.grpc.v1.cri".registry.configs 下面添加对应 core.harbor.domain 的配置信息，insecure_skip_verify = true 表示跳过安全校验，然后通过 plugins."io.containerd.grpc.v1.cri".registry.configs."core.harbor.domain".auth 配置 Harbor 镜像仓库的用户名和密码。
配置完成后重启 containerd：
$ systemctl restart containerd
现在我们使用 nerdctl 来进行登录：
$ nerdctl login -u admin core.harbor.domain
Enter Password:
ERRO[0004] failed to call tryLoginWithRegHost error="failed to call rh.Client.Do: Get \"https://harbor.k8s.local/v2/\": x509: certificate signed by unknown authority" i=0
FATA[0004] failed to call rh.Client.Do: Get "https://harbor.k8s.local/v2/": x509: certificate signed by unknown authority
[root@master1 ~]#
可以看到还是会报证书相关的错误，只需要添加一个 --insecure-registry 参数即可解决该问题：
$ nerdctl login -u admin --insecure-registry core.harbor.domain
Enter Password:
WARN[0004] skipping verifying HTTPS certs for "harbor.k8s.local"
WARNING: Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
制作一个es镜像：
在 Releases · medcl/elasticsearch-analysis-ik · GitHub 下载 es 的ik插件
unzip xx -d ik
FROM docker.elastic.co/elasticsearch/elasticsearch:8.8.2
# ik 下载地址 https://github.com/medcl/elasticsearch-analysis-ik/releases
COPY ik /usr/share/elasticsearch/plugins/ik
# COPY pinyin /usr/share/elasticsearch/plugins/pinyin
然后将该镜像重新 tag 成 Harbor 上的镜像地址：
harbor新增一个项目 es
nerdctl tag elasticsearch:latest core.harbor.domain/elasticsearch:latest再执行 push 命令即可将镜像推送到 Harbor 上：
nerdctl -n k8s.io push --insecure-registry core.harbor.domain/es/elasticsearch:latest$ nerdctl push --insecure-registry harbor.k8s.local/library/busybox:1.35.0
INFO[0000] pushing as a reduced-platform image (application/vnd.docker.distribution.manifest.list.v2+json, sha256:29fe0126b13c3ea2641ca42c450fa69583d212dbd9b7b623814977b5b0945726)
WARN[0000] skipping verifying HTTPS certs for "harbor.k8s.local"
index-sha256:29fe0126b13c3ea2641ca42c450fa69583d212dbd9b7b623814977b5b0945726: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:8cde9b8065696b65d7b7ffaefbab0262d47a5a9852bfd849799559d296d2e0cd: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:d8c0f97fc6a6ac400e43342e67d06222b27cecdb076cbf8a87f3a2a25effe81c: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 6.9 s total: 2.2 Ki (333.0 B/s)
推送完成后，我们就可以在 Portal 页面上看到这个镜像的信息了：
镜像 push 成功，同样可以测试下 pull：
$ nerdctl rmi harbor.k8s.local/library/busybox:1.35.0
Untagged: harbor.k8s.local/library/busybox:1.35.0@sha256:8c40df61d40166f5791f44b3d90b77b4c7f59ed39a992fd9046886d3126ffa68
Deleted: sha256:cf4ac4fc01444f1324571ceb0d4f175604a8341119d9bb42bc4b2cb431a7f3a5
$ nerdctl rmi busybox:1.35.0
Untagged: docker.io/library/busybox:1.35.0@sha256:8c40df61d40166f5791f44b3d90b77b4c7f59ed39a992fd9046886d3126ffa68
Deleted: sha256:cf4ac4fc01444f1324571ceb0d4f175604a8341119d9bb42bc4b2cb431a7f3a5
$ nerdctl pull --insecure-registry harbor.k8s.local/library/busybox:1.35.0
WARN[0000] skipping verifying HTTPS certs for "harbor.k8s.local"
harbor.k8s.local/library/busybox:1.35.0: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:29fe0126b13c3ea2641ca42c450fa69583d212dbd9b7b623814977b5b0945726: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:8cde9b8065696b65d7b7ffaefbab0262d47a5a9852bfd849799559d296d2e0cd: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:d8c0f97fc6a6ac400e43342e67d06222b27cecdb076cbf8a87f3a2a25effe81c: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fc0cda0e09ab32c72c61d272bb409da4e2f73165c7bf584226880c9b85438e63: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.7 s total: 2.2 Ki (3.2 KiB/s)
$ nerdctl images
REPOSITORY TAG IMAGE ID CREATED PLATFORM SIZE BLOB SIZE
harbor.k8s.local/library/busybox 1.35.0 29fe0126b13c 17 seconds ago linux/amd64 1.2 MiB 757.7 KiB
但是上面我们也可以看到单独使用 containerd 比如通过 nerdctl 或者 ctr 命令访问 Harbor 镜像仓库的时候即使跳过证书校验或者配置上 CA 证书也是会出现证书错误的，这个时候我们需要去跳过证书校验或者指定证书路径才行。
# 解决办法1.指定 -k 参数跳过证书校验。
$ ctr images pull --user admin:Harbor12345 -k harbor.k8s.local/library/busybox:1.35.0
# 解决办法2.指定CA证书、Harbor 相关证书文件路径。
$ ctr images pull --user admin:Harbor12345 --tlscacert ca.crt harbor.k8s.local/library/busybox:1.35.0
但是如果直接使用 ctrctl 则是有效的：
$ crictl pull harbor.k8s.local/library/busybox@sha256:29fe0126b13c3ea2641ca42c450fa69583d212dbd9b7b623814977b5b0945726
Image is up to date for sha256:d8c0f97fc6a6ac400e43342e67d06222b27cecdb076cbf8a87f3a2a25effe81c
如果想要在 Kubernetes 中去使用那么就需要将 Harbor 的认证信息以 Secret 的形式添加到集群中去：
$ kubectl create secret docker-registry harbor-auth --docker-server=https://harbor.k8s.local --docker-username=admin --docker-password=Harbor12345 --docker-email=info@ydzs.io -n default
然后我们使用上面的私有镜像仓库来创建一个 Pod：
# test-harbor.yaml
apiVersion: v1
kind: Pod
metadata:
name: harbor-registry-test
spec:
containers:
- name: test
image: harbor.k8s.local/library/busybox:1.35.0
args:
- sleep
- "3600"
imagePullSecrets:
- name: harbor-auth
创建后可以查看该 Pod 是否能正常获取镜像：
$ kubectl describe pod harbor-registry-test
Name: harbor-registry-test
Namespace: default
Priority: 0
Node: node1/192.168.0.107
Start Time: Thu, 07 Jul 2022 18:52:39 +0800
# ......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10s default-scheduler Successfully assigned default/harbor-registry-test to node1
Normal Pulling 10s kubelet Pulling image "harbor.k8s.local/library/busybox:1.35.0"
Normal Pulled 5s kubelet Successfully pulled image "harbor.k8s.local/library/busybox:1.35.0" in 4.670528883s
Normal Created 5s kubelet Created container test
Normal Started 5s kubelet Started container test
到这里证明上面我们的私有镜像仓库搭建成功了，大家可以尝试去创建一个私有的项目，然后创建一个新的用户，使用这个用户来进行 pull/push 镜像，Harbor 还具有其他的一些功能，比如镜像复制，Helm Chart 包托管等等，大家可以自行测试，感受下 Harbor 和官方自带的 registry 仓库的差别。
