kubevirt基于CDI创建虚拟机

CDI介绍

KubeVirt 的 Containerized Data Importer (CDI) 是一个 Kubernetes 原生的数据管理组件，专门为虚拟机 (VM) 提供存储支持，尤其在虚拟机的镜像管理和数据导入方面非常有用。CDI 的主要用途是帮助用户轻松地将外部数据源导入到 Kubernetes 集群中，并将这些数据转换为虚拟机可以使用的持久化存储卷 (Persistent Volume, PV)。
在这里插入图片描述
containerized-data-importer(CDI) 项目提供了一些设施，使持久卷声明(PVC) 能够通过DataVolumes用作 KubeVirt VM 的磁盘。 CDI 的三个主要用例是：

将磁盘映像从 Web 服务器或容器注册表导入到 DataVolume
将现有 PVC 克隆到 DataVolume
将本地磁盘镜像上传到DataVolume

在这里插入图片描述

本文档涉及第三个用例。因此，您应该在集群中安装 CDI、要上传的 VM 磁盘，并在路径中安装 virtctl。

前置要求

已准备可用的kubernetes集群

root@node40:~# kubectl get nodes -o wide
NAME     STATUS   ROLES           AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
node40   Ready    control-plane   161d   v1.29.3   192.168.72.40   <none>        Ubuntu 22.04.2 LTS   5.15.0-122-generic   containerd://1.7.15
node41   Ready    <none>          161d   v1.29.3   192.168.72.41   <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic    containerd://1.7.15
node42   Ready    <none>          161d   v1.29.3   192.168.72.42   <none>        Ubuntu 22.04.2 LTS   5.15.0-122-generic   containerd://1.7.15
root@node40:~#

已安装kubevirt以及virtctl命令行工具。

root@node40:~# virtctl version
Client Version: version.Info{GitVersion:"v1.4.0-alpha.0", GitCommit:"946f894f472b3a355ebb8eefaf89b871a06415ab", GitTreeState:"clean", BuildDate:"2024-09-12T11:50:12Z", GoVersion:"go1.22.6 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{GitVersion:"v1.4.0-alpha.0", GitCommit:"946f894f472b3a355ebb8eefaf89b871a06415ab", GitTreeState:"clean", BuildDate:"2024-09-12T13:21:26Z", GoVersion:"go1.22.6 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"}

已准备可用的默认storageclass

root@node40:~# kubectl get sc
NAME                 PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
longhorn (default)   driver.longhorn.io   Delete          Immediate              true                   10d
longhorn-static      driver.longhorn.io   Delete          Immediate              true                   60s

CDI安装

安装最新的 CDI 版本

export TAG=$(curl -s -w %{redirect_url} https://github.com/kubevirt/containerized-data-importer/releases/latest)
export VERSION=$(echo ${TAG##*/})
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml

确认安装成功

root@node40:~# kubectl -n cdi get pods
NAME                               READY   STATUS    RESTARTS        AGE
cdi-apiserver-55dd9447cb-ctsww     1/1     Running   0               5d22h
cdi-deployment-55c6d9fc49-88vf8    1/1     Running   14 (2d5h ago)   10d
cdi-operator-7f5bc68fc5-zp54v      1/1     Running   5 (2d5h ago)    5d22h
cdi-uploadproxy-5b76f7c876-9gkqs   1/1     Running   0               10d

公开 cdi-uploadproxy 服务

参考：https://github.com/kubevirt/containerized-data-importer/blob/main/doc/upload.md

$ cat cdi-uploadproxy-nodeport.yaml
apiVersion: v1
kind: Service
metadata:
  name: cdi-uploadproxy-nodeport
  namespace: cdi
  labels:
    cdi.kubevirt.io: "cdi-uploadproxy"
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 31001
      protocol: TCP
  selector:
    cdi.kubevirt.io: cdi-uploadproxy

应用yaml文件

kubectl apply -f cdi-uploadproxy-nodeport.yaml

确认存在NodePort类型的service：cdi-uploadproxy-nodeport

root@node40:~# kubectl -n cdi get svc
NAME                       TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
cdi-api                    ClusterIP   10.96.1.45    <none>        443/TCP         11d
cdi-prometheus-metrics     ClusterIP   10.96.3.131   <none>        8080/TCP        11d
cdi-uploadproxy            ClusterIP   10.96.0.118   <none>        443/TCP         11d
cdi-uploadproxy-nodeport   NodePort    10.96.2.193   <none>        443:31001/TCP   3h4m

上传镜像

CDI 支持 qemu 支持的raw和qcow2镜像格式。还可以使用可启动 ISO 映像，并将其视为raw映像。镜像可以使用gz或xz格式进行压缩。

下载镜像，以rocky linux cloud image为例：

wget https://dl.rockylinux.org/pub/rocky/9.4/images/x86_64/Rocky-9-GenericCloud-Base-9.4-20240609.1.x86_64.qcow2

使用virtctl上传镜像，确认已安装virtctl工具。

virtctl image-upload dv rocky-vm-disk \
--size=10Gi \
--image-path=/root/Rocky-9-GenericCloud-Base-9.4-20240609.1.x86_64.qcow2 \
--uploadproxy-url=https://192.168.72.40:31001/v1beta1/upload \
--insecure

参数说明：

virtctl image-upload:子命令用于上传一个磁盘镜像到 Kubernetes 集群内，通常会存储在一个 DataVolume 中。
dv rocky-vm-disk: dv 是 DataVolume 的缩写，表示要上传的镜像会创建或关联到一个名为 rocky-vm-disk 的 DataVolume 资源。DataVolume 是 CDI 提供的一个资源类型，用于管理虚拟机磁盘镜像的数据导入和上传。
--size=10Gi: 指定了要创建的虚拟磁盘的大小，这里是 10GiB。DataVolume 会申请一个大小为 10GiB 的持久化卷（PVC）来存储上传的镜像文件。
--image-path: 指定了本地磁盘上待上传的镜像文件路径。这里的文件是 Rocky Linux 9.4 的云基础镜像，格式为 qcow2，位于 /root 目录下。
--uploadproxy-url: 指定了 CDI 的上传代理服务 (uploadproxy) 的 URL，该 URL 用于上传镜像文件至 Kubernetes 集群。这个 URL 由 Kubernetes nodeport类型的服务暴露，使用节点NODE IP 地址以及 31001 端口访问。通过这个代理服务，virtctl 将本地镜像文件上传至集群中的 DataVolume。
--insecure: 表示在与 uploadproxy 服务通信时，不使用 TLS 证书验证。这在测试或非生产环境中很常见，目的是避免因为证书问题而导致上传失败。生产环境中建议使用带有证书验证的安全连接。

查看创建的cdi-upload pod，上传完成后该pod被自动清理

root@node40:~# kubectl get pods
NAME                                                    READY   STATUS    RESTARTS       AGE
cdi-upload-prime-d9d2776b-bda2-4852-a0f7-6e69c6582031   1/1     Running   0              3m22s

查看上传日志信息

root@node40:~# kubectl logs -f cdi-upload-prime-e2055ca6-92a0-42fa-89da-ba6211c9aa2a
I0929 07:40:34.705880       1 uploadserver.go:81] Running server on 0.0.0.0:8443
I0929 07:40:37.181186       1 uploadserver.go:438] Content type header is ""
I0929 07:40:37.181236       1 data-processor.go:348] Calculating available size
I0929 07:40:37.188220       1 data-processor.go:356] Checking out block volume size.
I0929 07:40:37.188243       1 data-processor.go:368] Request image size not empty.
I0929 07:40:37.188275       1 data-processor.go:373] Target size 10737418240.
I0929 07:40:37.188641       1 data-processor.go:247] New phase: TransferScratch
I0929 07:40:37.189035       1 util.go:96] Writing data...
I0929 07:41:08.076678       1 data-processor.go:247] New phase: Convert
I0929 07:41:08.076712       1 data-processor.go:253] Validating image
E0929 07:41:08.191970       1 prlimit.go:156] failed to kill the process; os: process already finished
I0929 07:41:08.192141       1 qemu.go:115] Running qemu-img with args: [convert -t writeback -p -O raw /scratch/tmpimage /dev/cdi-block-volume]
E0929 07:43:58.201878       1 prlimit.go:156] failed to kill the process; os: process already finished
I0929 07:43:58.202649       1 data-processor.go:247] New phase: Resize
I0929 07:43:58.205172       1 data-processor.go:247] New phase: Complete
I0929 07:43:58.206054       1 uploadserver.go:465] Wrote data to /dev/cdi-block-volume
I0929 07:43:58.206845       1 uploadserver.go:230] Shutting down http server after successful upload
I0929 07:43:58.208180       1 uploadserver.go:115] UploadServer successfully exited

确认上传成功

root@node40:~# virtctl image-upload dv rocky-vm-disk \
--size=10Gi \
--image-path=/root/Rocky-9-GenericCloud-Base-9.4-20240609.1.x86_64.qcow2 \
--uploadproxy-url=https://192.168.72.40:31001/v1beta1/upload \
--insecure
PVC default/rocky-vm-disk not found 
DataVolume default/rocky-vm-disk created
Waiting for PVC rocky-vm-disk upload pod to be ready...
Pod now ready
Uploading data to https://192.168.72.40:31001/v1beta1/upload

578.31 MiB / 578.31 MiB [----------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 2.88 MiB p/s 3m21s

Uploading data completed successfully, waiting for processing to complete, you can hit ctrl-c without interrupting the progress
Processing completed successfully
Uploading /root/Rocky-9-GenericCloud-Base-9.4-20240609.1.x86_64.qcow2 completed successfully
root@node40:~#

查看创建的pvc

root@node40:~# kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
rocky-vm-disk    Bound    pvc-0679a050-51d4-4c62-8d52-1b20dc2f9cfb   10Gi       RWO            longhorn       <unset>                 3m25s

查看创建的pv

root@node40:~# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-0679a050-51d4-4c62-8d52-1b20dc2f9cfb   10Gi       RWO            Delete           Bound    default/rocky-vm-disk   longhorn       <unset>                          3m53s

查看创建的datavolume

root@node40:~# kubectl get dv
NAME             PHASE       PROGRESS   RESTARTS   AGE
rocky-vm-disk   Succeeded   N/A                   4m17s

查看datavolume详细信息

root@node40:~# kubectl get dv rocky-vm-disk -o yaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  annotations:
    cdi.kubevirt.io/storage.usePopulator: "true"
  creationTimestamp: "2024-09-29T07:40:17Z"
  generation: 1
  name: rocky-vm-disk
  namespace: default
  resourceVersion: "5568675"
  uid: 54d1741c-169e-4919-933e-37abe416cb62
spec:
  contentType: kubevirt
  source:
    upload: {}
  storage:
    resources:
      requests:
        storage: 10Gi
status:
  claimName: rocky-vm-disk
  conditions:
  - lastHeartbeatTime: "2024-09-29T07:43:59Z"
    lastTransitionTime: "2024-09-29T07:43:59Z"
    message: PVC rocky-vm-disk Bound
    reason: Bound
    status: "True"
    type: Bound
  - lastHeartbeatTime: "2024-09-29T07:43:59Z"
    lastTransitionTime: "2024-09-29T07:43:59Z"
    status: "True"
    type: Ready
  - lastHeartbeatTime: "2024-09-29T07:43:58Z"
    lastTransitionTime: "2024-09-29T07:43:58Z"
    message: Upload Complete
    reason: Completed
    status: "False"
    type: Running
  phase: Succeeded
  progress: N/A

创建虚拟机

示例yaml文件

root@node40:~/cdi# cat rocky-vm.yaml 
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: rocky-vm
spec:
  runStrategy: Always
  template:
    metadata:
      labels:
        kubevirt.io/size: small
        kubevirt.io/domain: rocky-vm
    spec:
      domain:
        devices:
          disks:
            - name: datavolumevolume
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
          - name: default
            masquerade: {}
        resources:
          requests:
            memory: 1Gi
          limits:
            memory: 2Gi
      networks:
      - name: default
        pod: {}
      volumes:
        - name: datavolumevolume
          dataVolume:
            name: rocky-vm-disk
        - name: cloudinitdisk
          cloudInitNoCloud:
            userData: |-
              #cloud-config
              password: rocky
              chpasswd: { expire: False }

应用yaml文件

kubectl apply -f rocky-vm.yaml

查看创建的虚拟机

root@node40:~/cdi# kubectl get vm
NAME                      AGE     STATUS    READY
rocky-vm                  7m49s   Running   True
root@node40:~/cdi# 
root@node40:~/cdi# kubectl get vmi
NAME                      AGE     PHASE     IP             NODENAME   READY
rocky-vm                  7m52s   Running   100.64.1.111   node42     True
root@node40:~/cdi#

通过virtctl 访问虚拟机，默认账号密码为rocky/rocky

root@node40:~# virtctl console rocky-vm
Successfully connected to rocky-vm console. The escape sequence is ^]

rocky-vm login: rocky
Password: 
Last login: Sun Sep 29 07:52:19 on ttyS0
[rocky@rocky-vm ~]$ 
[rocky@rocky-vm ~]$ cat /etc/os-release 
NAME="Rocky Linux"
VERSION="9.4 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.4 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.4"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
[rocky@rocky-vm ~]$

虚拟机写入数据

[rocky@rocky-vm ~]$ echo $(date) > data.txt
[rocky@rocky-vm ~]$ 
[rocky@rocky-vm ~]$ cat data.txt 
Sun Sep 29 08:01:03 AM UTC 2024
[rocky@rocky-vm ~]$

删除虚拟机

root@node40:~# kubectl delete vmi rocky-vm
virtualmachineinstance.kubevirt.io "rocky-vm" deleted

等待虚拟机重建，验证持久化数据是否存在

root@node40:~# virtctl console rocky-vm
[rocky@rocky-vm ~]$ cat data.txt 
Sun Sep 29 08:01:03 AM UTC 2024
[rocky@rocky-vm ~]$

YAML文件示例

通过将 DataVolume 添加到 dataVolumeTemplates 列表，可以直接在 VM 规范中定义 DataVolume。下面是一个例子。

直接创建以下yaml文件：

root@node40:~/cdi# cat cirros-vm.yaml 
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: cirros-vm
  name: cirros-vm
spec:
  dataVolumeTemplates:
  - metadata:
      name: cirros-dv
    spec:
      storage:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: longhorn
      source:
        http:
          url: https://download.cirros-cloud.net/0.6.3/cirros-0.6.3-x86_64-disk.img
  runStrategy: Always
  template:
    metadata:
      labels:
        kubevirt.io/vm: cirros-vm
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: datavolumevolume
          - name: cloudinitdisk
            disk:
              bus: virtio
        machine:
          type: ""
        resources:
          limits:
            memory: 2Gi
          requests:
            memory: 1Gi
      terminationGracePeriodSeconds: 0
      volumes:
      - dataVolume:
          name: cirros-dv
        name: datavolumevolume
      - name: cloudinitdisk
        cloudInitNoCloud:
          userDataBase64: SGkuXG4=

应用yaml文件

kubectl apply -f cirros-vm.yaml

查看创建的虚拟机

root@node40:~/cdi# kubectl get vm
NAME        AGE    STATUS    READY
cirros-vm   7m2s   Running   True
rocky-vm    91m    Running   True

通过console访问虚拟机

root@node40:~/cdi# virtctl console cirros-vm
Successfully connected to cirros-vm console. The escape sequence is ^]

login as 'cirros' user. default password: 'gocubsgo'. use 'sudo' for root.
cirros-vm login: cirros
Password: 
$ 
$ cat /etc/os-release 
PRETTY_NAME="CirrOS 0.6.3"
NAME="CirrOS"
VERSION_ID="0.6.3"
ID=cirros
HOME_URL="https://cirros-cloud.net"
BUG_REPORT_URL="https://github.com/cirros-dev/cirros/issues"
$