ComfyUI 集成混元DIT（comfyui-hydit）

news2024/11/24 20:42:29

最近腾讯官方推出了ComfyUI插件comfyui-hydit 。是一个专门为腾讯的 Hunyuan-DiT 模型设计的自定义节点和工作流。本文主要介绍如何通过ComfyUI来运行腾讯新出的支持中文提示词的混元文生图大模型Hunyuan-DiT

环境准备

插件

从腾讯混元DIT 源码库获取插件源码：
https://github.com/Tencent/HunyuanDiT.git
从该仓库中获取：comfyui-hydit

模型

所有模型下载后保存在需要挂载的路径下。

所有模型下载后保存在需要挂载的路径下：/u01/workspace/models/hunyuan

克隆HunyuanDiT模型

有多个模型可选择，我们这里采用了第三个模型：
Tencent-Hunyuan/HunyuanDiT 基础模型
Tencent-Hunyuan/HunyuanDiT-Diffusers 加入Diffusers格式
Tencent-Hunyuan/Distillation 加速版

git clone https://huggingface.co/Tencent-Hunyuan/HunyuanDiT
git clone https://huggingface.co/Tencent-Hunyuan/Distillation

如果想要加速版，以上两个地址的文件都要下载
注意下载完Distillation 模型后，需要拷贝模型pytorch_model_distill.pt到： HunyuanDiT/t2i/model/目录下

下载OpenAI视觉模型

git clone https://hf-mirror.com/openai/clip-vit-large-patch14-336
:::danger
保存位置：/u01/workspace/models/clip-vit-large-patch14-336
源码修改方式请参照混元模型的部署过程。
:::

容器化部署ComfyUI

复制comfyui-hydit插件目录到ComfyUI custom_nodes目录：

├── AIGODLIKE-ComfyUI-Translation
├── comfyui_controlnet_aux
├── ComfyUI_Custom_Nodes_AlekPet
├── ComfyUI-Custom-Scripts
├── comfyui-hydit
├── ComfyUI-Manager
├── ComfyUI_TiledKSampler
├── ComfyUI_UltimateSDUpscale
├── Derfuu_ComfyUI_ModdedNodes
├── efficiency-nodes-comfyui
├── example_node.py.example
├── __pycache__
├── sdxl_prompt_styler
└── websocket_image_save.py

准备Dockerfile文件

准备Dockerfile可以安装预装插件和不预装两种不同的方式，如果不预装插件，可以把插件挂载到宿主机上等启动之在安装插件，本文采用的是预装的方式准备的Dockerfile

FROM pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Etc/UTC
ENV LANG=zh_CN.UTF-8
WORKDIR /app

RUN apt-get update 
RUN apt-get install -y libgl1-mesa-glx libglib2.0-0 libsm6 libxrender1 libxext6 git
RUN rm -rf /var/lib/apt/lists/*
ENV GIT_PYTHON_REFRESH=quiet 

RUN pip config set global.index-url http://mirrors.aliyun.com/pypi/simple
RUN pip config set install.trusted-host mirrors.aliyun.com

COPY ComfyUI/ /app/ComfyUI/
WORKDIR /app/ComfyUI

RUN pip install --use-pep517 -r requirements.txt
RUN pip install -r  custom_nodes/comfyui_controlnet_aux/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI-Manager/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI_Custom_Nodes_AlekPet/ArgosTranslateNode/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI_Custom_Nodes_AlekPet/DeepTranslatorNode/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI_Custom_Nodes_AlekPet/GoogleTranslateNode/requirements.txt
RUN pip install -r  custom_nodes/efficiency-nodes-comfyui/requirements.txt
RUN pip install -r  custom_nodes/comfyui-hydit/requirements.txt

EXPOSE 8188
CMD [ "python","main.py ","--port 8188"]

custom_nodes/comfyui-hydit/requirements.txt 修改：

#--extra-index-url https://pypi.ngc.nvidia.com
#--extra-index-url https://download.pytorch.org/whl/cu117
timm
diffusers
peft
protobuf
accelerate
loguru
sentencepiece
cuda-python
polygraphy
pandas
omegaconf
# torch>=2.3.0
# torchvision>=0.18.0
# torchaudio>=2.3.0
xformers>=0.0.26.post1
pytorch_lightning

执行构建

docker buile -t qingcloudtech/comfyui:v1.1 .

启动容器

docker run -it --gpus all \
-p 8188:8188 \
-v /u01/workspace/models/stablediffusion:/app/ComfyUI/models \
-v /u01/workspace/comfyui/output/:/app/ComfyUI/output \
- /u01/workspace/ComfyUI/extra_model_paths.yaml:/app/ComfyUI/extra_model_paths.yaml
qingcloudtech/comfyui:v1.5 python main.py

启动容器（docker-compose方式）

准备配置文件

version: '3.8'
services:
  comfyui:
    image: qingcloudtech/comfyui:v1.1
    container_name: comfyui
    command: python /app/ComfyUI/main.py
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - /u01/workspace/models/stablediffusion/:/app/ComfyUI/models/
      - /u01/workspace/comfyui/output/:/app/ComfyUI/output
      - /u01/workspace/ComfyUI/extra_model_paths.yaml:/app/ComfyUI/extra_model_paths.yaml
    network_mode: host

执行启动命令：

docker-compose up -d

浏览器访问：

演示访问：

1、浏览器访问：127.0.0.1:8188
2、从web界面中加载如下测试json：

{
  "last_node_id": 21,
  "last_link_id": 6,
  "nodes": [
    {
      "id": 19,
      "type": "DiffusersClipTextEncode",
      "pos": [
        74,
        277
      ],
      "size": {
        "0": 400,
        "1": 200
      },
      "flags": {},
      "order": 0,
      "mode": 0,
      "outputs": [
        {
          "name": "positive",
          "type": "STRINGC",
          "links": [
            5
          ],
          "shape": 3,
          "label": "positive"
        },
        {
          "name": "negative",
          "type": "STRINGC",
          "links": [
            6
          ],
          "shape": 3,
          "label": "negative"
        }
      ],
      "properties": {
        "Node name for S&R": "DiffusersClipTextEncode"
      },
      "widgets_values": [
        "castle,landscape,1girl,loli,silver hair,school_uniform,crystal_earrings,kind_smile,arm_support,in summer,mini_witch_hat,frilled,strappy_heels,book,\n",
        "错误的眼睛，糟糕的人脸，毁容，糟糕的艺术，变形，多余的肢体，模糊的颜色，模糊，重复，病态，残缺，"
      ]
    },
    {
      "id": 16,
      "type": "DiffusersModelMakeup",
      "pos": [
        526,
        393
      ],
      "size": {
        "0": 304.79998779296875,
        "1": 46
      },
      "flags": {},
      "order": 3,
      "mode": 0,
      "inputs": [
        {
          "name": "pipeline",
          "type": "PIPELINE",
          "link": 2,
          "label": "pipeline"
        },
        {
          "name": "scheduler",
          "type": "SCHEDULER",
          "link": 3,
          "label": "scheduler"
        }
      ],
      "outputs": [
        {
          "name": "MAKED_PIPELINE",
          "type": "MAKED_PIPELINE",
          "links": [
            4
          ],
          "shape": 3,
          "label": "MAKED_PIPELINE"
        }
      ],
      "properties": {
        "Node name for S&R": "DiffusersModelMakeup"
      }
    },
    {
      "id": 18,
      "type": "DiffusersSampler",
      "pos": [
        842,
        132
      ],
      "size": [
        315,
        454
      ],
      "flags": {},
      "order": 4,
      "mode": 0,
      "inputs": [
        {
          "name": "maked_pipeline",
          "type": "MAKED_PIPELINE",
          "link": 4,
          "label": "maked_pipeline"
        },
        {
          "name": "positive",
          "type": "STRINGC",
          "link": 5,
          "label": "positive"
        },
        {
          "name": "negative",
          "type": "STRINGC",
          "link": 6,
          "label": "negative"
        }
      ],
      "outputs": [
        {
          "name": "IMAGE",
          "type": "IMAGE",
          "links": [
            1
          ],
          "shape": 3,
          "label": "IMAGE"
        }
      ],
      "properties": {
        "Node name for S&R": "DiffusersSampler"
      },
      "widgets_values": [
        1,
        1024,
        1024,
        30,
        6,
        2705109163,
        "randomize"
      ]
    },
    {
      "id": 6,
      "type": "PreviewImage",
      "pos": [
        1186,
        181
      ],
      "size": [
        210,
        246
      ],
      "flags": {},
      "order": 5,
      "mode": 0,
      "inputs": [
        {
          "name": "images",
          "type": "IMAGE",
          "link": 1,
          "label": "图像"
        }
      ],
      "title": "Preview Image",
      "properties": {
        "Node name for S&R": "PreviewImage"
      }
    },
    {
      "id": 21,
      "type": "DiffusersPipelineLoader",
      "pos": [
        81,
        544
      ],
      "size": {
        "0": 315,
        "1": 190
      },
      "flags": {},
      "order": 1,
      "mode": 0,
      "outputs": [
        {
          "name": "PIPELINE",
          "type": "PIPELINE",
          "links": [
            2
          ],
          "shape": 3,
          "label": "PIPELINE"
        },
        {
          "name": "MODEL",
          "type": "MODEL",
          "links": null,
          "shape": 3,
          "label": "MODEL"
        },
        {
          "name": "CLIP",
          "type": "CLIP",
          "links": null,
          "shape": 3,
          "label": "CLIP"
        },
        {
          "name": "VAE",
          "type": "VAE",
          "links": null,
          "shape": 3,
          "label": "VAE"
        }
      ],
      "properties": {
        "Node name for S&R": "DiffusersPipelineLoader"
      },
      "widgets_values": [
        "ckpts",
        "pytorch_model_distill.pt",
        "disable",
        "diffusers"
      ]
    },
    {
      "id": 15,
      "type": "DiffusersSchedulerLoader",
      "pos": [
        73,
        129
      ],
      "size": {
        "0": 315,
        "1": 58
      },
      "flags": {},
      "order": 2,
      "mode": 0,
      "outputs": [
        {
          "name": "SCHEDULER",
          "type": "SCHEDULER",
          "links": [
            3
          ],
          "shape": 3,
          "label": "SCHEDULER"
        }
      ],
      "properties": {
        "Node name for S&R": "DiffusersSchedulerLoader"
      },
      "widgets_values": [
        "ddim"
      ]
    }
  ],
  "links": [
    [
      1,
      18,
      0,
      6,
      0,
      "IMAGE"
    ],
    [
      2,
      21,
      0,
      16,
      0,
      "PIPELINE"
    ],
    [
      3,
      15,
      0,
      16,
      1,
      "SCHEDULER"
    ],
    [
      4,
      16,
      0,
      18,
      0,
      "MAKED_PIPELINE"
    ],
    [
      5,
      19,
      0,
      18,
      1,
      "STRINGC"
    ],
    [
      6,
      19,
      1,
      18,
      2,
      "STRINGC"
    ]
  ],
  "groups": [],
  "config": {},
  "extra": {
    "ds": {
      "scale": 0.9090909090909098,
      "offset": {
        "0": 163.4918776991919,
        "1": 16.755363890844446
      }
    }
  },
  "version": 0.4
}