前言：最近在运行dalle2模型进行推断，本篇博客记录相关资料。

相关博客：超详细！DALL · E 文生图模型实践指南

---

---

1. 环境搭建和预训练模型准备

本文使用的代码仓库为：https://github.com/lucidrains/DALLE2-pytorch

环境搭建

pip install dalle2-pytorch

预训练模型下载

地址：https://huggingface.co/laion/DALLE2-PyTorch

2. 代码

DALLE2 for inference 完整推断流程如下（from cest_andre）：

import torch
from torchvision.transforms import ToPILImage
from dalle2_pytorch import DiffusionPrior, DiffusionPriorNetwork, OpenAIClipAdapter, Decoder, DALLE2
from dalle2_pytorch.train_configs import TrainDiffusionPriorConfig, TrainDecoderConfig


prior_config = TrainDiffusionPriorConfig.from_json_path("weights/prior_config.json").prior
prior = prior_config.create().cuda()

prior_model_state = torch.load("weights/prior_latest.pth")
prior.load_state_dict(prior_model_state, strict=True)

decoder_config = TrainDecoderConfig.from_json_path("weights/decoder_config.json").decoder
decoder = decoder_config.create().cuda()

decoder_model_state = torch.load("weights/decoder_latest.pth")["model"]

for k in decoder.clip.state_dict().keys():
    decoder_model_state["clip." + k] = decoder.clip.state_dict()[k]

decoder.load_state_dict(decoder_model_state, strict=True)

dalle2 = DALLE2(prior=prior, decoder=decoder).cuda()

images = dalle2(
    ['your prompt here'],
    cond_scale = 2.
).cpu()

print(images.shape)

for img in images:
    img = ToPILImage()(img)
    img.show()

3. BUG&DEBUG

URLError

报错信息如下：

Traceback (most recent call last):
  File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 1350, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/root/anaconda3/envs/ldm/lib/python3.8/http/client.py", line 1424, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/root/anaconda3/envs/ldm/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/root/anaconda3/envs/ldm/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/root/anaconda3/envs/ldm/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/newdata/SD/extra/dalle2_cest.py", line 11, in <module>
    prior = prior_config.create().cuda()
  File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/dalle2_pytorch/train_configs.py", line 185, in create
    clip = self.clip.create()
  File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/dalle2_pytorch/train_configs.py", line 122, in create
    return OpenAIClipAdapter(self.model)
  File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/dalle2_pytorch/dalle2_pytorch.py", line 313, in __init__
    openai_clip, preprocess = clip.load(name)
  File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/clip/clip.py", line 122, in load
    model_path = _download(_MODELS[name], download_root or os.path.expanduser("~/.cache/clip"))
  File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/clip/clip.py", line 59, in _download
    with urllib.request.urlopen(url) as source, open(download_target, "wb") as output:
  File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 1393, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py", line 1353, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 104] Connection reset by peer>

我使用的是https://github.com/lucidrains/DALLE2-pytorch这个网址。

找到 /root/anaconda3/envs/ldm/lib/python3.8/urllib/request.py 中对应的位置，我这里是第1349行，修改方式也在下面代码中一并给出。

try:
    h.request(req.get_method(), req.selector, req.data, headers,
              encode_chunked=req.has_header('Transfer-encoding'))
    time.sleep(0.5)  # 添加的一行
except OSError as err: # timeout error
    raise URLError(err)

RuntimeError

Traceback (most recent call last):
  File "/newdata/SD/extra/dalle2_cest.py", line 14, in <module>
    prior.load_state_dict(prior_model_state, strict=True)
  File "/root/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1604, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DiffusionPrior:
        Missing key(s) in state_dict: "net.null_text_encodings", "net.null_text_embeds", "net.null_image_embed". 
        Unexpected key(s) in state_dict: "net.null_text_embed". 

解决办法：load_state_dict()函数中的 strict=True 改为 strict=False，如下：

...
prior.load_state_dict(prior_model_state, strict=False)

decoder.load_state_dict(decoder_model_state, strict=False)
...

CUDA error

RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

解决方法：版本不匹配，更换与系统cuda相匹配的pytorch版本。比如我的cuda版本是12.0，可以使用如下命令安装pytorch：

pip install torch==2.0.0+cu118 torchvision==0.15.1+cu118 torchaudio==2.0.1+cu118 -f https://download.pytorch.org/whl/torch_stable.html

到这里，模型就可以完成推断过程啦～嘻嘻！

---

参考链接

1. https://github.com/lucidrains/DALLE2-pytorch/issues/282
2. python requests请求报错ConnectionError: (‘Connection aborted.‘, error(104, ‘Connection reset by peer‘))_铁朵斯提的博客-CSDN博客
3. GPU版本pytorch（Cuda12.1）清华源快速安装一步一步教！小白教学~_清华源安装torch-CSDN博客