​
Diffusion Autoencoders: Toward a Meaningful and Decodable Representation
扩散自编码器:面向有意义和可解码的表示
code：https://github.com/phizaz/diffae
A CVPR 2022 (ORAL) paper (paper, site, 5-min video)

---

Diffusion probabilistic models (DPMs) have achieved remarkable quality in image generation that rivals GANs’. But unlike GANs, DPMs use a set of latent variables that lack semantic meaning and cannot serve as a useful representation for other tasks. This paper explores the possibility of using DPMs for representation learning and seeks to extract a meaningful and decodable representation of an input image via autoencoding. Our key idea is to use a learnable encoder for discovering the high-level semantics, and a DPM as the decoder for modeling the remaining stochastic variations. Our method can encode any image into a two-part latent code where the first part is semantically meaningful and linear, and the second part captures stochastic details, allowing near-exact reconstruction. This capability enables challenging applications that currently foil GAN-based methods, such as attribute manipulation on real images. We also show that this two-level encoding improves denoising efficiency and naturally facilitates various downstream tasks including few-shot conditional sampling. Our novel latent space is more readily discriminative than StyleGAN-W (via inversion) when used to encode real input images.

Figure 1. Attribute manipulation and interpolation on real images.

Figure 2: Overview of our diffusion autoencoder.

Autoencoder由4部分组成， $Z_{sem}$ 捕获了高级语义【semantics】, while $X_T$ 捕获了低级的随机变化【stochastic variations / stochastic subcode】, 它俩一起可以精准的decoder回原始图像。

• a “semantic” encoder that maps the input image to the semantic subcode ($X_0\to Z_{sem}$),
• a conditional DDIM that acts both as a “stochastic” encoder ($X_0\to X_T$ )
• a decoder ($\left(\left({\mathbf{z}}_{\mathrm{s}\mathrm{e}\mathrm{m}},{\mathbf{x}}_T\right)\to X_0\right)$).
• 为了从autoencoder采样（为了无条件图像生成）, we fit a latent DDIM to the distribution of $Z_{sem}$and sample $\left({\mathbf{z}}_{\mathrm{s}\mathrm{e}\mathrm{m}},{\mathbf{x}}_T\sim \mathcal{N}(0,\mathbf{I})\right)$ for decoding.

Figure 3. 由改变随机子码$X_T$ 引起的变化和重建结果. 每一行对应一个不同的 $Z_{sem}$, 它完全改变了人，而改变随机子码 $X_T$ 只影响次要的细节。

Figure 5. 通过移动被线性分类器发现的 $Z_{sem}$ 的正向或逆向 $Z_{sem}$ 改变人脸的属性。

Figure 6. 逆向生成 $x_0$ at $t_{9,8,7,5,2,0}(T=10)$. 通过以 $Z_{sem}$ 为条件, 我们的方法生成与 $x_0$相似的图像更快。

写在最后： 其实这种通过改变latent的方式改变外观/一些属性的方法并不好，因为真实的效果很差，他们只展示效果比较好的，并且你在训练的时候需要标注大量的数据集，比如眼睛的位置、嘴巴的位置、…

​