利用梯度上升可视化卷积核
文章目录
- 前言
- 基本原理
- 版本和包
- 结果展示
- 简单绘图
- 修改源码绘图
- 方法一
- 方法二（推荐）
- 报错解决
- 总结
前言
基于梯度上升的可视化是一种常用的技术，用于理解卷积神经网络（CNN）中的卷积核是如何对输入图像进行特征提取的。该方法可以通过最大化卷积层输出的激活值来生成图像，从而使得卷积核对特定特征更加敏感。
基本原理
以下是基于梯度上升可视化卷积核的基本原理步骤：
- 选择目标卷积层和卷积核：首先，选择你希望可视化的目标卷积层和对应的卷积核。通常选择靠近网络顶部的卷积层，因为这些层对于更高级别的特征有更强的响应。
- 定义损失函数：为了最大化卷积层的输出激活值，需要定义一个损失函数。通常使用该卷积层的平均激活值作为损失函数。
- 随机生成图像：从随机噪声图像开始，作为初始输入图像。
- 前向传播：将生成的图像输入到CNN网络中，并进行前向传播，记录目标卷积层的输出激活值。
- 反向传播：计算损失函数对于输入图像的梯度，即目标卷积层输出激活值对于输入图像的影响。
- 更新输入图像：根据梯度信息更新输入图像，使得激活值增大。
- 重复步骤4-6：重复进行前向传播、反向传播和图像更新，直到达到预定的迭代次数或满足特定的停止条件。
- 可视化结果：最终生成的图像即为可视化结果，它代表了卷积核对于特定特征的响应模式。
通过这个过程，可以逐步调整输入图像，使得卷积核对于特定特征更加敏感。这种方法可以帮助我们理解卷积神经网络学习到的特征，以及卷积层如何对输入图像进行处理。
版本和包
python=3.7
torch=1.7
flashtorch=0.1.3
其中flashtorch就是用来求取中间层梯度上升的包
结果展示
简单绘图
官方代码如下所示：
（以vgg16为例子）
import torchvision.models as models
from flashtorch.activmax import GradientAscent
model = models.vgg16(pretrained=True)
g_ascent = GradientAscent(model.features)
# specify layer and filter info
conv5_1 = model.features[24]
conv5_1_filters = [45, 271, ]## 这里调节要展示的卷积核（绘出子图的个数），最大值是对应权重的通道数
vis = g_ascent.visualize(conv5_1, conv5_1_filters, title="VGG16: conv5_1",return_output=True)
以上代码，用vis即可画图。
vis =【output45,output271】的列表，每个都包含很多层，源代码是用的最后一层进行绘图（下图为源码截图）：
修改源码绘图
方法一
由于官方代码画不出来图，所以修改from flashtorch.activmax import GradientAscent 的源码，把plt返回即可：
找到GradientAscent函数（gradient_ascent.py) 第211行
修改为：
if return_output:
return self.output, self.plt
方法二（推荐）
最好的方法是：在同级目录下新建文件grad_as.py
直接把以下代码复制到grad_as.py里：
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
from flashtorch.utils import (apply_transforms,
format_for_plotting,
load_image,
standardize_and_clip)
class GradientAscent:
"""Provides an interface for activation maximization via gradient descent.
This class implements the gradient ascent algorithm in order to perform
activation maximization with convolutional neural networks (CNN).
`Activation maximization <https://pdfs.semanticscholar.org/65d9/94fb778a8d9e0f632659fb33a082949a50d3.pdf>`_
is one form of feature visualization that allows us to visualize what CNN
filters are "looking for", by applying each filter to an input image and
updating the input image so as to maximize the activation of the filter of
interest (i.e. treating it as a gradient ascent task with activation as the
loss). The implementation is inspired by `this demo <https://blog.keras.io/category/demo.html>`_
by Francois Chollet.
Args:
model: A neural network model from `torchvision.models
<https://pytorch.org/docs/stable/torchvision/models.html>`_,
typically without the fully-connected part of the network.
e.g. torchvisions.alexnet(pretrained=True).features
img_size (int, optional, default=224): The size of an input image to be
optimized.
lr (float, optional, default=1.): The step size (or learning rate) of
the gradient ascent.
use_gpu (bool, optional, default=False): Use GPU if set to True and
`torch.cuda.is_available()`.
""" # noqa
####################
# Public interface #
####################
def __init__(self, model, img_size=224, lr=1., use_gpu=False):
self.model = model
self._img_size = img_size
self._lr = lr
self._use_gpu = use_gpu
self.num_layers = len(list(self.model.named_children()))
self.activation = None
self.gradients = None
self.handlers = []
self.output = None
self.plt = None
@property
def lr(self):
return self._lr
@lr.setter
def lr(self, lr):
self._lr = lr
@property
def img_size(self):
return self._img_size
@img_size.setter
def img_size(self, img_size):
self._img_size = img_size
@property
def use_gpu(self):
return self._use_gpu
@use_gpu.setter
def use_gpu(self, use_gpu):
self._use_gpu = use_gpu
def optimize(self, layer, filter_idx, input_=None, num_iter=30):
"""Generates an image that maximally activates the target filter.
Args:
layer (torch.nn.modules.conv.Conv2d): The target Conv2d layer from
which the filter to be chosen, based on `filter_idx`.
filter_idx (int): The index of the target filter.
num_iter (int, optional, default=30): The number of iteration for
the gradient ascent operation.
Returns:
output (list of torch.Tensor): With dimentions
:math:`(num_iter, C, H, W)`. The size of the image is
determined by `img_size` attribute which defaults to 224.
"""
# Validate the type of the layer
if type(layer) != nn.modules.conv.Conv2d:
raise TypeError('The layer must be nn.modules.conv.Conv2d.')
# Validate filter index
num_total_filters = layer.out_channels
self._validate_filter_idx(num_total_filters, filter_idx)
# Inisialize input (as noise) if not provided
if input_ is None:
input_ = np.uint8(np.random.uniform(
150, 180, (self._img_size, self._img_size, 3)))
input_ = apply_transforms(input_, size=self._img_size)
if torch.cuda.is_available() and self.use_gpu:
self.model = self.model.to('cuda')
input_ = input_.to('cuda')
# Remove previous hooks if any
while len(self.handlers) > 0:
self.handlers.pop().remove()
# Register hooks to record activation and gradients
self.handlers.append(self._register_forward_hooks(layer, filter_idx))
self.handlers.append(self._register_backward_hooks())
# Inisialize gradients
self.gradients = torch.zeros(input_.shape)
# Optimize
return self._ascent(input_, num_iter)
def visualize(self, layer, filter_idxs=None, lr=1., num_iter=30,
num_subplots=4, figsize=(4, 4), title='Conv2d',
return_output=False):
"""Optimizes for the target layer/filter and visualizes the output.
A method that combines optimization and visualization. There are
mainly 3 types of operations, given a target layer:
1. If `filter_idxs` is provided as an integer, it optimizes for the
filter specified and plots the output.
2. If `filter_idxs` is provided as a list of integers, it optimizes for
all the filters specified and plots the output.
3. if `filter_idx` is not provided, i.e. None, it randomly chooses
`num_subplots` number of filters from the layer provided and
plots the output.
It also returns the output of the optimization, if specified with
`return_output=True`.
Args:
layer (torch.nn.modules.conv.Conv2d): The target Conv2d layer from
which the filter to be chosen, based on `filter_idx`.
filter_idxs (int or list of int, optional, default=None): The index
or indecies of the target filter(s).
lr (float, optional, default=.1): The step size of optimization.
num_iter (int, optional, default=30): The number of iteration for
the gradient ascent operation.
num_subplots (int, optional, default=4): The number of filters to
optimize for and visualize. Relevant in case 3 above.
figsize (tuple, optional, default=(4, 4)): The size of the plot.
Relevant in case 1 above.
title (str, optional default='Conv2d'): The title of the plot.
return_output (bool, optional, default=False): Returns the
output(s) of optimization if set to True.
Returns:
For a single optimization (i.e. case 1 above):
output (list of torch.Tensor): With dimentions
:math:`(num_iter, C, H, W)`. The size of the image is
determined by `img_size` attribute which defaults to 224.
For multiple optimization (i.e. case 2 or 3 above):
output (list of list of torch.Tensor): With dimentions
:math:`(num_subplots, num_iter, C, H, W)`. The size of the
image is determined by `img_size` attribute which defaults
to 224.
"""
self._lr = lr
if (type(filter_idxs) == int):
self._visualize_filter(layer,
filter_idxs,
num_iter=num_iter,
figsize=figsize,
title=title)
else:
num_total_filters = layer.out_channels
if filter_idxs is None:
num_subplots = min(num_total_filters, num_subplots)
filter_idxs = np.random.choice(range(num_total_filters),
size=num_subplots)
self._visualize_filters(layer,
filter_idxs,
num_iter,
len(filter_idxs),
title=title)
if return_output:
return self.output, self.plt
def deepdream(self, img_path, layer, filter_idx, lr=.1, num_iter=20,
figsize=(4, 4), title='DeepDream', return_output=False):
"""Creates DeepDream.
It applies the optimization on the image provided. The image is loaded
and made into a torch.Tensor that is compatible as the input to the
network.
Read the original blog post by Google for more information on
`DeepDream <https://ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html>`_.
Args:
img_path (str): A path to the image you want to apply DeepDream on
layer (torch.nn.modules.conv.Conv2d): The target Conv2d layer from
which the filter to be chosen, based on `filter_idx`.
filter_idx (int): The index of the target filter.
lr (float, optional, default=.1): The step size of optimization.
num_iter (int, optional, default=30): The number of iteration for
the gradient ascent operation.
figsize (tuple, optional, default=(4, 4)): The size of the plot.
Relevant in case 1 above.
title (str, optional default='Conv2d'): The title of the plot.
return_output (bool, optional, default=False): Returns the
output(s) of optimization if set to True.
Returns:
output (list of torch.Tensor): With dimentions
:math:`(num_iter, C, H, W)`. The size of the image is
determined by `img_size` attribute which defaults to 224.
""" # noqa
input_ = apply_transforms(load_image(img_path), self.img_size)
self._lr = lr
output = self.optimize(layer, filter_idx, input_, num_iter=num_iter)
plt.figure(figsize=figsize)
plt.axis('off')
plt.title(title)
plt.imshow(format_for_plotting(
standardize_and_clip(output[-1],
saturation=0.15,
brightness=0.7))); # noqa
if return_output:
return output
#####################
# Private interface #
#####################
def _register_forward_hooks(self, layer, filter_idx):
def _record_activation(module, input_, output):
self.activation = torch.mean(output[:, filter_idx, :, :])
return layer.register_forward_hook(_record_activation)
def _register_backward_hooks(self):
def _record_gradients(module, grad_in, grad_out):
if self.gradients.shape == grad_in[0].shape:
self.gradients = grad_in[0]
for _, module in self.model.named_modules():
if isinstance(module, nn.modules.conv.Conv2d) and \
module.in_channels == 3:
return module.register_backward_hook(_record_gradients)
def _ascent(self, x, num_iter):
output = []
for i in range(num_iter):
self.model(x)
self.activation.backward()
self.gradients /= (torch.sqrt(torch.mean(
torch.mul(self.gradients, self.gradients))) + 1e-5)
x = x + self.gradients * self._lr
output.append(x)
return output
def _validate_filter_idx(self, num_filters, filter_idx):
if not np.issubdtype(type(filter_idx), np.integer):
raise TypeError('Indecies must be integers.')
elif (filter_idx < 0) or (filter_idx > num_filters):
raise ValueError(f'Filter index must be between 0 and \
{num_filters - 1}.')
def _visualize_filter(self, layer, filter_idx, num_iter, figsize, title):
self.output = self.optimize(layer, filter_idx, num_iter=num_iter)
plt.figure(figsize=figsize)
plt.axis('off')
plt.title(title)
plt.imshow(format_for_plotting(
standardize_and_clip(self.output[-1],
saturation=0.15,
brightness=0.7))); # noqa
def _visualize_filters(self, layer, filter_idxs, num_iter, num_subplots,
title):
# Prepare the main plot
num_cols = 4
num_rows = int(np.ceil(num_subplots / num_cols))
fig = plt.figure(figsize=(16, num_rows * 5))
plt.title(title)
plt.axis('off')
self.output = []
# Plot subplots
for i, filter_idx in enumerate(filter_idxs):
output = self.optimize(layer, filter_idx, num_iter=num_iter)
self.output.append(output)
ax = fig.add_subplot(num_rows, num_cols, i+1)
ax.set_xticks([])
ax.set_yticks([])
ax.set_title(f'filter {filter_idx}')
ax.imshow(format_for_plotting(
standardize_and_clip(output[-1],
saturation=0.15,
brightness=0.7)))
plt.subplots_adjust(wspace=0, hspace=0); # noqa
self.plt = plt```
然后使用以下代码绘图：
import torchvision.models as models
from grad_as import GradientAscent
model = models.vgg16(pretrained=True) # 这里可以改成其它模型
g_ascent = GradientAscent(model.features)
# specify layer and filter info
conv5_1 = model.features[24] # 相应的这里也要改，可以用print(model)代码查看模型结构进行修改
conv5_1_filters = [45, 271, ]
vis,plt = g_ascent.visualize(conv5_1, conv5_1_filters, title="VGG16: conv5_1",return_output=True)
plt.show()
报错解决
报错如下
from importlib_resources import path
ImportError: cannot import name 'path' from 'importlib_resources'
找到报错处的源文件，尝试在代码中手动导入 path，而不是从 importlib_resources 直接导入：
from importlib_resources import path
替换为：
from importlib_resources import files
def path(package, resource):
return files(package).joinpath(resource)
这样，代码将使用 files 函数来导入 path，而不是直接从 importlib_resources 中导入。
总结
以上代码均经过本人亲测可用。
