最近，看到很多轻量化工作是基于RepVGG改进而来，决定重新回顾一下RepVGG，并在此记录一些理解与心得。

论文地址：https://arxiv.org/abs/2101.03697

Introduction

RepVGG通过结构重参数化思想，让训练网络的多路结构(多分支模型训练时的优势——性能高)转换为推理网络的单路结构(模型推理时的好处——速度快、省内存），从而达到推理速度快与模型性能高兼备的效果。

Method

根据Resnet与Inception等论文论证，与单路直筒结构相比，增加shortcut与muti-branch可以提高模型的性能，RepVGG在VGG的基础上做了相关实验，从下表中可以发现，Identity branch与1x1 branch能够提高准确率，但同时推理时间也延长了许多。

为了在保证模型精度的基础上，降低模型推理时间，RepVGG采用了结构重构的方法，即在训练时使用复杂结构，而测试时将复杂模型重构成单路直筒型模型，如下图所示。图B中，RepVGG在training时加入了shortcut和1x1 branch，而在test时，RepVGG重构成直筒型VGG模型，如图c。

Re-param for Plain Inference-time Model
如下图所示，我们的目的是将带有BN层的3x3卷积与1x1卷积以及Identity mapping融合成一个3x3卷积，如何融合呢？操作分为两部：1.融合conv与bn；2.合并卷积。

1.Merge BN and Conv together

2. Fuse 3x3conv 1x1conv and identity

假设输入特征尺寸为（1x2x3x3），输出特征尺寸为输入相同，stride=1，那么conv3x3的卷积过程如下图所示，conv的特征尺寸为（2x2x3x3），首先对输入扩边，padding=kernel_size // 2，然后从左上角开始滑动窗口，最后获得右边的输出特征。

conv_1x1的卷积相较于3x3卷积更为简单，只需要将卷积参数与对应的单个输入参数相乘再相加。

为了使conv_1x1可以与conv_3x3线性相加，我们可以将conv_1x1的1x1卷积核扩边成3x3的卷积核，这样形式上就可以变成conv_3x3，同时，结果与conv_1x1一致。

同样，Identity mapping可以看作特殊的conv_1x1，参数是固定的，即输入特征的某一层对应的卷积参数为1，其余均为0。我们可以按照上面介绍的conv_1x1与conv_3x3转化方式，将Identity mapping转化成conv_3x3的形式，如下图所示。

如此，我们就可以将带有BN层的3x3卷积与1x1卷积以及Identity mapping融合成一个3x3卷积。

code

我们看一下代码如何实现RepVGG，首先，我们看到deploy的控制Flag，当depoly为False时，模型处于training状态，结构没有重参化，当deploy为True时，模型处于test状态，需要对结构重参。

结构重参时，需要调用switch_to_deploy()，该函数的作用是调用self.get_equivalent_kernel_bias()获得重构后的kernel, bias， 对重构卷积赋值–self.rbr_reparam = nn.Conv2d，其中Conv2d的参数是kernel，偏移是bias，将参数detach脱离计算图，并删除其余卷积操作内存（否则无法正常运行）。

get_equivalent_kernel_bias(self):逻辑很简单，分为两部：1.调用self._fuse_bn_tensor()将conv与bn融合。需要注意的是，identity没有conv，只有BN操作，所以在self._fuse_bn_tensor()函数中需要构造卷积kernel，kernel_value[i, i % input_dim, 1, 1] = 1，需要注意分组卷积的形式。函数的返回值 return kernel3x3 + self._pad_1x1_to_3x3_tensor(kernel1x1) + kernelid, bias3x3 + bias1x1 + biasid

class RepVGGBlock(nn.Module):
    '''RepVGGBlock is a basic rep-style block, including training and deploy status
    This code is based on https://github.com/DingXiaoH/RepVGG/blob/main/repvgg.py
    '''
    def __init__(self, in_channels, out_channels, kernel_size=3,
                 stride=1, padding=1, dilation=1, groups=1, padding_mode='zeros', deploy=False, use_se=False):
        super(RepVGGBlock, self).__init__()
        """ Initialization of the class.
        Args:
            in_channels (int): Number of channels in the input image
            out_channels (int): Number of channels produced by the convolution
            kernel_size (int or tuple): Size of the convolving kernel
            stride (int or tuple, optional): Stride of the convolution. Default: 1
            padding (int or tuple, optional): Zero-padding added to both sides of
                the input. Default: 1
            dilation (int or tuple, optional): Spacing between kernel elements. Default: 1
            groups (int, optional): Number of blocked connections from input
                channels to output channels. Default: 1
            padding_mode (string, optional): Default: 'zeros'
            deploy: Whether to be deploy status or training status. Default: False
            use_se: Whether to use se. Default: False
        """
        self.deploy = deploy
        self.groups = groups
        self.in_channels = in_channels
        self.out_channels = out_channels

        assert kernel_size == 3
        assert padding == 1

        padding_11 = padding - kernel_size // 2

        self.nonlinearity = nn.ReLU()

        if use_se:
            raise NotImplementedError("se block not supported yet")
        else:
            self.se = nn.Identity()

        if deploy:
            self.rbr_reparam = nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, stride=stride,
                                         padding=padding, dilation=dilation, groups=groups, bias=True, padding_mode=padding_mode)

        else:
            self.rbr_identity = nn.BatchNorm2d(num_features=in_channels) if out_channels == in_channels and stride == 1 else None
            self.rbr_dense = conv_bn(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, stride=stride, padding=padding, groups=groups)
            self.rbr_1x1 = conv_bn(in_channels=in_channels, out_channels=out_channels, kernel_size=1, stride=stride, padding=padding_11, groups=groups)

    def forward(self, inputs):
        '''Forward process'''
        if hasattr(self, 'rbr_reparam'):
            return self.nonlinearity(self.se(self.rbr_reparam(inputs)))

        if self.rbr_identity is None:
            id_out = 0
        else:
            id_out = self.rbr_identity(inputs)

        return self.nonlinearity(self.se(self.rbr_dense(inputs) + self.rbr_1x1(inputs) + id_out))

    def get_equivalent_kernel_bias(self):
        kernel3x3, bias3x3 = self._fuse_bn_tensor(self.rbr_dense)
        kernel1x1, bias1x1 = self._fuse_bn_tensor(self.rbr_1x1)
        kernelid, biasid = self._fuse_bn_tensor(self.rbr_identity)
        return kernel3x3 + self._pad_1x1_to_3x3_tensor(kernel1x1) + kernelid, bias3x3 + bias1x1 + biasid

    def _pad_1x1_to_3x3_tensor(self, kernel1x1):
        if kernel1x1 is None:
            return 0
        else:
            return torch.nn.functional.pad(kernel1x1, [1, 1, 1, 1])

    def _fuse_bn_tensor(self, branch):
        if branch is None:
            return 0, 0
        if isinstance(branch, nn.Sequential):
            kernel = branch.conv.weight
            running_mean = branch.bn.running_mean
            running_var = branch.bn.running_var
            gamma = branch.bn.weight
            beta = branch.bn.bias
            eps = branch.bn.eps
        else:
            assert isinstance(branch, nn.BatchNorm2d)
            if not hasattr(self, 'id_tensor'):
                input_dim = self.in_channels // self.groups
                kernel_value = np.zeros((self.in_channels, input_dim, 3, 3), dtype=np.float32)
                for i in range(self.in_channels):
                    kernel_value[i, i % input_dim, 1, 1] = 1
                self.id_tensor = torch.from_numpy(kernel_value).to(branch.weight.device)
            kernel = self.id_tensor
            running_mean = branch.running_mean
            running_var = branch.running_var
            gamma = branch.weight
            beta = branch.bias
            eps = branch.eps
        std = (running_var + eps).sqrt()
        t = (gamma / std).reshape(-1, 1, 1, 1)
        return kernel * t, beta - running_mean * gamma / std

    def switch_to_deploy(self):
        if hasattr(self, 'rbr_reparam'):
            return
        kernel, bias = self.get_equivalent_kernel_bias()
        self.rbr_reparam = nn.Conv2d(in_channels=self.rbr_dense.conv.in_channels, out_channels=self.rbr_dense.conv.out_channels,
                                     kernel_size=self.rbr_dense.conv.kernel_size, stride=self.rbr_dense.conv.stride,
                                     padding=self.rbr_dense.conv.padding, dilation=self.rbr_dense.conv.dilation, groups=self.rbr_dense.conv.groups, bias=True)
        self.rbr_reparam.weight.data = kernel
        self.rbr_reparam.bias.data = bias
        for para in self.parameters():
            para.detach_()
        self.__delattr__('rbr_dense')
        self.__delattr__('rbr_1x1')
        if hasattr(self, 'rbr_identity'):
            self.__delattr__('rbr_identity')
        if hasattr(self, 'id_tensor'):
            self.__delattr__('id_tensor')
        self.deploy = True

Drawback

在RepVGG中存在一个致命的缺点，我们知道RepVGG是为了轻量化而生，但是结构重参导致模型参数方差较大，从而引起量化误差，实验证明，RepVGG通过训练后量化会将准确率降低20%，与此同时，由于特殊的training与test结构差异，RepVGG很难进行感知量化训练。为了对量化友好，Repopt提出了很好的思路解决这个问题，我们下篇继续介绍Repopt。