13.编码器的结构

news2025/4/22 14:26:16

从入门AI到手写Transformer-13.编码器的结构

13.编码器的结构
代码

整理自视频老袁不说话。

13.编码器的结构

$T r an s f or m er E n co d er : 输入 [b, n]$

$E mb e dd in g : - > [b, n, d]$
$P os i t i o na lE n co d er : - > [b, n, d]$
$Dro p o u t : - > [b, n, d]$
$E n co d er Bl oc k : [b, n, d] - > [b, n, d]$ 重复N次
- $M u lt ih e a d A tt e n t i o n : 3 * [b, n, d] - > [b, n, d]$
- $Dro p o u t : [b, n, d] - > [b, n, d]$
- $A dd N or m : 2 * [b, n, d] (Dro u p o u t 输出， M u lt ih e a d A tt e n t i o n 输入) - > [b, n, d]$
- $FFN : [b, n, d] - > [b, n, d]$
- $Dro p o u t : [b, n, d] - > [b, n, d]$
- $A dd N or m : 2 * [b, n, d] (Dro u p o u t 输出， FFN 输入) - > [b, n, d]$
  
  编码器结构
  
  多处执行Dropout

代码

import torch.nn as nn

class Embedding(nn.Module):
    def __init__(self,*args,**kwargs)->None:
        super().__init__(*args,**kwargs)
    def forward(self):
        print(self.__class__.__name__)
class PositionalEncoding(nn.Module):
    def __init__(self,*args,**kwargs)->None:
        super().__init__(*args,**kwargs)
    def forward(self):
        print(self.__class__.__name__)
class MultiheadAttention(nn.Module):
    def __init__(self,*args,**kwargs)->None:
        super().__init__(*args,**kwargs)
    def forward(self):
        print(self.__class__.__name__)
class Dropout(nn.Module):
    def __init__(self,*args,**kwargs)->None:
        super().__init__(*args,**kwargs)
    def forward(self):
        print(self.__class__.__name__)
class AddNorm(nn.Module):
    def __init__(self,*args,**kwargs)->None:
        super().__init__(*args,**kwargs)
    def forward(self):
        print(self.__class__.__name__)
class FFN(nn.Module):
    def __init__(self,*args,**kwargs)->None:
        super().__init__(*args,**kwargs)
    def forward(self):
        print(self.__class__.__name__)

class EncoderBlock(nn.Module):
    def __init__(self,*args, **kwargs)->None:
        super().__init__(*args,**kwargs)
        self.mha = MultiheadAttention()
        self.dropout1=Dropout()
        self.addnorm1=AddNorm()
        self.ffn=FFN()
        self.dropout2=Dropout()
        self.addnorm2 = AddNorm()
    def forward(self):
        self.mha()
        self.dropout1()
        self.addnorm1()
        self.ffn()
        self.dropout2()
        self.addnorm2()

class TransformerEncoder(nn.Module):
    def __init__(self,*args,**kwargs)->None:
        super().__init__(*args,**kwargs)
        self.embedding=Embedding() # 把序号转变为有语义信息的编码
        self.posenc=PositionalEncoding()
        self.dropout=Dropout()
        self.encblocks=nn.Sequential()
        for i in range(3):
            self.encblocks.add_module(str(i),EncoderBlock())
    def forward(self):
        self.embedding()
        self.posenc()
        self.dropout()
        for i,blk in enumerate(self.encblocks):
            print(i)
            blk()

te=TransformerEncoder()
te()

输出结果

Embedding
PositionalEncoding
Dropout
0
MultiheadAttention
Dropout
AddNorm
FFN
Dropout
AddNorm
1
MultiheadAttention
Dropout
AddNorm
FFN
Dropout
AddNorm
2
MultiheadAttention
Dropout
AddNorm
FFN
Dropout
AddNorm

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/2340159.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！