信息门户网站建设合同,如何制作app软件赚钱,企业网络营销的模式有哪些,wordpress百家号采集在Transformer模型中#xff0c;自注意力#xff08;Self-Attention#xff09;是核心组件#xff0c;用于捕捉输入序列中不同位置之间的关系。自注意力机制通过计算每个标记与其他所有标记之间的注意力权重#xff0c;然后根据这些权重对输入序列进行加权求和#xff0c…在Transformer模型中自注意力Self-Attention是核心组件用于捕捉输入序列中不同位置之间的关系。自注意力机制通过计算每个标记与其他所有标记之间的注意力权重然后根据这些权重对输入序列进行加权求和从而生成新的表示。下面是实现自注意力机制的代码及其详细说明。
自注意力机制的实现
1. 计算注意力得分Scaled Dot-Product Attention
自注意力机制的基本步骤包括以下几个部分
线性变换将输入序列通过三个不同的线性变换层得到查询Query、键Key和值Value矩阵。计算注意力得分通过点积计算查询与键的相似度再除以一个缩放因子通常是键的维度的平方根以稳定梯度。应用掩码在计算注意力得分后应用掩码如果有避免未来信息泄露用于解码器中的自注意力。计算注意力权重通过softmax函数将注意力得分转换为概率分布。加权求和使用注意力权重对值进行加权求和得到新的表示。
2. 多头注意力机制Multi-Head Attention
为了捕捉不同子空间的特征Transformer使用多头注意力机制。通过将查询、键和值分割成多个头每个头独立地计算注意力然后将所有头的输出连接起来并通过一个线性层进行组合。
自注意力机制代码实现
import torch
import torch.nn as nn
import torch.nn.functional as F# Scaled Dot-Product Attention
def scaled_dot_product_attention(query, key, value, maskNone):d_k query.size(-1)scores torch.matmul(query, key.transpose(-2, -1)) / torch.sqrt(torch.tensor(d_k, dtypetorch.float32))print(fScores shape: {scores.shape}) # (batch_size, num_heads, seq_length, seq_length)if mask is not None:scores scores.masked_fill(mask 0, float(-inf))attention_weights F.softmax(scores, dim-1)print(fAttention weights shape: {attention_weights.shape}) # (batch_size, num_heads, seq_length, seq_length)output torch.matmul(attention_weights, value)print(fOutput shape after attention: {output.shape}) # (batch_size, num_heads, seq_length, d_k)return output, attention_weights# Multi-Head Attention
class MultiHeadAttention(nn.Module):def __init__(self, d_model, num_heads):super(MultiHeadAttention, self).__init__()assert d_model % num_heads 0self.d_model d_modelself.num_heads num_headsself.d_k d_model // num_headsself.linear_query nn.Linear(d_model, d_model)self.linear_key nn.Linear(d_model, d_model)self.linear_value nn.Linear(d_model, d_model)self.linear_out nn.Linear(d_model, d_model)def forward(self, query, key, value, maskNone):batch_size query.size(0)# Linear projectionsquery self.linear_query(query)key self.linear_key(key)value self.linear_value(value)print(fQuery shape after linear: {query.shape}) # (batch_size, seq_length, d_model)print(fKey shape after linear: {key.shape}) # (batch_size, seq_length, d_model)print(fValue shape after linear: {value.shape}) # (batch_size, seq_length, d_model)# Split into num_headsquery query.view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)key key.view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)value value.view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)print(fQuery shape after split: {query.shape}) # (batch_size, num_heads, seq_length, d_k)print(fKey shape after split: {key.shape}) # (batch_size, num_heads, seq_length, d_k)print(fValue shape after split: {value.shape}) # (batch_size, num_heads, seq_length, d_k)# Apply scaled dot-product attentionx, attention_weights scaled_dot_product_attention(query, key, value, mask)# Concatenate headsx x.transpose(1, 2).contiguous().view(batch_size, -1, self.d_model)print(fOutput shape after concatenation: {x.shape}) # (batch_size, seq_length, d_model)# Final linear layerx self.linear_out(x)print(fOutput shape after final linear: {x.shape}) # (batch_size, seq_length, d_model)return x, attention_weights# 示例用法
d_model 512
num_heads 8
batch_size 64
seq_length 10# 假设输入是随机生成的张量
query torch.rand(batch_size, seq_length, d_model)
key torch.rand(batch_size, seq_length, d_model)
value torch.rand(batch_size, seq_length, d_model)# 创建多头注意力层
mha MultiHeadAttention(d_model, num_heads)
output, attention_weights mha(query, key, value)print(最终输出形状:, output.shape) # 最终输出形状: (batch_size, seq_length, d_model)
print(注意力权重形状:, attention_weights.shape) # 注意力权重形状: (batch_size, num_heads, seq_length, seq_length)每一步的形状解释 Linear Projections Query, Key, Value分别经过线性变换。形状[batch_size, seq_length, d_model] Split into Heads 将Query, Key, Value分割成多个头。形状[batch_size, num_heads, seq_length, d_k]其中d_k d_model // num_heads Scaled Dot-Product Attention 计算注意力得分Scores。形状[batch_size, num_heads, seq_length, seq_length]计算注意力权重Attention Weights。形状[batch_size, num_heads, seq_length, seq_length]使用注意力权重对Value进行加权求和。形状[batch_size, num_heads, seq_length, d_k] Concatenate Heads 将所有头的输出连接起来。形状[batch_size, seq_length, d_model] Final Linear Layer 通过一个线性层将连接的输出转换为最终的输出。形状[batch_size, seq_length, d_model]
通过这种方式我们可以清楚地看到每一步变换后的张量形状理解自注意力和多头注意力机制的具体实现细节。
代码说明
scaled_dot_product_attention实现了缩放点积注意力机制计算查询和键的点积应用掩码计算softmax然后使用权重对值进行加权求和。MultiHeadAttention实现了多头注意力机制包括线性变换、分割、缩放点积注意力和最后的线性变换。
多头注意力机制的细节
线性变换将输入序列通过线性层转换为查询、键和值的矩阵。分割头将查询、键和值的矩阵分割为多个头每个头的维度是[batch_size, num_heads, seq_length, d_k]。缩放点积注意力对每个头分别计算缩放点积注意力。连接头将所有头的输出连接起来得到[batch_size, seq_length, d_model]的张量。线性变换通过一个线性层将连接的输出转换为最终的输出。
