flash网站做seo优化好不好,成都网站推广哪家专业,华中农业大学基因编辑在线设计网站,如何申请网页域名文章目录 1. 背景介绍2. 数据加载和预处理3. 模型结构4. Loss计算5. 总结和讨论 1. 背景介绍
梳理了PersFormer 3D Lane这篇论文对应的开源代码。
2. 数据加载和预处理
数据组织方式参考#xff1a;自动驾驶三维车道线检测系列—OpenLane数据集介绍。
坐标系参考#xff… 文章目录 1. 背景介绍2. 数据加载和预处理3. 模型结构4. Loss计算5. 总结和讨论 1. 背景介绍
梳理了PersFormer 3D Lane这篇论文对应的开源代码。
2. 数据加载和预处理
数据组织方式参考自动驾驶三维车道线检测系列—OpenLane数据集介绍。
坐标系参考WaymoOpenLane的相机坐标系定义为X轴向前Y轴向左Z轴向上。
所以目前数据集中extrinsic如下所示 extrinsic: [[0.9999790376291948,0.0045643569339703614,-0.004592488211072773,1.5393679168564702],[-0.004561049760530192,0.9999893316554864,0.0007303425877435168,-0.023691101834458727],[0.004595772761080904,-0.0007093807107760264,0.9999891877674156,2.116014704116658],[0.0,0.0,0.0,1.0]]表示相机与车体坐标系之间的外参。证明当前相机坐标系与车体坐标系基本一样只不过坐标系原点不同。
真值中三维车道线坐标如下 xyz: [[17.382800978819017,17.50040078037587,...20.274071148194103,20.38195255123586,20.48338473286711,...21.931553979558124,22.035070660270055,22.225836627658275,...26.61997710857102,26.79194824101369,26.900410829029237,27.120094997627227,27.373557832034134,27.544735503944892,27.76517221715751,...34.04364108074635,34.34183952795403,34.44990326517201,34.64902798842567,34.80330435386684,34.97738513466869,35.10772807701519,35.21127993338897,...43.26603227123105,43.394400283321744,43.54402499844422],[2.106162267265404,2.117356716888622,2.1267196912979487,2.134409778081357,...2.0836507579014345,2.083507611187606,2.0835073709153624,2.0840386385354006,2.084208834336426,2.0839706956475896,...2.041385347080901,2.0409709291451565,2.040427413456658,2.039615358123066,...2.0134828598759036,2.0115237025717696,2.008972154232112],[-2.3179790375385325,-2.3071957428128735,...-2.6387563474505975,-2.6083380476462663,-2.615133499516616,-2.6078418903876996,...-2.8227286818859705,-2.7188114534133323,-2.7218183933456475,-2.7215278064851747,...-2.8628242116626312,-2.8972871689102906,-2.9289600110699885]],可以看出三维车道线真值的坐标系是在Camera坐标系之下的表示也就是Z值在相机坐标系的地面处。
初始化 超参数设定H_crop计算H_ipm2g计算H_g2ipm计算Anchor grid计算Anchor dim计算初始化OpenLane数据集 输入数据集文件夹输出X, Y和Z的offset以及image anchors算法内容 获取所有的.json文件即真值文件调整外参从Waymo车体坐标系变换到ground坐标系。此处重点解释下Waymo相机坐标系的定义与正常相机坐标系定义不一样Waymo是X轴朝前Y轴朝左Z轴向上。此时车体坐标系也是如此定义的。所以数据集中默认的外参旋转近似单位阵。而坐标系的原点不一样而已。而ground坐标系的定义是Y轴朝前X轴朝右Z轴向上。此外三维车道线结果是在Waymo相机坐标系下的。将三维车道线坐标从相机坐标系变换到ground坐标系将三维车道线坐标从ground坐标系变换到flat ground坐标系下计算offset作为预测回归的结果 模型构建模型训练数据加载 看起来模型训练数据加载时又做了很多上述数据初始化很多一样的步骤加载图像数据根据resize参数调整图像尺寸对图像进行归一化处理数据增强图像旋转以及Anchor旋转计算segmentation所需要的label直接利用车道线三维真值投影回图像得到segmentation的label在IPM上绘制三维车道线真值数据得到Seg Bev数据用于后续模型训练
最终数据处理这层为下游提供的数据包含如下
图像归一化后的数据Segmentation labelGround-truth lane (anchor)Ground-truth lane 2DCamera heightCamera pitch内参外参ground to camera数据增强矩阵Segmentation Bev map
3. 模型结构
模型整体框图如下图所示
Encoder Backbone选用ResNet输出特征listout_featListNeck选用一系列的二维卷积、BN和ReLU输出特征listneck_outShareEncoderFrontViewPathway输出frontview_features取frontview_features的最后一层特征frontview_final_feat执行Lane 2D的Attention输出laneatt_proposals_list PerspectiveTransformer输入包括原图、frontview_features和变换矩阵M 获取BEV中的2D网格将BEV网格利用M矩阵投影到IPMTransformer encoder: Self-attn - norm - cross-attn - norm - ffn - norm Query是query_embedBEV视角Value是frontview_featuresBev 2D gridIPM 2D grid首先BEV得self-attention之后是BEV与IPM和frontview_features的cross attention最后norm等操作输出feature 输出特征projs Bev Head 输入特征projs输出特征bev_feature算法内容 SingleTopViewPathwaySingleTopViewPathway经过一系列卷积操作 LanePredictionHead 输入特征bev_feature输出处理后的特征bev_feature算法内容 一系列卷积操作将张量 x 中特定的维度范围应用 Sigmoid 函数以便将这些值的值域从任意实数映射到 (0, 1) 区间内。这通常用于处理模型输出中的可见性或置信度分数使得这些分数可以被解释为概率。 模型最后输出Laneatt_proposals_list, out, cam_height, cam_pitch, pred_seg_bev_map, uncertainty_loss
4. Loss计算
函数调用
# 3D loss
loss_3d, loss_3d_dict criterion(output_net, gt, pred_hcam, gt_hcam, pred_pitch, gt_pitch)
# Add lane attion loss
loss_att, loss_att_dict model.module.laneatt_head.loss(laneatt_proposals_list, gt_laneline_img,cls_loss_weightargs.cls_loss_weight,reg_vis_loss_weightargs.reg_vis_loss_weight)loss_seg bceloss(pred_seg_bev_map, seg_bev_map)
# overall loss
loss self.compute_loss(args, epoch, loss_3d, loss_att, loss_seg, uncertainty_loss, loss_3d_dict, loss_att_dict)3D loss
class Laneline_loss_gflat_3D(nn.Module):Compute the loss between predicted lanelines and ground-truth laneline in anchor representation.The anchor representation is in flat ground space X, Y and real 3D Z. Visibility estimation is also included.The X Y and Z estimation will be transformed to real X, Y to compare with ground truth. An additional loss inX, Y space is expected to guide the learning of features to satisfy the geometry constraints between two spacesloss loss0 loss1 loss2 loss2loss0: cross entropy loss for lane point visibilityloss1: cross entropy loss for lane type classificationloss2: sum of geometric distance betwen 3D lane anchor points in X and Z offsetsloss3: error in estimating pitch and camera heightsdef __init__(self, batch_size, num_types, anchor_x_steps, anchor_y_steps, x_off_std, y_off_std, z_std, pred_camFalse, no_cudaFalse):super(Laneline_loss_gflat_3D, self).__init__()self.batch_size batch_sizeself.num_types num_typesself.num_x_steps anchor_x_steps.shape[0]self.num_y_steps anchor_y_steps.shape[0]self.anchor_dim 3*self.num_y_steps 1self.pred_cam pred_cam# prepare broadcast anchor_x_tensor, anchor_y_tensor, std_X, std_Y, std_Ztmp_zeros torch.zeros(self.batch_size, self.num_x_steps, self.num_types, self.num_y_steps)self.x_off_std torch.tensor(x_off_std.astype(np.float32)).reshape(1, 1, 1, self.num_y_steps) tmp_zerosself.y_off_std torch.tensor(y_off_std.astype(np.float32)).reshape(1, 1, 1, self.num_y_steps) tmp_zerosself.z_std torch.tensor(z_std.astype(np.float32)).reshape(1, 1, 1, self.num_y_steps) tmp_zerosself.anchor_x_tensor torch.tensor(anchor_x_steps.astype(np.float32)).reshape(1, self.num_x_steps, 1, 1) tmp_zerosself.anchor_y_tensor torch.tensor(anchor_y_steps.astype(np.float32)).reshape(1, 1, 1, self.num_y_steps) tmp_zerosself.anchor_x_tensor self.anchor_x_tensor/self.x_off_stdself.anchor_y_tensor self.anchor_y_tensor/self.y_off_stdif not no_cuda:self.z_std self.z_std.cuda()self.anchor_x_tensor self.anchor_x_tensor.cuda()self.anchor_y_tensor self.anchor_y_tensor.cuda()def forward(self, pred_3D_lanes, gt_3D_lanes, pred_hcam, gt_hcam, pred_pitch, gt_pitch)::param pred_3D_lanes: predicted tensor with size N x (ipm_w/8) x 3*(2*K1):param gt_3D_lanes: ground-truth tensor with size N x (ipm_w/8) x 3*(2*K1):param pred_pitch: predicted pitch with size N:param gt_pitch: ground-truth pitch with size N:param pred_hcam: predicted camera height with size N:param gt_hcam: ground-truth camera height with size N:return:sizes pred_3D_lanes.shape# reshape to N x ipm_w/8 x 3 x (3K1)pred_3D_lanes pred_3D_lanes.reshape(sizes[0], sizes[1], self.num_types, self.anchor_dim)gt_3D_lanes gt_3D_lanes.reshape(sizes[0], sizes[1], self.num_types, self.anchor_dim)# class prob N x ipm_w/8 x 3 x 1, anchor values N x ipm_w/8 x 3 x 2K, visibility N x ipm_w/8 x 3 x Kpred_class pred_3D_lanes[:, :, :, -1].unsqueeze(-1)pred_anchors pred_3D_lanes[:, :, :, :2*self.num_y_steps]pred_visibility pred_3D_lanes[:, :, :, 2*self.num_y_steps:3*self.num_y_steps]gt_class gt_3D_lanes[:, :, :, -1].unsqueeze(-1)gt_anchors gt_3D_lanes[:, :, :, :2*self.num_y_steps]gt_visibility gt_3D_lanes[:, :, :, 2*self.num_y_steps:3*self.num_y_steps]# cross-entropy loss for visibilityloss0 -torch.sum(gt_visibility*torch.log(pred_visibility torch.tensor(1e-9)) (torch.ones_like(gt_visibility) - gt_visibility torch.tensor(1e-9)) *torch.log(torch.ones_like(pred_visibility) - pred_visibility torch.tensor(1e-9)))/self.num_y_steps# cross-entropy loss for lane probabilityloss1 -torch.sum(gt_class*torch.log(pred_class torch.tensor(1e-9)) (torch.ones_like(gt_class) - gt_class) *torch.log(torch.ones_like(pred_class) - pred_class torch.tensor(1e-9)))# applying L1 norm does not need to separate X and Zloss2 torch.sum(torch.norm(gt_class*torch.cat((gt_visibility, gt_visibility), 3)*(pred_anchors-gt_anchors), p1, dim3))# compute loss in real 3D X, Y space, the transformation considers offset to anchor and normalization by stdpred_Xoff_g pred_anchors[:, :, :, :self.num_y_steps]pred_Z pred_anchors[:, :, :, self.num_y_steps:2*self.num_y_steps]gt_Xoff_g gt_anchors[:, :, :, :self.num_y_steps]gt_Z gt_anchors[:, :, :, self.num_y_steps:2*self.num_y_steps]pred_hcam pred_hcam.reshape(self.batch_size, 1, 1, 1)gt_hcam gt_hcam.reshape(self.batch_size, 1, 1, 1)pred_Xoff (1 - pred_Z * self.z_std / pred_hcam) * pred_Xoff_g - pred_Z * self.z_std / pred_hcam * self.anchor_x_tensorpred_Yoff -pred_Z * self.z_std / pred_hcam * self.anchor_y_tensorgt_Xoff (1 - gt_Z * self.z_std / gt_hcam) * gt_Xoff_g - gt_Z * self.z_std / gt_hcam * self.anchor_x_tensorgt_Yoff -gt_Z * self.z_std / gt_hcam * self.anchor_y_tensorloss3 torch.sum(torch.norm(gt_class * torch.cat((gt_visibility, gt_visibility), 3) *(torch.cat((pred_Xoff, pred_Yoff), 3) - torch.cat((gt_Xoff, gt_Yoff), 3)), p1, dim3))if not self.pred_cam:return loss0loss1loss2loss3loss4 torch.sum(torch.abs(gt_pitch-pred_pitch)) torch.sum(torch.abs(gt_hcam-pred_hcam))return loss0loss1loss2loss3loss4函数说明 输入预测的3D结果pred_3D_lanes真值gt_3D_lanes预测的Camera高度pred_hcam真值Camera高度gt_hcam预测的pitch pred_pitch和真值的pitch gt_pitch输出3D loss算法内容 计算visibility的loss计算lane probability的loss计算offset的3D loss计算pitch loss
5. 总结和讨论
梳理了PersFormer 3D Lane这篇论文对应的开源代码。
