Skip to content
The Chat Archive
gpt-4o-2024-05-13· OpenAI WildChat-4.8M (AI2)

测试时报错:Tracker: ceutrack ceutrackcoesot None , Sequence: dvSave-20220924154658 test…

  1. User

    测试时报错:Tracker: ceutrack ceutrack_coesot None , Sequence: dvSave-2022_09_24_15_46_58 test config: {'MODEL': {'PRETRAIN_FILE': 'mae_pretrain_vit_base.pth', 'RGB_PRETRAIN_FILE': 'mae_pretrain_vit_base.pth', 'EVENT_PRETRAIN_FILE': 'vim_s_midclstok_80p5acc.pth', 'EXTRA_MERGER': False, 'RETURN_INTER': False, 'RETURN_STAGES': [], 'BACKBONE': {'TYPE': 'vit_base_patch16_224_ce', 'STRIDE': 16, 'MID_PE': False, 'SEP_SEG': False, 'CAT_MODE': 'direct', 'MERGE_LAYER': 0, 'ADD_CLS_TOKEN': False, 'CLS_TOKEN_USE_MODE': 'ignore', 'CE_LOC': [3, 6, 9], 'CE_KEEP_RATIO': [1, 1, 1], 'CE_TEMPLATE_RANGE': 'CTR_POINT'}, 'HEAD': {'TYPE': 'CENTER', 'NUM_CHANNELS': 256}}, 'TRAIN': {'LR': 0.0001, 'WEIGHT_DECAY': 0.0001, 'EPOCH': 50, 'LR_DROP_EPOCH': 40, 'BATCH_SIZE': 32, 'NUM_WORKER': 4, 'OPTIMIZER': 'ADAMW', 'BACKBONE_MULTIPLIER': 0.1, 'GIOU_WEIGHT': 1, 'L1_WEIGHT': 14, 'FOCAL_WEIGHT': 1.0, 'FREEZE_LAYERS': [0], 'PRINT_INTERVAL': 50, 'VAL_EPOCH_INTERVAL': 1, 'GRAD_CLIP_NORM': 0.1, 'AMP': False, 'CE_START_EPOCH': 0, 'CE_WARM_EPOCH': 0, 'DROP_PATH_RATE': 0.1, 'SCHEDULER': {'TYPE': 'step', 'DECAY_RATE': 0.1, 'MILESTONES': [50, 80], 'GAMMA': 0.1}}, 'DATA': {'SAMPLER_MODE': 'causal', 'MEAN': [0.485, 0.456, 0.406], 'STD': [0.229, 0.224, 0.225], 'MAX_SAMPLE_INTERVAL': 100, 'TRAIN': {'DATASETS_NAME': ['COESOT'], 'DATASETS_RATIO': [1], 'SAMPLE_PER_EPOCH': 60000}, 'VAL': {'DATASETS_NAME': ['COESOT_VAL'], 'DATASETS_RATIO': [1], 'SAMPLE_PER_EPOCH': 5000}, 'SEARCH': {'SIZE': 256, 'FACTOR': 4.0, 'CENTER_JITTER': 3, 'SCALE_JITTER': 0.25, 'NUMBER': 1}, 'TEMPLATE': {'NUMBER': 1, 'SIZE': 128, 'FACTOR': 2.0, 'CENTER_JITTER': 0, 'SCALE_JITTER': 0}}, 'TEST': {'TEMPLATE_FACTOR': 2.0, 'TEMPLATE_SIZE': 128, 'SEARCH_FACTOR': 4.0, 'SEARCH_SIZE': 256, 'EPOCH': 50}} local variable 'event_pretrain' referenced before assignment Tracker: ceutrack ceutrack_coesot None , Sequence: dvSave-2022_09_17_23_02_54 test config: {'MODEL': {'PRETRAIN_FILE': 'mae_pretrain_vit_base.pth', 'RGB_PRETRAIN_FILE': 'mae_pretrain_vit_base.pth', 'EVENT_PRETRAIN_FILE': 'vim_s_midclstok_80p5acc.pth', 'EXTRA_MERGER': False, 'RETURN_INTER': False, 'RETURN_STAGES': [], 'BACKBONE': {'TYPE': 'vit_base_patch16_224_ce', 'STRIDE': 16, 'MID_PE': False, 'SEP_SEG': False, 'CAT_MODE': 'direct', 'MERGE_LAYER': 0, 'ADD_CLS_TOKEN': False, 'CLS_TOKEN_USE_MODE': 'ignore', 'CE_LOC': [3, 6, 9], 'CE_KEEP_RATIO': [1, 1, 1], 'CE_TEMPLATE_RANGE': 'CTR_POINT'}, 'HEAD': {'TYPE': 'CENTER', 'NUM_CHANNELS': 256}}, 'TRAIN': {'LR': 0.0001, 'WEIGHT_DECAY': 0.0001, 'EPOCH': 50, 'LR_DROP_EPOCH': 40, 'BATCH_SIZE': 32, 'NUM_WORKER': 4, 'OPTIMIZER': 'ADAMW', 'BACKBONE_MULTIPLIER': 0.1, 'GIOU_WEIGHT': 1, 'L1_WEIGHT': 14, 'FOCAL_WEIGHT': 1.0, 'FREEZE_LAYERS': [0], 'PRINT_INTERVAL': 50, 'VAL_EPOCH_INTERVAL': 1, 'GRAD_CLIP_NORM': 0.1, 'AMP': False, 'CE_START_EPOCH': 0, 'CE_WARM_EPOCH': 0, 'DROP_PATH_RATE': 0.1, 'SCHEDULER': {'TYPE': 'step', 'DECAY_RATE': 0.1, 'MILESTONES': [50, 80], 'GAMMA': 0.1}}, 'DATA': {'SAMPLER_MODE': 'causal', 'MEAN': [0.485, 0.456, 0.406], 'STD': [0.229, 0.224, 0.225], 'MAX_SAMPLE_INTERVAL': 100, 'TRAIN': {'DATASETS_NAME': ['COESOT'], 'DATASETS_RATIO': [1], 'SAMPLE_PER_EPOCH': 60000}, 'VAL': {'DATASETS_NAME': ['COESOT_VAL'], 'DATASETS_RATIO': [1], 'SAMPLE_PER_EPOCH': 5000}, 'SEARCH': {'SIZE': 256, 'FACTOR': 4.0, 'CENTER_JITTER': 3, 'SCALE_JITTER': 0.25, 'NUMBER': 1}, 'TEMPLATE': {'NUMBER': 1, 'SIZE': 128, 'FACTOR': 2.0, 'CENTER_JITTER': 0, 'SCALE_JITTER': 0}}, 'TEST': {'TEMPLATE_FACTOR': 2.0, 'TEMPLATE_SIZE': 128, 'SEARCH_FACTOR': 4.0, 'SEARCH_SIZE': 256, 'EPOCH': 50}} local variable 'event_pretrain' referenced before assignment

  2. Assistant

    从你的错误信息来看,问题是由未定义的本地变量event_pretrain导致的。这种情况通常发生在一个变量在赋值前被引用了。你需要检查代码中的变量定义和使用顺序,尤其是在涉及到配置文件和预训练文件这些部分。

    下面是一个可能导致该错误的示例代码片段及相应的修复办法:

    示例代码片段

    假设你有如下的代码用于加载预训练权重:

    def load_pretrained_weights(config):
        if config['MODEL']['EVENT_PRETRAIN_FILE']:
            event_pretrain = load_weights(config['MODEL']['EVENT_PRETRAIN_FILE'])
        # 使用event_pretrain变量
        some_function(event_pretrain)
    

    如果在config['MODEL']['EVENT_PRETRAIN_FILE']为空或未定义的情况下,程序会跳过对event_pretrain的赋值,导致后续对它的引用时报错。

    修复方法

    确保所有情况下event_pretrain都被正确赋值。可以在进入条件分支之前初始化变量:

    def load_pretrained_weights(config):
        event_pretrain = None  # 初始化变量
        if config['MODEL']['EVENT_PRETRAIN_FILE']:
            event_pretrain = load_weights(config['MODEL']['EVENT_PRETRAIN_FILE'])
        if event_pretrain is not None:
            some_function(event_pretrain)
        else:
            # 处理 event_pretrain 为 None 的情况,比如使用默认值或者报错
            handle_missing_event_pretrain()
    

    通过这种方式,无论条件是否成立,event_pretrain都会有一个初始值。不至于在引用时发生错误。

    此外,为了确保完整性,你应该在代码的其他部分也做类似的安全检查。确保变量在使用前总是被妥善初始化和赋值。

    希望这可以帮助你找到并解决问题。如果你仍有疑问,可以进一步检查代码的细节或者提供更多的代码片段以便更详细的诊断。

  3. User

    在全局定位时,应该设置什么查找名?

  4. Assistant

    在全局定位(global positioning)中特定查找名的设置取决于应用的具体领域和需求。这里给出不同场景下的一些常见查找名(keyword)的示例和解释:

    1. 容器化和编排环境(如Kubernetes):

      • Pod名称(Pod Name):唯一标识一个Pod。
      • 服务名称(Service Name):标识特定服务。
      • 命名空间(Namespace):将资源分隔到不同的逻辑组中。
      • 标签(Label)和选择器(Selector):用于过滤和选择特定的资源集合。
      • 节点名(Node Name):用于标识集群中的物理节点或虚拟节点。
    2. 云服务和虚拟环境:

      • 实例ID(Instance ID):云服务提供商分配给每个实例的唯一标识符。
      • 虚拟机名称(VM Name):便于用户管理和查找特定虚拟机的名称。
      • 区域和可用区(Region and Availability Zone):标识资源所在的地理区域和故障隔离区。
    3. 网络和设备管理:

      • IP地址(IP Address):唯一标识网络中的一个设备或主机。
      • MAC地址(MAC Address):网络接口卡的唯一硬件地址。
      • 主机名(Hostname):网络设备或计算机的唯一名称。
    4. 基于内容的资源定位:

      • 唯一资源名称(URN, Uniform Resource Name):用于唯一标识资源。
      • 统一资源定位符(URL, Uniform Resource Locator):具体定位资源的地址。
      • 文件路径和文件名(File Path and Filename):在文件系统中定位文件。
    5. 数据库和存储系统:

      • 表名(Table Name):数据库表的名字。
      • 主键(Primary Key):唯一标识数据库记录的键。
      • 索引(Index):加速查询的特定字段或组合字段。
      • 桶名(Bucket Name)(对于对象存储,如AWS S3):用于分组存储对象的容器名称。
    6. Web和API请求上下文:

      • 会话ID(Session ID):用于跟踪用户会话。
      • 请求ID(Request ID):用于唯一标识请求。
      • 用户ID(User ID):标识用户。

    查找名(Lookup Name)的设置原则:

    • 唯一性:确保查找名在所有相关资源中是唯一的,避免冲突。
    • 易于识别:查找名应该便于相关人员识别和理解。
    • 一致性:在整个系统中保持命名规则的一致性,便于维护和管理。
    • 简洁性:尽量简短但不失表达能力,避免过长的命名。

    具体应用场景中,你需要根据实际需求选择合适的查找名,并确保这些查找名能够高效准确地定位到目标资源。

  5. User

    上文中local variable ‘event_pretrain’ referenced before assignment,应该在train,test还是model中查找问题?

  6. Assistant

    从你提供的错误信息来看,错误是与event_pretrain变量在使用之前没有被正确的初始化和赋值有关。这个问题通常是在代码的逻辑中某个模块(比如train, test, or model)处理中没有正确处理配置文件或者变量赋值。

    基于提供的配置和错误信息,这里有几个步骤帮助你确定哪个模块可能出错,以及如何修复这个问题:

    1. 追踪变量的定义和使用:

    检查event_pretrain变量在代码中的定义和使用情况:

    • 初始化和赋值:在哪里初始化event_pretrain变量并给它赋值。
    • 使用:在哪里使用event_pretrain变量。

    2. 检查配置加载和处理:

    根据你的配置,重点检查如何处理模型中的预训练文件:

    • MODEL配置块:
     'MODEL': {
         'PRETRAIN_FILE': 'mae_pretrain_vit_base.pth',
         'RGB_PRETRAIN_FILE': 'mae_pretrain_vit_base.pth',
         'EVENT_PRETRAIN_FILE': 'vim_s_midclstok_80p5acc.pth',
         ...
     }
    

    3. 详细检查相关代码模块:

    Model模块:

    1. 加载模型参数时:
    event_pretrain = None  # 初始化变量
    
    if 'EVENT_PRETRAIN_FILE' in config['MODEL'] and config['MODEL']['EVENT_PRETRAIN_FILE']:
        event_pretrain = load_weights(config['MODEL']['EVENT_PRETRAIN_FILE'])
    
    # 确保后续代码在使用 event_pretrain 之前做出判断
    if event_pretrain is not None:
        # 使用 event_pretrain 变量
    else:
        # 处理 event_pretrain 未定义的情况或者报错
    

    Train模块和Test模块:

    代码通常在训练或测试开始时加载并初始化模型,所以你也应该检查这些模块:

    1. 在训练(train)代码中:

    确保训练模块在初始化模型时正确处理预训练文件:

    def train(config):
        model = initialize_model(config)
        # 初始化训练程序
        ...
    
    1. 在测试(test)代码中:

    同样确保测试模块处理正确:

    def test(config):
        model = initialize_model(config)
        # 初始化测试程序
        ...
    

    4. 综合检查:

    在上面这些模块中的各个部分检查event_pretrain的定义,确保在使用之前已经得到正确的初始化和赋值。此外,可能需要添加额外的错误处理逻辑,以便在变量未定义时提供明确的错误信息。

    5. 调试和日志记录:

    通过添加日志记录来调试,确保event_pretrain变量确实在各种情况下都得到了适当的赋值:

    import logging
    
    logging.basicConfig(level=logging.DEBUG)
    ...
    
    logging.debug(f"Event pretrain file: {config['MODEL'].get('EVENT_PRETRAIN_FILE', 'None')}")
    if 'EVENT_PRETRAIN_FILE' in config['MODEL']:
        event_pretrain = load_weights(config['MODEL']['EVENT_PRETRAIN_FILE'])
        logging.debug("Event pretrain weights successfully loaded.")
    else:
        logging.warning("No event pretrain file specified.")
    

    总结来说,你需要在代码中逐步检查 train, test, 和 model 模块中有关配置文件加载和预训练权重处理的部分,尤其要注意 event_pretrain 变量是如何初始化和赋值的,确保在引用该变量之前它已经被正确设置。

  7. User

    ① vit模型:""" Vision Transformer (ViT) in PyTorch A PyTorch implement of Vision Transformers as described in: 'An Image Is Worth 16 x 16 Words: Transformers for Image Recognition at Scale' - https://arxiv.org/abs/2010.11929 How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers - https://arxiv.org/abs/2106.10270 The official jax code is released and available at https://github.com/google-research/vision_transformer DeiT model defs and weights from https://github.com/facebookresearch/deit, paper DeiT: Data-efficient Image Transformers - https://arxiv.org/abs/2012.12877 Acknowledgments:

    • The paper authors for releasing code and weights, thanks!
    • I fixed my class token impl based on Phil Wang's https://github.com/lucidrains/vit-pytorch ... check it out for some einops/einsum fun
    • Simple transformer style inspired by Andrej Karpathy's https://github.com/karpathy/minGPT
    • Bert reference code checks against Huggingface Transformers and Tensorflow Bert Hacked together by / Copyright 2021 Ross Wightman

    Modified by Botao Ye """ import math import logging from functools import partial from collections import OrderedDict from copy import deepcopy

    import torch import torch.nn as nn import torch.nn.functional as F

    from timm.data import IMAGENET_DEFAULT_MEAN, IMAGENET_DEFAULT_STD, IMAGENET_INCEPTION_MEAN, IMAGENET_INCEPTION_STD from timm.models.helpers import build_model_with_cfg, named_apply, adapt_input_conv from timm.models.layers import Mlp, DropPath, trunc_normal_, lecun_normal_ from timm.models.registry import register_model

    from lib.models.layers.patch_embed import PatchEmbed from lib.models.ceutrack.base_backbone import BaseBackbone

    class Attention(nn.Module): def init(self, dim, num_heads=8, qkv_bias=False, attn_drop=0., proj_drop=0.): super().init() self.num_heads = num_heads head_dim = dim // num_heads self.scale = head_dim ** -0.5

        self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
        self.attn_drop = nn.Dropout(attn_drop)
        self.proj = nn.Linear(dim, dim)
        self.proj_drop = nn.Dropout(proj_drop)
    
    def forward(self, x, return_attention=False):
        B, N, C = x.shape
        qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
        q, k, v = qkv[0], qkv[1], qkv[2]  # make torchscript happy (cannot use tensor as tuple)
    
        attn = (q @ k.transpose(-2, -1)) * self.scale
        attn = attn.softmax(dim=-1)
        attn = self.attn_drop(attn)
    
        x = (attn @ v).transpose(1, 2).reshape(B, N, C)
        x = self.proj(x)
        x = self.proj_drop(x)
    
        if return_attention:
            return x, attn
        return x
    

    class Block(nn.Module):

    def __init__(self, dim, num_heads, mlp_ratio=4., qkv_bias=False, drop=0., attn_drop=0.,
                 drop_path=0., act_layer=nn.GELU, norm_layer=nn.LayerNorm):
        super().__init__()
        self.norm1 = norm_layer(dim)
        self.attn = Attention(dim, num_heads=num_heads, qkv_bias=qkv_bias, attn_drop=attn_drop, proj_drop=drop)
        # NOTE: drop path for stochastic depth, we shall see if this is better than dropout here
        self.drop_path = DropPath(drop_path) if drop_path > 0. else nn.Identity()
        self.norm2 = norm_layer(dim)
        mlp_hidden_dim = int(dim * mlp_ratio)
        self.mlp = Mlp(in_features=dim, hidden_features=mlp_hidden_dim, act_layer=act_layer, drop=drop)
    
    def forward(self, x, return_attention=False):
        if return_attention:
            feat, attn = self.attn(self.norm1(x), True)
            x = x + self.drop_path(feat)
            x = x + self.drop_path(self.mlp(self.norm2(x)))
            return x, attn
        else:
            x = x + self.drop_path(self.attn(self.norm1(x)))
            x = x + self.drop_path(self.mlp(self.norm2(x)))
            return x
    

    class VisionTransformer(BaseBackbone): """ Vision Transformer A PyTorch impl of : An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale - https://arxiv.org/abs/2010.11929 Includes distillation token & head support for DeiT: Data-efficient Image Transformers - https://arxiv.org/abs/2012.12877 """

    def __init__(self, img_size=224, patch_size=16, in_chans=3, num_classes=1000, embed_dim=768, depth=12,
                 num_heads=12, mlp_ratio=4., qkv_bias=True, representation_size=None, distilled=False,
                 drop_rate=0., attn_drop_rate=0., drop_path_rate=0., embed_layer=PatchEmbed, norm_layer=None,
                 act_layer=None, weight_init=''):
        """
        Args:
            img_size (int, tuple): input image size
            patch_size (int, tuple): patch size
            in_chans (int): number of input channels
            num_classes (int): number of classes for classification head
            embed_dim (int): embedding dimension
            depth (int): depth of transformer
            num_heads (int): number of attention heads
            mlp_ratio (int): ratio of mlp hidden dim to embedding dim
            qkv_bias (bool): enable bias for qkv if True
            representation_size (Optional[int]): enable and set representation layer (pre-logits) to this value if set
            distilled (bool): model includes a distillation token and head as in DeiT models
            drop_rate (float): dropout rate
            attn_drop_rate (float): attention dropout rate
            drop_path_rate (float): stochastic depth rate
            embed_layer (nn.Module): patch embedding layer
            norm_layer: (nn.Module): normalization layer
            weight_init: (str): weight init scheme
        """
        super().__init__()
        self.num_classes = num_classes
        self.num_features = self.embed_dim = embed_dim  # num_features for consistency with other models
        self.num_tokens = 2 if distilled else 1
        norm_layer = norm_layer or partial(nn.LayerNorm, eps=1e-6)
        act_layer = act_layer or nn.GELU
    
        self.patch_embed = embed_layer(
            img_size=img_size, patch_size=patch_size, in_chans=in_chans, embed_dim=embed_dim)
        num_patches = self.patch_embed.num_patches
    
        self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
        self.dist_token = nn.Parameter(torch.zeros(1, 1, embed_dim)) if distilled else None
        self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + self.num_tokens, embed_dim))
        self.pos_drop = nn.Dropout(p=drop_rate)
    
        dpr = [x.item() for x in torch.linspace(0, drop_path_rate, depth)]  # stochastic depth decay rule
        self.blocks = nn.Sequential(*[
            Block(
                dim=embed_dim, num_heads=num_heads, mlp_ratio=mlp_ratio, qkv_bias=qkv_bias, drop=drop_rate,
                attn_drop=attn_drop_rate, drop_path=dpr[i], norm_layer=norm_layer, act_layer=act_layer)
            for i in range(depth)])
        self.norm = norm_layer(embed_dim)
    
        # # Representation layer
        # if representation_size and not distilled:
        #     self.num_features = representation_size
        #     self.pre_logits = nn.Sequential(OrderedDict([
        #         ('fc', nn.Linear(embed_dim, representation_size)),
        #         ('act', nn.Tanh())
        #     ]))
        # else:
        #     self.pre_logits = nn.Identity()
        #
        # # Classifier head(s)
        # self.head = nn.Linear(self.num_features, num_classes) if num_classes > 0 else nn.Identity()
        # self.head_dist = None
        # if distilled:
        #     self.head_dist = nn.Linear(self.embed_dim, self.num_classes) if num_classes > 0 else nn.Identity()
    
        self.init_weights(weight_init)
    
    def init_weights(self, mode=''):
        assert mode in ('jax', 'jax_nlhb', 'nlhb', '')
        head_bias = -math.log(self.num_classes) if 'nlhb' in mode else 0.
        trunc_normal_(self.pos_embed, std=.02)
        if self.dist_token is not None:
            trunc_normal_(self.dist_token, std=.02)
        if mode.startswith('jax'):
            # leave cls token as zeros to match jax impl
            named_apply(partial(_init_vit_weights, head_bias=head_bias, jax_impl=True), self)
        else:
            trunc_normal_(self.cls_token, std=.02)
            self.apply(_init_vit_weights)
    
    def _init_weights(self, m):
        # this fn left here for compat with downstream users
        _init_vit_weights(m)
    
    @torch.jit.ignore()
    def load_pretrained(self, checkpoint_path, prefix=''):
        _load_weights(self, checkpoint_path, prefix)
    
    @torch.jit.ignore
    def no_weight_decay(self):
        return {'pos_embed', 'cls_token', 'dist_token'}
    
    def get_classifier(self):
        if self.dist_token is None:
            return self.head
        else:
            return self.head, self.head_dist
    
    def reset_classifier(self, num_classes, global_pool=''):
        self.num_classes = num_classes
        self.head = nn.Linear(self.embed_dim, num_classes) if num_classes > 0 else nn.Identity()
        if self.num_tokens == 2:
            self.head_dist = nn.Linear(self.embed_dim, self.num_classes) if num_classes > 0 else nn.Identity()
    

    def init_vit_weights(module: nn.Module, name: str = '', head_bias: float = 0., jax_impl: bool = False): """ ViT weight initialization * When called without n, head_bias, jax_impl args it will behave exactly the same as my original init for compatibility with prev hparam / downstream use cases (ie DeiT). * When called w/ valid n (module name) and jax_impl=True, will (hopefully) match JAX impl """ if isinstance(module, nn.Linear): if name.startswith('head'): nn.init.zeros(module.weight) nn.init.constant_(module.bias, head_bias) elif name.startswith('pre_logits'): lecun_normal_(module.weight) nn.init.zeros_(module.bias) else: if jax_impl: nn.init.xavier_uniform_(module.weight) if module.bias is not None: if 'mlp' in name: nn.init.normal_(module.bias, std=1e-6) else: nn.init.zeros_(module.bias) else: trunc_normal_(module.weight, std=.02) if module.bias is not None: nn.init.zeros_(module.bias) elif jax_impl and isinstance(module, nn.Conv2d): # NOTE conv was left to pytorch default in my original init lecun_normal_(module.weight) if module.bias is not None: nn.init.zeros_(module.bias) elif isinstance(module, (nn.LayerNorm, nn.GroupNorm, nn.BatchNorm2d)): nn.init.zeros_(module.bias) nn.init.ones_(module.weight)

    @torch.no_grad() def _load_weights(model: VisionTransformer, checkpoint_path: str, prefix: str = ''): """ Load weights from .npz checkpoints for official Google Brain Flax implementation """ import numpy as np

    def _n2p(w, t=True):
        if w.ndim == 4 and w.shape[0] == w.shape[1] == w.shape[2] == 1:
            w = w.flatten()
        if t:
            if w.ndim == 4:
                w = w.transpose([3, 2, 0, 1])
            elif w.ndim == 3:
                w = w.transpose([2, 0, 1])
            elif w.ndim == 2:
                w = w.transpose([1, 0])
        return torch.from_numpy(w)
    
    w = np.load(checkpoint_path)
    if not prefix and 'opt/target/embedding/kernel' in w:
        prefix = 'opt/target/'
    
    if hasattr(model.patch_embed, 'backbone'):
        # hybrid
        backbone = model.patch_embed.backbone
        stem_only = not hasattr(backbone, 'stem')
        stem = backbone if stem_only else backbone.stem
        stem.conv.weight.copy_(adapt_input_conv(stem.conv.weight.shape[1], _n2p(w[f'{prefix}conv_root/kernel'])))
        stem.norm.weight.copy_(_n2p(w[f'{prefix}gn_root/scale']))
        stem.norm.bias.copy_(_n2p(w[f'{prefix}gn_root/bias']))
        if not stem_only:
            for i, stage in enumerate(backbone.stages):
                for j, block in enumerate(stage.blocks):
                    bp = f'{prefix}block{i + 1}/unit{j + 1}/'
                    for r in range(3):
                        getattr(block, f'conv{r + 1}').weight.copy_(_n2p(w[f'{bp}conv{r + 1}/kernel']))
                        getattr(block, f'norm{r + 1}').weight.copy_(_n2p(w[f'{bp}gn{r + 1}/scale']))
                        getattr(block, f'norm{r + 1}').bias.copy_(_n2p(w[f'{bp}gn{r + 1}/bias']))
                    if block.downsample is not None:
                        block.downsample.conv.weight.copy_(_n2p(w[f'{bp}conv_proj/kernel']))
                        block.downsample.norm.weight.copy_(_n2p(w[f'{bp}gn_proj/scale']))
                        block.downsample.norm.bias.copy_(_n2p(w[f'{bp}gn_proj/bias']))
        embed_conv_w = _n2p(w[f'{prefix}embedding/kernel'])
    else:
        embed_conv_w = adapt_input_conv(
            model.patch_embed.proj.weight.shape[1], _n2p(w[f'{prefix}embedding/kernel']))
    model.patch_embed.proj.weight.copy_(embed_conv_w)
    model.patch_embed.proj.bias.copy_(_n2p(w[f'{prefix}embedding/bias']))
    model.cls_token.copy_(_n2p(w[f'{prefix}cls'], t=False))
    pos_embed_w = _n2p(w[f'{prefix}Transformer/posembed_input/pos_embedding'], t=False)
    if pos_embed_w.shape != model.pos_embed.shape:
        pos_embed_w = resize_pos_embed(  # resize pos embedding when different size from pretrained weights
            pos_embed_w, model.pos_embed, getattr(model, 'num_tokens', 1), model.patch_embed.grid_size)
    model.pos_embed.copy_(pos_embed_w)
    model.norm.weight.copy_(_n2p(w[f'{prefix}Transformer/encoder_norm/scale']))
    model.norm.bias.copy_(_n2p(w[f'{prefix}Transformer/encoder_norm/bias']))
    if isinstance(model.head, nn.Linear) and model.head.bias.shape[0] == w[f'{prefix}head/bias'].shape[-1]:
        model.head.weight.copy_(_n2p(w[f'{prefix}head/kernel']))
        model.head.bias.copy_(_n2p(w[f'{prefix}head/bias']))
    if isinstance(getattr(model.pre_logits, 'fc', None), nn.Linear) and f'{prefix}pre_logits/bias' in w:
        model.pre_logits.fc.weight.copy_(_n2p(w[f'{prefix}pre_logits/kernel']))
        model.pre_logits.fc.bias.copy_(_n2p(w[f'{prefix}pre_logits/bias']))
    for i, block in enumerate(model.blocks.children()):
        block_prefix = f'{prefix}Transformer/encoderblock_{i}/'
        mha_prefix = block_prefix + 'MultiHeadDotProductAttention_1/'
        block.norm1.weight.copy_(_n2p(w[f'{block_prefix}LayerNorm_0/scale']))
        block.norm1.bias.copy_(_n2p(w[f'{block_prefix}LayerNorm_0/bias']))
        block.attn.qkv.weight.copy_(torch.cat([
            _n2p(w[f'{mha_prefix}{n}/kernel'], t=False).flatten(1).T for n in ('query', 'key', 'value')]))
        block.attn.qkv.bias.copy_(torch.cat([
            _n2p(w[f'{mha_prefix}{n}/bias'], t=False).reshape(-1) for n in ('query', 'key', 'value')]))
        block.attn.proj.weight.copy_(_n2p(w[f'{mha_prefix}out/kernel']).flatten(1))
        block.attn.proj.bias.copy_(_n2p(w[f'{mha_prefix}out/bias']))
        for r in range(2):
            getattr(block.mlp, f'fc{r + 1}').weight.copy_(_n2p(w[f'{block_prefix}MlpBlock_3/Dense_{r}/kernel']))
            getattr(block.mlp, f'fc{r + 1}').bias.copy_(_n2p(w[f'{block_prefix}MlpBlock_3/Dense_{r}/bias']))
        block.norm2.weight.copy_(_n2p(w[f'{block_prefix}LayerNorm_2/scale']))
        block.norm2.bias.copy_(_n2p(w[f'{block_prefix}LayerNorm_2/bias']))
    

    def resize_pos_embed(posemb, posemb_new, num_tokens=1, gs_new=()): # Rescale the grid of position embeddings when loading from state_dict. Adapted from # https://github.com/google-research/vision_transformer/blob/00883dd691c63a6830751563748663526e811cee/vit_jax/checkpoint.py#L224 print('Resized position embedding: %s to %s', posemb.shape, posemb_new.shape) ntok_new = posemb_new.shape[1] if num_tokens: posemb_tok, posemb_grid = posemb[:, :num_tokens], posemb[0, num_tokens:] ntok_new -= num_tokens else: posemb_tok, posemb_grid = posemb[:, :0], posemb[0] gs_old = int(math.sqrt(len(posemb_grid))) if not len(gs_new): # backwards compatibility gs_new = [int(math.sqrt(ntok_new))] * 2 assert len(gs_new) >= 2 print('Position embedding grid-size from %s to %s', [gs_old, gs_old], gs_new) posemb_grid = posemb_grid.reshape(1, gs_old, gs_old, -1).permute(0, 3, 1, 2) posemb_grid = F.interpolate(posemb_grid, size=gs_new, mode='bilinear') posemb_grid = posemb_grid.permute(0, 2, 3, 1).reshape(1, gs_new[0] * gs_new[1], -1) posemb = torch.cat([posemb_tok, posemb_grid], dim=1) return posemb

    def checkpoint_filter_fn(state_dict, model): """ convert patch embedding weight from manual patchify + linear proj to conv""" out_dict = {} if 'model' in state_dict: # For deit models state_dict = state_dict['model'] for k, v in state_dict.items(): if 'patch_embed.proj.weight' in k and len(v.shape) < 4: # For old models that I trained prior to conv based patchification O, I, H, W = model.patch_embed.proj.weight.shape v = v.reshape(O, -1, H, W) elif k == 'pos_embed' and v.shape != model.pos_embed.shape: # To resize pos embedding when using model at different size from pretrained weights v = resize_pos_embed( v, model.pos_embed, getattr(model, 'num_tokens', 1), model.patch_embed.grid_size) out_dict[k] = v return out_dict

    def _create_vision_transformer(variant, pretrained=False, default_cfg=None, **kwargs): if kwargs.get('features_only', None): raise RuntimeError('features_only not implemented for Vision Transformer models.')

    model = VisionTransformer(**kwargs)
    
    if pretrained:
        if 'npz' in pretrained:
            model.load_pretrained(pretrained, prefix='')
        else:
            checkpoint = torch.load(pretrained, map_location="cpu")
            missing_keys, unexpected_keys = model.load_state_dict(checkpoint["model"], strict=False)
            print('Load pretrained model from: ' + pretrained)
    
    return model
    

    def vit_base_patch16_224(pretrained=False, **kwargs): """ ViT-Base model (ViT-B/16) from original paper (https://arxiv.org/abs/2010.11929). """ model_kwargs = dict( patch_size=16, embed_dim=768, depth=12, num_heads=12, **kwargs) model = _create_vision_transformer('vit_base_patch16_224_in21k', pretrained=pretrained, **model_kwargs) return model ② from easydict import EasyDict as edict import yaml

    """ Add default config for ceutrack. """ cfg = edict()

    MODEL

    cfg.MODEL = edict() cfg.MODEL.PRETRAIN_FILE = "mae_pretrain_vit_base.pth" cfg.MODEL.RGB_PRETRAIN_FILE = "mae_pretrain_vit_base.pth" cfg.MODEL.EVENT_PRETRAIN_FILE = "vim_s_midclstok_80p5acc.pth" cfg.MODEL.EXTRA_MERGER = False

    cfg.MODEL.RETURN_INTER = False cfg.MODEL.RETURN_STAGES = []

    MODEL.BACKBONE

    cfg.MODEL.BACKBONE = edict() cfg.MODEL.BACKBONE.TYPE = "vit_base_patch16_224" cfg.MODEL.BACKBONE.STRIDE = 16 cfg.MODEL.BACKBONE.MID_PE = False cfg.MODEL.BACKBONE.SEP_SEG = False cfg.MODEL.BACKBONE.CAT_MODE = 'direct' cfg.MODEL.BACKBONE.MERGE_LAYER = 0 cfg.MODEL.BACKBONE.ADD_CLS_TOKEN = False cfg.MODEL.BACKBONE.CLS_TOKEN_USE_MODE = 'ignore'

    cfg.MODEL.BACKBONE.CE_LOC = [] cfg.MODEL.BACKBONE.CE_KEEP_RATIO = [] cfg.MODEL.BACKBONE.CE_TEMPLATE_RANGE = 'ALL' # choose between ALL, CTR_POINT, CTR_REC, GT_BOX

    MODEL.HEAD

    cfg.MODEL.HEAD = edict() cfg.MODEL.HEAD.TYPE = "CENTER" cfg.MODEL.HEAD.NUM_CHANNELS = 256

    TRAIN

    cfg.TRAIN = edict() cfg.TRAIN.LR = 0.0001 cfg.TRAIN.WEIGHT_DECAY = 0.0001 cfg.TRAIN.EPOCH = 500 cfg.TRAIN.LR_DROP_EPOCH = 400 cfg.TRAIN.BATCH_SIZE = 16 cfg.TRAIN.NUM_WORKER = 8 cfg.TRAIN.OPTIMIZER = "ADAMW" cfg.TRAIN.BACKBONE_MULTIPLIER = 0.1 cfg.TRAIN.GIOU_WEIGHT = 2.0 cfg.TRAIN.L1_WEIGHT = 5.0 cfg.TRAIN.FOCAL_WEIGHT = 1.0 cfg.TRAIN.FREEZE_LAYERS = [0, ] cfg.TRAIN.PRINT_INTERVAL = 50 cfg.TRAIN.VAL_EPOCH_INTERVAL = 5 cfg.TRAIN.GRAD_CLIP_NORM = 0.1 cfg.TRAIN.AMP = False

    cfg.TRAIN.CE_START_EPOCH = 20 # candidate elimination start epoch cfg.TRAIN.CE_WARM_EPOCH = 80 # candidate elimination warm up epoch cfg.TRAIN.DROP_PATH_RATE = 0.1 # drop path rate for ViT backbone

    TRAIN.SCHEDULER

    cfg.TRAIN.SCHEDULER = edict()

    cfg.TRAIN.SCHEDULER.TYPE = 'Mstep' # 修改为Mstep策略

    cfg.TRAIN.SCHEDULER.DECAY_RATE = 0.1

    cfg.TRAIN.SCHEDULER.MILESTONES = [30, 50] # 学习率在第30和50个epoch衰减

    cfg.TRAIN.SCHEDULER.GAMMA = 0.1

    cfg.TRAIN.SCHEDULER = edict() cfg.TRAIN.SCHEDULER.TYPE = "step" cfg.TRAIN.SCHEDULER.DECAY_RATE = 0.1 cfg.TRAIN.SCHEDULER.MILESTONES = [50, 80] cfg.TRAIN.SCHEDULER.GAMMA = 0.1

    DATA

    cfg.DATA = edict() cfg.DATA.SAMPLER_MODE = "causal" # sampling methods cfg.DATA.MEAN = [0.485, 0.456, 0.406] cfg.DATA.STD = [0.229, 0.224, 0.225] cfg.DATA.MAX_SAMPLE_INTERVAL = 200

    DATA.TRAIN

    cfg.DATA.TRAIN = edict() cfg.DATA.TRAIN.DATASETS_NAME = ["LASOT", "GOT10K_vottrain"] cfg.DATA.TRAIN.DATASETS_RATIO = [1, 1] cfg.DATA.TRAIN.SAMPLE_PER_EPOCH = 60000

    DATA.VAL

    cfg.DATA.VAL = edict() cfg.DATA.VAL.DATASETS_NAME = ["GOT10K_votval"] cfg.DATA.VAL.DATASETS_RATIO = [1] cfg.DATA.VAL.SAMPLE_PER_EPOCH = 10000

    DATA.SEARCH

    cfg.DATA.SEARCH = edict() cfg.DATA.SEARCH.SIZE = 320 cfg.DATA.SEARCH.FACTOR = 5.0 cfg.DATA.SEARCH.CENTER_JITTER = 4.5 cfg.DATA.SEARCH.SCALE_JITTER = 0.5 cfg.DATA.SEARCH.NUMBER = 1

    DATA.TEMPLATE

    cfg.DATA.TEMPLATE = edict() cfg.DATA.TEMPLATE.NUMBER = 1 cfg.DATA.TEMPLATE.SIZE = 128 cfg.DATA.TEMPLATE.FACTOR = 2.0 cfg.DATA.TEMPLATE.CENTER_JITTER = 0 cfg.DATA.TEMPLATE.SCALE_JITTER = 0

    TEST

    cfg.TEST = edict() cfg.TEST.TEMPLATE_FACTOR = 2.0 cfg.TEST.TEMPLATE_SIZE = 128 cfg.TEST.SEARCH_FACTOR = 5.0 cfg.TEST.SEARCH_SIZE = 320 cfg.TEST.EPOCH = 500

    def _edict2dict(dest_dict, src_edict): if isinstance(dest_dict, dict) and isinstance(src_edict, dict): for k, v in src_edict.items(): if not isinstance(v, edict): dest_dict[k] = v else: dest_dict[k] = {} _edict2dict(dest_dict[k], v) else: return

    def gen_config(config_file): cfg_dict = {} _edict2dict(cfg_dict, cfg) with open(config_file, 'w') as f: yaml.dump(cfg_dict, f, default_flow_style=False)

    def _update_config(base_cfg, exp_cfg): if isinstance(base_cfg, dict) and isinstance(exp_cfg, edict): for k, v in exp_cfg.items(): if k in base_cfg: if not isinstance(v, dict): base_cfg[k] = v else: _update_config(base_cfg[k], v) else: raise ValueError("{} not exist in config.py".format(k)) else: return

    def update_config_from_file(filename, base_cfg=None): exp_config = None with open(filename) as f: exp_config = edict(yaml.safe_load(f)) if base_cfg is not None: _update_config(base_cfg, exp_config) else: _update_config(cfg, exp_config) ③ """ Basic ceutrack model. """ import math import os from typing import List

    import torch from torch import nn from torch.nn.modules.transformer import _get_clones

    from lib.models.layers.head import build_box_head from lib.models.ceutrack.vit import vit_base_patch16_224 from lib.models.ceutrack.vit_ce import vit_large_patch16_224_ce, vit_base_patch16_224_ce

    from lib.models.ceutrack.vit_ce_BACKUPS import vit_large_patch16_224_ce, vit_base_patch16_224_ce

    from lib.utils.box_ops import box_xyxy_to_cxcywh

    class CEUTrack(nn.Module): """ This is the base class for ceutrack """

    def __init__(self, transformer, box_head, aux_loss=False, head_type="CORNER"):
        """ Initializes the model.
        Parameters:
            transformer: torch module of the transformer architecture.
            aux_loss: True if auxiliary decoding losses (loss at each decoder layer) are to be used.
        """
        super().__init__()
        self.backbone = transformer
        self.box_head = box_head
    
        self.aux_loss = aux_loss
        self.head_type = head_type
        if head_type == "CORNER" or head_type == "CENTER":
            self.feat_sz_s = int(box_head.feat_sz)
            self.feat_len_s = int(box_head.feat_sz ** 2)
    
        if self.aux_loss:
            self.box_head = _get_clones(self.box_head, 6)
    
    def forward(self, template: torch.Tensor,
                search: torch.Tensor,
                event_template: torch.Tensor,        # torch.Size([4, 1, 19, 10000])
                event_search: torch.Tensor,          # torch.Size([4, 1, 19, 10000])
                ce_template_mask=None,
                ce_keep_rate=None,
                return_last_attn=False,
                ):
    
        # before feeding into backbone, we need to concat four vectors, or two two concat;
        x, aux_dict = self.backbone(z=template, x=search, event_z=event_template, event_x=event_search,
                                    ce_template_mask=ce_template_mask,
                                    ce_keep_rate=ce_keep_rate,
                                    return_last_attn=return_last_attn, )
    
        # Forward head
        feat_last = x
        if isinstance(x, list):
            feat_last = x[-1]
        out = self.forward_head(feat_last, None)
    
        out.update(aux_dict)
        out['backbone_feat'] = x
        return out
    
    def forward_head(self, cat_feature, gt_score_map=None):
        """
        cat_feature: output embeddings of the backbone, it can be (HW1+HW2, B, C) or (HW2, B, C)
        """
        # output the last 256
        # enc_opt = cat_feature[:, -self.feat_len_s:]  # encoder output for the search region (B, HW, C)  768*256
        # opt = (enc_opt.unsqueeze(-1)).permute((0, 3, 2, 1)).contiguous()
        # bs, Nq, C, HW = opt.size()
        # opt_feat = opt.view(-1, C, self.feat_sz_s, self.feat_sz_s)
    
        ## dual head   768+768)*256
        enc_opt1 = cat_feature[:, -self.feat_len_s:]
        enc_opt2 = cat_feature[:, :self.feat_len_s]
        enc_opt = torch.cat([enc_opt1, enc_opt2], dim=-1)
        opt = (enc_opt.unsqueeze(-1)).permute((0, 3, 2, 1)).contiguous()
        bs, Nq, C, HW = opt.size()
        opt_feat = opt.view(-1, C, self.feat_sz_s, self.feat_sz_s)
    
        if self.head_type == "CORNER":
            # run the corner head
            pred_box, score_map = self.box_head(opt_feat, True)
            outputs_coord = box_xyxy_to_cxcywh(pred_box)
            outputs_coord_new = outputs_coord.view(bs, Nq, 4)
            out = {'pred_boxes': outputs_coord_new,
                   'score_map': score_map,
                   }
            return out
    
        elif self.head_type == "CENTER":
            # run the center head
            score_map_ctr, bbox, size_map, offset_map = self.box_head(opt_feat, gt_score_map)
            # outputs_coord = box_xyxy_to_cxcywh(bbox)
            outputs_coord = bbox
            outputs_coord_new = outputs_coord.view(bs, Nq, 4)
            out = {'pred_boxes': outputs_coord_new,
                   'score_map': score_map_ctr,
                   'size_map': size_map,
                   'offset_map': offset_map}
            return out
        else:
            raise NotImplementedError
    

    ########### 根据配置文件 cfg.model.pretrain_file的值,决定是否加载预训练权重 #########

    结合vit_ce_Mamba模型修改 CEUTrack

    def build_ceutrack(cfg, training=True): current_dir = os.path.dirname(os.path.abspath(file)) # This is your Project Root pretrained_path = os.path.join(current_dir, 'pretrained_models') # 将加载预训练模型的逻辑扩展以包括RGB和Event预训练模型 if cfg.MODEL.PRETRAIN_FILE and ('CEUTrack' not in cfg.MODEL.PRETRAIN_FILE) and training: pretrained = os.path.join(pretrained_path, cfg.MODEL.PRETRAIN_FILE) rgb_pretrain = os.path.join(pretrained_path, cfg.MODEL.RGB_PRETRAIN_FILE) event_pretrain = os.path.join(pretrained_path, cfg.MODEL.EVENT_PRETRAIN_FILE) else: pretrained = '' rgb_pretrain = None # event_pretrain = None if cfg.MODEL.BACKBONE.TYPE == 'vit_base_patch16_224': backbone = vit_base_patch16_224(pretrained, drop_path_rate=cfg.TRAIN.DROP_PATH_RATE) hidden_dim = backbone.embed_dim patch_start_index = 1 elif cfg.MODEL.BACKBONE.TYPE == 'vit_base_patch16_224_ce': backbone = vit_base_patch16_224_ce(pretrained, rgb_pretrain=rgb_pretrain, event_pretrain=event_pretrain, drop_path_rate=cfg.TRAIN.DROP_PATH_RATE, ce_loc=cfg.MODEL.BACKBONE.CE_LOC, ce_keep_ratio=cfg.MODEL.BACKBONE.CE_KEEP_RATIO, ) hidden_dim = backbone.embed_dim * 2 # Update hidden_dim to be embed_dim * 2 patch_start_index = 1

    elif cfg.MODEL.BACKBONE.TYPE == 'vit_large_patch16_224_ce':
        backbone = vit_large_patch16_224_ce(pretrained, drop_path_rate=cfg.TRAIN.DROP_PATH_RATE,
                                            ce_loc=cfg.MODEL.BACKBONE.CE_LOC,
                                            ce_keep_ratio=cfg.MODEL.BACKBONE.CE_KEEP_RATIO,
                                            )
        hidden_dim = backbone.embed_dim
        patch_start_index = 1
    
    else:
        raise NotImplementedError
    

    def build_ceutrack(cfg, training=True):

    current_dir = os.path.dirname(os.path.abspath(file)) # This is your Project Root

    pretrained_path = os.path.join(current_dir, 'pretrained_models')

    if cfg.MODEL.PRETRAIN_FILE and ('CEUTrack' not in cfg.MODEL.PRETRAIN_FILE) and training:

    pretrained = os.path.join(pretrained_path, cfg.MODEL.PRETRAIN_FILE)

    else:

    pretrained = ''

    if cfg.MODEL.BACKBONE.TYPE == 'vit_base_patch16_224':

    backbone = vit_base_patch16_224(pretrained, drop_path_rate=cfg.TRAIN.DROP_PATH_RATE)

    hidden_dim = backbone.embed_dim

    patch_start_index = 1

    elif cfg.MODEL.BACKBONE.TYPE == 'vit_base_patch16_224_ce':

    backbone = vit_base_patch16_224_ce(pretrained, drop_path_rate=cfg.TRAIN.DROP_PATH_RATE,

    ce_loc=cfg.MODEL.BACKBONE.CE_LOC,

    ce_keep_ratio=cfg.MODEL.BACKBONE.CE_KEEP_RATIO,

    )

    # hidden_dim = backbone.embed_dim

    hidden_dim = backbone.embed_dim * 2

    patch_start_index = 1

    elif cfg.MODEL.BACKBONE.TYPE == 'vit_large_patch16_224_ce':

    backbone = vit_large_patch16_224_ce(pretrained, drop_path_rate=cfg.TRAIN.DROP_PATH_RATE,

    ce_loc=cfg.MODEL.BACKBONE.CE_LOC,

    ce_keep_ratio=cfg.MODEL.BACKBONE.CE_KEEP_RATIO,

    )

    hidden_dim = backbone.embed_dim

    patch_start_index = 1

    else:

    raise NotImplementedError

    backbone.finetune_track(cfg=cfg, patch_start_index=patch_start_index)
    
    box_head = build_box_head(cfg, hidden_dim)
    
    model = CEUTrack(
        backbone,
        box_head,
        aux_loss=False,
        head_type=cfg.MODEL.HEAD.TYPE,
    )
    # 将权重加载到模型中
    if 'CEUTrack' in cfg.MODEL.PRETRAIN_FILE and training:
        checkpoint = torch.load(cfg.MODEL.PRETRAIN_FILE, map_location="cpu")
        missing_keys, unexpected_keys = model.load_state_dict(checkpoint["net"], strict=False)
        print('Load pretrained model from: ' + cfg.MODEL.PRETRAIN_FILE)
    
    return model
    

    ④ DATA: MAX_SAMPLE_INTERVAL: 100 MEAN:

    • 0.485
    • 0.456
    • 0.406 SEARCH: CENTER_JITTER: 3 FACTOR: 4.0 SCALE_JITTER: 0.25 SIZE: 256 NUMBER: 1 STD:
    • 0.229
    • 0.224
    • 0.225 TEMPLATE: CENTER_JITTER: 0 FACTOR: 2.0 SCALE_JITTER: 0 SIZE: 128 TRAIN: DATASETS_NAME:
      • COESOT DATASETS_RATIO:
      • 1 SAMPLE_PER_EPOCH: 60000 VAL: DATASETS_NAME:
      • COESOT_VAL DATASETS_RATIO:
      • 1 SAMPLE_PER_EPOCH: 5000 MODEL: PRETRAIN_FILE: "mae_pretrain_vit_base.pth"

    添加RGB分支和Event分支的预训练模型

    RGB_PRETRAIN_FILE: "mae_pretrain_vit_base.pth" EVENT_PRETRAIN_FILE: "vim_s_midclstok_80p5acc.pth" EXTRA_MERGER: False RETURN_INTER: False BACKBONE: TYPE: vit_base_patch16_224_ce STRIDE: 16 CE_LOC: [3, 6, 9] CE_KEEP_RATIO: [1, 1, 1] CE_TEMPLATE_RANGE: 'CTR_POINT' HEAD: TYPE: CENTER NUM_CHANNELS: 256 TRAIN: BACKBONE_MULTIPLIER: 0.1 DROP_PATH_RATE: 0.1 CE_START_EPOCH: 0 # candidate elimination start epoch 20 CE_WARM_EPOCH: 0 # candidate elimination warm up epoch 50 BATCH_SIZE: 32 EPOCH: 50 GIOU_WEIGHT: 1 # 2.0 L1_WEIGHT: 14 # 5.0 FOCAL_WEIGHT: 1.0 # 1.5 GRAD_CLIP_NORM: 0.1 LR: 0.0001 LR_DROP_EPOCH: 40 NUM_WORKER: 4 # 4 OPTIMIZER: ADAMW PRINT_INTERVAL: 50 SCHEDULER: TYPE: step DECAY_RATE: 0.1

    TYPE: Mstep

    MILESTONES: [40, 60]

    GAMMA: 0.1

    VAL_EPOCH_INTERVAL: 1 WEIGHT_DECAY: 0.0001 #0.0001 AMP: False TEST: EPOCH: 50 SEARCH_FACTOR: 4.0 SEARCH_SIZE: 256 TEMPLATE_FACTOR: 2.0 TEMPLATE_SIZE: 128 ⑤ from functools import partial

    import torch import torch.nn as nn import torch.nn.functional as F from timm.models.vision_transformer import resize_pos_embed from timm.models.layers import DropPath, to_2tuple, trunc_normal_

    from lib.models.layers.patch_embed import PatchEmbed, PatchEmbed_event from lib.models.ceutrack.utils import combine_tokens, recover_tokens

    class BaseBackbone(nn.Module): def init(self): super().init()

        # for original ViT
        self.pos_embed = None
        self.pos_embed_event = PatchEmbed_event(in_chans=190, embed_dim=768, kernel_size=5, stride=1)
        self.img_size = [224, 224]
        self.patch_size = 16
        self.embed_dim = 384
    
        self.cat_mode = 'direct'
    
        self.pos_embed_z = None
        self.pos_embed_x = None
        # self.pos_embed_z_event = None
        # self.pos_embed_x_event = None
    
        self.template_segment_pos_embed = None
        self.search_segment_pos_embed = None
    
        self.return_inter = False
        self.return_stage = [2, 5, 8, 11]
    
        self.add_cls_token = False
        self.add_sep_seg = False
    
    def finetune_track(self, cfg, patch_start_index=1):
    
        search_size = to_2tuple(cfg.DATA.SEARCH.SIZE)
        template_size = to_2tuple(cfg.DATA.TEMPLATE.SIZE)
        new_patch_size = cfg.MODEL.BACKBONE.STRIDE
    
        self.cat_mode = cfg.MODEL.BACKBONE.CAT_MODE
        self.return_inter = cfg.MODEL.RETURN_INTER
        self.return_stage = cfg.MODEL.RETURN_STAGES
        self.add_sep_seg = cfg.MODEL.BACKBONE.SEP_SEG
    
        # resize patch embedding
        if new_patch_size != self.patch_size:
            print('Inconsistent Patch Size With The Pretrained Weights, Interpolate The Weight!')
            old_patch_embed = {}
            for name, param in self.patch_embed.named_parameters():
                if 'weight' in name:
                    param = nn.functional.interpolate(param, size=(new_patch_size, new_patch_size),
                                                      mode='bicubic', align_corners=False)
                    param = nn.Parameter(param)
                old_patch_embed[name] = param
            self.patch_embed = PatchEmbed(img_size=self.img_size, patch_size=new_patch_size, in_chans=3,
                                          embed_dim=self.embed_dim)
            self.patch_embed.proj.bias = old_patch_embed['proj.bias']
            self.patch_embed.proj.weight = old_patch_embed['proj.weight']
    
        # for patch embedding
        patch_pos_embed = self.pos_embed[:, patch_start_index:, :]
        patch_pos_embed = patch_pos_embed.transpose(1, 2)
        B, E, Q = patch_pos_embed.shape
        P_H, P_W = self.img_size[0] // self.patch_size, self.img_size[1] // self.patch_size
        patch_pos_embed = patch_pos_embed.view(B, E, P_H, P_W)
    
        # for search region
        H, W = search_size
        new_P_H, new_P_W = H // new_patch_size, W // new_patch_size
        search_patch_pos_embed = nn.functional.interpolate(patch_pos_embed, size=(new_P_H, new_P_W), mode='bicubic',
                                                           align_corners=False)
        search_patch_pos_embed = search_patch_pos_embed.flatten(2).transpose(1, 2)
    
        # for template region
        H, W = template_size
        new_P_H, new_P_W = H // new_patch_size, W // new_patch_size
        template_patch_pos_embed = nn.functional.interpolate(patch_pos_embed, size=(new_P_H, new_P_W), mode='bicubic',
                                                             align_corners=False)
        template_patch_pos_embed = template_patch_pos_embed.flatten(2).transpose(1, 2)
    
        self.pos_embed_z = nn.Parameter(template_patch_pos_embed)
        self.pos_embed_x = nn.Parameter(search_patch_pos_embed)
        # self.pos_embed_z_event = nn.Parameter(template_patch_pos_embed)
        # self.pos_embed_x_event = nn.Parameter(search_patch_pos_embed)
    
        # for cls token (keep it but not used)
        if self.add_cls_token and patch_start_index > 0:
            cls_pos_embed = self.pos_embed[:, 0:1, :]
            self.cls_pos_embed = nn.Parameter(cls_pos_embed)
    
        # separate token and segment token
        if self.add_sep_seg:
            self.template_segment_pos_embed = nn.Parameter(torch.zeros(1, 1, self.embed_dim))
            self.template_segment_pos_embed = trunc_normal_(self.template_segment_pos_embed, std=.02)
            self.search_segment_pos_embed = nn.Parameter(torch.zeros(1, 1, self.embed_dim))
            self.search_segment_pos_embed = trunc_normal_(self.search_segment_pos_embed, std=.02)
    
        # self.cls_token = None
        # self.pos_embed = None
    
        if self.return_inter:
            for i_layer in self.return_stage:
                if i_layer != 11:
                    norm_layer = partial(nn.LayerNorm, eps=1e-6)
                    layer = norm_layer(self.embed_dim)
                    layer_name = f'norm{i_layer}'
                    self.add_module(layer_name, layer)
    
    def forward_features(self, z, x, event_z, event_x):
        # N, L = event_z.shape[2], event_z.shape[3]
        event_x = self.pos_embed_event(event_x)     # B 768 256
        event_z = self.pos_embed_event(event_z)
        event_x += self.pos_embed_x
        event_z += self.pos_embed_z
        B, H, W = x.shape[0], x.shape[2], x.shape[3]
    
        x = self.patch_embed(x)
        z = self.patch_embed(z)
    
        if self.add_cls_token:
            cls_tokens = self.cls_token.expand(B, -1, -1)
            cls_tokens = cls_tokens + self.cls_pos_embed
    
        z += self.pos_embed_z
        x += self.pos_embed_x
    
        if self.add_sep_seg:
            x += self.search_segment_pos_embed
            z += self.template_segment_pos_embed
    
        x = combine_tokens(x, z, event_z, event_x, mode=self.cat_mode)
        if self.add_cls_token:
            x = torch.cat([cls_tokens, x], dim=1)
    
        x = self.pos_drop(x)
    
        for i, blk in enumerate(self.blocks):
            x = blk(x)
    
        lens_z = self.pos_embed_z.shape[1]
        lens_x = self.pos_embed_x.shape[1]
        x = recover_tokens(x, lens_z, lens_x, mode=self.cat_mode)
    
        aux_dict = {"attn": None}
        return self.norm(x), aux_dict
    
    def forward(self, z, x, event_z, event_x, **kwargs):
        """
        Joint feature extraction and relation modeling for the basic ViT backbone.
        Args:
            z (torch.Tensor): template feature, [B, C, H_z, W_z]
            x (torch.Tensor): search region feature, [B, C, H_x, W_x]
    
        Returns:
            x (torch.Tensor): merged template and search region feature, [B, L_z+L_x, C]
            attn : None
        """
        x, aux_dict = self.forward_features(z, x, event_z, event_x)
    
        return x, aux_dict ⑥ import importlib
    

    import os from collections import OrderedDict

    def create_default_local_file(): path = os.path.join(os.path.dirname(file), 'local.py')

    empty_str = '\'\''
    default_settings = OrderedDict({
        'workspace_dir': empty_str,
        'tensorboard_dir': 'self.workspace_dir + \'/tensorboard/\'',
        'pretrained_networks': 'self.workspace_dir + \'/pretrained_networks/\'',
        'coesot_dir': empty_str,
        'coesot_val_dir': empty_str,
        'fe108_dir':  empty_str,
        'fe108_val_dir':  empty_str,
        'visevent_dir': empty_str,
        'visevent_val_dir':  empty_str,
    })
    
    comment = {'workspace_dir': 'Base directory for saving network checkpoints.',
               'tensorboard_dir': 'Directory for tensorboard files.'}
    
    with open(path, 'w') as f:
        f.write('class EnvironmentSettings:\n')
        f.write('    def __init__(self):\n')
    
        for attr, attr_val in default_settings.items():
            comment_str = None
            if attr in comment:
                comment_str = comment[attr]
            if comment_str is None:
                f.write('        self.{} = {}\n'.format(attr, attr_val))
            else:
                f.write('        self.{} = {}    # {}\n'.format(attr, attr_val, comment_str))
    

    def create_default_local_file_ITP_train(workspace_dir, data_dir): path = os.path.join(os.path.dirname(file), 'local.py')

    empty_str = '\'\''
    default_settings = OrderedDict({
        'workspace_dir': workspace_dir,
        'tensorboard_dir': os.path.join(workspace_dir, 'tensorboard'),    # Directory for tensorboard files.
        'pretrained_networks': os.path.join(workspace_dir, 'pretrained_networks'),        'coesot_dir': 'self.workspace_dir + \'/data/\'',
        'coesot_val_dir': os.path.join(workspace_dir, 'data/COESOT/test'),
        'coesot_dir': os.path.join(workspace_dir, 'data/COESOT/train'),
        'fe108_dir':  os.path.join(workspace_dir, 'data/FE108/train'),
        'fe108_val_dir': os.path.join(workspace_dir, 'data/FE108/test'),
        'visevent_dir': os.path.join(workspace_dir, 'data/VisEvent/train'),
        'visevent_val_dir':  os.path.join(workspace_dir, 'data/VisEvent/test'),
    })
    
    comment = {'workspace_dir': 'Base directory for saving network checkpoints.',
               'tensorboard_dir': 'Directory for tensorboard files.'}
    
    with open(path, 'w') as f:
        f.write('class EnvironmentSettings:\n')
        f.write('    def __init__(self):\n')
    
        for attr, attr_val in default_settings.items():
            comment_str = None
            if attr in comment:
                comment_str = comment[attr]
            if comment_str is None:
                if attr_val == empty_str:
                    f.write('        self.{} = {}\n'.format(attr, attr_val))
                else:
                    f.write('        self.{} = \'{}\'\n'.format(attr, attr_val))
            else:
                f.write('        self.{} = \'{}\'    # {}\n'.format(attr, attr_val, comment_str))
    

    def env_settings(): env_module_name = 'lib.train.admin.local' try: env_module = importlib.import_module(env_module_name) return env_module.EnvironmentSettings() except: env_file = os.path.join(os.path.dirname(file), 'local.py')

        create_default_local_file()
        raise RuntimeError('YOU HAVE NOT SETUP YOUR local.py!!!\n Go to "{}" and set all the paths you need. Then try to run again.'.format(env_file))
    

    ⑦ import os import glob import torch import traceback from lib.train.admin import multigpu from torch.utils.data.distributed import DistributedSampler

    class BaseTrainer: """Base trainer class. Contains functions for training and saving/loading checkpoints. Trainer classes should inherit from this one and overload the train_epoch function."""

    def __init__(self, actor, loaders, optimizer, settings, lr_scheduler=None):
        """
        args:
            actor - The actor for training the network
            loaders - list of dataset loaders, e.g. [train_loader, val_loader]. In each epoch, the trainer runs one
                        epoch for each loader.
            optimizer - The optimizer used for training, e.g. Adam
            settings - Training settings
            lr_scheduler - Learning rate scheduler
        """
        self.actor = actor
        self.optimizer = optimizer
        self.lr_scheduler = lr_scheduler
        self.loaders = loaders
    
        self.update_settings(settings)
    
        self.epoch = 0
        self.stats = {}
    
        self.device = getattr(settings, 'device', None)
        if self.device is None:
            self.device = torch.device("cuda:0" if torch.cuda.is_available() and settings.use_gpu else "cpu")
    
        self.actor.to(self.device)
        self.settings = settings
    
    def update_settings(self, settings=None):
        """Updates the trainer settings. Must be called to update internal settings."""
        if settings is not None:
            self.settings = settings
    
        if self.settings.env.workspace_dir is not None:
            self.settings.env.workspace_dir = os.path.expanduser(self.settings.env.workspace_dir)
            '''2021.1.4 New function: specify checkpoint dir'''
            if self.settings.save_dir is None:
                self._checkpoint_dir = os.path.join(self.settings.env.workspace_dir, 'checkpoints')
            else:
                self._checkpoint_dir = os.path.join(self.settings.save_dir, 'checkpoints')
            print("checkpoints will be saved to %s" % self._checkpoint_dir)
    
            if self.settings.local_rank in [-1, 0]:
                if not os.path.exists(self._checkpoint_dir):
                    print("Training with multiple GPUs. checkpoints directory doesn't exist. "
                          "Create checkpoints directory")
                    os.makedirs(self._checkpoint_dir)
        else:
            self._checkpoint_dir = None
    
    def train(self, max_epochs, load_latest=False, fail_safe=True, load_previous_ckpt=False, distill=False):
        """Do training for the given number of epochs.
        args:
            max_epochs - Max number of training epochs,
            load_latest - Bool indicating whether to resume from latest epoch.
            fail_safe - Bool indicating whether the training to automatically restart in case of any crashes.
        """
    
        epoch = -1
        num_tries = 1
        for i in range(num_tries):
            try:
                if load_latest:
                    self.load_checkpoint()
                if load_previous_ckpt:
                    directory = '{}/{}'.format(self._checkpoint_dir, self.settings.project_path_prv)
                    self.load_state_dict(directory)
                if distill:
                    directory_teacher = '{}/{}'.format(self._checkpoint_dir, self.settings.project_path_teacher)
                    self.load_state_dict(directory_teacher, distill=True)
                for epoch in range(self.epoch+1, max_epochs+1):
                    self.epoch = epoch
    
                    self.train_epoch()
    
                    if self.lr_scheduler is not None:
                        if self.settings.scheduler_type != 'cosine':
                            self.lr_scheduler.step()
                        else:
                            self.lr_scheduler.step(epoch - 1)
                    # only save the last 10 checkpoints
                    save_every_epoch = getattr(self.settings, "save_every_epoch", False)
                    save_epochs = [79, 159, 239]
                    if epoch > (max_epochs - 1) or save_every_epoch or epoch % 5 == 0 or epoch in save_epochs or epoch > (max_epochs - 3):
                    # if epoch > (max_epochs - 10) or save_every_epoch or epoch % 100 == 0:
                        if self._checkpoint_dir:
                            if self.settings.local_rank in [-1, 0]:
                                self.save_checkpoint()
            except:
                print('Training crashed at epoch {}'.format(epoch))
                if fail_safe:
                    self.epoch -= 1
                    load_latest = True
                    print('Traceback for the error!')
                    print(traceback.format_exc())
                    print('Restarting training from last epoch ...')
                else:
                    raise
    
        print('Finished training!')
    
    def train_epoch(self):
        raise NotImplementedError
    
    def save_checkpoint(self):
        """Saves a checkpoint of the network and other variables."""
    
        net = self.actor.net.module if multigpu.is_multi_gpu(self.actor.net) else self.actor.net
    
        actor_type = type(self.actor).__name__
        net_type = type(net).__name__
        state = {
            'epoch': self.epoch,
            'actor_type': actor_type,
            'net_type': net_type,
            'net': net.state_dict(),
            'net_info': getattr(net, 'info', None),
            'constructor': getattr(net, 'constructor', None),
            'optimizer': self.optimizer.state_dict(),
            'stats': self.stats,
            'settings': self.settings
        }
    
        directory = '{}/{}'.format(self._checkpoint_dir, self.settings.project_path)
        print(directory)
        if not os.path.exists(directory):
            print("directory doesn't exist. creating...")
            os.makedirs(directory)
    
        # First save as a tmp file
        tmp_file_path = '{}/{}_ep{:04d}.tmp'.format(directory, net_type, self.epoch)
        torch.save(state, tmp_file_path)
    
        file_path = '{}/{}_ep{:04d}.pth.tar'.format(directory, net_type, self.epoch)
    
        # Now rename to actual checkpoint. os.rename seems to be atomic if files are on same filesystem. Not 100% sure
        os.rename(tmp_file_path, file_path)
    
    def load_checkpoint(self, checkpoint = None, fields = None, ignore_fields = None, load_constructor = False):
        """Loads a network checkpoint file.
    
        Can be called in three different ways:
            load_checkpoint():
                Loads the latest epoch from the workspace. Use this to continue training.
            load_checkpoint(epoch_num):
                Loads the network at the given epoch number (int).
            load_checkpoint(path_to_checkpoint):
                Loads the file from the given absolute path (str).
        """
    
        net = self.actor.net.module if multigpu.is_multi_gpu(self.actor.net) else self.actor.net
    
        actor_type = type(self.actor).__name__
        net_type = type(net).__name__
    
        if checkpoint is None:
            # Load most recent checkpoint
            checkpoint_list = sorted(glob.glob('{}/{}/{}_ep*.pth.tar'.format(self._checkpoint_dir,
                                                                             self.settings.project_path, net_type)))
            if checkpoint_list:
                checkpoint_path = checkpoint_list[-1]
            else:
                print('No matching checkpoint file found')
                return
        elif isinstance(checkpoint, int):
            # Checkpoint is the epoch number
            checkpoint_path = '{}/{}/{}_ep{:04d}.pth.tar'.format(self._checkpoint_dir, self.settings.project_path,
                                                                 net_type, checkpoint)
        elif isinstance(checkpoint, str):
            # checkpoint is the path
            if os.path.isdir(checkpoint):
                checkpoint_list = sorted(glob.glob('{}/*_ep*.pth.tar'.format(checkpoint)))
                if checkpoint_list:
                    checkpoint_path = checkpoint_list[-1]
                else:
                    raise Exception('No checkpoint found')
            else:
                checkpoint_path = os.path.expanduser(checkpoint)
        else:
            raise TypeError
    
        # Load network
        checkpoint_dict = torch.load(checkpoint_path, map_location='cpu')
    
        assert net_type == checkpoint_dict['net_type'], 'Network is not of correct type.'
    
        if fields is None:
            fields = checkpoint_dict.keys()
        if ignore_fields is None:
            ignore_fields = ['settings']
    
            # Never load the scheduler. It exists in older checkpoints.
        ignore_fields.extend(['lr_scheduler', 'constructor', 'net_type', 'actor_type', 'net_info'])
    
        # Load all fields
        for key in fields:
            if key in ignore_fields:
                continue
            if key == 'net':
                net.load_state_dict(checkpoint_dict[key])
            elif key == 'optimizer':
                self.optimizer.load_state_dict(checkpoint_dict[key])
            else:
                setattr(self, key, checkpoint_dict[key])
    
        # Set the net info
        if load_constructor and 'constructor' in checkpoint_dict and checkpoint_dict['constructor'] is not None:
            net.constructor = checkpoint_dict['constructor']
        if 'net_info' in checkpoint_dict and checkpoint_dict['net_info'] is not None:
            net.info = checkpoint_dict['net_info']
    
        # Update the epoch in lr scheduler
        if 'epoch' in fields:
            self.lr_scheduler.last_epoch = self.epoch
        # 2021.1.10 Update the epoch in data_samplers
            for loader in self.loaders:
                if isinstance(loader.sampler, DistributedSampler):
                    loader.sampler.set_epoch(self.epoch)
        return True
    
    def load_state_dict(self, checkpoint=None, distill=False):
        """Loads a network checkpoint file.
    
        Can be called in three different ways:
            load_checkpoint():
                Loads the latest epoch from the workspace. Use this to continue training.
            load_checkpoint(epoch_num):
                Loads the network at the given epoch number (int).
            load_checkpoint(path_to_checkpoint):
                Loads the file from the given absolute path (str).
        """
        if distill:
            net = self.actor.net_teacher.module if multigpu.is_multi_gpu(self.actor.net_teacher) \
                else self.actor.net_teacher
        else:
            net = self.actor.net.module if multigpu.is_multi_gpu(self.actor.net) else self.actor.net
    
        net_type = type(net).__name__
    
        if isinstance(checkpoint, str):
            # checkpoint is the path
            if os.path.isdir(checkpoint):
                checkpoint_list = sorted(glob.glob('{}/*_ep*.pth.tar'.format(checkpoint)))
                if checkpoint_list:
                    checkpoint_path = checkpoint_list[-1]
                else:
                    raise Exception('No checkpoint found')
            else:
                checkpoint_path = os.path.expanduser(checkpoint)
        else:
            raise TypeError
    
        # Load network
        print("Loading pretrained model from ", checkpoint_path)
        checkpoint_dict = torch.load(checkpoint_path, map_location='cpu')
    
        assert net_type == checkpoint_dict['net_type'], 'Network is not of correct type.'
    
        missing_k, unexpected_k = net.load_state_dict(checkpoint_dict["net"], strict=False)
        print("previous checkpoint is loaded.")
        print("missing keys: ", missing_k)
        print("unexpected keys:", unexpected_k)
    
        return True
    

    ,涉及到local variable ‘event_pretrain’ referenced before assignment报错的原因及修改方案,其中test.py是:import os import sys import argparse

    prj_path = os.path.join(os.path.dirname(file), '..') if prj_path not in sys.path: sys.path.append(prj_path)

    from lib.test.evaluation import get_dataset from lib.test.evaluation.running import run_dataset from lib.test.evaluation.tracker import Tracker

    CPU usage limit

    import os os.environ["KMP_BLOCKTIME"] = "0" os.environ["OMP_NUM_THREADS"] = "1" import torch torch.set_num_threads(1) import cv2 cv2.setNumThreads(0) cv2.ocl.setUseOpenCL(False)

    def run_tracker(tracker_name, tracker_param, run_id=None, dataset_name='otb', sequence=None, debug=0, threads=0, num_gpus=8): """Run tracker on sequence or dataset. args: tracker_name: Name of tracking method. tracker_param: Name of parameter file. run_id: The run id. dataset_name: Name of dataset sequence: Sequence number or name. debug: Debug level. threads: Number of threads. """

    dataset = get_dataset(dataset_name)
    
    if sequence is not None:
        dataset = [dataset[sequence]]
    
    trackers = [Tracker(tracker_name, tracker_param, dataset_name, run_id)]
    
    run_dataset(dataset, trackers, debug, threads, num_gpus=num_gpus)
    

    def main(): parser = argparse.ArgumentParser(description='Run tracker on sequence or dataset.') parser.add_argument('tracker_name', type=str, help='Name of tracking method.') parser.add_argument('tracker_param', type=str, help='Name of config file.') parser.add_argument('--runid', type=int, default=None, help='The run id.') parser.add_argument('--dataset_name', type=str, default='otb', help='Name of dataset.') parser.add_argument('--sequence', type=str, default=None, help='Sequence number or name.') parser.add_argument('--debug', type=int, default=0, help='Debug level.') parser.add_argument('--threads', type=int, default=0, help='Number of threads.') parser.add_argument('--num_gpus', type=int, default=8)

    args = parser.parse_args()
    
    try:
        seq_name = int(args.sequence)
    except:
        seq_name = args.sequence
    
    run_tracker(args.tracker_name, args.tracker_param, args.runid, args.dataset_name, seq_name, args.debug,
                args.threads, num_gpus=args.num_gpus)
    

    if name == 'main': # os.environ['CUDA_VISIBLE_DEVICES'] = "4" main() (中文回答)

  8. Assistant

    从你提供的代码中可以看到,event_pretrain 这个变量在被引用之前可能没有初始化或者赋值,导致程序在运行时出现 local variable 'event_pretrain' referenced before assignment 的错误。

    问题定位与修改方案:

    在 build_ceutrack 函数中有如下代码段处理预训练权重:

    # 结合vit_ce_Mamba模型修改 CEUTrack
    def build_ceutrack(cfg, training=True):
        current_dir = os.path.dirname(os.path.abspath(__file__))  # 这个是你的项目根目录
        pretrained_path = os.path.join(current_dir, 'pretrained_models')
        # 将加载预训练模型的逻辑扩展以包括RGB和Event预训练模型
        if cfg.MODEL.PRETRAIN_FILE and ('CEUTrack' not in cfg.MODEL.PRETRAIN_FILE) and training:
            pretrained = os.path.join(pretrained_path, cfg.MODEL.PRETRAIN_FILE)
            rgb_pretrain = os.path.join(pretrained_path, cfg.MODEL.RGB_PRETRAIN_FILE)
            event_pretrain = os.path.join(pretrained_path, cfg.MODEL.EVENT_PRETRAIN_FILE)
        else:
            pretrained = ''
            rgb_pretrain = None
            event_pretrain = None  # 这里需要初始化 event_pretrain 为 None
    
        if cfg.MODEL.BACKBONE.TYPE == 'vit_base_patch16_224':
            backbone = vit_base_patch16_224(pretrained, drop_path_rate=cfg.TRAIN.DROP_PATH_RATE)
            hidden_dim = backbone.embed_dim
            patch_start_index = 1
        elif cfg.MODEL.BACKBONE.TYPE == 'vit_base_patch16_224_ce':
            backbone = vit_base_patch16_224_ce(pretrained, rgb_pretrain=rgb_pretrain, event_pretrain=event_pretrain,
                                               drop_path_rate=cfg.TRAIN.DROP_PATH_RATE,
                                               ce_loc=cfg.MODEL.BACKBONE.CE_LOC,
                                               ce_keep_ratio=cfg.MODEL.BACKBONE.CE_KEEP_RATIO,
                                               )
            hidden_dim = backbone.embed_dim * 2  # 更新 hidden_dim 为 embed_dim 的 2 倍
            patch_start_index = 1
        elif cfg.MODEL.BACKBONE.TYPE == 'vit_large_patch16_224_ce':
            backbone = vit_large_patch16_224_ce(pretrained, drop_path_rate=cfg.TRAIN.DROP_PATH_RATE,
                                                ce_loc=cfg.MODEL.BACKBONE.CE_LOC,
                                                ce_keep_ratio=cfg.MODEL.BACKBONE.CE_KEEP_RATIO,
                                                )
            hidden_dim = backbone.embed_dim
            patch_start_index = 1
        else:
            raise NotImplementedError
    
        backbone.finetune_track(cfg=cfg, patch_start_index=patch_start_index)
    
        box_head = build_box_head(cfg, hidden_dim)
    
        model = CEUTrack(
            backbone,
            box_head,
            aux_loss=False,
            head_type=cfg.MODEL.HEAD.TYPE,
        )
    
        # 将权重加载到模型中
        if 'CEUTrack' in cfg.MODEL.PRETRAIN_FILE and training:
            checkpoint = torch.load(cfg.MODEL.PRETRAIN_FILE, map_location="cpu")
            missing_keys, unexpected_keys = model.load_state_dict(checkpoint["net"], strict=False)
            print('Load pretrained model from: ' + cfg.MODEL.PRETRAIN_FILE)
    
        return model
    

    修改方法:

    在原代码中,将 else 分支中对 event_pretrain 进行初始化。这可以保证在任何情况下,event_pretrain 都会有一个默认值。

    具体修改内容:

    1. 确保 event_pretrain 在所有条件下都初始化:
    def build_ceutrack(cfg, training=True):
        current_dir = os.path.dirname(os.path.abspath(__file__))  # 这个是你的项目根目录
        pretrained_path = os.path.join(current_dir, 'pretrained_models')
        # 将加载预训练模型的逻辑扩展以包括RGB和Event预训练模型
        if cfg.MODEL.PRETRAIN_FILE and ('CEUTrack' not in cfg.MODEL.PRETRAIN_FILE) and training:
            pretrained = os.path.join(pretrained_path, cfg.MODEL.PRETRAIN_FILE)
            rgb_pretrain = os.path.join(pretrained_path, cfg.MODEL.RGB_PRETRAIN_FILE)
            event_pretrain = os.path.join(pretrained_path, cfg.MODEL.EVENT_PRETRAIN_FILE)
        else:
            pretrained = ''
            rgb_pretrain = None
            event_pretrain = None  # 确保这里初始化为 None
    
        if cfg.MODEL.BACKBONE.TYPE == 'vit_base_patch16_224':
            backbone = vit_base_patch16_224(pretrained, drop_path_rate=cfg.TRAIN.DROP_PATH_RATE)
            hidden_dim = backbone.embed_dim
            patch_start_index = 1
        elif cfg.MODEL.BACKBONE.TYPE == 'vit_base_patch16_224_ce':
            backbone = vit_base_patch16_224_ce(pretrained, rgb_pretrain=rgb_pretrain, event_pretrain=event_pretrain,
                                               drop_path_rate=cfg.TRAIN.DROP_PATH_RATE,
                                               ce_loc=cfg.MODEL.BACKBONE.CE_LOC,
                                               ce_keep_ratio=cfg.MODEL.BACKBONE.CE_KEEP_RATIO,
                                               )
            hidden_dim = backbone.embed_dim * 2  # 更新 hidden_dim 为 embed_dim 的 2 倍
            patch_start_index = 1
        elif cfg.MODEL.BACKBONE.TYPE == 'vit_large_patch16_224_ce':
            backbone = vit_large_patch16_224_ce(pretrained, 
                                                drop_path_rate=cfg.TRAIN.DROP_PATH_RATE,
                                                ce_loc=cfg.MODEL.BACKBONE.CE_LOC,
                                                ce_keep_ratio=cfg.MODEL.BACKBONE.CE_KEEP_RATIO,
                                                )
            hidden_dim = backbone.embed_dim
            patch_start_index = 1
        else:
            raise NotImplementedError
    
        backbone.finetune_track(cfg=cfg, patch_start_index=patch_start_index)
    
        box_head = build_box_head(cfg, hidden_dim)
    
        model = CEUTrack(
            backbone,
            box_head,
            aux_loss=False,
            head_type=cfg.MODEL.HEAD.TYPE,
        )
    
        # 将权重加载到模型中
        if 'CEUTrack' in cfg.MODEL.PRETRAIN_FILE and training:
            checkpoint = torch.load(cfg.MODEL.PRETRAIN_FILE, map_location="cpu")
            missing_keys, unexpected_keys = model.load_state_dict(checkpoint["net"], strict=False)
            print('Load pretrained model from: ' + cfg.MODEL.PRETRAIN_FILE)
    
        return model
    

    总结:

    通过在所有条件下初始化 event_pretrain 变量,避免了使用未初始化变量的问题。这样的处理方式可以确保在不同的配置情况下,程序始终能够正常运行,不会因为变量未定义而导致运行错误。

  9. User

    那么在训练:import os import argparse import random

    def parse_args(): """ args for training. """ parser = argparse.ArgumentParser(description='Parse args for training') # for train parser.add_argument('--script', type=str, help='training script name') parser.add_argument('--config', type=str, default='baseline', help='yaml configure file name') parser.add_argument('--save_dir', type=str, help='root directory to save checkpoints, logs, and tensorboard') parser.add_argument('--mode', type=str, choices=["single", "multiple", "multi_node"], default="multiple", help="train on single gpu or multiple gpus") parser.add_argument('--nproc_per_node', type=int, help="number of GPUs per node") # specify when mode is multiple parser.add_argument('--use_lmdb', type=int, choices=[0, 1], default=0) # whether datasets are in lmdb format parser.add_argument('--script_prv', type=str, help='training script name') parser.add_argument('--config_prv', type=str, default='baseline', help='yaml configure file name') parser.add_argument('--use_wandb', type=int, choices=[0, 1], default=0) # whether to use wandb # for knowledge distillation parser.add_argument('--distill', type=int, choices=[0, 1], default=0) # whether to use knowledge distillation parser.add_argument('--script_teacher', type=str, help='teacher script name') parser.add_argument('--config_teacher', type=str, help='teacher yaml configure file name')

    # for multiple machines
    parser.add_argument('--rank', type=int, help='Rank of the current process.')
    parser.add_argument('--world-size', type=int, help='Number of processes participating in the job.')
    parser.add_argument('--ip', type=str, default='127.0.0.1', help='IP of the current rank 0.')
    parser.add_argument('--port', type=int, default='20000', help='Port of the current rank 0.')
    
    args = parser.parse_args()
    
    return args
    

    def main(): args = parse_args() if args.mode == "single": train_cmd = "python lib/train/run_training.py --script %s --config %s --save_dir %s --use_lmdb %d "
    "--script_prv %s --config_prv %s --distill %d --script_teacher %s --config_teacher %s --use_wandb %d"
    % (args.script, args.config, args.save_dir, args.use_lmdb, args.script_prv, args.config_prv, args.distill, args.script_teacher, args.config_teacher, args.use_wandb) elif args.mode == "multiple": train_cmd = "python -m torch.distributed.launch --nproc_per_node %d --master_port %d lib/train/run_training.py "
    "--script %s --config %s --save_dir %s --use_lmdb %d --script_prv %s --config_prv %s --use_wandb %d "
    "--distill %d --script_teacher %s --config_teacher %s"
    % (args.nproc_per_node, random.randint(10000, 50000), args.script, args.config, args.save_dir, args.use_lmdb, args.script_prv, args.config_prv, args.use_wandb, args.distill, args.script_teacher, args.config_teacher) elif args.mode == "multi_node": train_cmd = "python -m torch.distributed.launch --nproc_per_node %d --master_addr %s --master_port %d --nnodes %d --node_rank %d lib/train/run_training.py "
    "--script %s --config %s --save_dir %s --use_lmdb %d --script_prv %s --config_prv %s --use_wandb %d "
    "--distill %d --script_teacher %s --config_teacher %s"
    % (args.nproc_per_node, args.ip, args.port, args.world_size, args.rank, args.script, args.config, args.save_dir, args.use_lmdb, args.script_prv, args.config_prv, args.use_wandb, args.distill, args.script_teacher, args.config_teacher) else: raise ValueError("mode should be 'single' or 'multiple'.") os.system(train_cmd)

    if name == "main": main() 时,未将上文中提到的event_pretrain = None,会影响训练吗?

  10. Assistant

    在训练代码中,未将 event_pretrain = None 明确初始化确实会影响训练,尤其是如果你的训练代码依赖于 build_ceutrack 函数并且没有适当地处理 event_pretrain 变量的话。

    你提供的训练脚本是用来解析训练参数并生成相应的训练命令的。其中所生成的命令将调用 lib/train/run_training.py 来实际执行训练。因此,我们还需要确保 run_training.py 中也正确处理了 event_pretrain 变量的初始化。

    针对之前问题的解决方案是确保代码中的所有引用 event_pretrain 的地方都已经正确初始化和赋值。为了稳健地处理训练过程中的变量引用问题,建议你做以下步骤:

    步骤1:确保初始化

    在你之前的 build_ceutrack 函数中已经有了一个修复:

    else:
        pretrained = ''
        rgb_pretrain = None
        event_pretrain = None  # 确保变量 event_pretrain 初始化
    

    确保你在所有涉及到的代码模块都做了类似的初始化。

    步骤2:检查 run_training.py

    确保 run_training.py 文件中也正确处理了 event_pretrain 变量。这是因为 run_training.py 将通过解析你的传入配置文件并调用 build_ceutrack 函数来构建模型并开始训练。

    假设 run_training.py 包含以下内容:

    import argparse
    from lib.models.ceutrack_model import build_ceutrack
    
    def main():
        parser = argparse.ArgumentParser(description="Train model")
        parser.add_argument('--script', type=str)
        parser.add_argument('--config', type=str, default='baseline')
        parser.add_argument('--save_dir', type=str)
        parser.add_argument('--use_lmdb', type=int, default=0)
        parser.add_argument('--script_prv', type=str)
        parser.add_argument('--config_prv', type=str, default='baseline')
        parser.add_argument('--distill', type=int, default=0)
        parser.add_argument('--script_teacher', type=str)
        parser.add_argument('--config_teacher', type=str)
        parser.add_argument('--use_wandb', type=int, default=0)
    
        args = parser.parse_args()
    
        # 加载配置文件
        config = load_config(args.config)
    
        # 构建模型
        model = build_ceutrack(config)
    
        # 开始训练
        train(model, config)
    
    if __name__ == "__main__":
        main()
    

    确保在 build_ceutrack 函数被调用时,它总是能够正确初始化event_pretrain。

    步骤3:测试和验证

    以单卡和多卡训练模式运行几次训练命令,确保 event_pretrain 的初始化不会导致任何代码路径的错误。例如:

    1. 单卡模式:
    python train.py --script my_script --config my_config --save_dir /path/to/save --mode single
    
    1. 多卡模式:
    python train.py --script my_script --config my_config --save_dir /path/to/save --mode multiple --nproc_per_node 4
    
    1. 多节点模式:
    python train.py --script my_script --config my_config --save_dir /path/to/save --mode multi_node --world-size 2 --rank 0 --ip 127.0.0.1 --port 20000
    

    总结:确保在所有可能情况下,变量 event_pretrain 都已正确初始化和赋值,可以有效避免未定义变量导致的运行时错误。通过系统检查和测试,可以确保训练过程中的稳定性和鲁棒性。

  11. User

    分析一下:import math import torch import torch.nn as nn import torch.nn.functional as F from einops import rearrange import numbers from lib.models.mamba_fetrack.mamba_cross_simple import Mamba

    将输入张量从(b,c,h,w)重排为(b,h*w,c),常用于将图像张量转换为序列格式

    def to_3d(x): return rearrange(x, 'b c h w -> b (h w) c')

    将序列张量从(b,h*w,c)重排回(b,c,h,w),这是to_3d的反向操作,通常在完成序列处理之后用于恢复原始形状。

    def to_4d(x, h, w): return rearrange(x, 'b (h w) c -> b c h w', h=h, w=w)

    无偏LayerNorm

    实现了不包含偏置项的层标准化,计算输入的方差,用于对输入进行规范化处理,再通过权重进行缩放。

    class BiasFree_LayerNorm(nn.Module): def init(self, normalized_shape): super(BiasFree_LayerNorm, self).init() if isinstance(normalized_shape, numbers.Integral): normalized_shape = (normalized_shape,) normalized_shape = torch.Size(normalized_shape)

        assert len(normalized_shape) == 1
    
        self.weight = nn.Parameter(torch.ones(normalized_shape))
        self.normalized_shape = normalized_shape
    
    def forward(self, x):
        sigma = x.var(-1, keepdim=True, unbiased=False)
        return x / torch.sqrt(sigma+1e-5) * self.weight
    

    包含偏置的层标准化

    class WithBias_LayerNorm(nn.Module): def init(self, normalized_shape): super(WithBias_LayerNorm, self).init() if isinstance(normalized_shape, numbers.Integral): normalized_shape = (normalized_shape,) normalized_shape = torch.Size(normalized_shape)

        assert len(normalized_shape) == 1
    
        self.weight = nn.Parameter(torch.ones(normalized_shape))
        self.bias = nn.Parameter(torch.zeros(normalized_shape))
        self.normalized_shape = normalized_shape
    
    def forward(self, x):
        mu = x.mean(-1, keepdim=True)
        sigma = x.var(-1, keepdim=True, unbiased=False)
        return (x - mu) / torch.sqrt(sigma+1e-5) * self.weight + self.bias
    

    对2种不同的层标准化进行的封装 LayerNorm_type参数决定了使用哪种层标准化

    同时通过先把二维图像数据转换为3维序列,应用层标准化再转换回原来的2维结构。

    class LayerNorm(nn.Module): def init(self, dim, LayerNorm_type): super(LayerNorm, self).init() if LayerNorm_type =='BiasFree': self.body = BiasFree_LayerNorm(dim) else: self.body = WithBias_LayerNorm(dim)

    def forward(self, x):
        h, w = x.shape[-2:]
        return to_4d(self.body(to_3d(x)), h, w)
    

    和 # Copyright (c) 2023, Tri Dao, Albert Gu.

    import math from typing import Optional

    import torch import torch.nn as nn import torch.nn.functional as F from torch import Tensor

    from einops import rearrange, repeat

    try: from causal_conv1d import causal_conv1d_fn, causal_conv1d_update except ImportError: causal_conv1d_fn, causal_conv1d_update = None

    try: from mamba_ssm.ops.selective_scan_interface import selective_scan_fn, mamba_inner_fn, bimamba_inner_fn, mamba_inner_fn_no_out_proj except ImportError: # selective_scan_fn, mamba_inner_fn, bimamba_inner_fn, mamba_inner_fn_no_out_proj = None, None, None, None, None selective_scan_fn, mamba_inner_fn, bimamba_inner_fn, mamba_inner_fn_no_out_proj = None, None, None, None try: from mamba_ssm.ops.triton.selective_state_update import selective_state_update except ImportError: selective_state_update = None

    try: from mamba_ssm.ops.triton.layernorm import RMSNorm, layer_norm_fn, rms_norm_fn except ImportError: RMSNorm, layer_norm_fn, rms_norm_fn = None, None, None

    class Mamba(nn.Module): def init( self, d_model, d_state=16, d_conv=4, expand=2, dt_rank="auto", dt_min=0.001, dt_max=0.1, dt_init="random", dt_scale=1.0, dt_init_floor=1e-4, conv_bias=True, bias=False, use_fast_path=True, # Fused kernel options layer_idx=None, device=None, dtype=None, bimamba_type="v2", if_devide_out=False, init_layer_scale=None, ): factory_kwargs = {"device": device, "dtype": dtype} super().init() self.d_model = d_model self.d_state = d_state self.d_conv = d_conv self.expand = expand self.d_inner = int(self.expand * self.d_model) self.dt_rank = math.ceil(self.d_model / 16) if dt_rank == "auto" else dt_rank self.use_fast_path = use_fast_path self.layer_idx = layer_idx self.bimamba_type = bimamba_type self.if_devide_out = if_devide_out

        self.init_layer_scale = init_layer_scale
        if init_layer_scale is not None:
            self.gamma = nn.Parameter(init_layer_scale * torch.ones((d_model)), requires_grad=True)
    
        self.in_proj = nn.Linear(self.d_model, self.d_inner * 2, bias=bias, **factory_kwargs)
    
        self.conv1d = nn.Conv1d(
            in_channels=self.d_inner,
            out_channels=self.d_inner,
            bias=conv_bias,
            kernel_size=d_conv,
            groups=self.d_inner,
            padding=d_conv - 1,
            **factory_kwargs,
        )
    
        self.activation = "silu"
        self.act = nn.SiLU()
    
        self.x_proj = nn.Linear(
            self.d_inner, self.dt_rank + self.d_state * 2, bias=False, **factory_kwargs
        )
        self.dt_proj = nn.Linear(self.dt_rank, self.d_inner, bias=True, **factory_kwargs)
    
        # Initialize special dt projection to preserve variance at initialization
        dt_init_std = self.dt_rank**-0.5 * dt_scale
        if dt_init == "constant":
            nn.init.constant_(self.dt_proj.weight, dt_init_std)
        elif dt_init == "random":
            nn.init.uniform_(self.dt_proj.weight, -dt_init_std, dt_init_std)
        else:
            raise NotImplementedError
    
        # Initialize dt bias so that F.softplus(dt_bias) is between dt_min and dt_max
        dt = torch.exp(
            torch.rand(self.d_inner, **factory_kwargs) * (math.log(dt_max) - math.log(dt_min))
            + math.log(dt_min)
        ).clamp(min=dt_init_floor)
        # Inverse of softplus: https://github.com/pytorch/pytorch/issues/72759
        inv_dt = dt + torch.log(-torch.expm1(-dt))
        with torch.no_grad():
            self.dt_proj.bias.copy_(inv_dt)
        # Our initialization would set all Linear.bias to zero, need to mark this one as _no_reinit
        self.dt_proj.bias._no_reinit = True
    
        # S4D real initialization
        A = repeat(
            torch.arange(1, self.d_state + 1, dtype=torch.float32, device=device),
            "n -> d n",
            d=self.d_inner,
        ).contiguous()
        A_log = torch.log(A)  # Keep A_log in fp32
        self.A_log = nn.Parameter(A_log)
        self.A_log._no_weight_decay = True
    
        # D "skip" parameter
        self.D = nn.Parameter(torch.ones(self.d_inner, device=device))  # Keep in fp32
        self.D._no_weight_decay = True
    
        # bidirectional
        if bimamba_type == "v1":
            A_b = repeat(
                torch.arange(1, self.d_state + 1, dtype=torch.float32, device=device),
                "n -> d n",
                d=self.d_inner,
            ).contiguous()
            A_b_log = torch.log(A_b)  # Keep A_b_log in fp32
            self.A_b_log = nn.Parameter(A_b_log)
            self.A_b_log._no_weight_decay = True
        elif bimamba_type == "v2" or bimamba_type=='v3':
            A_b = repeat(
                torch.arange(1, self.d_state + 1, dtype=torch.float32, device=device),
                "n -> d n",
                d=self.d_inner,
            ).contiguous()
            A_b_log = torch.log(A_b)  # Keep A_b_log in fp32
            self.A_b_log = nn.Parameter(A_b_log)
            self.A_b_log._no_weight_decay = True 
    
            self.conv1d_b = nn.Conv1d(
                in_channels=self.d_inner,
                out_channels=self.d_inner,
                bias=conv_bias,
                kernel_size=d_conv,
                groups=self.d_inner,
                padding=d_conv - 1,
                **factory_kwargs,
            )
    
            self.x_proj_b = nn.Linear(
                self.d_inner, self.dt_rank + self.d_state * 2, bias=False, **factory_kwargs
            )
            self.dt_proj_b = nn.Linear(self.dt_rank, self.d_inner, bias=True, **factory_kwargs)
    
            self.D_b = nn.Parameter(torch.ones(self.d_inner, device=device))  # Keep in fp32
            self.D_b._no_weight_decay = True
            self.in_proj_extra = nn.Linear(self.d_model, self.d_inner * 2, bias=bias, **factory_kwargs)
    
        self.out_proj = nn.Linear(self.d_inner, self.d_model, bias=bias, **factory_kwargs)
    
    def forward(self, hidden_states, inference_params=None,extra_emb=None):
        """
        hidden_states: (B, L, D)
        Returns: same shape as hidden_states
        """
        batch, seqlen, dim = hidden_states.shape
    
        conv_state, ssm_state = None, None
        if inference_params is not None:
            conv_state, ssm_state = self._get_states_from_cache(inference_params, batch)
            if inference_params.seqlen_offset > 0:
                # The states are updated inplace
                out, _, _ = self.step(hidden_states, conv_state, ssm_state)
                return out
        if extra_emb is None:
            # We do matmul and transpose BLH -> HBL at the same time
            xz = rearrange(
                self.in_proj.weight @ rearrange(hidden_states, "b l d -> d (b l)"),
                "d (b l) -> b d l",
                l=seqlen,
            )
            if self.in_proj.bias is not None:
                xz = xz + rearrange(self.in_proj.bias.to(dtype=xz.dtype), "d -> d 1")
        else:
            xz = rearrange(
                self.in_proj.weight @ rearrange(hidden_states, "b l d -> d (b l)"),
                "d (b l) -> b d l",
                l=seqlen,
            )
            if self.in_proj.bias is not None:
                xz = xz + rearrange(self.in_proj.bias.to(dtype=xz.dtype), "d -> d 1")
            extra_emb = rearrange(
                self.in_proj_extra.weight @ rearrange(extra_emb, "b l d -> d (b l)"),
                "d (b l) -> b d l",
                l=seqlen,
            )
            if self.in_proj_extra.bias is not None:
                extra_emb = extra_emb + rearrange(self.in_proj_extra.bias.to(dtype=xz.dtype), "d -> d 1")
        A = -torch.exp(self.A_log.float())  # (d_inner, d_state)
        # In the backward pass we write dx and dz next to each other to avoid torch.cat
        if self.use_fast_path and inference_params is None:  # Doesn't support outputting the states
            if self.bimamba_type == "v1":
                A_b = -torch.exp(self.A_b_log.float())
                out = bimamba_inner_fn(
                    xz,
                    self.conv1d.weight,
                    self.conv1d.bias,
                    self.x_proj.weight,
                    self.dt_proj.weight,
                    self.out_proj.weight,
                    self.out_proj.bias,
                    A,
                    A_b,
                    None,  # input-dependent B
                    None,  # input-dependent C
                    self.D.float(),
                    delta_bias=self.dt_proj.bias.float(),
                    delta_softplus=True,
                )    
            elif self.bimamba_type == "v2":
                A_b = -torch.exp(self.A_b_log.float())
                out = mamba_inner_fn_no_out_proj(
                    xz,
                    self.conv1d.weight,
                    self.conv1d.bias,
                    self.x_proj.weight,
                    self.dt_proj.weight,
                    A,
                    None,  # input-dependent B
                    None,  # input-dependent C
                    self.D.float(),
                    delta_bias=self.dt_proj.bias.float(),
                    delta_softplus=True,
                )
                out_b = mamba_inner_fn_no_out_proj(
                    xz.flip([-1]),
                    self.conv1d_b.weight,
                    self.conv1d_b.bias,
                    self.x_proj_b.weight,
                    self.dt_proj_b.weight,
                    A_b,
                    None,
                    None,
                    self.D_b.float(),
                    delta_bias=self.dt_proj_b.bias.float(),
                    delta_softplus=True,
                )
                # F.linear(rearrange(out_z, "b d l -> b l d"), out_proj_weight, out_proj_bias)
                if not self.if_devide_out:
                    out = F.linear(rearrange(out + out_b.flip([-1]), "b d l -> b l d"), self.out_proj.weight, self.out_proj.bias)
                else:
                    out = F.linear(rearrange(out + out_b.flip([-1]), "b d l -> b l d") / 2, self.out_proj.weight, self.out_proj.bias)
            elif self.bimamba_type == "v3":
                A_b = -torch.exp(self.A_b_log.float())
                out = mamba_inner_fn_no_out_proj(
                    xz,
                    self.conv1d.weight,
                    self.conv1d.bias,
                    self.x_proj.weight,
                    self.dt_proj.weight,
                    A,
                    None,  # input-dependent B
                    None,  # input-dependent C
                    self.D.float(),
                    delta_bias=self.dt_proj.bias.float(),
                    delta_softplus=True,
                )
                out_b = mamba_inner_fn_no_out_proj(
                    # extra_emb同上文中xz.flip不同
                    extra_emb,
                    self.conv1d_b.weight,
                    self.conv1d_b.bias,
                    self.x_proj_b.weight,
                    self.dt_proj_b.weight,
                    A_b,
                    None,
                    None,
                    self.D_b.float(),
                    delta_bias=self.dt_proj_b.bias.float(),
                    delta_softplus=True,
                )
                # F.linear(rearrange(out_z, "b d l -> b l d"), out_proj_weight, out_proj_bias)
                if not self.if_devide_out:             #True
                    out = F.linear(rearrange(out + out_b, "b d l -> b l d"), self.out_proj.weight, self.out_proj.bias)
                else:                                  #False
                    out = F.linear(rearrange(out + out_b, "b d l -> b l d") / 2, self.out_proj.weight, self.out_proj.bias)
    
            else:           #False
                out = mamba_inner_fn(
                    xz,
                    self.conv1d.weight,
                    self.conv1d.bias,
                    self.x_proj.weight,
                    self.dt_proj.weight,
                    self.out_proj.weight,
                    self.out_proj.bias,
                    A,
                    None,  # input-dependent B
                    None,  # input-dependent C
                    self.D.float(),
                    delta_bias=self.dt_proj.bias.float(),
                    delta_softplus=True,
                )
        else:            #False
            x, z = xz.chunk(2, dim=1)
            # Compute short convolution
            if conv_state is not None:
                # If we just take x[:, :, -self.d_conv :], it will error if seqlen < self.d_conv
                # Instead F.pad will pad with zeros if seqlen < self.d_conv, and truncate otherwise.
                conv_state.copy_(F.pad(x, (self.d_conv - x.shape[-1], 0)))  # Update state (B D W)
            if causal_conv1d_fn is None:
                x = self.act(self.conv1d(x)[..., :seqlen])
            else:
                assert self.activation in ["silu", "swish"]
                x = causal_conv1d_fn(
                    x=x,
                    weight=rearrange(self.conv1d.weight, "d 1 w -> d w"),
                    bias=self.conv1d.bias,
                    activation=self.activation,
                )
    
            # We're careful here about the layout, to avoid extra transposes.
            # We want dt to have d as the slowest moving dimension
            # and L as the fastest moving dimension, since those are what the ssm_scan kernel expects.
            x_dbl = self.x_proj(rearrange(x, "b d l -> (b l) d"))  # (bl d)
            dt, B, C = torch.split(x_dbl, [self.dt_rank, self.d_state, self.d_state], dim=-1)
            dt = self.dt_proj.weight @ dt.t()
            dt = rearrange(dt, "d (b l) -> b d l", l=seqlen)
            B = rearrange(B, "(b l) dstate -> b dstate l", l=seqlen).contiguous()
            C = rearrange(C, "(b l) dstate -> b dstate l", l=seqlen).contiguous()
            assert self.activation in ["silu", "swish"]
            y = selective_scan_fn(
                x,
                dt,
                A,
                B,
                C,
                self.D.float(),
                z=z,
                delta_bias=self.dt_proj.bias.float(),
                delta_softplus=True,
                return_last_state=ssm_state is not None,
            )
            if ssm_state is not None:
                y, last_state = y
                ssm_state.copy_(last_state)
            y = rearrange(y, "b d l -> b l d")
            out = self.out_proj(y)
        if self.init_layer_scale is not None:           #False
                out = out * self.gamma    
        return out
    
    def step(self, hidden_states, conv_state, ssm_state):
        dtype = hidden_states.dtype
        assert hidden_states.shape[1] == 1, "Only support decoding with 1 token at a time for now"
        xz = self.in_proj(hidden_states.squeeze(1))  # (B 2D)
        x, z = xz.chunk(2, dim=-1)  # (B D)
    
        # Conv step
        if causal_conv1d_update is None:
            conv_state.copy_(torch.roll(conv_state, shifts=-1, dims=-1))  # Update state (B D W)
            conv_state[:, :, -1] = x
            x = torch.sum(conv_state * rearrange(self.conv1d.weight, "d 1 w -> d w"), dim=-1)  # (B D)
            if self.conv1d.bias is not None:
                x = x + self.conv1d.bias
            x = self.act(x).to(dtype=dtype)
        else:
            x = causal_conv1d_update(
                x,
                conv_state,
                rearrange(self.conv1d.weight, "d 1 w -> d w"),
                self.conv1d.bias,
                self.activation,
            )
    
        x_db = self.x_proj(x)  # (B dt_rank+2*d_state)
        dt, B, C = torch.split(x_db, [self.dt_rank, self.d_state, self.d_state], dim=-1)
        # Don't add dt_bias here
        dt = F.linear(dt, self.dt_proj.weight)  # (B d_inner)
        A = -torch.exp(self.A_log.float())  # (d_inner, d_state)
    
        # SSM step
        if selective_state_update is None:
            # Discretize A and B
            dt = F.softplus(dt + self.dt_proj.bias.to(dtype=dt.dtype))
            dA = torch.exp(torch.einsum("bd,dn->bdn", dt, A))
            dB = torch.einsum("bd,bn->bdn", dt, B)
            ssm_state.copy_(ssm_state * dA + rearrange(x, "b d -> b d 1") * dB)
            y = torch.einsum("bdn,bn->bd", ssm_state.to(dtype), C)
            y = y + self.D.to(dtype) * x
            y = y * self.act(z)  # (B D)
        else:
            y = selective_state_update(
                ssm_state, x, dt, A, B, C, self.D, z=z, dt_bias=self.dt_proj.bias, dt_softplus=True
            )
    
        out = self.out_proj(y)
        return out.unsqueeze(1), conv_state, ssm_state
    
    def allocate_inference_cache(self, batch_size, max_seqlen, dtype=None, **kwargs):
        device = self.out_proj.weight.device
        conv_dtype = self.conv1d.weight.dtype if dtype is None else dtype
        conv_state = torch.zeros(
            batch_size, self.d_model * self.expand, self.d_conv, device=device, dtype=conv_dtype
        )
        ssm_dtype = self.dt_proj.weight.dtype if dtype is None else dtype
        # ssm_dtype = torch.float32
        ssm_state = torch.zeros(
            batch_size, self.d_model * self.expand, self.d_state, device=device, dtype=ssm_dtype
        )
        return conv_state, ssm_state
    
    def _get_states_from_cache(self, inference_params, batch_size, initialize_states=False):
        assert self.layer_idx is not None
        if self.layer_idx not in inference_params.key_value_memory_dict:
            batch_shape = (batch_size,)
            conv_state = torch.zeros(
                batch_size,
                self.d_model * self.expand,
                self.d_conv,
                device=self.conv1d.weight.device,
                dtype=self.conv1d.weight.dtype,
            )
            ssm_state = torch.zeros(
                batch_size,
                self.d_model * self.expand,
                self.d_state,
                device=self.dt_proj.weight.device,
                dtype=self.dt_proj.weight.dtype,
                # dtype=torch.float32,
            )
            inference_params.key_value_memory_dict[self.layer_idx] = (conv_state, ssm_state)
        else:
            conv_state, ssm_state = inference_params.key_value_memory_dict[self.layer_idx]
            # TODO: What if batch size changes between generation, and we reuse the same states?
            if initialize_states:
                conv_state.zero_()
                ssm_state.zero_()
        return conv_state, ssm_state
    

    class Block(nn.Module): def init( self, dim, mixer_cls, norm_cls=nn.LayerNorm, fused_add_norm=False, residual_in_fp32=False ): """ Simple block wrapping a mixer class with LayerNorm/RMSNorm and residual connection"

        This Block has a slightly different structure compared to a regular
        prenorm Transformer block.
        The standard block is: LN -> MHA/MLP -> Add.
        [Ref: https://arxiv.org/abs/2002.04745]
        Here we have: Add -> LN -> Mixer, returning both
        the hidden_states (output of the mixer) and the residual.
        This is purely for performance reasons, as we can fuse add and LayerNorm.
        The residual needs to be provided (except for the very first block).
        """
        super().__init__()
        self.residual_in_fp32 = residual_in_fp32
        self.fused_add_norm = fused_add_norm
        self.mixer = mixer_cls(dim)
        self.norm = norm_cls(dim)
        if self.fused_add_norm:
            assert RMSNorm is not None, "RMSNorm import fails"
            assert isinstance(
                self.norm, (nn.LayerNorm, RMSNorm)
            ), "Only LayerNorm and RMSNorm are supported for fused_add_norm"
    
    def forward(
        self, hidden_states: Tensor, residual: Optional[Tensor] = None, inference_params=None
    ):
        r"""Pass the input through the encoder layer.
    
        Args:
            hidden_states: the sequence to the encoder layer (required).
            residual: hidden_states = Mixer(LN(residual))
        """
        if not self.fused_add_norm:
            residual = (hidden_states + residual) if residual is not None else hidden_states
            hidden_states = self.norm(residual.to(dtype=self.norm.weight.dtype))
            if self.residual_in_fp32:
                residual = residual.to(torch.float32)
        else:
            fused_add_norm_fn = rms_norm_fn if isinstance(self.norm, RMSNorm) else layer_norm_fn
            hidden_states, residual = fused_add_norm_fn(
                hidden_states,
                self.norm.weight,
                self.norm.bias,
                residual=residual,
                prenorm=True,
                residual_in_fp32=self.residual_in_fp32,
                eps=self.norm.eps,
            )
        hidden_states = self.mixer(hidden_states, inference_params=inference_params)
        return hidden_states, residual
    
    def allocate_inference_cache(self, batch_size, max_seqlen, dtype=None, **kwargs):
        return self.mixer.allocate_inference_cache(batch_size, max_seqlen, dtype=dtype, **kwargs)
    

    class DWMamba(nn.Module): def init( self, d_model, d_state=16, d_conv=4, expand=2, dt_rank="auto", dt_min=0.001, dt_max=0.1, dt_init="random", dt_scale=1.0, dt_init_floor=1e-4, conv_bias=True, bias=False, use_fast_path=True, # Fused kernel options layer_idx=None, device=None, dtype=None, bimamba_type="v2", if_devide_out=False, init_layer_scale=None, ): factory_kwargs = {"device": device, "dtype": dtype} super().init() self.d_model = d_model self.d_state = d_state self.d_conv = d_conv self.expand = expand self.d_inner = int(self.expand * self.d_model) self.dt_rank = math.ceil(self.d_model / 16) if dt_rank == "auto" else dt_rank self.use_fast_path = use_fast_path self.layer_idx = layer_idx self.bimamba_type = bimamba_type self.if_devide_out = if_devide_out

        self.init_layer_scale = init_layer_scale
        if init_layer_scale is not None:
            self.gamma = nn.Parameter(init_layer_scale * torch.ones((d_model)), requires_grad=True)
    
        self.in_proj = nn.Linear(self.d_model, self.d_inner * 2, bias=bias, **factory_kwargs)
    
        self.conv1d = nn.Conv1d(
            in_channels=self.d_inner,
            out_channels=self.d_inner,
            bias=conv_bias,
            kernel_size=d_conv,
            groups=self.d_inner,
            padding=d_conv - 1,
            **factory_kwargs,
        )
    
        self.activation = "silu"
        self.act = nn.SiLU()
    
        self.x_proj = nn.Linear(
            self.d_inner, self.dt_rank + self.d_state * 2, bias=False, **factory_kwargs
        )
        self.dt_proj = nn.Linear(self.dt_rank, self.d_inner, bias=True, **factory_kwargs)
    
        # Initialize special dt projection to preserve variance at initialization
        dt_init_std = self.dt_rank**-0.5 * dt_scale
        if dt_init == "constant":
            nn.init.constant_(self.dt_proj.weight, dt_init_std)
        elif dt_init == "random":
            nn.init.uniform_(self.dt_proj.weight, -dt_init_std, dt_init_std)
        else:
            raise NotImplementedError
    
        # Initialize dt bias so that F.softplus(dt_bias) is between dt_min and dt_max
        dt = torch.exp(
            torch.rand(self.d_inner, **factory_kwargs) * (math.log(dt_max) - math.log(dt_min))
            + math.log(dt_min)
        ).clamp(min=dt_init_floor)
        # Inverse of softplus: https://github.com/pytorch/pytorch/issues/72759
        inv_dt = dt + torch.log(-torch.expm1(-dt))
        with torch.no_grad():
            self.dt_proj.bias.copy_(inv_dt)
        # Our initialization would set all Linear.bias to zero, need to mark this one as _no_reinit
        self.dt_proj.bias._no_reinit = True
    
        # S4D real initialization
        A = repeat(
            torch.arange(1, self.d_state + 1, dtype=torch.float32, device=device),
            "n -> d n",
            d=self.d_inner,
        ).contiguous()
        A_log = torch.log(A)  # Keep A_log in fp32
        self.A_log = nn.Parameter(A_log)
        self.A_log._no_weight_decay = True
    
        # D "skip" parameter
        self.D = nn.Parameter(torch.ones(self.d_inner, device=device))  # Keep in fp32
        self.D._no_weight_decay = True
    
        # bidirectional
        if bimamba_type == "v1":
            A_b = repeat(
                torch.arange(1, self.d_state + 1, dtype=torch.float32, device=device),
                "n -> d n",
                d=self.d_inner,
            ).contiguous()
            A_b_log = torch.log(A_b)  # Keep A_b_log in fp32
            self.A_b_log = nn.Parameter(A_b_log)
            self.A_b_log._no_weight_decay = True
        elif bimamba_type == "v2" or bimamba_type=='v3':
            A_b = repeat(
                torch.arange(1, self.d_state + 1, dtype=torch.float32, device=device),
                "n -> d n",
                d=self.d_inner,
            ).contiguous()
            A_b_log = torch.log(A_b)  # Keep A_b_log in fp32
            self.A_b_log = nn.Parameter(A_b_log)
            self.A_b_log._no_weight_decay = True 
    
            self.conv1d_b = nn.Conv1d(
                in_channels=self.d_inner,
                out_channels=self.d_inner,
                bias=conv_bias,
                kernel_size=d_conv,
                groups=self.d_inner,
                padding=d_conv - 1,
                **factory_kwargs,
            )
    
            self.x_proj_b = nn.Linear(
                self.d_inner, self.dt_rank + self.d_state * 2, bias=False, **factory_kwargs
            )
            self.dt_proj_b = nn.Linear(self.dt_rank, self.d_inner, bias=True, **factory_kwargs)
    
            self.D_b = nn.Parameter(torch.ones(self.d_inner, device=device))  # Keep in fp32
            self.D_b._no_weight_decay = True
            self.in_proj_extra = nn.Linear(self.d_model, self.d_inner * 2, bias=bias, **factory_kwargs)
    
        self.out_proj = nn.Linear(self.d_inner, self.d_model, bias=bias, **factory_kwargs)
    
    def forward(self, hidden_states, inference_params=None,extra_emb=None):
        """
        hidden_states: (B, L, D)
        Returns: same shape as hidden_states
        """
        batch, seqlen, dim = hidden_states.shape
    
        conv_state, ssm_state = None, None
        if inference_params is not None:
            conv_state, ssm_state = self._get_states_from_cache(inference_params, batch)
            if inference_params.seqlen_offset > 0:
                # The states are updated inplace
                out, _, _ = self.step(hidden_states, conv_state, ssm_state)
                return out
        if extra_emb is None:
            # We do matmul and transpose BLH -> HBL at the same time
            xz = rearrange(
                self.in_proj.weight @ rearrange(hidden_states, "b l d -> d (b l)"),
                "d (b l) -> b d l",
                l=seqlen,
            )
            if self.in_proj.bias is not None:
                xz = xz + rearrange(self.in_proj.bias.to(dtype=xz.dtype), "d -> d 1")
        else:
            xz = rearrange(
                self.in_proj.weight @ rearrange(hidden_states, "b l d -> d (b l)"),
                "d (b l) -> b d l",
                l=seqlen,
            )
            if self.in_proj.bias is not None:
                xz = xz + rearrange(self.in_proj.bias.to(dtype=xz.dtype), "d -> d 1")
            extra_emb = rearrange(
                self.in_proj_extra.weight @ rearrange(extra_emb, "b l d -> d (b l)"),
                "d (b l) -> b d l",
                l=seqlen,
            )
            if self.in_proj_extra.bias is not None:
                extra_emb = extra_emb + rearrange(self.in_proj_extra.bias.to(dtype=xz.dtype), "d -> d 1")
        A = -torch.exp(self.A_log.float())  # (d_inner, d_state)
        # In the backward pass we write dx and dz next to each other to avoid torch.cat
        if self.use_fast_path and inference_params is None:  # Doesn't support outputting the states
            # mamba3个版本
            # v1版本涉及双向处理
            if self.bimamba_type == "v1":
                # 创建指数函数,用于双向处理
                A_b = -torch.exp(self.A_b_log.float())
                # 调用bimamba_inner_fn进行双向数据处理
                out = bimamba_inner_fn(
                    xz,
                    self.conv1d.weight,
                    self.conv1d.bias,
                    self.x_proj.weight,
                    self.dt_proj.weight,
                    self.out_proj.weight,
                    self.out_proj.bias,
                    A,
                    A_b,
                    None,  # input-dependent B
                    None,  # input-dependent C
                    self.D.float(),
                    delta_bias=self.dt_proj.bias.float(),
                    delta_softplus=True,
                )    
            # 版本2提供单向和反向处理,不直接使用输出投影合并结果
            # 对每个方向分别计算状态更新,反转flip输入以模拟反向处理
            # 结果通过加权和合并,然后应用线性输出投影;
    
            elif self.bimamba_type == "v2":
                A_b = -torch.exp(self.A_b_log.float())
                out = mamba_inner_fn_no_out_proj(
                    xz,
                    self.conv1d.weight,
                    self.conv1d.bias,
                    self.x_proj.weight,
                    self.dt_proj.weight,
                    A,
                    None,  # input-dependent B
                    None,  # input-dependent C
                    self.D.float(),
                    delta_bias=self.dt_proj.bias.float(),
                    delta_softplus=True,
                )
                out_b = mamba_inner_fn_no_out_proj(
                    xz.flip([-1]),
                    self.conv1d_b.weight,
                    self.conv1d_b.bias,
                    self.x_proj_b.weight,
                    self.dt_proj_b.weight,
                    A_b,
                    None,
                    None,
                    self.D_b.float(),
                    delta_bias=self.dt_proj_b.bias.float(),
                    delta_softplus=True,
                )
                # F.linear(rearrange(out_z, "b d l -> b l d"), out_proj_weight, out_proj_bias)
                if not self.if_devide_out:
                    out = F.linear(rearrange(out + out_b.flip([-1]), "b d l -> b l d"), self.out_proj.weight, self.out_proj.bias)
                else:
                    out = F.linear(rearrange(out + out_b.flip([-1]), "b d l -> b l d") / 2, self.out_proj.weight, self.out_proj.bias)
            # 基于v2版本,加入对额外输入的并行处理
            # 将extra_emb作为额外的上下文信息被处理,并与主输入并行处理
            # 不反转extra_emb表示它可能与主序列的方向一致
            # 结果合并后通过一个线性层来产生最终输出,允许融合来自两个不同源信息
            # conv1d和conv1d_b 用于提取局部特征
            # 状态映射层 x_proj 和 x_proj_b 用于将局部特征映射到状态更新形式
            # 时间步映射 dt_proj 和 dt_proj_b 用于控制状态更新的时间步长大小
            # 状态递推参数 A 和 A_b 用于控制状态递推的动态变化
            # 直通参数D和D_b用于控制状态更新的跳跃大小,在更新时直接与输入相加,维持某种输入特征的直通。
            elif self.bimamba_type == "v3":
                A_b = -torch.exp(self.A_b_log.float())
                out = mamba_inner_fn_no_out_proj(
                    xz,
                    self.conv1d.weight,
                    self.conv1d.bias,
                    self.x_proj.weight,
                    self.dt_proj.weight,
                    A,
                    None,  # input-dependent B
                    None,  # input-dependent C
                    self.D.float(),
                    delta_bias=self.dt_proj.bias.float(),
                    delta_softplus=True,
                )
                out_b = mamba_inner_fn_no_out_proj(
                    extra_emb,
                    self.conv1d_b.weight,
                    self.conv1d_b.bias,
                    self.x_proj_b.weight,
                    self.dt_proj_b.weight,
                    A_b,
                    None,
                    None,
                    self.D_b.float(),
                    delta_bias=self.dt_proj_b.bias.float(),
                    delta_softplus=True,
                )
                # F.linear(rearrange(out_z, "b d l -> b l d"), out_proj_weight, out_proj_bias)
                if not self.if_devide_out:
                    out = F.linear(rearrange(out + out_b, "b d l -> b l d"), self.out_proj.weight, self.out_proj.bias)
                else:
                    out = F.linear(rearrange(out + out_b, "b d l -> b l d") / 2, self.out_proj.weight, self.out_proj.bias)
    
            else:
                out = mamba_inner_fn(
                    xz,
                    self.conv1d.weight,
                    self.conv1d.bias,
                    self.x_proj.weight,
                    self.dt_proj.weight,
                    self.out_proj.weight,
                    self.out_proj.bias,
                    A,
                    None,  # input-dependent B
                    None,  # input-dependent C
                    self.D.float(),
                    delta_bias=self.dt_proj.bias.float(),
                    delta_softplus=True,
                )
        else:
            x, z = xz.chunk(2, dim=1)
            # Compute short convolution
            if conv_state is not None:
                # If we just take x[:, :, -self.d_conv :], it will error if seqlen < self.d_conv
                # Instead F.pad will pad with zeros if seqlen < self.d_conv, and truncate otherwise.
                conv_state.copy_(F.pad(x, (self.d_conv - x.shape[-1], 0)))  # Update state (B D W)
            if causal_conv1d_fn is None:
                x = self.act(self.conv1d(x)[..., :seqlen])
            else:
                assert self.activation in ["silu", "swish"]
                x = causal_conv1d_fn(
                    x=x,
                    weight=rearrange(self.conv1d.weight, "d 1 w -> d w"),
                    bias=self.conv1d.bias,
                    activation=self.activation,
                )
    
            # We're careful here about the layout, to avoid extra transposes.
            # We want dt to have d as the slowest moving dimension
            # and L as the fastest moving dimension, since those are what the ssm_scan kernel expects.
            x_dbl = self.x_proj(rearrange(x, "b d l -> (b l) d"))  # (bl d)
            dt, B, C = torch.split(x_dbl, [self.dt_rank, self.d_state, self.d_state], dim=-1)
            dt = self.dt_proj.weight @ dt.t()
            dt = rearrange(dt, "d (b l) -> b d l", l=seqlen)
            B = rearrange(B, "(b l) dstate -> b dstate l", l=seqlen).contiguous()
            C = rearrange(C, "(b l) dstate -> b dstate l", l=seqlen).contiguous()
            assert self.activation in ["silu", "swish"]
            y = selective_scan_fn(
                x,
                dt,
                A,
                B,
                C,
                self.D.float(),
                z=z,
                delta_bias=self.dt_proj.bias.float(),
                delta_softplus=True,
                return_last_state=ssm_state is not None,
            )
            if ssm_state is not None:
                y, last_state = y
                ssm_state.copy_(last_state)
            y = rearrange(y, "b d l -> b l d")
            out = self.out_proj(y)
        if self.init_layer_scale is not None:
                out = out * self.gamma    
        return out
    
    def step(self, hidden_states, conv_state, ssm_state):
        dtype = hidden_states.dtype
        assert hidden_states.shape[1] == 1, "Only support decoding with 1 token at a time for now"
        xz = self.in_proj(hidden_states.squeeze(1))  # (B 2D)
        x, z = xz.chunk(2, dim=-1)  # (B D)
    
        # Conv step
        if causal_conv1d_update is None:
            conv_state.copy_(torch.roll(conv_state, shifts=-1, dims=-1))  # Update state (B D W)
            conv_state[:, :, -1] = x
            x = torch.sum(conv_state * rearrange(self.conv1d.weight, "d 1 w -> d w"), dim=-1)  # (B D)
            if self.conv1d.bias is not None:
                x = x + self.conv1d.bias
            x = self.act(x).to(dtype=dtype)
        else:
            x = causal_conv1d_update(
                x,
                conv_state,
                rearrange(self.conv1d.weight, "d 1 w -> d w"),
                self.conv1d.bias,
                self.activation,
            )
    
        x_db = self.x_proj(x)  # (B dt_rank+2*d_state)
        dt, B, C = torch.split(x_db, [self.dt_rank, self.d_state, self.d_state], dim=-1)
        # Don't add dt_bias here
        dt = F.linear(dt, self.dt_proj.weight)  # (B d_inner)
        A = -torch.exp(self.A_log.float())  # (d_inner, d_state)
    
        # SSM step
        if selective_state_update is None:
            # Discretize A and B
            dt = F.softplus(dt + self.dt_proj.bias.to(dtype=dt.dtype))
            dA = torch.exp(torch.einsum("bd,dn->bdn", dt, A))
            dB = torch.einsum("bd,bn->bdn", dt, B)
            ssm_state.copy_(ssm_state * dA + rearrange(x, "b d -> b d 1") * dB)
            y = torch.einsum("bdn,bn->bd", ssm_state.to(dtype), C)
            y = y + self.D.to(dtype) * x
            y = y * self.act(z)  # (B D)
        else:
            y = selective_state_update(
                ssm_state, x, dt, A, B, C, self.D, z=z, dt_bias=self.dt_proj.bias, dt_softplus=True
            )
    
        out = self.out_proj(y)
        return out.unsqueeze(1), conv_state, ssm_state
    
    def allocate_inference_cache(self, batch_size, max_seqlen, dtype=None, **kwargs):
        device = self.out_proj.weight.device
        conv_dtype = self.conv1d.weight.dtype if dtype is None else dtype
        conv_state = torch.zeros(
            batch_size, self.d_model * self.expand, self.d_conv, device=device, dtype=conv_dtype
        )
        ssm_dtype = self.dt_proj.weight.dtype if dtype is None else dtype
        # ssm_dtype = torch.float32
        ssm_state = torch.zeros(
            batch_size, self.d_model * self.expand, self.d_state, device=device, dtype=ssm_dtype
        )
        return conv_state, ssm_state
    
    def _get_states_from_cache(self, inference_params, batch_size, initialize_states=False):
        assert self.layer_idx is not None
        if self.layer_idx not in inference_params.key_value_memory_dict:
            batch_shape = (batch_size,)
            conv_state = torch.zeros(
                batch_size,
                self.d_model * self.expand,
                self.d_conv,
                device=self.conv1d.weight.device,
                dtype=self.conv1d.weight.dtype,
            )
            ssm_state = torch.zeros(
                batch_size,
                self.d_model * self.expand,
                self.d_state,
                device=self.dt_proj.weight.device,
                dtype=self.dt_proj.weight.dtype,
                # dtype=torch.float32,
            )
            inference_params.key_value_memory_dict[self.layer_idx] = (conv_state, ssm_state)
        else:
            conv_state, ssm_state = inference_params.key_value_memory_dict[self.layer_idx]
            # TODO: What if batch size changes between generation, and we reuse the same states?
            if initialize_states:
                conv_state.zero_()
                ssm_state.zero_()
        return conv_state, ssm_state
    中关于crossmamba的作用,以及输入和输出是啥?
    
  12. Assistant

    CrossMamba的作用:

    CrossMamba是一种用于处理序列数据的复杂深度学习模型,基于多层Perceiver和混合卷积-状态空间模型(State Space Models,SSM)。它的目标是利用更复杂的时间关系来提高模型处理序列任务的能力。

    这个模型由多个部分组成,包括输入投影、卷积层、状态空间模块、激活函数和输出投影等。这些部分共同作用,实现对输入序列的特征提取、序列建模以及最终的预测。

    输入和输出:

    • 输入:输入是一个三维张量 hidden_states,形状为 (B, L, D),其中:

      • B:表示批处理大小(batch size)
      • L:表示序列长度(sequence length)
      • D:表示特征维度(feature dimension)
    • 输出:输出是和输入同样形状的三维张量,形状为 (B, L, D)。输出张量在经过数层处理后,保留了原输入的形状,但特征已经被丰富和转换,用于解决具体的任务(如分类、回归等)。

    代码分析:

    下面分析每个关键部分的作用:

    1. 模型初始化:

      • 初始化各类参数,如维度、卷积核大小、状态大小等。
      • 创建输入投影层、卷积层、投影层、状态空间模块等。
    2. forward 函数: 主要执行前向传播,包含以下几个步骤:

      • 状态初始化:检查是否需要从缓存中获取状态,如果需要就从缓存中获取状态。
      • 输入投影:对输入进行线性投影,以获得特征表示。
      • 卷积操作:通过卷积层提取局部特征。
      • 状态更新:
        • 计算 x 的线性投影,得到时间步 dt、状态更新 B 和 C。
        • 使用状态空间模型(SSM)更新状态。
      • 双向处理:
        • 若为双向处理,则会根据 bimamba_type 进行不同的方法选择,并进行双向的状态更新及组合处理。
      • 输出投影:对更新后的状态特征进行线性投影,得到最终输出。
    class Mamba(nn.Module):
        def __init__(self, d_model, d_state=16, d_conv=4, expand=2, dt_rank="auto", dt_min=0.001, 
                     dt_max=0.1, dt_init="random", dt_scale=1.0, dt_init_floor=1e-4, conv_bias=True, 
                     bias=False, use_fast_path=True, layer_idx=None, device=None, dtype=None, 
                     bimamba_type="v2", if_devide_out=False, init_layer_scale=None):
            factory_kwargs = {"device": device, "dtype": dtype}
            super().__init__()
            
            self.d_model = d_model
            self.d_state = d_state
            self.d_conv = d_conv
            self.expand = expand
            self.d_inner = int(self.expand * self.d_model)
            self.dt_rank = math.ceil(self.d_model / 16) if dt_rank == "auto" else dt_rank
            self.use_fast_path = use_fast_path
            self.layer_idx = layer_idx
            self.bimamba_type = bimamba_type
            self.if_devide_out = if_devide_out
    
            self.init_layer_scale = init_layer_scale
            if init_layer_scale is not None:
                self.gamma = nn.Parameter(init_layer_scale * torch.ones((d_model)), requires_grad=True)
    
            self.in_proj = nn.Linear(self.d_model, self.d_inner * 2, bias=bias, **factory_kwargs)
    
            self.conv1d = nn.Conv1d(
                in_channels=self.d_inner,
                out_channels=self.d_inner,
                bias=conv_bias,
                kernel_size=d_conv,
                groups=self.d_inner,
                padding=d_conv - 1,
                **factory_kwargs,
            )
    
            self.activation = "silu"
            self.act = nn.SiLU()
    
            self.x_proj = nn.Linear(
                self.d_inner, self.dt_rank + self.d_state * 2, bias=False, **factory_kwargs
            )
            self.dt_proj = nn.Linear(self.dt_rank, self.d_inner, bias=True, **factory_kwargs)
    
            # Initialize special dt projection to preserve variance at initialization
            dt_init_std = self.dt_rank**-0.5 * dt_scale
            if dt_init == "constant":
                nn.init.constant_(self.dt_proj.weight, dt_init_std)
            elif dt_init == "random":
                nn.init.uniform_(self.dt_proj.weight, -dt_init_std, dt_init_std)
            else:
                raise NotImplementedError
    
            # Initialize dt bias so that F.softplus(dt_bias) is between dt_min and dt_max
            dt = torch.exp(
                torch.rand(self.d_inner, **factory_kwargs) * (math.log(dt_max) - math.log(dt_min))
                + math.log(dt_min)
            ).clamp(min=dt_init_floor)
            inv_dt = dt + torch.log(-torch.expm1(-dt))  # Inverse of softplus
            with torch.no_grad():
                self.dt_proj.bias.copy_(inv_dt)
            self.dt_proj.bias._no_reinit = True
    
            # S4D real initialization
            A = repeat(
                torch.arange(1, self.d_state + 1, dtype=torch.float32, device=device),
                "n -> d n",
                d=self.d_inner,
            ).contiguous()
            A_log = torch.log(A)  # Keep A_log in fp32
            self.A_log = nn.Parameter(A_log)
            self.A_log._no_weight_decay = True
    
            # Skip parameter D
            self.D = nn.Parameter(torch.ones(self.d_inner, device=device))  # Keep in fp32
            self.D._no_weight_decay = True
    
            # Bidirectional logic
            if bimamba_type == "v1":
                A_b = repeat(
                    torch.arange(1, self.d_state + 1, dtype=torch.float32, device=device),
                    "n -> d n",
                    d=self.d_inner,
                ).contiguous()
                A_b_log = torch.log(A_b)  # Keep A_b_log in fp32
                self.A_b_log = nn.Parameter(A_b_log)
                self.A_b_log._no_weight_decay = True
            elif bimamba_type == "v2" or bimamba_type == 'v3':
                A_b = repeat(
                    torch.arange(1, self.d_state + 1, dtype=torch.float32, device=device),
                    "n -> d n",
                    d=self.d_inner,
                ).contiguous()
                A_b_log = torch.log(A_b)  # Keep A_b_log in fp32
                self.A_b_log = nn.Parameter(A_b_log)
                self.A_b_log._no_weight_decay = True 
    
                self.conv1d_b = nn.Conv1d(
                    in_channels=self.d_inner,
                    out_channels=self.d_inner,
                    bias=conv_bias,
                    kernel_size=d_conv,
                    groups=self.d_inner,
                    padding=d_conv - 1,
                    **factory_kwargs,
                )
    
                self.x_proj_b = nn.Linear(
                    self.d_inner, self.dt_rank + self.d_state * 2, bias=False, **factory_kwargs
                )
                self.dt_proj_b = nn.Linear(self.dt_rank, self.d_inner, bias=True, **factory_kwargs)
    
                self.D_b = nn.Parameter(torch.ones(self.d_inner, device=device))  # Keep in fp32
                self.D_b._no_weight_decay = True
                self.in_proj_extra = nn.Linear(self.d_model, self.d_inner * 2, bias=bias, **factory_kwargs)
    
            self.out_proj = nn.Linear(self.d_inner, self.d_model, bias=bias, **factory_kwargs)
    
        def forward(self, hidden_states, inference_params=None, extra_emb=None):
            """
            hidden_states: (B, L, D)
            Returns: same shape as hidden_states
            """
            batch, seqlen, dim = hidden_states.shape
    
            conv_state, ssm_state = None, None
            if inference_params is not None:
                conv_state, ssm_state = self._get_states_from_cache(inference_params, batch)
                if inference_params.seqlen_offset > 0:
                    out, _, _ = self.step(hidden_states, conv_state, ssm_state)
                    return out
    
            if extra_emb is None:
                xz = rearrange(
                    self.in_proj.weight @ rearrange(hidden_states, "b l d -> d (b l)"),
                    "d (b l) -> b d l",
                    l=seqlen,
                )
                if self.in_proj.bias is not None:
                    xz = xz + rearrange(self.in_proj.bias.to(dtype=xz.dtype), "d -> d 1")
            else:
                xz = rearrange(
                    self.in_proj.weight @ rearrange(hidden_states, "b l d -> d (b l)"),
                    "d (b l) -> b d l",
                    l=seqlen,
                )
                if self.in_proj.bias is not None:
                    xz = xz + rearrange(self.in_proj.bias.to(dtype=xz.dtype), "d -> d 1")
                extra_emb = rearrange(
                    self.in_proj_extra.weight @ rearrange(extra_emb, "b l d -> d (b l)"),
                    "d (b l) -> b d l",
                    l=seqlen,
                )
                if self.in_proj_extra.bias is not None:
                    extra_emb = extra_emb + rearrange(self.in_proj_extra.bias.to(dtype=xz.dtype), "d -> d 1")
    
            A = -torch.exp(self.A_log.float())  # (d_inner, d_state)
    
            # Fast path (uses custom kernels and skips certain computations)
            if self.use_fast_path and inference_params is None:
                if self.bimamba_type == "v1":
                    A_b = -torch.exp(self.A_b_log.float())
                    out = bimamba_inner_fn(
                        xz,
                        self.conv1d.weight,
                        self.conv1d.bias,
                        self.x_proj.weight,
                        self.dt_proj.weight,
                        self.out_proj.weight,
                        self.out_proj.bias,
                        A,
                        A_b,
                        None,
                        None,
                        self.D.float(),
                        delta_bias=self.dt_proj.bias.float(),
                        delta_softplus=True,
                    )
                elif self.bimamba_type == "v2":
                    A_b = -torch.exp(self.A_b_log.float())
                    out = mamba_inner_fn_no_out_proj(
                        xz,
                        self.conv1d.weight,
                        self.conv1d.bias,
                        self.x_proj.weight,
                        self.dt_proj.weight,
                        A,
                        None,
                        None,
                        self.D.float(),
                        delta_bias=self.dt_proj.bias.float(),
                        delta_softplus=True,
                    )
                    out_b = mamba_inner_fn_no_out_proj(
                        xz.flip([-1]),
                        self.conv1d_b.weight,
                        self.conv1d_b.bias,
                        self.x_proj_b.weight,
                        self.dt_proj_b.bias,
                        A_b,
                        None,
                        None,
                        self.D_b.float(),
                        delta_bias=self.dt_proj_b.bias.float(),
                        delta_softplus=True,
                    )
                    if not self.if_devide_out:
                        out = F.linear(rearrange(out + out_b.flip([-1]), "b d l -> b l d"), self.out_proj.weight, self.out_proj.bias)
                    else:
                        out = F.linear(rearrange(out + out_b.flip([-1]), "b d l -> b l d") / 2, self.out_proj.weight, self.out_proj.bias)
                elif self.bimamba_type == "v3":
                    A_b = -torch.exp(self.A_b_log.float())
                    out = mamba_inner_fn_no_out_proj(
                        xz,
                        self.conv1d.weight,
                        self.conv1d.bias,
                        self.x_proj.weight,
                        self.dt_proj.weight,
                        A,
                        None,
                        None,
                        self.D.float(),
                        delta_bias=self.dt_proj.bias.float(),
                        delta_softplus=True,
                    )
                    out_b = mamba_inner_fn_no_out_proj(
                        extra_emb,
                        self.conv1d_b.weight,
                        self.conv1d_b.bias,
                        self.x_proj_b.weight,
                        self.dt_proj_b.bias,
                        A_b,
                        None,
                        None,
                        self.D_b.float(),
                        delta_bias=self.dt_proj_b.bias.float(),
                        delta_softplus=True,
                    )
                    if not self.if_devide_out:
                        out = F.linear(rearrange(out + out_b, "b d l -> b l d"), self.out_proj.weight, self.out_proj.bias)
                    else:
                        out = F.linear(rearrange(out + out_b, "b d l -> b l d") / 2, self.out_proj.weight, self.out_proj.bias)
                else:
                    out = mamba_inner_fn(
                        xz,
                        self.conv1d.weight,
                        self.conv1d.bias,
                        self.x_proj.weight,
                        self.dt_proj.weight,
                        self.out_proj.weight,
                        self.out_proj.bias,
                        A,
                        None,
                        None,
                        self.D.float(),
                        delta_bias=self.dt_proj.bias.float(),
                        delta_softplus=True,
                    )
            else:
                x, z = xz.chunk(2, dim=1)
    
                if conv_state is not None:
                    conv_state.copy_(F.pad(x, (self.d_conv - x.shape[-1], 0)))
    
                if causal_conv1d_fn is None:
                    x = self.act(self.conv1d(x)[..., :seqlen])
                else:
                    x = causal_conv1d_fn(
                        x=x,
                        weight=rearrange(self.conv1d.weight, "d 1 w -> d w"),
                        bias=self.conv1d.bias,
                        activation=self.activation,
                    )
    
                x_dbl = self.x_proj(rearrange(x, "b d l -> (b l) d"))
                dt, B, C = torch.split(x_dbl, [self.dt_rank, self.d_state, self.d_state], dim=-1)
                dt = self.dt_proj.weight @ dt.t()
                dt = rearrange(dt, "d (b l) -> b d l", l=seqlen)
                B = rearrange(B, "(b l) dstate -> b dstate l", l=seqlen).contiguous()
                C = rearrange(C, "(b l) dstate -> b dstate l", l=seqlen).contiguous()
    
                y = selective_scan_fn(
                    x,
                    dt,
                    A,
                    B,
                    C,
                    self.D.float(),
                    z=z,
                    delta_bias=self.dt_proj.bias.float(),
                    delta_softplus=True,
                    return_last_state=ssm_state is not None,
                )
                if ssm_state is not None:
                    y, last_state = y
                    ssm_state.copy_(last_state)
                y = rearrange(y, "b d l -> b l d")
                out = self.out_proj(y)
    
            if self.init_layer_scale is not None:
                out = out * self.gamma    
    
            return out
    
        # 加速
        def step(self, hidden_states, conv_state, ssm_state):
            dtype = hidden_states.dtype
            assert hidden_states.shape[1] == 1, "Only support decoding with 1 token at a time for now"
            xz = self.in_proj(hidden_states.squeeze(1))  # (B 2D)
            x, z = xz.chunk(2, dim=-1)  # (B D)
    
            if causal_conv1d_update is None:
                conv_state.copy_(torch.roll(conv_state, shifts=-1, dims=-1))
                conv_state[:, :, -1] = x
                x = torch.sum(conv_state * rearrange(self.conv1d.weight, "d 1 w -> d w"), dim=-1)
                if self.conv1d.bias is not None:
                    x = x + self.conv1d.bias
                x = self.act(x).to(dtype=dtype)
            else:
                x = causal_conv1d_update(
                    x,
                    conv_state,
                    rearrange(self.conv1d.weight, "d 1 w -> d w"),
                    self.conv1d.bias,
                    self.activation,
                )
    
            x_db = self.x_proj(x)
            dt, B, C = torch.split(x_db, [self.dt_rank, self.d_state, self.d_state], dim=-1)
            dt = F.linear(dt, self.dt_proj.weight)
            A = -torch.exp(self.A_log.float())
    
            if selective_state_update is None:
                dt = F.softplus(dt + self.dt_proj.bias.to(dtype=dt.dtype))
                dA = torch.exp(torch.einsum("bd,dn->bdn", dt, A))
                dB = torch.einsum("bd,bn->bdn", dt, B)
                ssm_state.copy_(ssm_state * dA + rearrange(x, "b d -> b d 1") * dB)
                y = torch.einsum("bdn,bn->bd", ssm_state.to(dtype), C)
                y = y + self.D.to(dtype) * x
                y = y * self.act(z)
            else:
                y = selective_state_update(
                    ssm_state, x, dt, A, B, C, self.D, z=z, dt_bias=self.dt_proj.bias, dt_softplus=True
                )
    
            out = self.out_proj(y)
            return out.unsqueeze(1), conv_state, ssm_state
    
        def allocate_inference_cache(self, batch_size, max_seqlen, dtype=None, **kwargs):
            device = self.out_proj.weight.device
            conv_dtype = self.conv1d.weight.dtype if dtype is None else dtype
            conv_state = torch.zeros(batch_size, self.d_model * self.expand, self.d_conv, device=device, dtype=conv_dtype)
            ssm_dtype = self.dt_proj.weight.dtype if dtype is None else dtype
            ssm_state = torch.zeros(batch_size, self.d_model * self.expand, self.d_state, device=device, dtype=ssm_dtype)
            return conv_state, ssm_state
    
        def _get_states_from_cache(self, inference_params, batch_size, initialize_states=False):
            assert self.layer_idx is not None
            if self.layer_idx not in inference_params.key_value_memory_dict:
                conv_state = torch.zeros(
                    batch_size,
                    self.d_model * self.expand,
                    self.d_conv,
                    device=self.conv1d.weight.device,
                    dtype=self.conv1d.weight.dtype,
                )
                ssm_state = torch.zeros(
                    batch_size,
                    self.d_model * self.expand,
                    self.d_state,
                    device=self.dt_proj.weight.device,
                    dtype=self.dt_proj.weight.dtype,
                )
                inference_params.key_value_memory_dict[self.layer_idx] = (conv_state, ssm_state)
            else:
                conv_state, ssm_state = inference_params