^jrdZddlZddlmZddlmZddlmZedeGd d eZd gZ y) zZamba model configurationN)strict)PreTrainedConfig)auto_docstringzZyphra/Zamba-7B-v1) checkpointcHeZdZUdZdZdgZdddZdZee d<d Z e e d <d Z ee d <d Z ed ze d<dZee d<dZee d<dZee d<d Zed ze d<dZee d<dZee d<dZee d<dZee d<dZee d<dZee d<d Ze e d <d!Zee d"<d#Zed ze d$<d!Zed ze d%<dZeeezd ze d&<d'Z ee d(<d)Z!eeze d*<d+Z"ee d,<d-Z#ee d.<d Z$e e d/<dZ%ee d0<d-Z&ee d1<dZ'ee d2<d3Z(eeze d4<d5Z)ee d6<d7Z*ee d8<d9Z+ee d:<d Z,e e d;<dZ.d?Z/d@Z0xZ1S)A ZambaConfiga! attention_hidden_size (`int`, *optional*): Dimension of the hidden representations of the inputs to the Attention layer. attention_head_dim (`int`, *optional*): Dimension of the attention head in the Transformer decoder. n_mamba_heads (`int`, *optional*, defaults to 2): Number of mamba heads for each mamba layer. hidden_mamba_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the mamba layer. num_logits_to_keep (`int` or `None`, *optional*, defaults to 1): Number of prompt logits to calculate during generation. If `None`, all logits will be calculated. If an integer value, only last `num_logits_to_keep` logits will be calculated. Default is 1 because only the logits of the last prompt token are needed for generation. For long sequences, the logits for the entire sequence may use a lot of memory so, setting `num_logits_to_keep=1` will reduce memory footprint significantly. attn_layer_period (`int`, *optional*, defaults to 6): Once in this many layers, we will have a shared attention layer attn_layer_offset (`int`, *optional*, defaults to 4): Offset of the shared attention layer use_mamba_kernels (`bool`, *optional*, defaults to `True`): Flag indicating whether or not to use the fast mamba kernels. These are available only if `mamba-ssm` and `causal-conv1d` are installed, and the mamba modules are running on a CUDA device. Raises ValueError if `True` and kernels are not available mamba_dt_rank (`Union[int,str]`, *optional*, defaults to `"auto"`): Rank of the mamba discretization projection matrix. `"auto"` means that it will default to `math.ceil(self.hidden_size / 16)` zambapast_key_valueslayers_block_typeattention_head_dim) layer_typeshead_dimi} vocab_sizeTtie_word_embeddingsi hidden_sizeNattention_hidden_sizei:intermediate_sizeLnum_hidden_layersnum_attention_headsnum_key_value_heads n_mamba_headsgelu hidden_actsiluhidden_mamba_actg{Gz?initializer_rangegh㈵> rms_norm_eps use_cachenum_logits_to_keepr pad_token_id bos_token_id eos_token_idimax_position_embeddingsgattention_dropoutattn_layer_periodattn_layer_offsetuse_mamba_kernels mamba_d_state mamba_d_conv mamba_expandauto mamba_dt_rankgMbP? time_step_ming? time_step_maxg-C6?time_step_floormamba_conv_biasFmamba_proj_biasc |jxsd|jz|_|jxsd|jz|jz|_|jdk(r"t j |jdz n |j|_|j|j|j|j|_ t|4di|y)Nrr2r)rrr rr3mathceil_layers_block_typerr+r-r super __post_init__)selfkwargs __class__s x/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/zamba/configuration_zamba.pyr?zZambaConfig.__post_init__\s%)%?%?%W1tGWGWCW""&"9"9"mQAQAQ=QUYUmUm=mAEASASW]A]TYYt'7'7"'<=cgcucu!%!8!8  " "D$:$:Dr`sM .3#/0Y"Y1Yx /rD