^jdZddlmZddlmZddlmZddlmZeedd Gd d eZ eed d Gd deZ edeGddeZ dgZ y)zDBRX model configuration)strict)PreTrainedConfig)RopeParameters)auto_docstringz4This config is used to instantiate attention layers.z$transformers-community/dbrx-instruct) custom_intro checkpointcReZdZUdZdZdZeezed<dZ eezdzed<dZ eed<y) DbrxAttentionConfigaz attn_pdrop (`float`, *optional*, defaults to 0.0): The dropout probability for the attention layers. clip_qkv (`float`, *optional*): If set, clip the queries, keys, and values in the attention layer to this value. kv_n_heads (`int`, *optional*, defaults to 1): For grouped_query_attention only, allow user to specify number of kv heads. attn_config attn_pdropNclip_qkv kv_n_heads) __name__ __module__ __qualname____doc__base_config_keyrfloatint__annotations__rrv/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/dbrx/configuration_dbrx.pyr r s9 $O!J !#'HcEkD 'Jrr z6This config is used to instantiate feedforward layers.ceZdZUdZdZdZeed<dZe dzed<dZ eed<d Z eed <d Z eed <dZ edzed <dZeed<dZedzed<fdZxZS) DbrxFFNConfiga ffn_act_fn (`dict`, *optional*, defaults to `None`): A dict specifying activation function for the FFN. The dict should have a key 'name' with the value being the name of the activation function along with any additional keyword arguments. If `None`, then set to `{"name": "silu"}`. ffn_hidden_size (`int`, *optional*, defaults to 3584): The hidden size of the feedforward network. moe_num_experts (`int`, *optional*, defaults to 4): The number of experts in the mixture of experts layer. moe_top_k (`int`, *optional*, defaults to 1): The number of experts to use in the mixture of experts layer. moe_jitter_eps (`float`, *optional*, defaults to `None`): If not `None`, the jitter epsilon for the mixture of experts layer. moe_loss_weight (`float`, *optional*, defaults to 0.01): The loss weight for the mixture of experts layer. moe_normalize_expert_weights (`float`, *optional*, defaults to 1.0): The normalization factor for the expert weights. ffn_configi hidden_sizeN ffn_act_fniffn_hidden_sizemoe_num_expertsr moe_top_kmoe_jitter_epsg{Gz?moe_loss_weightg?moe_normalize_expert_weightsc |j ddi|_dD]}||vs|j|t|dk7rtd|t |di|y)Nnamesilu) model_typeattn_implementationexperts_implementationtransformers_version _commit_hash torch_dtypedtyperzFound unknown kwargs=r)r!poplen ValueErrorsuper __post_init__)selfkwargsk __class__s rr7zDbrxFFNConfig.__post_init__Qso ?? "%v.DO AF{ 1   v;! 5fY78 8 ''r)rrrrrr rrr!dictr"r$r%r&rr'r(r7 __classcell__r;s@rrr-sv &#OK"Jt "OSOSIs#'NEDL'!OU!14 %$,4((rr)r ceZdZUdZdZeedZdddddZd Z e d ze d<d Z e d ze d<d Z e d ze d<d Ze d ze d<d Ze e d<dZed ze d<dZed ze d<d Zeezd ze d<d Zeezd ze d<dZee d<dZee d<dZed ze d<d Zeezd ze d<d Ze d ze d<d Ze d ze d<d Ze ee zd ze d<dZ ee d<fdZ!d Z"xZ#S)! DbrxConfiga max_seq_len (`int`, *optional*, defaults to 2048): The maximum sequence length of the model. attn_config (`dict`, *optional*): A dictionary used to configure the model's attention module. ffn_config (`dict`, *optional*): A dictionary used to configure the model's FFN module. Example: ```python >>> from transformers import DbrxConfig, DbrxModel >>> # Initializing a Dbrx configuration >>> configuration = DbrxConfig(n_layers=2, d_model=256, n_heads=8, vocab_size=128) >>> # Initializing a model (with random weights) from the configuration >>> model = DbrxModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` dbrx)r rn_headsd_modeln_layers max_seq_len)num_attention_headsr num_hidden_layersmax_position_embeddingsiNi} vocab_sizer resid_pdrop emb_pdropr rT use_cacheg{Gz?initializer_rangeFoutput_router_logitsrope_parameters pad_token_id bos_token_id eos_token_idtie_word_embeddingsc |jt|_n4t|jtrtdi|j|_|jt |_n4t|jtrt di|j|_|jj |_t|$di|y)Nr) r r isinstancer<rrrnum_key_value_headsr6r7)r8r9r;s rr7zDbrxConfig.__post_init__s    #24D  (($ /2FT5E5EFD  ?? "+oDO  .+>doo>DO#'#3#3#>#>  ''rc2|jr tdy)zOPart of `@strict`-powered validation. Validates the architecture of the config.z5tie_word_embeddings is not supported for DBRX models.N)rUr5)r8s rvalidate_architecturez DbrxConfig.validate_architectures  # #TU U $r)$rrrrr,r r sub_configs attribute_maprCrrrBrDrErKrLrrMr r<rrNboolrOrPrQrrRrSrTlistrUr7rZr=r>s@rr@r@fs6.J"5]SK( '#0 MGS4ZGS4ZHcDj"Kt"J #K#!Iut|!59K$t+d29.2J $t+2It#u#(-$+-48O^d*T18#L#*##L#*#+/L#S /D(/ %% (Vrr@N) rhuggingface_hub.dataclassesrconfiguration_utilsrmodeling_rope_utilsrutilsrr rr@__all__rrrrds.31#G5*  "I51($1(  1(hABDV!DVCDVN .r