^j+dZddlmZddlmZddlmZddlmZedeGd d eZ edeGd d eZ d gZ y )zMpt configuration)Literal)strict)PreTrainedConfig)auto_docstringzmosaicml/mpt-7b) checkpointceZdZUdZdZdZeded<dZe ed<dZ e ed <d Z e d zed <d Ze d zed <d Zeed<d Zeed<d Zeed<dZeed<dZe ed<y )MptAttentionConfigaD attn_type (`str`, *optional*, defaults to `"multihead_attention"`): type of attention to use. Options: `"multihead_attention"`, `"multiquery_attention"`. attn_pdrop (`float`, *optional*, defaults to `0.0`): The dropout probability for the attention layers. attn_impl (`str`, *optional*, defaults to `"torch"`): The attention implementation to use. One of `"torch"`, `"flash"`, or `"triton"`. clip_qkv (`float`, *optional*): If not `None`, clip the queries, keys, and values in the attention layer to this value. softmax_scale (`float`, *optional*): If not `None`, scale the softmax in the attention layer by this value. If `None`, will default to `1/sqrt(hidden_size)`. prefix_lm (`bool`, *optional*, defaults to `False`): Whether the model should operate as a Prefix LM. This requires passing an extra `prefix_mask` argument which indicates which tokens belong to the prefix. Tokens in the prefix can attend to one another bi-directionally. Tokens outside the prefix use causal attention. qk_ln (`bool`, *optional*, defaults to `False`): Whether to apply layer normalization to the queries and keys in the attention layer. attn_uses_sequence_id (`bool`, *optional*, defaults to `False`): Whether to restrict attention to tokens that have the same token_type_ids. When the model is in `train` mode, this requires passing an extra *token_type_ids* argument which indicates which sub-sequence each token belongs to. Defaults to `False` meaning any provided *token_type_ids* will be ignored. alibi (`bool`, *optional*, defaults to `True`): Whether or not to use the alibi bias instead of positional embedding. alibi_bias_max (`int`, *optional*, defaults to 8): The maximum value of the alibi bias. attn_configmultihead_attention)r multiquery_attention attn_typer attn_pdroptorch attn_implNclip_qkv softmax_scaleF prefix_lmqk_lnattn_uses_sequence_idTalibialibi_bias_max)__name__ __module__ __qualname____doc__base_config_keyrr__annotations__rintrstrrfloatrrboolrrrrt/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/mpt/configuration_mpt.pyr r s8$OH]IwDE]JIs!Hedl!"&M54<&ItE4"'4'E4NCr%r ceZdZUdZdZdeiZddddZdZe e d<d Z e e d<d Z e e d<d Z e e d <dZe e d <dZe e d<dZee ze d<dZee d<dZee ze d<dZee d<dZeezdze d<dZee d<dZeezdze d<dZee d<dZee d<dZee d<d Zee d!<d"Zee d#<dZ ee d$<dZ!e dze d%<dZ"e dze d&<dZ#e e$e zdze d'<fd(Z%xZ&S)) MptConfigaZ expansion_ratio (`int`, *optional*, defaults to 4): The ratio of the up/down scale in the MLP. max_seq_len (`int`, *optional*, defaults to 2048): The maximum sequence length of the model. layer_norm_epsilon (`float`, *optional*, defaults to 1e-05): The epsilon to use in the layer normalization layers. learned_pos_emb (`bool`, *optional*, defaults to `True`): Whether to use learned positional embeddings. attn_config (`dict`, *optional*): A dictionary used to configure the model's attention module. init_device (`str`, *optional*, defaults to `"cpu"`): The device to use for parameter initialization. Defined for backward compatibility logit_scale (`float`, *optional*): If not None, scale the logits by this value. no_bias (`bool`, *optional*, defaults to `True`): Whether to use bias in all linear layers. embedding_fraction (`float`, *optional*, defaults to 1.0): The fraction to scale the gradients of the embedding layer by. norm_type (`str`, *optional*, defaults to `"low_precision_layernorm"`): Type of layer norm to use. All MPT models uses the same layer norm implementation. Defined for backward compatibility. Example: ```python >>> from transformers import MptConfig, MptModel >>> # Initializing a Mpt configuration >>> configuration = MptConfig() >>> # Initializing a model (with random weights) from the configuration >>> model = MptModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` mptr n_headsd_modeln_layers)num_attention_heads hidden_sizenum_hidden_layersiexpansion_ratio max_seq_leni vocab_sizeg resid_pdropgh㈵>layer_norm_epsilon emb_pdropTlearned_pos_embNcpu init_device logit_scaleno_biasg?embedding_fractionlow_precision_layernorm norm_typeF use_cacheg{Gz?initializer_rangetie_word_embeddings pad_token_id bos_token_id eos_token_idc |jt|_n4t|jtrtdi|j|_t |di|y)Nr$)r r isinstancedictsuper __post_init__)selfkwargs __class__s r&rKzMptConfig.__post_init__sP    #13D  (($ /1ED4D4DED  ''r%)'rrrr model_typer sub_configs attribute_mapr+r rr*r,r3r4r5r6r"r7r8r9r#r rIr;r!r<r=r>r@rArBrCrDrErFlistrK __classcell__)rNs@r&r(r(EsH%NJ "45K( 'M GSGSHcOSKJ"K" $$ Ius{  OT 48K**T18K&*Kt#*GT ##.Is.It#u# $$#L#*##L#*#+/L#S /D(/((r%r(N) rtypingrhuggingface_hub.dataclassesrconfiguration_utilsrutilsrr r(__all__r$r%r&rYsx.3#,-()(.(V,-L( L(.L(^ -r%