^j<TddlmZmZddlmZddlmZddlmZm Z ddl m Z e je ZedeGd d eZedeGd d eZedeGd deZedeGddeZgdZy))AnyLiteral)strict)PreTrainedConfig)auto_docstringlogging)intervalzgoogle/gemma-4-e2b-it) checkpointcJeZdZUdZdZdZeed<dZeed<dZ eed<d Z e ed <d Z e eeeefzed <d Zeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<d Zeed!<ed"d#$d%&Zeed'<fd(ZxZS))Gemma4AudioConfiga subsampling_conv_channels (`list[int]`, defaults to `[128, 32]`): Channel sizes for the convolutional layers in the Sub-sample Convolution Projection. residual_weight (`float`, defaults to `0.5`): Scaling applied to hidden_states prior to combining with the residual in the feedforward. attention_chunk_size (`int`, defaults to `12`): The sub-sequence size for attention processing. attention_context_left (`int`, defaults to `13`): The leftward context size for the attention chunk. attention_context_right (`int`, defaults to `0`): The rightward context size for the attention chunk. attention_logit_cap (`float`, defaults to `50.0`): Cap applied to attention weights. attention_invalid_logits_value (`float`, defaults to `1e-9`): Value to use for invalid logits in attention. use_clipped_linears (`bool`, defaults to `True`): If true, apply clipping to the Linear layers, drawing bounds from the model checkpoint. gradient_clipping (`float`, defaults to `1e10`): Clipping value used to stabilize extremely large gradient values. output_proj_dims (`int`, defaults to `1536`): Dimension of the final linear projection from `hidden_size` to the model's output. gemma4_audioi hidden_size num_hidden_layersnum_attention_headssilu hidden_act) subsampling_conv_channelsconv_kernel_sizeg?residual_weightattention_chunk_size attention_context_leftrattention_context_rightgI@attention_logit_capgeattention_invalid_logits_valueTuse_clipped_linearsư> rms_norm_epsg _Bgradient_clippingioutput_proj_dimsg?)minmax{Gz?)defaultinitializer_rangec t|jtrt|j|_t |di|y)N) isinstancertuplelistsuper __post_init__selfkwargs __class__s z/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/gemma4/configuration_gemma4.pyr3zGemma4AudioConfig.__post_init__Ns6 d44e <-1$2P2P-QD * '') __name__ __module__ __qualname____doc__ model_typerint__annotations__rrrstrrr1r0rrfloatrrrr r!r"boolr$r%r&r r,r3 __classcell__r7s@r8r r s. JKs  J>GtCy5c?:Fc OU "#""$C$#$S$!%%,2"E2 $$L%#u# c 9xCS9$GuG((r9r c eZdZUdZdZdgZddddddddddddd Zdddd d d dd Zd gdgfddgdgfdgdgfdZdZ e e d<dZ e e d<dZ e e d<dZe e d<dZe e d<dZe e d<dZe e d<d Zee d!<d"Ze e d#<d$Zee d%<d&Zee d'<d(Zee d)<d*Ze d+ze d,<d-Ze ee zd+ze d.<d/Ze d+ze d0<d(Zee d1<d+Ze d+ze d2<d3Z!ee d4<d5Z"e ezd+ze d6<d7Z#e e d8<d+Z$eed+ze d9<d+Z%ed+ze d:<d+Z&e'd;d+ze d<<dZ(e e d=<dZ)e e d><d+Z*e d+ze d?<d7Z+e e d@<d3Z,ee dA<d*Z-e e dB<d3Z.ee dC<d3Z/ee dD<d+Z0e d+ze dE<d+Z1e d+ze dF<d+Z2e d+ze dG<fdHZ3dIZ4xZ5S)JGemma4TextConfigaS use_bidirectional_attention (`str`, *optional*): Controls bidirectional attention behavior. When set to `"vision"`, vision tokens attend bidirectionally while text tokens use causal attention. When set to `"all"`, all tokens use bidirectional attention. vocab_size_per_layer_input (`int`, defaults to 262144): Vocabulary size for the per-layer input embeddings (PLE). Used by models with per-layer residual streams where a smaller embedding is added at each decoder layer. hidden_size_per_layer_input (`int`, defaults to 256): Per-layer hidden dimension for the PLE system. The actual embedding weight has shape `[vocab_size_per_layer_input, num_hidden_layers * hidden_size_per_layer_input]` because all layers are packed into a single table. See the [Gemma4](https://huggingface.co/docs/transformers/main/en/model_doc/gemma4#per-layer-embeddings-ple) docs for a description of the full PLE pipeline. num_global_key_value_heads (`int`, *optional*): Number of key-value heads for global (full) attention layers. If `None`, defaults to `num_key_value_heads`. global_head_dim (`int`, defaults to 512): Dimension of each attention head in global (full) attention layers. attention_k_eq_v (`bool`, defaults to `False`): Whether keys and values share the same projection weights. When `True`, the key projection output is reused as the value projection. num_kv_shared_layers (`int`, defaults to 0): Number of consecutive decoder layers that share the same key-value projections. A value of 0 means no sharing (each layer has independent KV projections). enable_moe_block (`bool`, defaults to `False`): Whether to enable Mixture-of-Experts (MoE) blocks in the decoder layers. When `True`, eligible layers will use a sparse MoE feed-forward network. use_double_wide_mlp (`bool`, defaults to `False`): Whether to use a double-width MLP with fused gate and up projections. top_k_experts (`int`, *optional*): Number of experts activated per token in MoE layers. Only used when `enable_moe_block=True`. moe_intermediate_size (`int`, *optional*): Intermediate (hidden) size of each expert's feed-forward network in MoE layers. Only used when `enable_moe_block=True`. gemma4_textpast_key_valuescolwisereplicated_with_grad_allreducerowwisepacked_colwisemoe_tp_experts) zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.q_normzlayers.*.self_attn.k_normzlayers.*.self_attn.o_projlayers.*.mlp.gate_projlayers.*.mlp.up_projlayers.*.mlp.down_projlayers.*.experts.gate_up_projlayers.*.experts.down_projlayers.*.experts ep_router grouped_gemm)rOrPrQzlayers.*.routerrRrSrT input_ids inputs_embeds hidden_statesattention_mask) embed_tokenslayersnormi vocab_sizei ri$intermediate_sizerrrnum_key_value_headshead_dimgelu_pytorch_tanhhidden_activationmax_position_embeddingsr*r,r#r$T use_cacherN pad_token_id eos_token_id bos_token_idtie_word_embeddingsrope_parametersFattention_biasr'attention_dropoutisliding_window layer_typesfinal_logit_softcapping)allvisionuse_bidirectional_attentionvocab_size_per_layer_inputhidden_size_per_layer_inputnum_global_key_value_headsglobal_head_dimattention_k_eq_vnum_kv_shared_layersenable_moe_blockuse_double_wide_mlp num_experts top_k_expertsmoe_intermediate_sizec |jdk(rd|_|jdzdz|_|j=d}t |j Dcgc]}t |dz|zrdndc}|_|jr<|jdx}dk7r(tjd |d d|jd<d d d ddddd}|j||_ t|,di|ycc}w)NrvFrmrksliding_attentionfull_attentionz/Last layer must use `full_attention`, but got `z*`. Forcing last layer to `full_attention`.r+g@ rope_type rope_theta proportionalg?g.A)rpartial_rotary_factorr)rrr.) rx is_causalrsrtrangerrCloggerwarningrpr2r3)r5r6sliding_window_patternilast_layer_typedefault_rope_paramsr7s r8r3zGemma4TextConfig.__post_init__s  + +u 4"DN#'#6#6!#;q"@D     #%& "t556 (,QU6L,L'M#Scc D    D4D4DR4H!HM] ] NNA/ARR|} $4D  R 09!Q,:UYituf     '#6D  ''% sC/c |S)Nr.)r5r6s r8convert_rope_params_to_dictz,Gemma4TextConfig.convert_rope_params_to_dicts r9)6r:r;r<r=r>keys_to_ignore_at_inferencebase_model_tp_planbase_model_ep_planbase_model_pp_planr^r?r@rr_rrrbrdrfrArhr,rBr$rirCrjrlr1rnrorpdictrqrrrsrtrurxrryrzr{r|r}r~rrrrrr3rrDrEs@r8rGrGUsV#JJ#4"5%.%.%.%E%E%."+ )"+)9&/,  #, )"+&)7&4, &(9:#%568IJ!"_$56 JK!s!s    Hc0s0#*S*#u#L%It L#* +,L#S /D(, L#* $$#'OTD[' ND ,/sU{T)/NC$(KcT!(,0UT\0CG!9D!@G&--'**-1d 1OS"d" !#!"d" %%"Kt" $M3:$(,3:,(6r9rGc ReZdZUdZdZdddddddddd ZdZdZee d <d Z ee d <d Z ee d <dZ ee d<dZ ee d<dZee d<dZee d<dZee d<dZee d<dZedze d<dZedze d<dZedze d<dZee d <d Zee d!<d"Zee d#<dZee d$<dZee d%<d&Zee d'<fd(ZxZ S))Gemma4VisionConfiga pooling_kernel_size (`int`, *optional*): Spatial pooling kernel size applied after patchification. position_embedding_size (`int`, defaults to 10240): Maximum number of position embeddings for the vision encoder. Controls the size of the learned 2D position embedding table used by the patch embedder. use_clipped_linears (`bool`, defaults to `False`): Whether to use weight-clipped linear layers. When enabled, linear layer weights are clamped to a fixed range during the forward pass to improve numerical stability. standardize (`bool`, defaults to `False`): If true, applies a bias and scale to the soft tokens returned from the pooler. gemma4_visionrJrKrL) z!encoder.layers.*.self_attn.q_projz!encoder.layers.*.self_attn.k_projz!encoder.layers.*.self_attn.v_projz!encoder.layers.*.self_attn.q_normz!encoder.layers.*.self_attn.k_normz!encoder.layers.*.self_attn.o_projzencoder.layers.*.mlp.gate_projzencoder.layers.*.mlp.up_projzencoder.layers.*.mlp.down_projY@iri r_rrrrb@rdrerfr#r$rgrhFNrqr'rrrprpooling_kernel_size patch_sizei(position_embedding_sizer" standardizer*r,c P|j ddd|_t|di|y)Nr+rrr.)rpr2r3r4s r8r3z Gemma4VisionConfig.__post_init__s,    '1:%#PD  ''r9)!r:r;r<r=r>r default_thetarr?r@r_rrrbrdrfrAr$rBrhrqrCrrrprrrrr"rr,r3rDrEs@r8rrs !J-6-6-6-M-M-6*3(1*3 MK!s!s!!!!Hc0s0L%#*S*"'ND4K'&)ut|)#'OTD['  J#,S, %%K#u#((r9rcNeZdZUdZdZeeedZdZ ee e e fzdze d<dZee e e fzdze d<dZee e e fzdze d<dZedze d <d Zedze d <d Zedze d <dZedze d<dZedze d<dZedze d<dZedze d<dZedze d<dZee d<fdZxZS) Gemma4Configag boi_token_id (`int`, *optional*, defaults to 255999): The begin-of-image token index to wrap the image prompt. eoi_token_id (`int`, *optional*, defaults to 258882): The end-of-image token index to wrap the image prompt. boa_token_id (`int`, *optional*, defaults to 256000): The begin-of-audio token index to wrap the audio prompt. eoa_token_index (`int`, *optional*, defaults to 258883): The end-of-audio token index to wrap the audio prompt. Example: ```python >>> from transformers import ( >>> Gemma4AudioConfig, >>> Gemma4Config, >>> Gemma4ForConditionalGeneration, >>> Gemma4TextConfig, >>> Gemma4VisionConfig, >>> ) >>> # Initializing a Gemma 4 Audio config. >>> audio_config = Gemma4AudioConfig() >>> # Initializing a Gemma 4 Text config. >>> text_config = Gemma4TextConfig() >>> # Initializing a Gemma 4 vision config. >>> vision_config = Gemma4VisionConfig() >>> # Initializing a Gemma 4 config similar to google/gemma-4-e2b-it >>> configuration = Gemma4Config(text_config, vision_config, audio_config) >>> # Initializing a model from the google/gemma-4-e2b-it configuration >>> model = Gemma4ForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```gemma4) text_config vision_config audio_configNrrri boi_token_idiB eoi_token_idi@image_token_idiDvideo_token_idi boa_token_idiCeoa_token_indexiAaudio_token_idr*r,Troc B|j%t|_tjdn4t |jt rtdi|j|_|j tjdt |j t rtdi|j |_|jtjdt |jt rtdi|j|_t|,di|y)Nz4text_config is None. Using default Gemma4TextConfig.zHvision_config is None. Gemma4Model.vision_tower will not be initialized.zFaudio_config is None. Gemma4Model.audio_tower will not be initialized.r.) rrGrinfor/rrrrr r2r3r4s r8r3zGemma4Config.__post_init__Ws    #/1D  KKN O (($ //C$2B2BCD     % KKb c d(($ /!3!Id6H6H!ID     $ KK` a d'' . 1 FD4E4E FD  ''r9)r:r;r<r=r>rGrr sub_configsrrrArr@rrrr?rrrrrrr,rBrorCr3rDrEs@r8rrs& PJ'+)K =AK!DcN2T9@@DM%S#X6=D>BL#d38n4t;B&L#*&&L#*&!(NC$J(!(NC$J(&L#*&")OS4Z)!(NC$J(&*ut|* $$((r9r)r rrGrN)typingrrhuggingface_hub.dataclassesrconfiguration_utilsrutilsrr utils.type_validatorsr get_loggerr:rr rGrr__all__r.r9r8rs .3,-   H %235((5(45(p23F'F4FR233()3(43(l23N(#N(4N(b Zr9