^j`ddlZddlmZddlmZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZmZmZdd lmZmZdd lmZdd lmZmZmZmZmZddlmZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&ejNe(Z) d2dejTdejVdejVdejVdejVdzde,dzde,fdZ-GddejTZ.GddeZ/eGd d!eZ0ed"#Gd$d%e0Z1Gd&d'ejTZ2ed(#eGd)d*eZ3ed+#Gd,d-e0Z4ed.#Gd/d0e0e Z5gd1Z6y)3N)Callable) dataclass)nn)ACT2FN)Cache)GenerationMixin)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)merge_with_config_defaults)capture_outputs) AutoModel) VoxtralConfigVoxtralEncoderConfigmodulequerykeyvalueattention_maskscalingdropoutc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)Nrr)dimptrainingr) sizetorchmatmul transposer functionalsoftmaxr#r* contiguous) rrrr r!r"r#kwargs attn_weights attn_outputs w/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/voxtral/modeling_voxtral.pyeager_attention_forwardr6.s**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$c,eZdZdZ ddedededededed edzd edzffd Zd e jd edefdZ dde jde jdzdede e je jdze e jdzffdZ xZS)VoxtralAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsr# is_decoderbias is_causal layer_idxconfigc zt |||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ |/|r-tjd|jjd||_t!j"||d|_t!j"||||_t!j"||||_t!j"||||_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r&zInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Fr=)super__init__r:r;r#head_dimr@ ValueErrorr"r<r>logger warning_once __class____name__r?rLineark_projv_projq_projout_proj) selfr:r;r#r<r=r>r?r@rIs r5rDzVoxtralAttention.__init__Ks  "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"     *4>>+B+B*CD,,  #ii 95A ii 94@ ii 94@  )YTB r7tensorseq_lenbszc|j|||j|jjddj S)Nrr)viewr;rEr.r1)rPrQrRrSs r5_shapezVoxtralAttention._shapess7{{3GQQRSUVWbbddr7 hidden_statesr!output_attentionsreturnc 0|j\}}}|j|j||jz||}|j|j |d|} |j|j |d|} t j|jjt} | ||| | |f|jsdn |jd|d|\} } | j||dj} |j| } | | fS)z#Input shape: Batch x Time x Channelr%?)r#r"rX)r+rVrNr"rLrMr get_interfacer@_attn_implementationr6r*r#reshaper1rO)rPrWr!rXr2rStgt_len_ query_states key_states value_statesattention_interfacer4r3s r5forwardzVoxtralAttention.forwardvs(,,.Wa{{4;;}#= #LgWZ[ [[]!;RE {{4;;}#=r3G (?(M(M KK , ,.E) %8      % $}}C$,,/ %  % ! \"))#w;FFH mmK0 L((r7)r[FTFNNNF)rJ __module__ __qualname____doc__intfloatboolrrDr,TensorrVtuplerf __classcell__rIs@r5r9r9Hs G   $'+&C&C&C &C  &C  &C&C:&C$&CPeU\\eCece /3"' ')||') t+') ') u||U\\D0% 2E2LL M ')r7r9c~eZdZdeffd Zdej dej deedej fdZ xZ S)VoxtralEncoderLayerr@cht||j|_t |j|j |j ||_tj|j|_ |j|_ t|j|_|j|_tj |j|j"|_tj |j"|j|_tj|j|_y)N)r:r;r#r@)rCrDd_modelr:r9encoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normr#ractivation_function activation_fnactivation_dropoutrKencoder_ffn_dimfc1fc2final_layer_normrPr@rIs r5rDzVoxtralEncoderLayer.__init__s )nn44,,   %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r7rWr!r2rYc |}|j|}|jd||d|\}}tjj ||j|j }||z}|}|j |}|j|j|}tjj ||j|j }|j|}tjj ||j|j }||z}|jtjk(rEtj|jjdz }tj || |}|S)a> Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. )rWr!r(i)minmax)rzrxrr/r#r*rr|rr}rdtyper,float16finforclamp)rPrWr!r2residualra clamp_values r5rfzVoxtralEncoderLayer.forwardsT!11-@ )4>> ')   q  --mt||VZVcVc-d  =0  --m< **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0   %-- /++m&9&9:>>EK!KK K<[YMr7) rJrhrirrDr,rnrrrfrprqs@r5rsrssL=}=$"||" "+, "  "r7rscFeZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ dZdZy)VoxtralPreTrainedModelr@model)audiotextTNpast_key_values)rJrhrir__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_cache_class_supports_attention_backend_can_compile_fullgraphrr7r5rrsL (&*##4"5N "&!r7rz: The Voxtral encoder, which is a Whisper encoder.  custom_introceZdZUdZeed<dZdZdgZe e dZ deffd Z dZ d ejfd Zd ejfd Zee dd eed eezfdZdej4fdZxZS)VoxtralEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`VoxtralEncoderLayer`]. Args: config: VoxtralEncoderConfig r@input_featuresrrs) attentionsrWcbt|||j|_|j|_|j }|j |_|j|_|jrtj|nd|_ tj|j |dd|_tj||ddd|_tj |j||_|j"j%dtj&t)|j*Dcgc] }t-|c}|_tj0|j |_tj4dd|_d|_|j;ycc}w) Nr\rr) kernel_sizepaddingr)rstriderF)r)rCrDr#encoder_layerdrop layerdropru num_mel_binsmax_source_positionsscale_embeddingmathsqrt embed_scalerConv1dconv1conv2 Embeddingembed_positionsrequires_grad_ ModuleListrangeencoder_layersrslayersry layer_norm AvgPool1d avg_poolergradient_checkpointing post_init)rPr@r:rarIs r5rDzVoxtralEncoder.__init__s6  ~~ 11NN "//$*$?$?!393I3I499Y/sYYt00)TUV YYy)1VWX !||D,E,EyQ ++E2mm%PVPePeJf$gQ%8%@$gh ,,v~~6,,q3&+# %hs6F,cJ|jD] }d|_ d|_yrg) parameters requires_grad_requires_grad)rPparams r5_freeze_parametersz!VoxtralEncoder._freeze_parameterss(__& (E"'E  (#r7rYc|jSNr)rPs r5get_input_embeddingsz#VoxtralEncoder.get_input_embeddingss zzr7r c||_yrr)rPr s r5set_input_embeddingsz#VoxtralEncoder.set_input_embeddings!s  r7r2c |jj|jjdz|jjdz}|j d|k7r"t d|d|j dd|d|j|jjj|jjj}tjj|j|}tjj|j |}|jddd }|jj}||zj|j}tjj!||j |j" }t%|j&D]\}} | || }|j)|}t+| S) a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Voxtral does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rr%z7Voxtral expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .rdevicerrr()r!)last_hidden_state)r@rrrrshaperFtoweightrrrr/gelupermuterr#r* enumeraterrr ) rPrr!r2expected_seq_length inputs_embeds embed_posrWidx encoder_layers r5rfzVoxtralEncoder.forward$s(#kk>>ARARSTAUUX\XbXbXiXijkXll    #': :IJ]I^^jkykkAClDkEErsFrGGHI (**1B1B1H1HQUQ[Q[QbQbQiQi*j **4::n+EF  **4::m+DE %--aA6 ((// &266}7J7JK  --mt||VZVcVc-d "+DKK"8  C)-M   6 )+  r7 input_lengthsc6|dz dzdz}|dz dzdz}||fS)zs Computes the output length of the convolutional layers and the output length of the audio encoder rrr)rProutput_lengthss r5 _get_feat_extract_output_lengthsz/VoxtralEncoder._get_feat_extract_output_lengthsTs7'*q014 '!+1A5n,,r7r)rJrhrirjrrmain_input_namerrr9rs_can_record_outputsrDrrModulerrrrrrror rfr, LongTensorrrprqs@r5rrs  ! &O./&, 32$ bii")) + +, + + + +  + \-e>N>N-r7rc*eZdZdeffd ZdZxZS)VoxtralMultiModalProjectorr@cft|tj|jj |j jd|_t|j|_ tj|j j|j jd|_ yNFrB) rCrDrrK audio_configintermediate_size text_config hidden_sizelinear_1rprojector_hidden_actactlinear_2rs r5rDz#VoxtralMultiModalProjector.__init__^sz  &"5"5"G"GI[I[IgIgnst &556 &"4"4"@"@&BTBTB`B`glm r7cl|j|}|j|}|j|}|Sr)rrr)rPaudio_featuresrWs r5rfz"VoxtralMultiModalProjector.forwardds2 n5 /  m4 r7)rJrhrirrDrfrprqs@r5rr]sn}n r7rzL Base class for Voxtral outputs, with hidden states and attentions. c:eZdZUdZdZej dzed<y)VoxtralModelOutputWithPastzg audio_hidden_states (`torch.FloatTensor`, *optional*): Projected audio hidden states. Naudio_hidden_states)rJrhrirjrr, FloatTensorrrr7r5rrks 59**T18r7rz The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a Llama language model, without a language modeling head. ceZdZfdZeeddejdee de e zfdZ dejd ejd ejfd Zee ddejd zdejd zd ejd zdejd zded zd ejd zded zdee de ezfdZxZS) VoxtralModelct||tj|j|_tj|j |_t||_ |jyr) rCrDr from_configr audio_towerrlanguage_modelrmulti_modal_projectorrrs r5rDzVoxtralModel.__init__sY  $001D1DE'33F4F4FG%?%G" r7zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.rrr2rYc |j|fddi|}|j}|jd|jjj }|j |}||_|S)aa input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] return_dictTr%)rrr_r@rrr pooler_output)rPrr2 audio_outputsr audio_embedss r5get_audio_featureszVoxtralModel.get_audio_featuressn)((TTTVT +==199"dkk>V>V>h>hi112EF &2 #r7 input_idsrrc.|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd}|jdj|j|j }t||j|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. rr%rz6Audio features and audio tokens do not match, tokens: z , features: )rr,rQr@audio_token_idlongrallsumr unsqueeze expand_asrrnumel)rPrrrspecial_audio_maskn_audio_tokensn_audio_featuress r5get_placeholder_maskz!VoxtralModel.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2/99"=GG VYYZgZnZno , - 3 3 59M9M9O OD^DTT`aq`r s "!r7Nr! position_idsr use_cachec ||j|}d} |^|\|j|dj} |j||| } |j | | j |j }|jd|||||d|} t| j| j| j| j| S)NT)r)rr)r!rrrr)rrrWrrr) rrrrmasked_scatterrrrrrrrWr) rPrrr!rrrrr2rr outputss r5rfzVoxtralModel.forwards  7D557 BM  %)*?22>t2TbbL"&!:!:|";" *889K\__]j]q]qMrsM+>4+>+>, )%+' ,  , *%77#33!//)) ,   r7)NNNNNNN)rJrhrirDrrr,rrrror rrrrnrrmrrfrprqs@r5rrzs\w#//;ABT;U + +&"))":?:K:K"]b]n]n"0.237.204(,26!%' ##d*' ))D0'  t+ ' &&- '  ' ((4/' $;' +,'  + +' ' r7rzs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a Llama language model. c^eZdZdgZfdZdZee ddejdzdejdzdejdzdejdzd e dzd ejdzd ejdzd e dzd eejzdeedeezfdZfdZxZS)VoxtralForConditionalGenerationrct||t||_t j |j j|j jd|_ |jyr) rCrDrrrrKrr vocab_sizelm_headrrs r5rDz(VoxtralForConditionalGeneration.__init__sS  !&) yy!3!3!?!?ASASA^A^ejk  r7c:|jj|i|Sr)rr)rPargsr2s r5rz2VoxtralForConditionalGeneration.get_audio_featuress,tzz,,d=f==r7Nrrr!rrrlabelsrlogits_to_keepr2rYc |jd|||||||d| } | j} t| tr t | dn| } |j | dd| ddf}d}|4|j d|||jjjd| }t||| j| j| jS)aj Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```)rrr!rrrrN)logitsrr)lossrrrWrr)rr isinstancerkslicer loss_functionr@rrr rrWr)rPrrr!rrrrrrr2rrW slice_indicesrr s r5rfz'VoxtralForConditionalGeneration.forwards`$**  ))%+'     11 8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD&#33!//))   r7c|jdd}|jdd}t| |i|}|s|jdds||d<|S)Nris_first_iterationFrT)popgetrCprepare_inputs_for_generation)rPrr2rr& model_inputsrIs r5r)z=VoxtralForConditionalGeneration.prepare_inputs_for_generationAs] $4d;#ZZ(.237.204(,26*.!%-.I ##d*I ))D0I  t+ I &&- I  I ((4/I   4'I $;I ell*I +,I  ' 'I I V  r7r)rrrr)Nr[)7rcollections.abcr dataclassesrr,r activationsr cache_utilsr generationr modeling_layersr modeling_outputsr r r modeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrutils.output_capturingrautorconfiguration_voxtralrr get_loggerrJrGrrnrlr6r9rsrrrrrr__all__rr7r5r<s, $! ! )9kkF&jj75F   H %!% II% <<% % << % LL4' % T\ %%4U)ryyU)p545p "_ " " m-+m-  m-`    9!89  9 ` )`  ` F e&