^j:ddlmZddlZddlmZddlmZddlmZddlm Z ddl m Z m Z m Z dd lmZdd lmZmZmZmZdd lmZdd lmZd dlmZd dlmZmZmZmZddl m!Z!GddeZ"GddeZ#GddeZ$edGddeZ%GddejLZ'edeGdd e Z(ed!Gd"d#e$Z)ed$Gd%d&e$e Z*gd'Z+y)() dataclassN)nn)ACT2FN)Cache)GenerationMixin)BaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)Unpack)TransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)merge_with_config_defaults)capture_outputs) AutoModel)Qwen2AudioAttentionQwen2AudioEncoderQwen2AudioEncoderLayerQwen2AudioPreTrainedModel) VoxtralConfigc eZdZy)VoxtralAttentionN__name__ __module__ __qualname__v/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/voxtral/modular_voxtral.pyrr+r"rc eZdZy)VoxtralEncoderLayerNrr!r"r#r&r&/r$r"r&c eZdZdZdZdZdZdZy)VoxtralPreTrainedModelTN)rrr _supports_flex_attn_supports_cache_class_supports_attention_backend_can_compile_fullgraph_no_split_modulesr!r"r#r(r(3s  "&!r"r(z: The Voxtral encoder, which is a Whisper encoder.  custom_introcJeZdZeedZee ddee de e zfdZ y)VoxtralEncoder) attentions hidden_statesNkwargsreturnc |jj|jjdz|jjdz}|j d|k7r"t d|d|j dd|d|j|jjj|jjj}tjj|j|}tjj|j |}|jddd }|jj}||zj|j}tjj!||j |j" }t%|j&D]\}} | || }|j)|}t+| S) a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Voxtral does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rz7Voxtral expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .dtypedevicerr)ptraining)attention_mask)last_hidden_state)configmax_source_positionsconv1strideconv2shape ValueErrortoweightr:r;r functionalgelupermuteembed_positionsdropoutr= enumeratelayers layer_normr ) selfinput_featuresr>r4expected_seq_length inputs_embeds embed_posr3idx encoder_layers r#forwardzVoxtralEncoder.forwardGs(#kk>>ARARSTAUUX\XbXbXiXijkXll    #': :IJ]I^^jkykkAClDkEErsFrGGHI (**1B1B1H1HQUQ[Q[QbQbQiQi*j **4::n+EF  **4::m+DE %--aA6 ((// &266}7J7JK  --mt||VZVcVc-d "+DKK"8  C)-M   6 )+  r"N) rrr rr&_can_record_outputsrrr r tupler rXr!r"r#r1r1<sS',  + +, + + + +  + r"r1c*eZdZdeffd ZdZxZS)VoxtralMultiModalProjectorr@cft|tj|jj |j jd|_t|j|_ tj|j j|j jd|_ yNF)bias) super__init__rLinear audio_configintermediate_size text_config hidden_sizelinear_1rprojector_hidden_actactlinear_2rQr@ __class__s r#rbz#VoxtralMultiModalProjector.__init__xsz  &"5"5"G"GI[I[IgIgnst &556 &"4"4"@"@&BTBTB`B`glm r"cl|j|}|j|}|j|}|SrY)rhrjrk)rQaudio_featuresr3s r#rXz"VoxtralMultiModalProjector.forward~s2 n5 /  m4 r")rrr rrbrX __classcell__rms@r#r]r]wsn}n r"r]zL Base class for Voxtral outputs, with hidden states and attentions. c:eZdZUdZdZej dzed<y)VoxtralModelOutputWithPastzg audio_hidden_states (`torch.FloatTensor`, *optional*): Projected audio hidden states. Naudio_hidden_states)rrr __doc__rttorch FloatTensor__annotations__r!r"r#rsrss 59**T18r"rsz The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a Llama language model, without a language modeling head. ceZdZfdZeeddejdee de e zfdZ dejd ejd ejfd Zee ddejd zdejd zd ejd zdejd zded zd ejd zded zdee de ezfdZxZS) VoxtralModelct||tj|j|_tj|j |_t||_ |jyrY) rarbr from_configrd audio_towerrflanguage_modelr]multi_modal_projector post_initrls r#rbzVoxtralModel.__init__sY  $001D1DE'33F4F4FG%?%G" r"zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.r.rRr4r5c |j|fddi|}|j}|jd|jjj }|j |}||_|S)aa input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] return_dictTr7)r}r?reshaper@rdrer pooler_output)rQrRr4 audio_outputsrt audio_embedss r#get_audio_featureszVoxtralModel.get_audio_featuressn)((TTTVT +==199"dkk>V>V>h>hi112EF &2 #r" input_idsrTroc.|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd}|jdj|j|j }t||j|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. r9r7rz6Audio features and audio tokens do not match, tokens: z , features: )get_input_embeddingsrvtensorr@audio_token_idlongr;allsumrE unsqueeze expand_asrGrnumel)rQrrTrospecial_audio_maskn_audio_tokensn_audio_featuress r#get_placeholder_maskz!VoxtralModel.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2/99"=GG VYYZgZnZno , - 3 3 59M9M9O OD^DTT`aq`r s "!r"Nr> position_idspast_key_values use_cachec ||j|}d} |^|\|j|dj} |j||| } |j | | j |j }|jd|||||d|} t| j| j| j| j| S)NT)r)rTro)r>rrrTr)r?rr3r2rtr!) rrrrmasked_scatterrGr;r~rsr?rr3r2) rQrrRr>rrrTrr4rroutputss r#rXzVoxtralModel.forwards  7D557 BM  %)*?22>t2TbbL"&!:!:|";" *889K\__]j]q]qMrsM+>4+>+>, )%+' ,  , *%77#33!//)) ,   r")NNNNNNN)rrr rbrrrvrwr r r[r r LongTensorrTensorrboolrsrXrprqs@r#rzrzs\w#//;ABT;U + +&"))":?:K:K"]b]n]n"0.237.204(,26!%' ##d*' ))D0'  t+ ' &&- '  ' ((4/' $;' +,'  + +' ' r"rzzs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a Llama language model. c^eZdZdgZfdZdZee ddejdzdejdzdejdzdejdzd e dzd ejdzd ejdzd e dzd eejzdeedeezfdZfdZxZS)VoxtralForConditionalGenerationrLct||t||_t j |j j|j jd|_ |jyr_) rarbrzmodelrrcrfrg vocab_sizelm_headrrls r#rbz(VoxtralForConditionalGeneration.__init__sS  !&) yy!3!3!?!?ASASA^A^ejk  r"c:|jj|i|SrY)rr)rQargsr4s r#rz2VoxtralForConditionalGeneration.get_audio_features s,tzz,,d=f==r"NrrRr>rrrTlabelsrlogits_to_keepr4r5c |jd|||||||d| } | j} t| tr t | dn| } |j | dd| ddf}d}|4|j d|||jjjd| }t||| j| j| jS)aj Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```)rrRr>rrrTrN)logitsrr)lossrrr3r2r!)rr? isinstanceintslicer loss_functionr@rfrr rr3r2)rQrrRr>rrrTrrrr4rr3 slice_indicesrrs r#rXz'VoxtralForConditionalGeneration.forwards`$**  ))%+'     11 8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD&#33!//))   r"c|jdd}|jdd}t| |i|}|s|jdds||d<|S)NrRis_first_iterationFrT)popgetraprepare_inputs_for_generation)rQrr4rRr model_inputsrms r#rz=VoxtralForConditionalGeneration.prepare_inputs_for_generation[s] $4d;#ZZ(.237.204(,26*.!%-.I ##d*I ))D0I  t+ I &&- I  I ((4/I   4'I $;I ell*I +,I  ' 'I I V  r"r)r(r1rzr), dataclassesrrvr activationsr cache_utilsr generationrmodeling_outputsr r r processing_utilsr utilsr rrr utils.genericrutils.output_capturingrautor qwen2_audio.modeling_qwen2_audiorrrrconfiguration_voxtralrrr&r(r1Moduler]rsrzr__all__r!r"r#rs2 " ! ) 'aa75 1 *  0 6 3 &3  3 l    9!89  9 ` )`  ` F e&