^j7>ddlmZddlZddlmZddlmZddlmZddlm Z ddl m Z m Z dd l mZdd lmZmZmZmZdd lmZdd lmZd dlmZmZd dlmZmZmZmZd dl m!Z!m"Z"ddl#m$Z$ejJe&Z'Gdde!Z(Gdde"Z)GddeZ*eGddeZ+edeGdde Z,edGd d!eZ-Gd"d#eZ.ed$Gd%d&eZ/ed'Gd(d)eZ0gd*Z1y)+) dataclassN)nn)ACT2FN)Cache)create_bidirectional_mask)BaseModelOutputWithPooling ModelOutput)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs)Qwen2AudioEncoderQwen2AudioPreTrainedModel)VoxtralForConditionalGeneration VoxtralModelVoxtralModelOutputWithPastVoxtralMultiModalProjector)WhisperAttentionWhisperEncoderLayer)AudioFlamingo3Configc eZdZy)AudioFlamingo3AttentionN__name__ __module__ __qualname__/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/audioflamingo3/modular_audioflamingo3.pyrr.r$rc eZdZy)AudioFlamingo3EncoderLayerNrr#r$r%r(r(2r&r$r(ceZdZdZy)AudioFlamingo3PreTrainedModelTN)r r!r"_supports_attention_backendr#r$r%r*r*6s"&r$r*c eZdZy)!AudioFlamingo3ModelOutputWithPastNrr#r$r%r-r-:sr$r-zZ Base class for AudioFlamingo3 causal language model (or autoregressive) outputs.  custom_introceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) $AudioFlamingo3CausalLMOutputWithPastaV loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. audio_hidden_states (`torch.FloatTensor`, *optional*): Hidden states of the audio encoder after projection. Nlosslogitspast_key_values hidden_states attentionsaudio_hidden_states)r r!r"__doc__r2torch FloatTensor__annotations__r3r4rr5tupler6r7r#r$r%r1r1?s &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r$r1zT The audio model from AudioFlamingo3 without any head or projection on top. c zeZdZdZeedZee dde jde jdzde e zfdZ y) AudioFlamingo3EncoderzY AudioFlamingo3 encoder: Whisper encoder, average pool (time/2), then LayerNorm. )r5r6Ninput_featuresinput_features_maskreturnc |jddz dzdz}|jd}|dz dzdz}tj||j|dddfk}t j j|j|}t j j|j|}|jddd}||jjz}t j j||j|j}t|j ||}|j"D]>} |jxr"tj$g|j&k} | r6| ||}@|jddd}|j)|jddd}|j+|}t-| S) ap Args: input_features (`torch.FloatTensor` of shape `(batch_size, feature_size, sequence_length)`): Log-Mel features extracted from raw audio. Use the processor/feature extractor to compute and pad these features from waveform input. input_features_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. rrdeviceNr)ptraining)config inputs_embedsattention_mask)last_hidden_state)shapesumr9arangerEr functionalgeluconv1conv2permuteembed_positionsweightdropoutrGrrHlayersrand layerdrop avg_pooler layer_normr ) selfr?r@kwargsseq_leninput_features_lengthsrIr5rJlayerdrops r%forwardzAudioFlamingo3Encoder.forwardhs("''+a/A59!4!8!8!<"81"<!BQ!F#ll7>;P;PQTjklnrkrTss **4::n+EF  **4::m+DE %--aA6 &(<(<(C(CC  --mt||VZVcVc-d 2;;'. [[ EE==DUZZ^dnn%DD %m^ D  E &--aA6  6>>q!QG  6 )+  r$N)r r!r"r8r(r_can_record_outputsrrr9Tensorr<r rbr#r$r%r>r>Ysg  4-  483  3 #\\D03 + + 3  3 r$r>c(eZdZdZdeffd ZxZS)!AudioFlamingo3MultiModalProjectorz Audio adaptor (small MLP) that projects AudioFlamingo3Encoder features to the LLM embedding space so they can replace `` tokens. rHct|tj|jj |j j |j|_t|j|_ tj|j j |j j |j|_ y)N)bias) super__init__rLinear audio_config hidden_size text_configprojector_biaslinear_1rprojector_hidden_actactlinear_2r\rH __class__s r%rkz*AudioFlamingo3MultiModalProjector.__init__s     + +V-?-?-K-KRXRgRg  &556    * *F,>,>,J,JQWQfQf  r$)r r!r"r8rrk __classcell__rvs@r%rgrgs  3  r$rgz The AudioFlamingo3 model (fine-tuned Whisper encoder, multi-modal projector, Qwen2 language model), without a language modeling head. ceZdZdZdZdZfdZeedde jde jde e deezfd Zee dd e j"dzde jdzde jdzd e jdzd e j"dzd edzde jdzdedzde e fdZxZS)AudioFlamingo3ModelNc$t||yrc)rjrkrus r%rkzAudioFlamingo3Model.__init__s  r$zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.r.r?r@r]rAc |j|f|dd|}|j|j}|jdj t j }|jj|\}}t j|jd|jdddf|dddfk} || j |j|_ |S)a1 input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padded feature indices. T)r@ return_dictrCrrDN) audio_towermulti_modal_projectorrKrMtor9long _get_feat_extract_output_lengthsrNrLrE pooler_output) r\r?r@r] audio_output audio_embeds input_lengths_ post_lengths valid_masks r%get_audio_featuresz&AudioFlamingo3Model.get_audio_featuress"(t''  0CQU Y_ 11,2P2PQ ,//366uzzB **KKMZ<\\,"4"4Q"7 @S@STUY[\U\]`lmnptmt`uu %1*-- @S@S2T%U "r$ input_idsrJ position_idsr4rI use_cachec ||j|}d} |_|]|j||dj} |j||| } |j | | j |j }|jd|||||d| } t| j| j| j| j| S)z input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padding feature indices. NT)r})rIaudio_features)rIrJrr4r)rKr4r5r6r7r#) get_input_embeddingsrrget_placeholder_maskmasked_scatterrrElanguage_modelr-rKr4r5r6) r\rr?r@rJrr4rIrr]rspecial_audio_maskoutputss r%rbzAudioFlamingo3Model.forwards$  7D557 BM  %)*?22>CVdh2iwwL"&!:!:|";" *889K\__]j]q]qMrsM%$%% ')%+    1%77#33!//)) ,   r$)NNNNNNNN)r r!r"_tp_plan_pp_plan_keep_in_fp32_modules_strictrkrr r9r:rer r r<r r LongTensorrboolrbrwrxs@r%rzrzsWHH#' !w))#\\+,  + + 4.23737.204(,26!%, ##d*, ))D0, #\\D0 ,  t+ , &&- , , ((4/, $;, +,, , r$rzz The AudioFlamingo3 model which consists of a fine-tuned Whisper encoder, a multi-modal projector and a Qwen2 language model. ceZdZdZfdZdZee ddejdzdejdzdejdzdejdzdejdzd e dzd ejdzd ejdzd e dzd eejzdeedeezfdZddde ffdZxZS)&AudioFlamingo3ForConditionalGenerationNcdt||t||_|j yrc)rjrkrzmodel post_initrus r%rkz/AudioFlamingo3ForConditionalGeneration.__init__s&  (0  r$c >|jj||fi|Src)rr)r\r?r@r]s r%rz9AudioFlamingo3ForConditionalGeneration.get_audio_featuress",tzz,,^=P[TZ[[r$rr?r@rJrr4rIlabelsrlogits_to_keepr]rAc |jd|||||||| d| } | j} t| tr t | dn| }|j | dd|ddf}d}|4|j d|||jjjd| }t||| j| j| j| jS)a input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padding feature indices. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Example: ```python >>> from transformers import AudioFlamingo3ForConditionalGeneration, AutoProcessor >>> model_id = "nvidia/audio-flamingo-3-hf" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = AudioFlamingo3ForConditionalGeneration.from_pretrained(model_id, device_map="auto") ```)rr?r@rJrr4rIrN)r3r vocab_size)r2r3r4r5r6r7r#)rrK isinstanceintslicelm_head loss_functionrHrorr1r4r5r6r7)r\rr?r@rJrr4rIrrrr]rr5 slice_indicesr3r2s r%rbz.AudioFlamingo3ForConditionalGeneration.forwards>$**  ) 3)%+'     11 8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD4#33!//)) ' ; ;   r$F)is_first_iterationrc|jdd}|jdd}t||i|}|s|jdds|||d<|||d<|S)Nr?r@rF)poprjprepare_inputs_for_generationget)r\rargsr]r?r@ model_inputsrvs r%rzDAudioFlamingo3ForConditionalGeneration.prepare_inputs_for_generation\st$4d;$jj)>Ewrz)2 dataclassesrr9r activationsr cache_utilsr masking_utilsrmodeling_outputsr r processing_utilsr utilsr r rr utils.genericrutils.output_capturingr qwen2_audio.modeling_qwen2_audiorrvoxtral.modeling_voxtralrrrrwhisper.modeling_whisperrrconfiguration_audioflamingo3r get_loggerr loggerrr(r*r-r1r>rgrzr__all__r#r$r%rsW " ! 6G&RR75 M>   H % .  !4 '$='  (B     9;9  9( ? -?  ? D (B " T ,T  T n U-LU  Up r$