^jm8ddlZddlmZddlmZddlZddlmZddlmZddl m Z m Z ddl m Z dd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZddlmZddlmZmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+e!jXe-Z. d6dej^dej`dej`dej`dej`dzde1dzde1fdZ2Gddej^Z3Gd d!eZ4eGd"d#eZ5eGd$d%eZ6ed&'eGd(d)eZ7ed*'Gd+d,e5Z8Gd-d.ej^Z9ed/'Gd0d1e5Z:ed2'Gd3d4e5e Z;gd5ZL==((6??([L,,|U3K''1-88:K  $$c:eZdZdZ ddedededededed edzd edzffd Z dd e jd e jdzde dzde jdzdede e dee je jdzee jdzffdZxZS)AudioFlamingo3Attentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsr& is_decoderbias is_causal layer_idxconfigc zt |||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ |/|r-tjd|jjd||_t!j"||d|_t!j"||||_t!j"||||_t!j"||||_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)zInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Fr@)super__init__r=r>r&head_dimrC ValueErrorr%r?rAlogger warning_once __class____name__rBrLineark_projv_projq_projout_proj) selfr=r>r&r?r@rArBrCrLs r8rGz AudioFlamingo3Attention.__init__Ms  "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"     *4>>+B+B*CD,,  #ii 95A ii 94@ ii 94@  )YTB r: hidden_stateskey_value_statespast_key_valuesr$output_attentionsr5returnc |du}|jdd}g|d|j} |j||jzj | j ddj } |it|trY|jj|j} |r&d|j|j<|j}n |j}||n|} |rK|rI rG|j|jj} |j|jj }n|dd|j"|jf}|j%| j |j ddj } |j'| j |j ddj }| |j)| ||j\} }t+j,|j.j0t2}||| | ||f|j4sdn |j6d|d |\}}|j8g|dj }|j;|}||fS) z#Input shape: Batch x Time x ChannelNr(rrTr?)r&r%rW)shaperHrQr%viewr1r4 isinstancer is_updatedgetrBcross_attention_cacheself_attention_cachelayerskeysvaluesr>rOrPupdater get_interfacerC_attn_implementationr9r-r&reshaperR)rSrTrUrVr$rWr5is_cross_attention input_shape hidden_shape query_statesr_current_states key_states value_stateskv_shapeattention_interfacer7r6s r8forwardzAudioFlamingo3Attention.forwardus`.T9#))#2. 88b8$--8  M2T\\AGG U__`acdeppr   &:oGZ+[(3377GJ!=A**4>>:"1"G"G"1"F"F.>-I)} /j(//?DDJ*11$..AHHL $ADNNDMMJH^499(CMMaQRS^^`J;;~6;;HEOOPQSTU``bL*+:+A+A*l\`\j\j+k( L(?(M(M KK , ,.E) %8      % $}}C$,,/ %  % ! \*k));;;;FFH mmK0 L((r:)rZFTFNN)NNNF)rM __module__ __qualname____doc__intfloatboolrrGr/Tensorrrr tuplers __classcell__rLs@r8r<r<Js&G   $.2&C&C&C &C  &C  &C&C:&C%t+&CV15(,.2"' H)||H) ,,-H) H)  t+ H)  H)-.H) u||U\\D0% 2E2LL MH)r:r<c~eZdZdeffd Zdej dej deedej fdZ xZ S)AudioFlamingo3EncoderLayerrCcht||j|_t |j|j |j ||_tj|j|_ |j|_ t|j|_|j|_tj |j|j"|_tj |j"|j|_tj|j|_y)N)r=r>r&rC)rFrGd_modelr=r<encoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normr&ractivation_function activation_fnactivation_dropoutrNencoder_ffn_dimfc1fc2final_layer_normrSrCrLs r8rGz#AudioFlamingo3EncoderLayer.__init__s 0nn44,,   %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r:rTr$r5rXc |}|j|}|jd||d|\}}tjj ||j|j }||z}|}|j |}|j|j|}tjj ||j|j }|j|}tjj ||j|j }||z}|jtjk(rEtj|jjdz }tj || |}|S)a> Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. )rTr$r+i)minmax)rrrr2r&r-rrrrrdtyper/float16finforclamp)rSrTr$r5residual_ clamp_values r8rsz"AudioFlamingo3EncoderLayer.forwardsT!11-@ )4>> ')   q  --mt||VZVcVc-d  =0  --m< **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0   %-- /++m&9&9:>>EK!KK K<[YMr:) rMrtrurrGr/rzrrrsr|r}s@r8rrsM=3=$"||" "+, "  "r:rc<eZdZUeed<dZdZdZdgZdgZ dZ dZ dZ y)AudioFlamingo3PreTrainedModelrCmodel)audiotextTr<rVN) rMrtrur__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_attention_backendrr:r8rrs=  (&*#23#4"5N"&r:rc:eZdZUdZdZej dzed<y)!AudioFlamingo3ModelOutputWithPastzg audio_hidden_states (`torch.FloatTensor`, *optional*): Projected audio hidden states. Naudio_hidden_states)rMrtrurvrr/ FloatTensorrrr:r8rrs 59**T18r:rzZ Base class for AudioFlamingo3 causal language model (or autoregressive) outputs.  custom_introceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) $AudioFlamingo3CausalLMOutputWithPastaV loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. audio_hidden_states (`torch.FloatTensor`, *optional*): Hidden states of the audio encoder after projection. NlosslogitsrVrT attentionsr)rMrtrurvrr/rrrrVrrTr{rrrr:r8rrs &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r:rzT The audio model from AudioFlamingo3 without any head or projection on top. c eZdZUdZeed<dZdZdgZe e dZ deffd Z dZ d ejfd Zd ejfd Zee ddej*dej*d zd eezfdZdej2fdZxZS)AudioFlamingo3EncoderzY AudioFlamingo3 encoder: Whisper encoder, average pool (time/2), then LayerNorm. rCinput_featuresrr)rTrcbt|||j|_|j|_|j }|j |_|j|_|jrtj|nd|_ tj|j |dd|_tj||ddd|_tj |j||_|j"j%dtj&t)|j*Dcgc] }t-|c}|_tj0|j |_tj4dd|_d|_|j;ycc}w) Nr[rr) kernel_sizepaddingr)rstriderF)r)rFrGr&encoder_layerdrop layerdropr num_mel_binsmax_source_positionsscale_embeddingmathsqrt embed_scalerConv1dconv1conv2 Embeddingembed_positionsrequires_grad_ ModuleListrangeencoder_layersrrcr layer_norm AvgPool1d avg_poolergradient_checkpointing post_init)rSrCr=rrLs r8rGzAudioFlamingo3Encoder.__init__>s7  ~~ 11NN "//$*$?$?!393I3I499Y/sYYt00)TUV YYy)1VWX !||D,E,EyQ ++E2mmQVW]WlWlQm$nA%?%G$no ,,v~~6,,q3&+# %os6F,cJ|jD] }d|_ d|_y)NF) parameters requires_grad_requires_grad)rSparams r8_freeze_parametersz(AudioFlamingo3Encoder._freeze_parametersWs(__& (E"'E  (#r:rXc|jSNr)rSs r8get_input_embeddingsz*AudioFlamingo3Encoder.get_input_embeddings\s zzr:r#c||_yrr)rSr#s r8set_input_embeddingsz*AudioFlamingo3Encoder.set_input_embeddings_s  r:Ninput_features_maskc |jddz dzdz}|jd}|dz dzdz}tj||j|dddfk}t j j|j|}t j j|j|}|jddd}||jjz}t j j||j|j}t|j ||}|j"D]>} |jxr"tj$g|j&k} | r6| ||}@|jddd}|j)|jddd}|j+|}t-| S) ap Args: input_features (`torch.FloatTensor` of shape `(batch_size, feature_size, sequence_length)`): Log-Mel features extracted from raw audio. Use the processor/feature extractor to compute and pad these features from waveform input. input_features_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding feature indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. r(rrdeviceNrr+)rC inputs_embedsr$)last_hidden_state)r\sumr/arangerrr2gelurrpermuterweightr&r-r rCrcrandrrrr) rSrrr5seq_leninput_features_lengthsrrTr$layerdrops r8rszAudioFlamingo3Encoder.forwardbs("''+a/A59!4!8!8!<"81"<!BQ!F#ll7>;P;PQTjklnrkrTss **4::n+EF  **4::m+DE %--aA6 &(<(<(C(CC  --mt||VZVcVc-d 2;;'. [[ EE==DUZZ^dnn%DD %m^ D  E &--aA6  6>>q!QG  6 )+  r: input_lengthsc6|dz dzdz}|dz dzdz}||fS)zs Computes the output length of the convolutional layers and the output length of the audio encoder rrr)rSroutput_lengthss r8 _get_feat_extract_output_lengthsz6AudioFlamingo3Encoder._get_feat_extract_output_lengthss7'*q014 '!+1A5n,,r:r)rMrtrurvrrmain_input_namerrrr<_can_record_outputsrGrrModulerrrrr/rzr{rrs LongTensorrr|r}s@r8rr)s  ('&O564- :2$ bii")) 483  3 #\\D03 + + 3  3 l-e>N>N-r:rc.eZdZdZdeffd ZdZxZS)!AudioFlamingo3MultiModalProjectorz Audio adaptor (small MLP) that projects AudioFlamingo3Encoder features to the LLM embedding space so they can replace `` tokens. rCct|tj|jj |j j |j|_t|j|_ tj|j j |j j |j|_ y)NrE) rFrGrrN audio_config hidden_size text_configprojector_biaslinear_1rprojector_hidden_actactlinear_2rs r8rGz*AudioFlamingo3MultiModalProjector.__init__s     + +V-?-?-K-KRXRgRg  &556    * *F,>,>,J,JQWQfQf  r:cl|j|}|j|}|j|}|Sr)rrr)rSaudio_featuresrTs r8rsz)AudioFlamingo3MultiModalProjector.forwards2 n5 /  m4 r:)rMrtrurvrrGrsr|r}s@r8rrs  3 r:rz The AudioFlamingo3 model (fine-tuned Whisper encoder, multi-modal projector, Qwen2 language model), without a language modeling head. ceZdZdZdZdZfdZeedde jde jde e deezfd Zd e j"d e jd e jfd Zee dd e j"dzde jdzde jdzde jdzde j"dzdedzd e jdzdedzde e deezfdZxZS)AudioFlamingo3ModelNct||tj|j|_tj|j |_t||_ |jyr) rFrGr from_configr audio_towerrlanguage_modelrmulti_modal_projectorrrs r8rGzAudioFlamingo3Model.__init__sY  $001D1DE'33F4F4FG%Fv%N" r:zThis method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.rrrr5rXc |j|f|dd|}|j|j}|jdj t j }|jj|\}}t j|jd|jdddf|dddfk} || j |j|_ |S)a1 input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padded feature indices. T)r return_dictr(rrN) rrrrtor/longrrr\r pooler_output) rSrrr5 audio_output audio_embedsrr post_lengths valid_masks r8get_audio_featuresz&AudioFlamingo3Model.get_audio_featuress"(t''  0CQU Y_ 11,2P2PQ ,//366uzzB **KKMZ<\\,"4"4Q"7 @S@STUY[\U\]`lmnptmt`uu %1*-- @S@S2T%U "r: input_idsrrc.|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd}|jdj|j|j }t||j|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )rrr(rz6Audio features and audio tokens do not match, tokens: z , features: )rr/tensorrCaudio_token_idr rallrr\ unsqueeze expand_asr rnumel)rSrrrspecial_audio_maskn_audio_tokensn_audio_featuress r8get_placeholder_maskz(AudioFlamingo3Model.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2/99"=GG VYYZgZnZno , - 3 3 59M9M9O OD^DTT`aq`r s "!r:r$ position_idsrV use_cachec ||j|}d} |_|]|j||dj} |j||| } |j | | j |j }|jd|||||d| } t| j| j| j| j| S)z input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padding feature indices. NT)r )rr)rr$rrVr)rrVrTrrr) rrr rmasked_scatterr rrrrrVrTr) rSrrrr$rrVrrr5rroutputss r8rszAudioFlamingo3Model.forwards$  7D557 BM  %)*?22>CVdh2iwwL"&!:!:|";" *889K\__]j]q]qMrsM%$%% ')%+    1%77#33!//)) ,   r:)NNNNNNNN)rMrtru_tp_plan_pp_plan_keep_in_fp32_modules_strictrGrrr/rrzrrr{rrrrrryrrsr|r}s@r8rrsHH#' w))#\\+,  + + 4"))":?:K:K"]b]n]n"0.23737.204(,26!%, ##d*, ))D0, #\\D0 ,  t+ , &&- , , ((4/, $;, +,,  2 2, , r:rz The AudioFlamingo3 model which consists of a fine-tuned Whisper encoder, a multi-modal projector and a Qwen2 language model. ceZdZdgZdZfdZdZee dde jdzde jdzde jdzde jdzd e jdzd e dzd e jdzd e jdzd edzdee jzdeedeezfdZdddeffdZxZS)&AudioFlamingo3ForConditionalGenerationrNct||t||_t j |j j|j jd|_ |jy)NFrE) rFrGrrrrNrr vocab_sizelm_headrrs r8rGz/AudioFlamingo3ForConditionalGeneration.__init__<sS  (0 yy!3!3!?!?ASASA^A^ejk  r:c >|jj||fi|Sr)rr)rSrrr5s r8rz9AudioFlamingo3ForConditionalGeneration.get_audio_featuresBs",tzz,,^=P[TZ[[r:rrrr$rrVrlabelsrlogits_to_keepr5rXc |jd|||||||| d| } | j} t| tr t | dn| }|j | dd|ddf}d}|4|j d|||jjjd| }t||| j| j| j| jS)a input_features_mask (`torch.Tensor` of shape `(batch_size, feature_sequence_length)`): Mask to avoid performing attention on padding feature indices. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Example: ```python >>> from transformers import AudioFlamingo3ForConditionalGeneration, AutoProcessor >>> model_id = "nvidia/audio-flamingo-3-hf" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = AudioFlamingo3ForConditionalGeneration.from_pretrained(model_id, device_map="auto") ```)rrrr$rrVrrN)rr,r))rrrVrTrrr)rrr^rwslicer* loss_functionrCrr)rrVrTrr)rSrrrr$rrVrr,rr-r5r"rT slice_indicesrrs r8rsz.AudioFlamingo3ForConditionalGeneration.forwardEs>$**  ) 3)%+'     11 8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD4#33!//)) ' ; ;   r:F)is_first_iterationr2c|jdd}|jdd}t||i|}|s|jdds|||d<|||d<|S)NrrrF)poprFprepare_inputs_for_generationr`)rSr2argsr5rr model_inputsrLs r8r5zDAudioFlamingo3ForConditionalGeneration.prepare_inputs_for_generationst$4d;$jj)>EwrKs, $! !5)6B9``F&jj75[   H %!% II% <<% % << % LL4' % T\ %%4s)biis)l5!;5p 'O ' ' 9(?9 9  9;9  9( r-9r-  r-j . p 7p  p f W-JOW  Wt r: