^j( ddlmZddlZddlmZddlmZddlm Z m Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lmZmZmZmZdd lmZdd lmZddlm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0ejbe2Z3edeGddeZ4Gdde)Z5Gdde*Z6Gdde'Z7Gdd e$Z8Gd!d"e%Z9ed#eGd$d%eZ:eGd&d'e(e:Z;Gd(d)ejxZ=ed*Gd+d,e&e Z>Gd-d.ejxZ?eGd/d0e(Z@ed1Gd2d3e:e0ZAgd4ZBy)5) dataclassN)initialization)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)Unpack) ModelOutputauto_docstringcan_return_tuplelogging)merge_with_config_defaults)is_torchdynamo_compiling)capture_outputs) AutoModel)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModel LlamaRMSNormLlamaRotaryEmbeddingTransformersKwargs) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dfdzed<dZ e ej dfdzed<dZej dzed <dZej dzed <dZe dzed <dZe ej dfdzed <dZe ej dfdzed <dZej dzed<y)CsmOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r&torch FloatTensor__annotations__r'r(rr)tupler*r+r,r-r.r/r0n/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/csm/modular_csm.pyr%r%1s8&*D%  d ")'+FE   $+$(OUT\(:>M5**C/047>7;Je'',-4;37))D0759%++d2926!54<6HLu'8'8#'=!>!ELEIeE$5$5s$:;dBI.2M5$$t+2r:r%c eZdZy) CsmRMSNormNr1r2r3r9r:r;r=r=br:r=c eZdZy)CsmRotaryEmbeddingNr>r9r:r;rArAfr?r:rAc eZdZy)CsmMLPNr>r9r:r;rCrCjr?r:rCc eZdZy) CsmAttentionNr>r9r:r;rErEnr?r:rEc eZdZy)CsmDecoderLayerNr>r9r:r;rGrGrr?r:rGz[ The bare Csm Model outputting raw hidden-states without any specific head on top. ceZdZUeed<dZdZdZdgZdgZ dZ dZ dZ dZ eedZej$fdZxZS) CsmPreTrainedModelconfigmodel)audiotextTrGr()r)r*ct||t|trV|j}t |dz D]8}t j|jd|jj:yt|tr_t j|jtj|jj|jj zyy)Nrg)meanstd)super _init_weights isinstanceCsmCodebooksHead num_codebooksrangeinitnormal_weightrJinitializer_rangeCsmBackboneModelEmbeddingscopy_audio_tokens_offsetsr5arange vocab_size)selfmodulerUi __class__s r;rRz CsmPreTrainedModel._init_weightss f% f. /"00M=1,- Y V]]$++:W:WX Y  : ; JJv22ELLAZAZ4[^b^i^i^t^t4t u*>ARAR*RU[UpUpq')yy1L1LfN`N`gl'm$r:N input_idsbackbone_last_hidden_stateattention_mask position_idsr( inputs_embeds use_cachekwargsreturnc |!tstjdd}|du|duz r td|r|t |j }||j nd} ||jdn|jd} | |jn |j} tj| | | z| }|rtj|dz d} | |jz} |j|| z}|ddk(}| ||dddf<n!ts|rtjd |j|}t!|j |||| }|}|j#d}|j%|| }|j&d|j j(D]}||f|||||d |}|j+|}t-||r| Sd S)aJ backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.)rJrrdevice)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.)rJrrr(r)r)rrr(rposition_embeddings)last_hidden_stater()rlogger warning_once ValueErrorrrJget_seq_lengthshaperr5r^clampr_rzwarningr}r unsqueeze rotary_emblayersnum_hidden_layersnormr )r`rrrrr(rrrpast_seen_tokensinputs_seq_lengthr codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr)r decoder_layers r;forwardzCsmDepthDecoderModel.forwards'&  #,D,F   w  L  -t"; <Z[ [ 0*$++>O?N?Z?99;`a6C6OM//2U^UdUdefUg)6)B%% HXHX||$46FIZ6Zcij  !KK q(8a@M"T__4F --i&.@AM+7?a+? ()5&@ ad#/16RNNQ44]C (;;')+%  & $--a0 "oom,oW![[)H4;;+H+HI M)*) /#$7 M  -0 &+/8O  >B  r:)NNNNNNN)r1r2r3r!r7rvrrrr5 LongTensorr6Tensorrboolr rr8r rrorps@r;rrrrs !!n  .2?C.204(,26!%N ##d*N %*$5$5$<N  t+ N &&- N  N ((4/N $;N +,N  ( (N  N r:rrc&eZdZfdZddZxZS)rTct|||_tjt j |jdz |||_y)Nr)rQrvrUrw Parameterr5emptyrY)r`r|rUr_rcs r;rvzCsmCodebooksHead.__init__s? *ll5;;t/A/AA/E{T^#_` r:c |dz }|j|}t|jdDcgc]9}tjj |dd|ddf||j ;}}tj|d}|Scc}w)Nrrdim) rYrVrrw functionallinearTr5stack)r`r)codebook_indicescodebook_weight codebook_idxs r;rzCsmCodebooksHead.forwards+a/++&67!&o&;&;A&> ?  MM q,/A!BOT`DaDcDc d   Mq9   s>BNr1r2r3rvrrorps@r;rTrTsa r:rTa$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) ceZdZdZdZdZfdZ ddejde dzde dzdejdzdejdzde dzf fd Z ee ddejdzd ejdzdej dzd ejdzde dzdejdzd ejdzd e dzde ej zdeedeezfdZxZS)CsmDepthDecoderForCausalLMNct|||`t|j|j |j |_t||_ yr) rQrvlm_headrTr|rUr_codebooks_headrrrKr~s r;rvz#CsmDepthDecoderForCausalLM.__init__sE  L.v/A/A6CWCWY_YjYjk)&1 r:rnext_sequence_lengthr(rris_first_iterationc xt ||||||fi|}|s|jd|jd|S)Nrr)rQprepare_inputs_for_generationpop) r`rrr(rrrr model_inputsrcs r;rz8CsmDepthDecoderForCausalLM.prepare_inputs_for_generationsRw< +_nm _e "   9 : (r:rrlabelsrlogits_to_keeprrc ||jnd} ||jdn|jd} | |jn |j} tj| | | z}|j d |||||||d| }|d}t | tr | dk(r tdd}nt| d}n| }|j|dd|ddf||}|j}d}|B|dddfj}|jd |d|jj|d| }t|||j|j |j"S) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nrrr)rrrrr(rr.)r'rr_ shift_labels)r&r'r(r)r*r9)rrrr5r^rKrSintslicer contiguous loss_functionrJr_r r(r)r*)r`rrrrr(rrrrrrseq_lenrroutputsr) slice_indicesr'r&rs r;rz"CsmDepthDecoderForCausalLM.forward3s0@O?Z?99;`a,9,E-%%a(9??[\K])6)B%% HXHX <<?BRR$**  'A)%+'      nc *" %a %~ot < *M$$]1mQ3F%GIYZgIhi""$  !#qr'?557L%4%%dt{{7M7M\hlrD&#33!//))   r:)NNNNF) NNNNNNNNr)r1r2r3_tied_weights_keys_tp_plan_pp_planrvr5rrrr6rrrrrr rr8r rrorps@r;rr sHH2,0(,2626*/##"Dj  ((4/  ((4/ !4K,.2?C.204(,26*.!%-.A ##d*A %*$5$5$<A  t+ A &&- A  A ((4/A   4'A $;A ell*A +,A  ' 'A A r:rc$eZdZfdZdZxZS)r[ct|tj|j|j z|j |_|jdtj|j|j zdy)Nr]F) persistent) rQrvrwrxrU codebook_sizer|embed_audio_tokensregister_bufferr5r^r~s r;rvz#CsmBackboneModelEmbeddings.__init__zsn "$,,0D0DvG[G[0[^d^p^p"q  "ELL1E1E$FI]I]$]jo  r:cf|j||jz}|jd}|S)Nrr)rr]sum)r`rrs r;rz"CsmBackboneModelEmbeddings.forwards6// Dr:c "t|di|S)a& input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) r9)rQr)r` super_kwargsrcs r;rzCsmBackboneModel.forwards w...r:) r1r2r3rvrrrrrorps@r;rrs,?  /  /r:rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. ceZdZddiZfdZdZdZefdZfdZ dd e jdzd e jdzd e jdzd e jdzd e jdzf dZ dd e jdedzdedzde jdzde j dzf fd Zee dd e jdzd e jdzde jdzd e jdzde jdzdedzde j dzd e jdzdedzdee jzdeed eezfdZxZS)CsmForConditionalGenerationz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightct|||j|_tj|j |jd|_tj|j|j |_ tj||_ tj|j|_t!j"|j$|_|j)yrt)rQrvr_rwr{r|rrxtext_vocab_sizeembed_text_tokensr _from_configbackbone_modelrdepth_decoder_config depth_decoderr from_config codec_config codec_model post_initr~s r;rvz$CsmForConditionalGeneration.__init__s   ++yy!3!3V5F5FUS !#f.D.DfFXFX!Y.;;FC7DDVE`E`a$001D1DE r:c.|jjSrrrz)r`s r;get_input_embeddingsz0CsmForConditionalGeneration.get_input_embeddingss""///r:c&||j_yrr)r`values r;set_input_embeddingsz0CsmForConditionalGeneration.set_input_embeddingss+0(r:c|jddrt ||i|\}}nt ||i|}d}t|}t |j j Dcic]\}}|j|r||d|} }}t |jj jddi| | D]}t|j ||zd|vr|fS|Scc}}w)Noutput_loading_infoFdepth_decoder__from_model_config) getrQfrom_pretrainedlenvarsgeneration_configitems startswithrupdatedelattr) clsargsrrK loading_infoprefix prefix_lenattrrdepth_decoder_attrsrcs r;rz+CsmForConditionalGeneration.from_pretraineds ::+U 3"''"94"J6"J E<G+TzQCsmForConditionalGeneration._merge_input_ids_with_input_values..s&Orrxx{&Os)rdtypeiTas_tuple)rr)$rrwrpaddiffr5r^maxrexpandrrrnziprVrrencode audio_codes transposeappendrget_audio_codes_maskrJaudio_token_idrrzonesrUlongcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrnonzero)r`rrrrr audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsrb start_idxend_idx audio_batch codec_outputs codebook_idsmax_audio_framesr batched_audio_token_idsaudio_codes_maskraudio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxss r;"_merge_input_ids_with_input_valuesz>CsmForConditionalGeneration._merge_input_ids_with_input_valuess*..y9  ##%==#4#45I6#R 01E1JKPPRM)-!*;q$A "O9O&P (,(8(8(?(? @U@UVW@X(Y '4'@'@'J'J1b'Q )00aA BB$'&O=N&O#O */++`qrZ\R]]&&rAq!5EQR 5S+TUr+'$(#3#3#H#HIZ#[ ! \$"[[77N(N: ..;;rr(rrc 0t |d |||||d|}|}|jdk(rn|jd]|j ||jd|jd|jd}|j |d|ddd|S) N)rrr(rrrrrrr)rrrr)rrrr9)rQrndimrr9r) r`rrr(rrrr merged_inputsrcs r;rz9CsmForConditionalGeneration.prepare_inputs_for_generation1sw< !5+)'      Y^^q%8\=M=Mo=^=f CC##ZZ7%+ZZ0F%Gzz(+ DM   "/"@MZbLcrvw r:rrrrc |/|jdk(r |j||||} | d}| d}d}|jd|||||| d| } | d}t| tr t | dn| }|j |dd|ddf}d}d}d}d}||dddddf}|jd|||jjd| }|ddddddfd k(jd }||d d|jjdz f}tjj|d d}|jd}||d|ddz ddf}||}|j d||| d|d| }|j"}||z}t%||||| j&| j(| j*| |j,nd| |j&nd| |j(nd||j* Sd S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be inferred from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrrr)rrrr(rrr)r'rr_rrr r.r )rTr)rrr return_dictr) r&r0r+r'r(r)r*r,r-r.r/r9)r;r9rrSrrrrrJr_allrUrwrrr$rr&r%r(r)r*r')r`rrrrrr(rrrrrr<backbone_outputsbackbone_hidden_statesrbackbone_logitsr&r0r+depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss r;rz#CsmForConditionalGeneration.forwardPsd  Y^^q%8 CC<)=vM*/:M"8,FI.4.. )%+'   "2!!48B>SV8W~ot4]k ,,'=aPQ>Q'RS ! $  $Q1WoO.D..&4;;KaKaekM "!Q(+t388R8@@J&,Z&8>] @Y@Y\]@]>]9]&^ #&(mm&7&78OQW_`&7&a ##++T+:J*@APZ[\P]`aPacdAd*e '#)*#5 $6D$6$6%1+F# + %  % !"7!;!;  #55D '1",<<*88'22AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%E  qu  r:)NNNN) NNNNNNNNNr)r1r2r3rrvrr classmethodrrr5rr9rrrr6rrrrr rr8r%rrorps@r;rrsD @Aj0141*.,048&* PB<<$&PBllT)PB$llT1 PB  t# PB   PBj,0(,2626 ##"Dj  ((4/  ((4/ >.2,0.24804(,26*.!%-.Y ##d*Y llT)Y  t+ Y $llT1 Y &&- Y Y ((4/Y   4'Y $;Y ell*Y +,Y  " "Y Y r:r)rIrrrrr)C dataclassesrr5torch.nnrwrrW cache_utilsrr generationr masking_utilsr modeling_outputsr r modeling_utilsr processing_utilsr utilsrrrr utils.genericrutils.import_utilsrutils.output_capturingrautorllama.modeling_llamarrrrrrrrconfiguration_csmr r!generation_csmr" get_loggerr1rr%r=rArCrErGrIrrModulerTrr[rr__all__r9r:r;r_s" &.)/O-&KK7:5   @.   H %  '3 '3  '3V   -  X  >  '  vv  v<Y :'9Y Y xryy(d !1?d d N  /z//0 F "46HF  F R  r: