^jmddlmZddlmZddlmZddlZddlmZddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZmZdd lmZdd lmZdd lmZmZddlmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6m7Z7ddl8m9Z9e+jte;Z<e)deGdde'Z=edGdd ej|Z?Gd!d"ej|Z@Gd#d$ej|ZAd%ZBed&dKd'ZCd(ejd)eEd*ejfd+ZF dLd,ej|d-ejd.ejd/ejd0ejdzd1eGd2eGd3e%e(fd4ZHeeCGd5d6ej|ZIGd7d8eZJe)d9e)Gd:d;e#ZKe)Gd<d=eKZLGd>d?ej|ZMe)d@GdAdBeKeZNGdCdDej|ZOe)GdEdFeKZPe)dGGdHdIeKe9ZQgdJZRy)M)Callable) dataclass)OptionalN)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)is_torchdynamo_compiling)capture_outputs) AutoModel) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dfdzed<dZ e ej dfdzed<dZej dzed <dZej dzed <dZe dzed <dZe ej dfdzed <dZe ej dfdzed <dZej dzed<y)CsmOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r*torch FloatTensor__annotations__r+r,r r-tupler.r/r0r1r2r3r4o/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/csm/modeling_csm.pyr)r)3s8&*D%  d ")'+FE   $+$(OUT\(:>M5**C/047>7;Je'',-4;37))D0759%++d2926!54<6HLu'8'8#'=!>!ELEIeE$5$5s$:;dBI.2M5$$t+2r>r)RMSNormcheZdZddeddffd Zdej dej fdZdZxZ S) CsmRMSNormepsreturnNct|tjt j ||_||_y)z9 CsmRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parameterr9onesweightvariance_epsilon)self hidden_sizerC __class__s r?rGzCsmRMSNorm.__init__es1 ll5::k#:; #r>r-c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)Nr!T)keepdim) dtypetor9float32powmeanrsqrtrLrK)rMr- input_dtypevariances r?forwardzCsmRMSNorm.forwardmsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r>c^t|jjd|jS)Nz, eps=)r<rKshaperLrMs r? extra_reprzCsmRMSNorm.extra_reprts* ))*+6$2G2G1HIIr>)gư>) r5r6r7floatrGr9Tensorr[r_ __classcell__rOs@r?rBrBcs7$$$$;U\\;ell;Jr>rBc eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)CsmRotaryEmbeddinginv_freqNconfigct||j|_|j|_||_|j j d|_|j}|jdk7rt|j}||j |\}|_ |jd|d|jd|jdy)N rope_typedefaultrfF persistentoriginal_inv_freq) rFrGmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrgrope_parametersricompute_default_rope_parametersrattention_scalingregister_bufferclone)rMrgdevice rope_init_fnrfrOs r?rGzCsmRotaryEmbedding.__init__{s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L($( ZeD 0(..2BuUr>rvz torch.deviceseq_lenrDz torch.Tensorc|jd}t|ddxs|j|jz}d}d|t j d|dtj j|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr!rSrvrS) rqgetattrrNnum_attention_headsr9arangeint64rTr`)rgrvrxbasedimattention_factorrfs r?rrz2CsmRotaryEmbedding.compute_default_rope_parameterss&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k )))r>cN|jddddfjj|jdddj |j }|dddddfj}t |j jtr/|j jdk7r|j jnd}t|d5|j|jzjdd}tj||fd }|j|jz}|j|jz} dddj |j  j |j fS#1swYAxYw) NrrQr#mpscpuF) device_typeenabledr!rr|)rfr`expandr]rTrv isinstancetypestrr transposer9catcosrssinrS) rMx position_idsinv_freq_expandedposition_ids_expandedrfreqsembrrs r?r[zCsmRotaryEmbedding.forwardsR!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk  U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BFF$NNNN)r5r6r7r9rar;r$rG staticmethodrintr<r`rrno_gradrr[rbrcs@r?rerexsllVyV #'+/"*D *(*t* ~u$ % **:U]]_ < rec$eZdZfdZdZxZS)CsmMLPct|||_|j|_|j|_t j |j|j|j|_t j |j|j|j|_ t j |j|j|j|_ t|j|_ y)Nbias)rFrGrgrNintermediate_sizerHLinearmlp_bias gate_projup_proj down_projr hidden_actact_fnrMrgrOs r?rGzCsmMLP.__init__s  !--!'!9!94#3#3T5K5KRXRaRabyy!1!143I3IPVP_P_` 4#9#94;K;KRXRaRabV../ r>c|j|j|j||j|z}|Sr)rrrr)rMrrs r?r[zCsmMLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r>r5r6r7rGr[rbrcs@r?rrs 0r>rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..NrQr!r)r]r9r)rx1x2s r? rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r>rotary_pos_embc|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkrr unsqueeze_dimq_embedk_embeds r?apply_rotary_pos_embrsY& -- &C -- &C3w;q>C/0G3w;q>C/0G G r>r-n_reprDc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r#N)r]rreshape)r-rbatchnum_key_value_headsslenr{s r? repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr>modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||j dd|z} || |z} t j j| dtjj|j} t j j| ||j} tj| | } | j ddj} | | fS)Nr!rrQ)rrS)ptrainingr#)rnum_key_value_groupsr9matmulrrH functionalsoftmaxrUrTrSrr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r?eager_attention_forwardrs3 ; ;ceZdZdZdedeffd Z d dejde ejejfdzdejdzd e dzd e e d e ejejff d Z xZS) CsmAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrg layer_idxcdt|||_||_t |d|j |j z|_|j |jz|_ |jdz|_ |j|_ d|_ tj|j |j |jz|j|_tj|j |j|jz|j|_tj|j |j|jz|j|_tj|j |jz|j |j|_y)Nr{gTr)rFrGrgrr~rNrr{rrrattention_dropout is_causalrHrattention_biasq_projk_projv_projo_projrMrgrrOs r?rGzCsmAttention.__init__sM  " F4F4F&JdJd4de $*$>$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf  r>Nr-position_embeddingsrr,rrDc |jdd}g|d|j}|j|j|j dd}|j |j|j dd} |j |j|j dd} |\} } t|| | | \}} | |j| | |j\} } tj|jjt} | ||| | |f|jsdn |j |j"d|\}}|j$g|dj'}|j)|}||fS)NrQr#r!)rr)r]r{rviewrrrrupdaterr get_interfacerg_attn_implementationrrrrrrr)rMr-rrr,r input_shape hidden_shape query_statesrrrrattention_interfacerrs r?r[zCsmAttention.forward*s$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST &S#7 jRUWZ#[ j  &'6'='=j,X\XfXf'g $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r>r)r5r6r7r8r$rrGr9rar<r rrr[rbrcs@r?rrsG y S 4IM.2(, &)||&)#5<<#=>E&) t+ &)  &) +, &) u||U\\) *&)r>rceZdZdedeffd Z ddejdejdzdejdzde dzd e dzd e ejejfdzd e e d ejfd ZxZS)CsmDecoderLayerrgrc t||j|_t|||_t ||_t|j|j|_ t|j|j|_ y)N)rgrrC) rFrGrNr self_attnrmlprB rms_norm_epsinput_layernormpost_attention_layernormrs r?rGzCsmDecoderLayer.__init__Tsk !--%VyI&>)&*<*<&BUBUV(263E3E6K^K^(_%r>Nr-rrr, use_cacherrrDc |}|j|}|jd||||||d|\}} ||z}|}|j|}|j|}||z}|S)N)r-rrr,rrr=)rrrr) rMr-rrr,rrrresidual_s r?r[zCsmDecoderLayer.forward^s!,,]; )4>> ')%+ 3    q!=0 !55mD /  =0 r>)NNNFN)r5r6r7r$rrGr9ra LongTensorr boolr<rrr[rbrcs@r?rrSs`y`S`/304(,!&HL|| t+&&-    $; #5<<#=>E+, r>rz[ The bare Csm Model outputting raw hidden-states without any specific head on top. ceZdZUeed<dZdZdZdgZdgZ dZ dZ dZ dZ eedZej$fdZxZS) CsmPreTrainedModelrgmodel)audiotextTrr,)r-r.ct||t|trV|j}t |dz D]8}t j|jd|jj:yt|tr_t j|jtj|jj|jj zyy)Nr#r)rWstd)rF _init_weightsrCsmCodebooksHead num_codebooksrangeinitnormal_rKrginitializer_rangeCsmBackboneModelEmbeddingscopy_audio_tokens_offsetsr9r vocab_size)rMrrirOs r?rz CsmPreTrainedModel._init_weightss f% f. /"00M=1,- Y V]]$++:W:WX Y  : ; JJv22ELLAZAZ4[^b^i^i^t^t4t u)r5r6r7r$r;base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr9rrrbrcs@r?rr~st (&*#*+#4"5N""&(" U]]_vvr>rceZdZUeed<fdZeee dde jdzde jdzde jdzde jdzde dzd e jdzd edzd eed eezfd ZxZS)CsmDepthDecoderModelrgc rt|||j|_|j|_t j |j|jz|j|_ t jt|jDcgc]}t||c}|_t|j |j"|_t'||_d|_t j,|j|j d|_|j1ycc}w)NrrgFr)rFrG pad_token_id padding_idxrrH Embeddingrbackbone_hidden_size embed_tokens ModuleListrnum_hidden_layersrlayersrBrNrnormre rotary_embgradient_checkpointingrinputs_embeds_projector post_initrs r?rGzCsmDepthDecoderModel.__init__s  !.. ++LL&*>*>ARAR*RU[UpUpqmmAFvG_G_A` aI_VY / a  v11v7J7JK ,F;&+#')yy1L1LfN`N`gl'm$  bsD4N input_idsbackbone_last_hidden_staterrr, inputs_embedsrrrDc |!tstjdd}|du|duz r td|r|t |j }||j nd} ||jdn|jd} | |jn |j} tj| | | z| }|rtj|dz d} | |jz} |j|| z}|ddk(}| ||dddf<n!ts|rtjd |j|}t!|j |||| }|}|j#d}|j%|| }|j&d|j j(D]}||f|||||d |}|j+|}t-||r| Sd S)aJ backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.rrr#rv)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.rgr-rr,rr)rrr,rrlast_hidden_stater,)rlogger warning_once ValueErrorr rgget_seq_lengthr]rvr9rclamprr"warningr)rrr'r%r$r&r)rMr+r,rrr,r-rrpast_seen_tokensinputs_seq_lengthrv codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr-r decoder_layers r?r[zCsmDepthDecoderModel.forwards'&  #,D,F   w  L  -t"; <Z[ [ 0*$++>O?N?Z?99;`a6C6OM//2U^UdUdefUg)6)B%% HXHX||$46FIZ6Zcij  !KK q(8a@M"T__4F --i&.@AM+7?a+? ()5&@ ad#/16RNNQ44]C (;;')+%  & $--a0 "oom,oW![[)H4;;+H+HI M)*) /#$7 M  -0 &+/8O  >B  r>)NNNNNNN)r5r6r7r%r;rGrr rr9rr:rar rrrr<rr[rbrcs@r?rrs !!  .2?C.204(,26!%N ##d*N %*$5$5$<N  t+ N &&- N  N ((4/N $;N +,N  ( (N  N r>rc&eZdZfdZddZxZS)rct|||_tjt j |jdz |||_y)Nr#)rFrGrrHrIr9emptyrK)rMrNrrrOs r?rGzCsmCodebooksHead.__init__ s? *ll5;;t/A/AA/E{T^#_` r>c |dz }|j|}t|jdDcgc]9}tjj |dd|ddf||j ;}}tj|d}|Scc}w)Nr#rr) rKrr]rHrlinearTr9stack)rMr-codebook_indicescodebook_weight codebook_idxs r?r[zCsmCodebooksHead.forwards+a/++&67!&o&;&;A&> ?  MM q,/A!BOT`DaDcDc d   Mq9   s>Brrrcs@r?rr sa r>ra$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) ceZdZdZdZdZfdZee dde jdzde jdzde jdzde jdzde dzde jdzd e jdzd edzd ee jzd eed eezfdZ dde jdedzde dzde jdzde jdzdedzf fd ZxZS)CsmDepthDecoderForCausalLMNct||t||_|j|_t |j |j|j|_|jyr) rFrGrrrrrNrcodebooks_headr*rs r?rGz#CsmDepthDecoderForCausalLM.__init__*sY  )&1  ++.v/A/A6CWCWY_YjYjk r>r+r,rrr,r-labelsrlogits_to_keeprrDc ||jnd} ||jdn|jd} | |jn |j} tj| | | z}|j d |||||||d| }|d}t | tr | dk(r tdd}nt| d}n| }|j|dd|ddf||}|j}d}|B|dddfj}|jd |d|jj|d| }t|||j|j |j"S) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Nrr#r/)r+r,rrr,r-r.)r+rPr shift_labels)r*r+r,r-r.r=)r8r]rvr9rrrrslicerOr loss_functionrgrrr,r-r.)rMr+r,rrr,r-rPrrQrr;rxrvrIoutputsr- slice_indicesr+r*rSs r?r[z"CsmDepthDecoderForCausalLM.forward3s0@O?Z?99;`a,9,E-%%a(9??[\K])6)B%% HXHX <<?BRR$**  'A)%+'      nc *" %a %~ot < *M$$]1mQ3F%GIYZgIhi""$  !#qr'?557L%4%%dt{{7M7M\hlrD&#33!//))   r>next_sequence_lengthis_first_iterationc xt ||||||fi|}|s|jd|jd|S)Nr,r)rFprepare_inputs_for_generationpop) rMr+rXr,rr-rYr model_inputsrOs r?r[z8CsmDepthDecoderForCausalLM.prepare_inputs_for_generationxsRw< +_nm _e "   9 : (r>) NNNNNNNNr)NNNNF)r5r6r7_tied_weights_keys_tp_plan_pp_planrGrrr9rr:rar rrrrr<rr[r[rbrcs@r?rMrMsHH.2?C.204(,26*.!%-.A ##d*A %*$5$5$<A  t+ A &&- A  A ((4/A   4'A $;A ell*A +,A  ' 'A A L,0(,2626*/##"Dj  ((4/  ((4/ !4Kr>rMc$eZdZfdZdZxZS)r ct|tj|j|j z|j |_|jdtj|j|j zdy)Nr Frk) rFrGrHr r codebook_sizerNembed_audio_tokensrtr9rrs r?rGz#CsmBackboneModelEmbeddings.__init__sn "$,,0D0DvG[G[0[^d^p^p"q  "ELL1E1E$FI]I]$]jo  r>cf|j||jz}|jd}|S)Nr!r)rdr sum)rMr+r-s r?r[z"CsmBackboneModelEmbeddings.forwards6// Drrcs@r?r r s  r>r ceZdZfdZeee d dejdzdejdzdejdzde dzdejdzde dzd e ed efd ZxZS) CsmBackboneModelc t|||j|_|j|_t ||_tjt|jDcgc]}t||c}|_ t|j|j|_t#||_d|_|j)ycc}w)NrrF)rFrGrrrr r"rHr#rr$rr%rBrNrr&rer'r(r*rs r?rGzCsmBackboneModel.__init__s  !.. ++6v>mmAFvG_G_A` aI_VY / a  v11v7J7JK ,F;&+#  bs)CNr+rrr,r-rrrDc <|du|duz r td||j|}|r|t|j}|V||j nd}t j |jd|j|z}|jd}t|j||||} |} |j| |} |jd|jjD]} | | f| | |||d |} |j| } t| | S) a& input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Nz:You must specify exactly one of input_ids or inputs_embedsrrr#r/r1r2)rrrr,rr3)r7r"r rgr8r9rr]rvrrr'r%r$r&r) rMr+rrr,r-rrr;r@r-rrAs r?r[zCsmBackboneModel.forwardsL2  -t"; <YZ Z  *.*;*;I*FM 0*$++>O  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L(;;')+%  & "oom,oW![[)H4;;+H+HI M)*$7) /# M  -0 &++  r>)NNNNNN)r5r6r7rGrr rr9rrar r:rrrrr[rbrcs@r?rhrhs  .2.204(,26!%> ##d*>  t+> &&- >  > ((4/ > $;> +,>  !>  > r>rhz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. ceZdZddiZfdZdZdZefdZfdZ dd e jdzd e jdzd e jdzd e jdzd e jdzf dZ dd e jdedzdedzde jdzde j dzf fd Zee dd e jdzd e jdzde jdzd e jdzde jdzdedzde j dzd e jdzdedzdee jzdeed eezfdZxZS)CsmForConditionalGenerationz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightct|||j|_tj|j |jd|_tj|j|j |_ tj||_ tj|j|_t!j"|j$|_|j)y)NFr)rFrGrrHrrNlm_headr text_vocab_sizeembed_text_tokensrh _from_configbackbone_modelrMdepth_decoder_config depth_decoderr" from_config codec_config codec_modelr*rs r?rGz$CsmForConditionalGeneration.__init__s   ++yy!3!3V5F5FUS !#f.D.DfFXFX!Y.;;FC7DDVE`E`a$001D1DE r>c.|jjSrrrr"r^s r?get_input_embeddingsz0CsmForConditionalGeneration.get_input_embeddingss""///r>c&||j_yrry)rMrs r?set_input_embeddingsz0CsmForConditionalGeneration.set_input_embeddings s+0(r>c|jddrt ||i|\}}nt ||i|}d}t|}t |j j Dcic]\}}|j|r||d|} }}t |jj jddi| | D]}t|j ||zd|vr|fS|Scc}}w)Noutput_loading_infoFdepth_decoder__from_model_config) getrFfrom_pretrainedlenvarsgeneration_configitems startswithrtrdelattr) clsargsrr loading_infoprefix prefix_lenattrrdepth_decoder_attrsrOs r?rz+CsmForConditionalGeneration.from_pretrained s ::+U 3"''"94"J6"J E<G+TNr+ input_valuesinput_values_cutoffsrPrDc|j|}|Utjj|d}||dk\j }||dkD}t j |j|jjt|d}||jdk}t j5g}t||D]\} } | | dk\} t| jddz D]r} | | } | | dz} | d| | f}|j j#|jd}|j$j'dd}|j)|dttd|D}t j*|Dcgc]6}tjj|ddd||jdz f8c}}|j j-|}ddd|j.j0}||k(}|j2j5}|||<t j6dd|j.j8f|jt j: |j.j<z}|j2j5|j?d}||j.j@k(}|jC|jEd||<|j|jdjCdd|j.j8}||||<|||<|d k(jGd }d ||d|dddf<|}||dScc}w#1swYxYw)a Merges the input_ids and input_values to produce a single inputs_embeds tensor: 1 - Infers the codec model on the input_values to retrieve codebook token. 2 - Embeds codebook tokens and places them at the correct positions in the inputs_embeds tensor. 3 - If labels are provided, expands them to match codebook dimensions and position the target codebook tokens in the inputs_embeds tensor. Args: input_ids (`torch.Tensor` of shape `(batch_size, sequence_length)`): The input ids to embed. input_values (`torch.Tensor` of shape `(batch_size, channels, audio_sequence_length)`): The audio input values to embed. input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`): The cutoffs of the audio input values relative to its batch index, padded with -1 when no audio. Nr#rrr/rQr#.c3:K|]}|jdyw)rN)r]).0els r? zQCsmForConditionalGeneration._merge_input_ids_with_input_values..as&Orrxx{&Osr}iTas_tuple)r-rP)$rprHrpaddiffr9rmaxrvrrrrziprr]rwencode audio_codesrappendrHget_audio_codes_maskrgaudio_token_idrrr"rJrlongcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrfnonzero)rMr+rrrPr- audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsr start_idxend_idx audio_batch codec_outputs codebook_idsmax_audio_framesrbatched_audio_token_idsaudio_codes_maskraudio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxss r?"_merge_input_ids_with_input_valuesz>CsmForConditionalGeneration._merge_input_ids_with_input_values1s*..y9  ##%==#4#45I6#R 01E1JKPPRM)-!*;q$A "O9O&P (,(8(8(?(? @U@UVW@X(Y '4'@'@'J'J1b'Q )00aA BB$'&O=N&O#O */++`qrZ\R]]&&rAq!5EQR 5S+TUr+'$(#3#3#H#HIZ#[ ! \$"[[77N(N: ..;;rXr,rr-c 0t |d |||||d|}|}|jdk(rn|jd]|j ||jd|jd|jd}|j |d|ddd|S) N)r+rXr,rr-r!r-rrrP)r+rrrP)r-rPr+r=)rFr[ndimrrr) rMr+rXr,rr-rr] merged_inputsrOs r?r[z9CsmForConditionalGeneration.prepare_inputs_for_generationsw< !5+)'      Y^^q%8\=M=Mo=^=f CC##ZZ7%+ZZ0F%Gzz(+ DM   "/"@MZbLcrvw r>rrrQrc |/|jdk(r |j||||} | d}| d}d}|jd|||||| d| } | d}t| tr t | dn| }|j |dd|ddf}d}d}d}d}||dddddf}|jd|||jjd| }|ddddddfd k(jd }||d d|jjdz f}tjj|d d}|jd}||d|ddz ddf}||}|j d||| d|d| }|j"}||z}t%||||| j&| j(| j*| |j,nd| |j&nd| |j(nd||j* Sd S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_values_cutoffs (`torch.Tensor` of shape `(batch_size, max_num_audio)`, *optional*): Specify the end positions of audio segments within each batch entry, relative to the concatenated audio input. If a batch entry has fewer segments than the maximum, it is padded with -1. For example, in a batch of 2 sequences where the first contains 2 audio segments of length l1, and the second contains 1 audio segment of length l2, the input_values_cutoffs would be: [[l1, 2 * l1], [l2, -1]]. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[config.audio_token_id, -100, -101]`. Requires targeted `input_values` to be provided as audio tokens will be inferred from it using the `codec_model`. - `config.audio_token_id` indicates an audio frames (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) Such labels can be prepared using `output_labels=True` when calling [`CsmProcessor`]. logits_to_keep (`int` or `torch.Tensor`, *optional*): Kept for compatibility. Does not support another value than: 1. `0`, which is equivalent to keeping all logits, used in the training regime 2. `1`, which is equivalent to keeping only the last logit, used in the generation regime Example: ```python >>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nr!r-rP)r+rrr,r-rr)r+rPrr#rrQr.r)rTr)r+r,r return_dictrP) r*r4r/r+r,r-r.r0r1r2r3r=)rrrrrrrTrnrUrgrallrrHrrrrtr*r)r,r-r.r+)rMr+rrrrr,r-rPrrQrrbackbone_outputsbackbone_hidden_statesrWbackbone_logitsr*r4r/depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss r?r[z#CsmForConditionalGeneration.forwardsd  Y^^q%8 CC<)=vM*/:M"8,FI.4.. )%+'   "2!!48B>SV8W~ot4]k ,,'=aPQ>Q'RS ! $  $Q1WoO.D..&4;;KaKaekM "!Q(+t388R8@@J&,Z&8>] @Y@Y\]@]>]9]&^ #&(mm&7&78OQW_`&7&a ##++T+:J*@APZ[\P]`aPacdAd*e '#)*#5 $6D$6$6%1+F# + %  % !"7!;!;  #55D '1",<<*88'22AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%E  qu  r>)NNNN) NNNNNNNNNr)r5r6r7r^rGrzr| classmethodrrr9rarrrr r:r[rrrrrr<r)r[rbrcs@r?rlrlsD @Aj0141*.,048&* PB<<$&PBllT)PB$llT1 PB  t# PB   PBj,0(,2626 ##"Dj  ((4/  ((4/ >.2,0.24804(,26*.!%-.Y ##d*Y llT)Y  t+ Y $llT1 Y &&- Y Y ((4/Y   4'Y $;Y ell*Y +,Y  " "Y Y r>rl)rrhrrMrl)r#)r)Scollections.abcr dataclassesrtypingrr9torch.nnrHrr activationsr cache_utilsr r generationr integrationsr r r masking_utilsrmodeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrrutils.import_utilsrutils.output_capturingr autor"configuration_csmr$r%generation_csmr& get_loggerr5r5r)ModulerBrerrrrarrr`rrrrrrrMr rhrl__all__r=r>r?rs*%! &!.)ff/9OKF&__G:5?.   H %  '3 '3  '3TY'JJ(J(><>