^jddZddlZddlmZddlmZddlmZmZddl Z ddl m Z ddl m Z ddl mZdd lmZdd lmZmZdd lmZmZdd lmZdd lmZddlmZmZmZm Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2m3Z3e,jhe5Z6e+deGddeZ7e+deGddeZ8e+deGd d!eZ9e+eGd"d#eZ:Gd$d%e jvZ<Gd&d'e jvZ=Gd(d)e jvZ>Gd*d+e jvZ?d,Z@dNd-ZAGd.d/e jvZBd0e jd1eDd2e jfd3ZE dOd4e jvd5e jd6e jd7e jd8e jdzd9eFd:eFd;e(e*fd<ZGGd=d>e jvZHGd?d@eZIe+GdAdBe&ZJGdCdDeJeZKe+GdEdFeJZLe+dGGdHdIeJeZMe+dJGdKdLeJeZNgdMZOy)PzPyTorch Moshi model.N)Callable) dataclass)AnyOptional)CrossEntropyLoss)initialization)ACT2FN)Cache DynamicCache)GenerationConfigGenerationMixin)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast ModelOutputSeq2SeqLMOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)maybe_autocast) AutoModel) MoshiConfigMoshiDepthConfigzK Outputs of [`MoshiForConditionalConditionalGeneration.generate`]. ) custom_introceZdZUdZdZej dzed<dZejdzed<dZ ejdzed<dZ e ejdzed<dZe ejdzed<dZejdzed<dZe e ejdzed <dZe e ejdzed <dZedzed <dZejdzed <y) (MoshiConditionalGenerationGenerateOutputa audio_sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, 1, sequence_length)`, *optional*): The generated audio waveforms. sequences (`torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`): The generated text sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. sequences_scores (`torch.FloatTensor` of shape `(batch_size*num_return_sequences)`, *optional*, returned when `output_scores=True`): Final beam scores of the generated `sequences`. scores (`tuple(torch.FloatTensor)` *optional*, returned when `output_scores=True`): Beam transition scores for each vocabulary token at each generation step. Beam transition scores consisting of log probabilities of tokens conditioned on log softmax of previously generated tokens in this beam. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size*num_beams, config.vocab_size)`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. beam_indices (`torch.LongTensor`, *optional*, returned when `output_scores=True`): Beam indices of generated token id at each generation step. `torch.LongTensor` of shape `(batch_size*num_return_sequences, sequence_length)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size*num_beams*num_return_sequences, generated_length, hidden_size)`. past_key_values (`Cache`, *optional*, returned when `use_cache=True`): Contains the model cache, used to speed up decoding. Different models have a different cache format, check the model's documentation. Usually, a [`~cache_utils.Cache`] instance. audio_codes (`torch.LongTensor` of shape `(batch_size*num_return_sequences, num_codeooks, sequence_length)`, *optional*): The generated audio codes. Returned if `return_audio_codes=True`. Intermediate audio "tokens" which transforms to `audio_sequences` once passed through the audio decoder. Naudio_sequences sequencessequences_scoresscoreslogits beam_indices attentions hidden_statespast_key_values audio_codes)__name__ __module__ __qualname____doc__r&torchTensor__annotations__r' LongTensorr( FloatTensorr)tupler*r+r,r-r.r r/s/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/moshi/modeling_moshi.pyr%r%,sB,0OU\\D(/)-Iu$&-15e''$.5.2FE%## $t +2.2FE%## $t +2,0L%""T)09=JeE--./$6=<@M5u0012T9@$(OUT\(+/K!!D(/r;r%z% `MoshiForCausalLM` outputs. ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e dzed<dZ e ej dfdzed<dZe ej dfdzed <y) MoshiCausalLMOutputWithPasta% loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. Nlossr*last_hidden_stater..r-r,)r0r1r2r3r?r4r8r6r*r@r.r r-r9r,r:r;r<r>r>`s &*D%  d ")'+FE   $+26u((4/6$(OUT\(:>M5**C/047>7;Je'',-4;r;r>z2 `MoshiForConditionalGeneration` outputs. ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e dzed<dZ e ej dfdzed<dZe ej dfdzed <dZej dzed <dZej dzed <dZe dzed <dZe ej dfdzed <dZe ej dfdzed<y)(MoshiConditionalGenerationOutputWithPasta' loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `text_labels` is provided): Text language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the text language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `audio_labels` is provided): Audio language modeling loss (for next-token prediction). audio_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the audio language modeling heads. depth_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Past key-values of the depth decoder. depth_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Hidden states of the depth decoder depth_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Depth decoder's Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. Nr?r*r@r..r-r, depth_loss audio_logitsdepth_past_key_valuesdepth_hidden_statesdepth_attentions)r0r1r2r3r?r4r8r6r*r@r.r r-r9r,rCrDrErFrGr:r;r<rBrB{s.&*D%  d ")'+FE   $+26u((4/6$(OUT\(:>M5**C/047>7;Je'',-4;+/J!!D(/-1L%##d*1*.54<.@Du00#56=D=AeE--s23d:Ar;rBceZdZUdZdZej dzed<dZejdzed<dZ ejdzed<dZ ej dzed<y)MoshiUnconditionalInputa) input_ids (`torch.Tensor `of shape `(batch_size, sequence_length), *optional*): The sequence used as a text prompt for the generation. user_audio_codes (`torch.Tensor `of shape `(batch_size, num_codebooks, sequence_length), *optional*): The audio codes used as audio user prompt for the generation. Has priority over `user_input_values` and represents the audio "tokens" of `user_input_values` once passed through the audio encoder. moshi_audio_codes (`torch.Tensor `of shape `(batch_size, num_codebooks, sequence_length), *optional*): The audio codes used as audio Moshi prompt for the generation. Has priority over `moshi_input_values` and represents the audio "tokens" of `moshi_input_values` once passed through the audio encoder. attention_mask (`torch.LongTensor`) of shape `(batch_size, sequence_length)`, *optional*): Attention mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: 1 for tokens that are **not masked**, 0 for tokens that are **masked**. N input_idsuser_audio_codesmoshi_audio_codesattention_mask) r0r1r2r3rJr4r7r6rKr5rLrMr:r;r<rIrIs[ *.Iu$&-,0ellT)0-1u||d*1.2NE$$t+2r;rIc<eZdZddedeffd ZdZdZdZxZ S) MoshiRMSNormdimepsct|||_tjt j ||_yN)super__init__rQnn Parameterr4onesweight)selfrPrQ __class__s r<rUzMoshiRMSNorm.__init__s- ll5::c?3 r;c|tj|jdjdd|jzzS)NrT)keepdim)r4rsqrtpowmeanrQ)rZxs r<_normzMoshiRMSNorm._norms45;;quuQx}}R}>IJJJr;c|j|j}||jjz}|j|SrS)rcfloatrYtype_as)rZrboutputs r<forwardzMoshiRMSNorm.forwards<AGGI&$++++--~~a  r;c^t|jjd|jS)Nz, eps=)r9rYshaperQrZs r< extra_reprzMoshiRMSNorm.extra_reprs' ))*+6$((<) r0r1r2intrerUrcrhrl __classcell__r[s@r<rOrOs&4C4e4 K! =r;rOc&eZdZfdZddZxZS)MoshiFlexibleLinearct|tjt j ||||_yrS)rTrUrVrWr4randnrY)rZ input_size output_size num_layersr[s r<rUzMoshiFlexibleLinear.__init__s+ ll5;;z; #ST r;c|!tj|jd|n |j}|jdddddddddf}tj|dddddddf|}|j dS)a `MoshiFlexibleLinear` creates one linear layer per codebook. There's multiple ways to use it. In the default case, `sequence_length=num_layers`, so each element of the sequence will be matmul to the weights corresponding to its index on the sequence. For more advanced cases, one can specify which codebook's layer(s) to use with `layer_idx`. If `layer_idx` indicates a single integer, all of the element of the sequence will be matmul to this single codebook's layer. But if `layer_idx` is a tensor of shape `(seq_length,)`, it will matmul each i-th element of the input sequence to the corresponding layer `weight[i]`. Args: x (`torch.FloatTensor): input to the layer of shape `(batch, num_layers, embed_dim)` or of shape `(batch, seq_length, embed_dim)` layer_idx (`torch.Tensor`, *optional*): Can be used to specify which codebook's layers(s) to use. If it's a tensor of shape `(seq_length,)`, will matmul each element of the sequence to the corresponding weights. But if `layer_idx` is a tensor of shape `(seq_length,)`, it will matmul each i-th element of the input sequence to the corresponding layer `weight[i]`. Nrr r)r4 index_selectrY transposematmulsqueeze)rZrb layer_idxselected_weightss r<rhzMoshiFlexibleLinear.forwards(MVLa5--dkk1iHgkgrgr,55a;D!QMJ LL1aq=)+; <yy|r;rSr0r1r2rUrhrnros@r<rqrqsU r;rqc(eZdZdfd ZddZxZS) MoshiLinearct|||_|stj||d|_yt ||||_y)NFbias)rv)rTrUuse_flexible_linearrVLinearlinearrq)rZ input_dim output_dim num_codebooksrr[s r<rUzMoshiLinear.__init__s> #6 "))IzFDK-iP]^DKr;c`|jr|j||S|j|SrS)rr)rZrbr|s r<rhzMoshiLinear.forwards*  # #;;q), ,;;q> !r;FrSr~ros@r<rrs_"r;rc eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)MoshiRotaryEmbeddinginv_freqNconfigct||j|_|j|_||_|j j d|_|j}|jdk7rt|j}||j |\}|_ |jd|d|jd|jdy)N rope_typedefaultrF) persistentoriginal_inv_freq) rTrUmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrrope_parametersrcompute_default_rope_parametersrattention_scalingregister_bufferclone)rZrdevice rope_init_fnrr[s r<rUzMoshiRotaryEmbedding.__init__ s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L($( ZeD 0(..2BuUr;rz torch.deviceseq_lenreturnz torch.Tensorc|jd}t|ddxs|j|jz}d}d|t j d|dtj j|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rrdtyperr) rgetattr hidden_sizenum_attention_headsr4arangeint64tore)rrrbaserPattention_factorrs r<rz4MoshiRotaryEmbedding.compute_default_rope_parameterss&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k )))r;cN|jddddfjj|jdddj |j }|dddddfj}t |j jtr/|j jdk7r|j jnd}t|d5|j|jzjdd}tj||fd }|j|jz}|j|jz} dddj |j  j |j fS#1swYAxYw) Nrr]r mpscpuF) device_typeenabledrrPr)rreexpandrjrr isinstancetypestrrryr4catcosrsinr) rZrb position_idsinv_freq_expandedposition_ids_expandedrfreqsembrrs r<rhzMoshiRotaryEmbedding.forward:sR!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk  U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BFF$rS)NNN)r0r1r2r4r5r6r!rU staticmethodrrmr9rerno_gradrrhrnros@r<rr sllV{V %)+/"*d"*(*t* ~u$ % **:U]]_ <