^jW8ddlmZddlmZddlmZddlmZddlm Z m Z m Z ddl m ZddlmZdd lmZdd lmZdd lmZmZmZdd lmZdd lmZddlmZddlm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(e#rddlZGdde jRZ*e!GddeZ+eGddeZ,Gdde jRZ-dZ.dZ/e!d Gd!d"e+Z0e!d# eGd$d%eZ1e!d& Gd'd(e+eZ2gd)Z3y)*)Callable) dataclass)pi)Optional)Tensorbroadcast_tensorsnn)initialization)ACT2FN)Cache)GenerationMixin)BaseModelOutputWithPastBaseModelOutputWithPooling ModelOutput)ROPE_INIT_FUNCTIONS)PreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torch_availabletorch_compilable_check) AutoModel)MusicFlamingoConfigNc eZdZUdZej ed<ddeffd Ze ddedzde dde dzd e d e ffd Zejd ede d e eeffd ZdZxZS)MusicFlamingoRotaryEmbeddingaRotary time embedding module used by MusicFlamingo checkpoints. This is a checkpoint-faithful integration, not a direct implementation of the RoTE formulation described in (Goel et al., 2024): https://arxiv.org/abs/2410.12109. It applies axial rotary embeddings over the window index within each audio sample and the encoder time index within each window, then modulates both axes with absolute timestamps in seconds. inv_freqNconfigc t||j|_|j|_||_|j j d|_|j}|jdk7rt|j}||j |\}|_ |jd|d|jd|jd|j|j}|jd|dy)N rope_typedefaultr F) persistentoriginal_inv_freqposition_angles)super__init__max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr!rope_parametersr#compute_default_rope_parametersrattention_scalingregister_bufferclone_compute_position_anglesr )selfr!device rope_init_fnr r' __class__s /var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/musicflamingo/modeling_musicflamingo.pyr)z%MusicFlamingoRotaryEmbedding.__init__9s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L($( ZeD 0(..2BuU77 F .ERr4z torch.deviceseq_lenreturnz torch.Tensorcn|jd}|jjdd}t|ddxs|j|jz}t ||z}d}d|t jd|dt jj|t j |z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetapartial_rotary_factorg?head_dimNrrdtyper4r@) r-getgetattr hidden_sizenum_attention_headsinttorcharangeint64tofloat) r!r4r9baser=r>dimattention_factorr s r7r.zU]]_(&(3(5;P((&8r8rcteZdZUeed<dZdZdZdZdgZ dZ dZ dZ e jfdZxZS)MusicFlamingoPreTrainedModelr!model)audiotextTNpast_key_valuesct||t|tr<|j |j }t j|j|yyre) r( _init_weights isinstancerr2r initcopy_r')r3module buffer_valuer6s r7rwz*MusicFlamingoPreTrainedModel._init_weightssH f% f: ;!::6??KL JJv--| <