^j&ddlZddlmZddlmZddlZddlmZddlmZ ddl m Z ddl m Z mZmZdd lmZmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3e#jhe5Z6e!deGddeZ7GddejpZ9GddejpZ:GddejpZ;d Z<ed!dQd"Z=d#ej|d$e?d%ej|fd&Z@ dRd'ejpd(ej|d)ej|d*ej|d+ej|dzd,eAd-eAd.ee fd/ZBee=Gd0d1ejpZCGd2d3ejpZDGd4d5eZEe!Gd6d7eZFe!d8Gd9d:eFZGeGd;deHZIe!d?Gd@dAeFeZJGdBdCejpZKGdDdEejpZLeGdFdGeZMe!dHGdIdJeFe2ZNGdKdLeLZOe!dMGdNdOe3eNZPgdPZQy)SN)Callable) dataclass)nn)initialization)ACT2FN) CompileConfigGenerationMixinGenerationMode)use_kernel_func_from_hubuse_kernelized_func)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingCausalLMOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)capture_outputs) AutoModel)ParakeetCTCConfigParakeetEncoderConfigParakeetRNNTConfigParakeetTDTConfig)ParakeetRNNTDecoderCacheParakeetRNNTGenerationMixinParakeetTDTGenerationMixinz Extends [~modeling_outputs.BaseModelOutputWithPooling] to include the output attention mask since sequence length is not preserved in the model's forward. ) custom_introc:eZdZUdZdZej dzed<y)ParakeetEncoderModelOutputa attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices after sequence compression. Returned because the sequence length may differ from the input sequence length. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Nattention_mask)__name__ __module__ __qualname____doc__r*torchTensor__annotations__y/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/parakeet/modeling_parakeet.pyr)r)/s+/NELL4'.r3r)ceZdZUejed<d deffd Ze d dedzdejfdZ ejdejfdZ xZ S) $ParakeetEncoderRelPositionalEncodinginv_freqNconfigct||j|_||_|j ||}|j d|dy)Ndevicer7F) persistent)super__init__max_position_embeddingsr8.compute_default_relative_positional_parametersregister_buffer)selfr8r;r7 __class__s r4r>z-ParakeetEncoderRelPositionalEncoding.__init__FsN '-'E'E$ FFvV\F] ZeDr3returncd}d|tjd|jdtjj |tj |jz zz }|S)Ng@?rrdtype)r;rH)r/arange hidden_sizeint64tofloat)r8r;baser7s r4r@zSParakeetEncoderRelPositionalEncoding.compute_default_relative_positional_parametersMsd   Q 2 2AU[[ILLTZbgbmbmLn$$%  r3 hidden_statesc|jd}tj|dz | d|j}|jddddfj j |jdddj|j}|ddddfj }t|jjtr/|jjdk7r|jjnd}t|d5|j |j zjdd }|j}|j} tj|| gd } | j g| jdd d} ddd j|j" S#1swY%xYw) Nrr:rmpscpuF) device_typeenabledrdimrG)shaper/rIr;r7rMexpandrL isinstancetypestrr transposesincosstackreshaperH) rBrO seq_length position_idsinv_freq_expandedposition_ids_expandedrTfreqsr_r` pos_embeds r4forwardz,ParakeetEncoderRelPositionalEncoding.forward\s"((+ ||JNZKML`L`a MM$4- ( . . 0 7 7 8K8KA8NPRTU V Y YZgZnZn o !-T4] ; A A C-..33S9m>R>R>W>W[`>`  % %   U C E&,,.1F1L1L1NNYYZ[]^_E))+C))+C S#JB7I) ))D9??3B+?DDI  E||-"5"5|66 E Es B F::GNNN) r+r,r-r/r0r1r!r> staticmethodr@no_gradri __classcell__rCs@r4r6r6CsxllE4E/3 %,    U]]_7U\\77r3r6c*eZdZdeffd ZdZxZS)ParakeetEncoderFeedForwardr8c`t|tj|j|j |j |_t|j|_ tj|j |j|j |_ |j|_ y)Nbias) r=r>rLinearrJintermediate_sizeattention_biaslinear1r hidden_act activationlinear2activation_dropoutrBr8rCs r4r>z#ParakeetEncoderFeedForward.__init__vs| yy!3!3V5M5MTZTiTij  !2!23yy!9!96;M;MTZTiTij "(";";r3c|j|j|}tjj ||j |j }|j|}|S)Nptraining)rzrxr functionaldropoutr|rr{)rBrOs r4riz"ParakeetEncoderFeedForward.forward}sU ](CD  --mt?V?Vaeanan-o  ]3 r3r+r,r-r!r>rirnros@r4rqrqus<4<r3rqc.eZdZddeffd ZddZxZS) ParakeetEncoderConvolutionModuler8c 6t||j}|&|j}tt |dd|_n#|d}t|jdd|_|dz dz|_tj|d|zddd|j |_ tj|||d|j||j |_ tj||_tj||ddd|j |_y) z Args: config (ParakeetEncoderConfig): Configuration for the model. module_config (dict): Configuration for the module (e.g., encoder or decoder). Nrysilu kernel_sizerzrrr)rstridepaddingrt)rrgroupsrt)r=r>rJconv_kernel_sizergetattrrzgetrrConv1dconvolution_biaspointwise_conv1depthwise_conv BatchNorm1dnormpointwise_conv2)rBr8 module_configchannelsrrCs r4r>z)ParakeetEncoderConvolutionModule.__init__s %%   11K$WV\6%JKDO' 6K$]%6%6|V%LMDO#aA- !yy a(l!QVMdMd !ii   LL(( NN8, !yy hAaI`I` r3c|jdd}|j|}tjj |d}|c|j t jk(rt j|d}nt j|dk(d}|j|d}|j|}|j|}|j|}|j|}|jddS)aY Compute convolution module. Args: hidden_states (`torch.Tensor` of shape `(batch, time, channels)`): Input tensor. attention_mask (`torch.Tensor` of shape `(batch, 1, time, time)`): Attention mask. Returns: `torch.Tensor`: Output tensor of shape `(batch, time, channels)`. rrrV)r^rrrglurHr/boolall masked_fillrrrzr)rBrOr*all_masked_rowss r4riz(ParakeetEncoderConvolutionModule.forwards&//15 ,,];  ))-Q)?   %##uzz1"'))^O"C"'))n.C,D!"L)55osKM++M:  -0  6 ,,]; &&q!,,r3rjrros@r4rrs  4  D"-r3rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..NrQrrV)rYr/cat)xx1x2s r4 rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r3rotary_pos_embc|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qkr`r_ unsqueeze_dimq_embedk_embeds r4apply_rotary_pos_embrsY& -- &C -- &C3w;q>C/0G3w;q>C/0G G r3rOn_reprDc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rYrZrb)rOrbatchnum_key_value_headsslenhead_dims r4 repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr3modulequerykeyvaluer*scalingrkwargsc t||j}t||j} tj||j dd|z} || |z} t j j| dtjj|j} t j j| ||j} tj| | } | j ddj} | | fS)NrrrQrWrHrr)rnum_key_value_groupsr/matmulr^rrsoftmaxfloat32rLrHrr contiguous) rrrrr*rrr key_states value_states attn_weights attn_outputs r4eager_attention_forwardrs3 ; ;r8rrrJnum_attention_headsrrrrattention_dropout is_causalrrurwq_projk_projv_projo_projrelative_k_proj Parameterr/zerosbias_ubias_vrBr8rrCs r4r>z!ParakeetEncoderAttention.__init__s  " F4F4F&JdJd4de $*$>$>&B\B\$\!}}d* !'!9!9ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf   "yy););V=W=WZ^ZgZg=gnstll5;;v/I/I4==#YZ ll5;;v/I/I4==#YZ r3NrOposition_embeddingsr*rrDc |jdd}|\}}||d|jf}|j|j|j dd} |j |j|j dd} |j |j|j dd} tj|jjt} | |jjd|jjd|jz} | |jjd|jjd|jz}|j|}|j|d|jj|j}||j!ddddz}|j#|}|dd|f}||j$z}|)|j'|j)t+d}| |f| | | ||j,sdn |j.|j$d |\}}|j0g|dj3}|j5|}||fS) NrQrrrr.z-infr)rrrr*rr)rYrrviewr^rrr get_interfacer8_attn_implementationrrrrrpermute _rel_shiftr masked_fill_ logical_notrMrrrbrr)rBrOrr*r input_shape batch_sizerc hidden_shape query_statesrrattention_interfacequery_states_with_bias_uquery_states_with_bias_vrelative_key_states matrix_bdrrs r4riz ParakeetEncoderAttention.forward3sj$))#2. !, J"JDMMB {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST (?(M(M KK , ,.E) $0$++2B2B t{{..4==3 $  $0$++2B2B t{{..4==3 $  #223FG166z2t{{GfGfhlhuhuv-/B/J/J1aQRTU/VV OOI. c;J;./  ,  %"..~/I/I/KUSY][I%8  % *$#}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r3c|j\}}}}tjj|d}|j ||d|}|ddddddfj ||||}|S)ztRelative position shift for Shaw et al. style attention. See appendix B of https://huggingface.co/papers/1901.02860.)rr)padrQNr)rYrrrr)rBattention_scoresr num_heads query_lengthposition_lengths r4rz#ParakeetEncoderAttention._rel_shiftlst?O?U?U< I|_==,,-=6,J+00YLY+Aq!"H5:::yR^`opr3rj)r+r,r-r.r!intr>r/r0rrtuplerirrnros@r4rrs~[4[[B/3 7)||7)#\\D07) t+ 7) +, 7) u||U\\) * 7)r r3rceZdZdeffd Zdej dejfdZ d dej dej fdZ xZ S) ParakeetEncoderSubsamplingConv2Dr8c t||j|_|j|_|j |_|jdz dz|_ttj|j|_ tj|_|j j#tj$d|j|j|j |j|j j#tj&t)|jdz D]}|j j#tj$|j|j|j|j |j|j|j j#tj$|j|jd|j j#tj&|j*|j |jzz}tj,|j |z|j.d|_y)Nrr)rrr)rrrrrTrs)r=r>subsampling_conv_kernel_sizersubsampling_conv_stridersubsampling_conv_channelsrrrmathlog2subsampling_factor num_layersr ModuleListlayersappendConv2dReLUrange num_mel_binsrurJlinear)rBr8i out_lengthrCs r4r>z)ParakeetEncoderSubsamplingConv2D.__init__vs !>>44 88 ((1,2 dii(A(ABCmmo   IIaD4D4DT[[bfbnbn o  2779%t*+ *A KK   MMMM $ 0 0;; LL==   KK  ryy STU V KK  rwwy ) *"((T[[$//-IJ ii @ @: MvOaOahlm r3 input_lengths conv_layerct|drR|jdk7rC|j}|jd}|jd}||dz|dz|z |zdz}|S|S)Nr)rrrr)hasattrrrr)rBrr rrroutput_lengthss r4_get_output_lengthz3ParakeetEncoderSubsamplingConv2D._get_output_lengthsx :x (Z->->&-H ((G$003K&&q)F+gaj871:E SX^^abbN! !r3input_featuresr*c|jd}||jdnd}|jD]}||}t|tj s&|)|j ||}|jd}tj||j|dddfk}||ddddddfz}|jddj|jd|jdd}|j|}|S)NrrQrr:r)rsumrr[rrr rYr/rIr;r^rbr)rBrr*rOcurrent_lengthslayercurrent_seq_length channel_masks r4riz(ParakeetEncoderSubsamplingConv2D.forwards&003 4B4N.,,R0TX[[ @E!-0M%+0J"&"9"9/5"Q%2%8%8%;"LL!3N!??  @&//15==m>Q>QRS>TVcViVijkVlnpq  M2 r3rj) r+r,r-r!r>r/r0rrr rirnros@r4rrusI!n4!nF  ")) ellELLr3rc eZdZd dededzffd Z d dejdejdzdejdzdee d ejf d Z xZ S) ParakeetEncoderBlockNr8rct|d|_t||_t |||_t||_t||_ tj|j|_ tj|j|_tj|j|_tj|j|_tj|j|_y)NF)r=r>gradient_checkpointingrq feed_forward1r self_attnrconv feed_forward2r LayerNormrJnorm_feed_forward1 norm_self_att norm_convnorm_feed_forward2norm_outrs r4r>zParakeetEncoderBlock.__init__s &+#7?1&)D4V< 7?"$,,v/A/A"B\\&*<*<=f&8&89"$,,v/A/A"B V%7%78 r3rOr*rrrDc x|}|j|j|}|d|zz}|j|}|jd|||d|\}}||z}|j |j ||} || z}|j |j|} |d| zz}|j|}|S)Ng?)rOr*r)r*r2) rrrrrr rr!r") rBrOr*rrresidualnormalized_hidden_statesr_ conv_output ff2_outputs r4rizParakeetEncoderBlock.forwards!**4+B+B=+QR  3#66 #'#5#5m#D ' 2) 3    Q & 3 ii} =ni] % 3 ''(?(? (NO %j(88  m4 r3rjrk) r+r,r-r!rr>r/r0rrrirnros@r4rrsx 94 9t 9$/337 || t+#\\D0  +,   r3rceZdZUeed<dZdZdZdZdgZ dZ dZ dZ dZ dZdZeedZej(fd Zd ej,fd Zdd ej,ded zfdZxZS)ParakeetPreTrainedModelr8modelraudioTrF)rO attentionsct||t|jdd}t |t rEt j|jd|t j|jd|yt |tr<|j|j}t j|j|yy)Ninitializer_rangeg{Gz?r)meanstd)r= _init_weightsrr8r[rinitnormal_rrr6r@copy_r7)rBrr1 buffer_valuerCs r4r2z%ParakeetPreTrainedModel._init_weightss f%dkk#6= f6 7 LLSc : LLSc :  D E!PPQWQ^Q^_L JJv 5Fr3rct|jd|j}|j}|j}t t j |j}|dz dzdz}||z }|}t|D]Q} tj|jtj|z|dz}tj|}S|jtjS)Nencoder_configrrrGrF)rr8rrrrrrrr/divrLrMfloor) rBrr8rrr all_paddingsadd_padlengthsr&s r4_get_subsampling_output_lengthz6ParakeetPreTrainedModel._get_subsampling_output_length s .> L$AA 77>#D#DEF #aA-1 ,z" +Aii  = GPSVVGkk'*G +zz z**r3Nr* target_lengthc|j|jd}||n|j}tj||j |dddfk}|S)z Convert the input attention mask to its subsampled form. `target_length` sets the desired output length, useful when the attention mask length differs from `sum(-1).max()` (i.e., when the longest sequence in the batch is padded) rQNr:)r>rmaxr/rIr;)rBr*r?r  max_lengths r4_get_output_attention_maskz2ParakeetPreTrainedModel._get_output_attention_masksc <<^=O=OPR=ST&3&?]^EWEWEY j9N9NOR`abdhahRiir3rj)r+r,r-r r1base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_no_split_modules_supports_flat_attention_mask_supports_sdpa_supports_flex_attn_supports_flash_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsr/rmr2r0r>rrCrnros@r4r*r*s &O&*#/0$(!N!!"&-. U]]_ 6 6+ELL+"  VY\`V` r3r*z{ The Parakeet Encoder model, based on the [Fast Conformer architecture](https://huggingface.co/papers/2305.05084). ceZdZUeed<dZdeffd Zeee e d de jde jdzde deed ef d ZxZS) ParakeetEncoderr8encoderc t||||_d|_|j|_|j |_|j |_|jrtj|jnd|_ t||_ t||_t!j"t%|j&Dcgc]}t)||c}|_|j-ycc}w)NFrF)r=r>r8rrdropout_positions layerdrop scale_inputrsqrtrJ input_scaler subsamplingr6encode_positionsrrrnum_hidden_layersrr post_initrs r4r>zParakeetEncoder.__init__1s   &+#~~ !'!9!9))>> from transformers import AutoProcessor, ParakeetEncoder >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> encoder = ParakeetEncoder.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> encoder_outputs = encoder(**inputs) >>> print(encoder_outputs.last_hidden_state.shape) ``` rNrr?rQrFT)r*r)last_hidden_stater*)rYrXrZrrrrrTrCrYrrZr^rr/randrUr)r) rBrr*r]rrOr output_mask encoder_layerto_dropdropout_probabilitys r4rizParakeetEncoder.forwardDsF((H %(8(88 "33MB --mt||VZVcVc-d  mm33 4#9#9DMM4   %99.XeXkXklmXn9oK(2215<0JOd;??,  jn  r3)NT)r+r,r-r!r1rDr>rrrrr/r0rrrrrirnros@r4rQrQ(s "!!4&/3&* B  B  t+B  $ B +, B  B  B r3rQceZdZUdZej ed<dZeejdzed<dZ eeejdzed<dZ eeejdzed<y)ParakeetCTCGenerateOutputaz Outputs of Parakeet CTC model generation. Args: sequences (`torch.LongTensor` of shape `(batch_size, sequence_length)`): The generated sequences. The second dimension (sequence_length) is either equal to `max_length` or shorter if all batches finished early due to the `eos_token_id`. logits (`tuple(torch.FloatTensor)` *optional*, returned when `output_logits=True`): Unprocessed prediction scores of the language modeling head (scores for each vocabulary token before SoftMax) at each generation step. Tuple of `torch.FloatTensor` with up to `max_new_tokens` elements (one element for each generated token), with each tensor of shape `(batch_size, config.vocab_size)`. attentions (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_attentions=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, num_heads, generated_length, sequence_length)`. hidden_states (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `output_hidden_states=True`): Tuple (one element for each generated token) of tuples (one element for each layer of the decoder) of `torch.FloatTensor` of shape `(batch_size, generated_length, hidden_size)`. sequencesNlogitsr-rO) r+r,r-r.r/ LongTensorr1rir FloatTensorr-rOr2r3r4rgrgsm&.2FE%## $t +29=JeE--./$6=<@M5u0012T9@r3rgc"eZdZdZfdZxZS)ParakeetGenerateOutputz` Deprecated alias for ParakeetCTCGenerateOutput. Use ParakeetCTCGenerateOutput instead. cNt||i|tjdy)Nz`ParakeetGenerateOutput` is deprecated and removed starting from version 5.11.0; please use `ParakeetCTCGenerateOutput` instead.)r=r>logger warning_once)rBargsrrCs r4r>zParakeetGenerateOutput.__init__s) $)&) O r3)r+r,r-r.r>rnros@r4rmrms  r3rmzS Parakeet Encoder with a Connectionist Temporal Classification (CTC) head. cXeZdZUeed<deffd Zee d dejdejdzdejdzde e de f d Z ej ddejdejdzd ed edzde e deej$zf d ZxZS)ParakeetForCTCr8ct||tj|j|_t j|jj|jd|_ |jy)Nrr) r=r>r from_configr8rRrrrJ vocab_sizectc_headr\r}s r4r>zParakeetForCTC.__init__sY   ,,V-B-BC  &"7"7"C"CVEVEVdef  r3Nrr*labelsrrDc ||jdd|jd||d|}|j}|j|j ddj dd}d}|+|j j d} ||jjk7} | j d} |j| } tjj|dtjj d d} tjj j#d 5tjj%| | | | |jj|jj&|jj( }dddt+|||j,|j. S#1swY,xYw)a Example: ```python >>> from transformers import AutoProcessor, ParakeetForCTC >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> outputs = model(**inputs) >>> print(outputs.loss) ```Nr]Trr*rrrQrrF)rU)blank reduction zero_infinity)lossrirOr-r2) setdefaultrRr`rwr^r*rr8 pad_token_id masked_selectrr log_softmaxr/rbackendscudnnflagsctc_lossctc_loss_reductionctc_zero_infinityrrOr-)rBrr*rxrencoder_outputsrOrir~encoder_lengths labels_masktarget_lengthsflattened_targets log_probss r4rizParakeetForCTC.forwards:     5t <&$,, ))   (99 }66q!<=GG1M  -<<@@DO!DKK$<$<>> from transformers import AutoProcessor, ParakeetForCTC >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-ctc-1.1b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForCTC.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"], text=ds[0]["text"]) >>> predicted_ids = model.generate(**inputs) >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True) >>> print(transcription) ``` T return_dictrzrQrVrr_)rhrir-rOr2) get_compiled_call__call__riargmaxrCrYr8rrgr-rO) rBrr*rrr model_forwardoutputsrhs r4generatezParakeetForCTC.generate sBCQB\..~>bfbobo $}"/# ))# # NN))b)1   %!<<^[d[j[jkl[m<nN)-)A)AI~o & ",#~~"--%33  r3rk)NFN)r+r,r-r r1r>rrr/r0rrrrirmrr rgrjrrnros@r4rsrss 0/3&* B  B  t+B  t# B +, B  B B HU]]_/3(-/3 9 9 t+9"& 9 &, 9 +, 9 #U%5%5 599r3rscneZdZdZdeffd Z d dejdedzdejfdZ xZ S) ParakeetRNNTDecoderz'LSTM-based prediction network For RNN-Tr8cxt||j|_tj|j |j |_tj|j |j |jd|_ tj|j |j |_ y)NT) input_sizerJr batch_first) r=r>blank_token_idr Embeddingrvdecoder_hidden_size embeddingLSTMnum_decoder_layerslstmrudecoder_projectorr}s r4r>zParakeetRNNTDecoder.__init__Ks $33f&7&79S9STGG112200   "$6+E+EvGaGa!br3N input_idscacherDc|>|dddf|jk(}|jr|jr |jS|j |}|8|j}|s|j ||j |jf}nd}|j||\}\}} |j|} |(rnd} |j| || | |jS| S)NrQ)mask) ris_initializedrrrlazy_initialization hidden_state cell_staterrupdate) rBrr blank_mask embeddingswas_initializedhidden_cell_states lstm_outputrrdecoder_outputrs r4rizParakeetRNNTDecoder.forwardWs  "1b5)T-@-@@J## (8{{"^^I.   #22O"))*5"'"4"4e6F6F!G !% 26))JHZ2[/ /lJ// <  "1J;tD LLzL M;; r3rj) r+r,r-r.r"r>r/rjr$r0rirnros@r4rrHsJ1 c1 c26##($.  r3rceZdZdZdeffd ZdejdejdeejejffdZ xZ S)ParakeetRNNTJointNetworkzPJoint network that combines encoder and decoder outputs to predict token logits.r8ct|t|j|_t j |j|j|_ |j|_yrj) r=r>rryrzrrurrvheadr}s r4r>z!ParakeetRNNTJointNetwork.__init__{sK  !2!23IIf88&:K:KL  ++r3decoder_hidden_statesencoder_hidden_statesrDcL|j||z}|j|Srj)rzr)rBrr joint_outputs r4riz ParakeetRNNTJointNetwork.forwards( 'r/r0rrirnros@r4rrxsNZ,1, '$||' %||' u||U\\) * 'r3rcveZdZUdZdZej dzed<dZej dzed<dZ e dzed<y)ParakeetRNNTOutputa Output of the Parakeet RNN-T forward pass. Args: loss (`torch.FloatTensor`, *optional*): RNN-T loss, returned when `labels` are provided. logits (`torch.FloatTensor`): Joint token logits. Shape is `(batch, T, U+1, vocab)` for training or `(batch, 1, 1, vocab)` for single-step inference. decoder_cache (`ParakeetRNNTDecoderCache`, *optional*): Decoder LSTM cache containing hidden state, cell state, and last output. Nr~ri decoder_cache) r+r,r-r.r~r/rkr1rirr$r2r3r4rrsC &*D%  d ")'+FE   $+59M+d29r3rz? Parakeet Encoder with an RNN-T (RNN Transducer) head. ceZdZUeed<dgZejgZdeffd Z e dde jde jdzde edefd Zee dde jdzde jdzd e j$dzd edzd edzd eee j,zdzde jdzde edefdZxZS)ParakeetForRNNTr8rc`t||tj|j|_t j|jj|j|_ t||_ t||_|j|_|j!yrj)r=r>rrur8rRrrurJrencoder_projectorrdecoderrjointmax_symbols_per_stepr\r}s r4r>zParakeetForRNNT.__init__s}   ,,V-B-BC !#6+@+@+L+LfNhNh!i*62 -f5 $*$?$?! r3Nrr*rrDc p|jd||d|}|j|j|_|S)Nrzr2)rRrr` pooler_output)rBrr*rrs r4get_audio_featuresz"ParakeetForRNNT.get_audio_featuressJ'$,, ))   )-(>(>?`?`(a%r3decoder_input_idsruse_decoder_cacherrxc ||jd ||d|}|r|t|j}|j||} |j |j dddddddf| dddddddfj d} d} ||jjd} |jd | dddt| jf|| ||jjk7jd|jjd|} t| | |j|j |j|j |S) a? decoder_input_ids (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Decoder input token ids for single-step inference. decoder_cache (`ParakeetRNNTDecoderCache`, *optional*): Decoder LSTM cache. When provided and initialized, the cached `decoder_output` is reused (e.g. during blank-skipping) instead of running the decoder. When `input_ids` is provided, the decoder runs and the cache is updated in-place. use_decoder_cache (`bool`, *optional*): Whether to use a decoder cache. When `True` and `decoder_cache` is `None`, a new cache is created automatically during the forward pass. encoder_outputs (`tuple(torch.FloatTensor)`, *optional*): Pre-computed encoder outputs (last_hidden_state, pooler_output, hidden_states, attentions, attention_mask). Can be a tuple or `ParakeetEncoderModelOutput`. Example: ```python >>> from transformers import AutoProcessor, ParakeetForRNNT >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-rnnt-0.6b" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForRNNT.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> outputs = model(**inputs) ``` NrzrrrrrQ)rirx logit_lengths label_lengthsrr~rir`rrOr-rr2)rr$r8rrrsqueezer*r loss_functionrrArrr`rOr-) rBrr*rrrrrxrrrir~rs r4rizParakeetForRNNT.forwardsvX  "5d55--O !64T[[AM $ ->m T"1"?"?1dA "N"74A "F  '!*    +::>>rBM%4%%a!;3}'8'8':#;!;;<+%)C)CCHHL#{{99   D"-??)77)77&11'  r3rjNNNNNNN)r+r,r-r"r1rHr GREEDY_SEARCH_supported_generation_modesr>rr/r0rrr)rrrjr$rrrkrrirnros@r4rrsc ./#1#?#?"@1/3    t+ +,  $  /3.2599=)-X\&*N  t+N  t+N !++d2 N 0$6 N  $; N 4eErrurrvlen durationsrr}s r4r>z ParakeetTDTJointNetwork.__init__s?  IIf88&:K:KcRXRbRbNc:cd r3)r+r,r-r.r#r>rnros@r4rrs e0eer3rzG Parakeet Encoder with a TDT (Token Duration Transducer) head. c eZdZUeed<deffd Zee ddejdzdejdzdejdzde dzde dzd e eejzdzd ejdzd eed efd ZxZS)ParakeetForTDTr8cdt||t||_|j yrj)r=r>rrr\r}s r4r>zParakeetForTDT.__init__(s&  ,V4  r3Nrr*rrrrrxrrDc ||jd ||d|}|r|t|j}|j||} |j |j dddddddf| dddddddfj d} d} ||jd | dd|jjf| d|jjdf||jjd||jjk7jd|jj|jjd|} t| | |j|j |j |j"| S) a? decoder_input_ids (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Decoder input token ids for single-step inference. decoder_cache (`ParakeetRNNTDecoderCache`, *optional*): Decoder LSTM cache. When provided and initialized, the cached `decoder_output` is reused (e.g. during blank-skipping) instead of running the decoder. When `input_ids` is provided, the decoder runs and the cache is updated in-place. use_decoder_cache (`bool`, *optional*): Whether to use a decoder cache. When `True` and `decoder_cache` is `None`, a new cache is created automatically during the forward pass. encoder_outputs (`tuple(torch.FloatTensor)`, *optional*): Pre-computed encoder outputs (last_hidden_state, pooler_output, hidden_states, attentions, attention_mask). Can be a tuple or `ParakeetEncoderModelOutput`. Example: ```python >>> from transformers import AutoProcessor, ParakeetForTDT >>> from datasets import load_dataset, Audio >>> model_id = "nvidia/parakeet-tdt-0.6b-v3" >>> processor = AutoProcessor.from_pretrained(model_id) >>> model = ParakeetForTDT.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.cast_column("audio", Audio(sampling_rate=processor.feature_extractor.sampling_rate)) >>> inputs = processor(ds[0]["audio"]["array"]) >>> outputs = model(**inputs) ``` Nrzrrr.rQ) token_logitsduration_logitsrxrrrrrr2)rr$r8rrrrrrvr*rrrrrr`rOr-) rBrr*rrrrrxrrrir~s r4rizParakeetForTDT.forward.sX  "5d55--O !64T[[AM $ ->m T"1"?"?1dA "N"74A "F  '!*    %4%% #C)A4;;+A+A)A$AB &sDKK,B,B,D'D E-<<@@D%)A)AAFFrJ#{{99++//  D"-??)77)77&11'  r3r)r+r,r-r#r1r>rrr/r0rjr$rr)rrkrrrrirnros@r4rr s 0 /3.2599=)-X\&*O  t+O  t+O !++d2 O 0$6 O  $; O 4eErs$* $! &!HHI9[[F&__G5ssrr   H %   /!; /  //7299/7d  E-ryyE-P(*+,2 UU\\ U# U%,, U&% II% <<% % << % LL4' %  %%'(%2)*_ ryy_ +_ DBryyBJ,5,^;o;;| ] -]  ] @ A A A4   6      K,oK  K\-"))-`'ryy'$ :3: :& n -/Jn  n b e6 e Z /Z  Z z pr3