^jAfdZddlZddlZddlmZddlmZddlZddl Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlm Z m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3ddl4m5Z5dZ6dZ7e3jpe9Z:dZ;e/deGdde-Z< dide=e>e>fde?de>d e jdzd!e>d"ejf d#ZBdjd$e=d%e>d&ejdzfd'ZCGd(d)eZDGd*d+eZEGd,d-eZFGd.d/e jZHGd0d1e jZIGd2d3e jZJGd4d5e jZK dkd6e jd7e jd8e jd9e jd e jdzd:e?dzd;e?dd?e jZNGd@dAe jZOGdBdCeZPGdDdEeZQGdFdGe jZRGdHdIe jZSGdJdKe jZTGdLdMe jZUGdNdOe jZVGdPdQe jZWe/GdRdSe(ZXe/GdTdUeXZYe/dVGdWdXeXZZe/dYGdZd[eXZ[e/d\Gd]d^eXZ\e/Gd_d`eXZ]Gdadbe jZ^Gdcdde jZ_e/deGdfdgeXZ`gdhZay)lzPyTorch Wav2Vec2 model.N)Callable) dataclass) load_file)nn)CrossEntropyLoss)initialization)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)create_bidirectional_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel*get_torch_context_manager_or_global_device)Unpack) ModelOutputTransformersKwargsauto_docstring cached_filecheck_torch_load_is_safeis_peft_availablelogging)Wav2Vec2Configzadapter.{}.binzadapter.{}.safetensorsza Output type of [`Wav2Vec2ForPreTraining`], with potential hidden states and attentions. ) custom_introc^eZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed <dZej dzed <y) Wav2Vec2ForPreTrainingOutputa loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. contrastive_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The contrastive loss (L_m) as stated in the [official paper](https://huggingface.co/papers/2006.11477). diversity_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentionscontrastive_lossdiversity_loss)__name__ __module__ __qualname____doc__r'torch FloatTensor__annotations__r(r)r*r+tupler,r-r.y/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.pyr&r&Bs$&*D%  d ")15e''$.5;? 1 1D 8?6:5,,t3:59M5**+d2926Je''(4/615e''$.5/3NE%%,3r8r&shape mask_prob mask_lengthattention_mask min_masksreturnc|\}dkr tdkDrtdddtjjdj fd}|-|j j djnt|Dcgc]}c}}tj|ft } g} |} | d k(r| S|D]} || } tjjtj| dz z | d }t|d k(rdz }n|d }tj|tj| | z tj |zg}| j#|tj$| } tj&| dddddf|| f} | j)|| z} tjddddf}tj&||| fj)|| z}| |z} | j+dz kDrdz | | dz kD<tj,| | dd| Scc}w) an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. r!z&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkDrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedr!r)intmax) input_lengthnum_masked_spanepsilonr<r;r>sequence_lengths r9compute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spansoi,6DwNOoy9 [ (? :- batch_sizerI_ input_lengths spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrErFspec_aug_mask_idxdummy_mask_idxoffsetsrGrHs `` ` @@r9_compute_mask_indicesrmes0#(JQABB_$]^i]j&&7q :  iinnQ$$&G  $  % ##B'..0',Z'8 9!o 9HHj/:$GM1/Ba%5 1,?II,, IIlkAo6 7RW-   !Q &-q0N.q1NNN (;o(MUWU]U] ^ao o p  !!"34/52"451a:&5H+(V,33J@SVa@abii $T4]3Goog 4G'UV^^'+5G,g5/A"55GVYZGZ-!0CCDm%7B? w:s$ I+features_shape num_negativesmask_time_indicescd|\}}tj|}tj|||ftj}||j t ntj |t }t|D]}||jdz }|||} tjtj|dzdddf|dz|f} tjjd||dz|f} | | | k\xxdz cc<| | ||||<||xx||zz cc<|S)z> Sample `num_negatives` vectors from feature vectors. )r:rLNrKr!r)size) rOrZrWr^astyperXr]rVrTrarPrandint) rnrorprdrHsequence_length_rangesampled_negative_indices batch_idxhighmapped_masked_indicesfeature_indicessampled_indicess r9_sample_negative_indicesr|sT#1JIIo6 "xxz?M.Zbdbjbjk+<*G  &RWWUckoMp:& K  +//1A5 56G 6R S//"))D1H*=ag*FPQS`Hab))++At4!8]:S+T?:;q@;MbbqLr +,=i,HI !+y?/JJ+ K $#r8c&eZdZdfd ZdZxZS)Wav2Vec2NoLayerNormConvLayercdt||dkDr|j|dz nd|_|j||_t j |j|j|j||j||j|_ t|j|_ y)Nrr! kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__s r9rz%Wav2Vec2NoLayerNormConvLayer.__init__s !#Car8cV|jdkDr|ddddd|j f}|SNr)rrs r9rzWav2Vec2SamePadLayer.forwardxs6    ")!Q0F43F3F2F0F*FGMr8rrs@r9rrssKr8rc.eZdZdZfdZdZdZxZS)Wav2Vec2FeatureEncoderz.Construct the features from raw audio waveformc t||jdk(rDt|dgt |j dz Dcgc]}t ||dzc}z}nV|jdk(r.t |j Dcgc]}t||}}ntd|jdtj||_ d|_ d |_ ycc}wcc}w) Ngroupr)rr!layerz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT)rrfeat_extract_normrrVnum_feat_extract_layersr~rrNr ModuleList conv_layersgradient_checkpointing_requires_grad)rrirrs r9rzWav2Vec2FeatureEncoder.__init__s   # #w .5fqIJNSTZTrTruvTvNwNIJ,Va!eDNK % % 0HMfNlNlHmCD*6A>K01I1I0JJst ==5&+#"Ns C" C'cJ|jD] }d|_ d|_yNF) parameters requires_gradrrparams r9_freeze_parametersz)Wav2Vec2FeatureEncoder._freeze_parameterss(__& (E"'E  (#r8c|dddf}|jr|jrd|_|jD] }||} |S)NT)rtrainingrr)r input_valuesr+ conv_layers r9rzWav2Vec2FeatureEncoder.forwardsP$QW-    4==*.M '** 6J&}5M 6r8)r/r0r1r2rrrrrs@r9rr~s8#&$ r8rc$eZdZfdZdZxZS)Wav2Vec2FeatureProjectionc4t|tj|jd|j |_tj|jd|j|_ tj|j|_ y)NrJeps) rrrrrlayer_norm_epsrLinearr projectionDropoutfeat_proj_dropoutdropoutrrrs r9rz"Wav2Vec2FeatureProjection.__init__sf ,,vr':@U@UV))FOOB$79K9KLzz&":":; r8cp|j|}|j|}|j|}||fSr)rrr)rr+norm_hidden_statess r9rz!Wav2Vec2FeatureProjection.forwards:!__];(:;  ]3 000r8rrs@r9rrs < 1r8rmodulequerykeyvaluescalingrkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)NrJr#rr)prr!) rrr3matmulrr functionalsoftmaxrr contiguous) rrrrr=rrr attn_weights attn_outputs r9eager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r8c(eZdZdZ ddedededededed edzffd Z dd e jd e jdzd e jdzdedzde e de e je jdze e jdzff dZxZS)Wav2Vec2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsr is_decoderr is_causalrc t|||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ tj||||_ tj||||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)r)rrrrrhead_dimrrNrrr rrk_projv_projq_projout_proj) rrrrrrr rrs r9rzWav2Vec2Attention.__init__s "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r8r+key_value_statesr=output_attentionsrr?c |du}|jdd}g|d|j}|j|j|j dd} |r|n|} g| jddd|j} |j | j| j dd} |j | j| j dd} tj|jjt}||| | | |f|jsdn |j|j|d|\}}|jg|dj!}|j#|}||dfS)z#Input shape: Batch x Time x ChannelNrJr!r#)rrr)r:r rviewrr r r get_interfacer_attn_implementationrrrrrbrr)rr+rr=rris_cross_attention input_shape hidden_shape query_statescurrent_stateskv_shape key_states value_statesattention_interfacerrs r9rzWav2Vec2Attention.forwards.T9$))#2. 88b8$--8 {{=166|DNNqRST -?)]B^))#2.BBDMMB[[055h?II!QO {{>277AKKAqQ (?(M(M KK , ,.E) %8      % $}}C$,,LL/ %  % ! \*k));;;;FFH mmK0 L$..r8)rFTFN)NNF)r/r0r1r2rCfloatrXr"rr3Tensorrrr6rrrs@r9rrsG  (,CCC C  C  CC%CD15.2). 0/||0/ ,,-0/ t+ 0/  $; 0/-.0/ u||U\\D0% 2E2LL M0/r8rc$eZdZfdZdZxZS)Wav2Vec2FeedForwardct|tj|j|_tj |j|j|_ t|jtrt|j|_n|j|_tj |j|j|_tj|j |_yr)rrrractivation_dropoutintermediate_dropoutrrintermediate_sizeintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrs r9rzWav2Vec2FeedForward.__init__(s $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?r8c|j|}|j|}|j|}|j|}|j |}|Sr)r(r,r&r-r/rs r9rzWav2Vec2FeedForward.forward5sX// > 00? 11-@ ))-8 ++M: r8rrs@r9r#r#'s @r8r#c&eZdZfdZddZxZS)Wav2Vec2EncoderLayerct|t|j|j|j d||_tj|j|_ tj|j|j|_ t||_tj|j|j|_y)NFrrrrrr)rrrrnum_attention_headsattention_dropout attentionrrr.rrrrr# feed_forwardfinal_layer_normrs r9rzWav2Vec2EncoderLayer.__init__@s *((00,,  zz&"7"78 ,,v'9'9v?T?TU/7 " V-?-?VEZEZ [r8c|}|j|||\}}}|j|}||z}|j|}||j|z}|j |}|f}|r||fz }|SNr=r)r7rrr8r9rr+r=r attn_residualrreoutputss r9rzWav2Vec2EncoderLayer.forwardOs% )- .L]*8* & |Q ]3 % 5  6 %(9(9-(HH --m<  "    &Gr8rrrs@r9r2r2?s \r8r2cfeZdZfdZ ddej dej dzdefdZxZS)#Wav2Vec2EncoderLayerStableLayerNormct|t|j|j|j d||_tj|j|_ tj|j|j|_ t||_tj|j|j|_t#|ddt%||_yd|_y)NFr4radapter_attn_dim)rrrrr5r6r7rrr.rrrrr#r8r9getattrWav2Vec2AttnAdapterLayer adapter_layerrs r9rz,Wav2Vec2EncoderLayerStableLayerNorm.__init__ds *((00,,  zz&"7"78 ,,v'9'9v?T?TU/7 " V-?-?VEZEZ [ 6-t 4 @!9&!AD !%D r8Nr+r=rc$|}|j|}|j|||\}}}|j|}||z}||j|j |z}|j ||j |z}|f}|r||fz }|Sr;)rr7rr8r9rFr=s r9rz+Wav2Vec2EncoderLayerStableLayerNorm.forwardws &  6 )- .L]*8* & |Q ]3 % 5 %(9(9$:O:OP]:^(__    ))D,>,>},MMM "    &Gr8r) r/r0r1rr3r!rXrrrs@r9rArAcs>&,/3"' || t+ r8rAc reZdZfdZ d dej dej dzdededef dZxZ S) Wav2Vec2Encoderct|||_t||_t j |j|j|_ t j|j|_ t jt|jDcgc] }t!|c}|_d|_ycc}wNrF)rrrrpos_conv_embedrrrrrrr.rrrVnum_hidden_layersr2layersrrrrers r9rzWav2Vec2Encoder.__init__s  =fE,,v'9'9v?T?TUzz&"7"78 mm5QWQiQiKj$ka%9&%A$kl &+#%l!CNr+r=routput_hidden_states return_dictc|rdnd}|rdnd}|5|jdjdd|jd}d||<t|j||}|j |} || j |jz}|j|}|j|}txs t|} |jD]j} |r||fz}tjg} |jxr| |jj k} | r| r| |||}|d}| rd}|sb|dfz}l|r||fz}|st#d |||fDSt%||| S) Nr7rJr!r#rr inputs_embedsr=r<NNc3&K|] }|| ywrr7.0vs r9 z*Wav2Vec2Encoder.forward..mq_`_lmlast_hidden_stater+r,) unsqueezerepeatr:r rrLtodevicerrr r rNr3rQr layerdropr6rrr+r=rrQrRall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusrdropout_probabilityskip_the_layer layer_outputss r9rzWav2Vec2Encoder.forwards#7BD$5b4  %$2$<$(>}?S?S(TT  6  ]3 02R6LT6R [[ PE#$58H$H!#(**R. !]]Z/BT[[EZEZ/ZN![ %!.Te! !.a 0  ,  &9]1=M.r\r]r^)r`rar:r rrLrr r rNr3rQrrdrr6rres r9rz&Wav2Vec2EncoderStableLayerNorm.forwards#7BD$5b4  %$2$<$AQAQ Q+00o1Mt`d`m`moqr !oob)..z?BO J&&r8r) r/r0r1r2r staticmethodrrrrs@r9rvrv%s& (  #'r8rvc$eZdZfdZdZxZS)Wav2Vec2Adapterct|jjk7rTt j jj|_t jj|_ndx|_|_t jfdtjD|_ j|_ y)Nc34K|]}tywr)Wav2Vec2AdapterLayer)rYrers r9r[z+Wav2Vec2Adapter.__init__..|s#kQ$8$@#ks)rroutput_hidden_sizerrrprojrproj_layer_normrrVnum_adapter_layersrNrdrs `r9rzWav2Vec2Adapter.__init__rs   $ $(:(: : &"4"4f6O6OPDI#%<<0I0I#JD /3 3DI,mm#k%PVPiPiJj#kk ))r8ch|j.|j"|j|}|j|}|jdd}|jD]D}tj j }|j r||jkDs=||}F|jdd}|Sr)rrrrNrOrPrrd)rr+rlayerdrop_probs r9rzWav2Vec2Adapter.forwards 99 T%9%9%E IIm4M 00?M%//15 [[ 5EYY--/N==^dnn%D %m 4  5 &//15 r8rrs@r9rrqs  *r8rc$eZdZfdZdZxZS)rct|tj|jd|jz|j |j d|_y)Nr#r!)rr)rrrrradapter_kernel_sizeadapter_striderrs r9rzWav2Vec2AdapterLayer.__init__sJ II  % % )) )  & &((   r8cj|j|}tjj|d}|S)Nr!r)rrrglurs r9rzWav2Vec2AdapterLayer.forwards/ -0  ))-Q)? r8rrs@r9rrs  r8rc>eZdZfdZdej fdZxZS)rEct||j|_|j|_t j|j |_t j|j |j|_ t j|_ t j|j|j |_ y)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)rrrC input_dimr hidden_dimrrnormrlinear_1ReLUact_fnlinear_2rs r9rz!Wav2Vec2AttnAdapterLayer.__init__s 00 ,,LL1  $//4>>B ggi  $..$//B r8r+c|j|}|j|}|j|}|j|}|Sr)rrrrrs r9rz Wav2Vec2AttnAdapterLayer.forwards@ -0  m4  M2  m4 r8)r/r0r1rr3r4rrrs@r9rErEs CU%6%6r8rEceZdZUeed<dZdZdZdZdZ dZ dZ e jfdZdde jezd edzfd Z dd ed e jfd ZdZdZddefdZxZS)Wav2Vec2PreTrainedModelrwav2vec2raudioTc t||t|tr5|jj |j j yt|trutj|jjddtj|jjtj|jyt|t rtj|j"jddt%j&d|j"j(d|j"j*zz ztj,|j"jdyt|t.rt%j&d|j0j2z }tj|j0j| |tj|j0j| |yt|t4j6rtj8|j|j`t%j&|j:|j*|j(dzz }tj|j| |yyy)zInitialize the weightsrr!)rstdrr#)abN)r _init_weightsr)Wav2Vec2ForPreTraining project_hidreset_parameters project_qrvinitnormal_r~rzeros_runiform_r}rrmathsqrtr in_channels constant_rr in_featuresrrkaiming_normal_r)rrkrs r9rz%Wav2Vec2PreTrainedModel._init_weightss f% f4 5    / / 1    - - /  = > LL++22! D KK**// 0 MM&,, -  ? @ LL "" !v{{'>'>q'AFKKD[D['["\]]  NN6;;++Q /  9 : !f//;;; MM&++00QB! <  *   /{{&IIfmmv/A/AFDVDVWXDY/YZ[ fkkaR15'+r8Nrf add_adaptercT||jjn|}d}t|jj|jjD]\}}||||}|rBt |jj D] }||d|jj}"|S)zH Computes the output length of the convolutional layers c>tj||z |ddzS)Nfloor) rounding_moder!)r3divrErrs r9_conv_out_lengthzRWav2Vec2PreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths"99\K7wWZ[[ [r8r!)rrziprrrVrr)rrfrrrrres r9 _get_feat_extract_output_lengthsz8Wav2Vec2PreTrainedModel._get_feat_extract_output_lengthss 2=1Ddkk--+  \ $'t{{'>'> @W@W#X Q K,]KPM Q 4;;99: _ 04;;C]C] ^  _r8feature_vector_lengthr=c|jddddf}|j||}|jtj}|j d}tj ||f|j|j}d|tj|j d|j|dz f<|jdgjdjdgj}|S)NrJrrr)rLrcr!)rc) cumsumrrbr3longr:rWrLrcrZfliprX)rrr=rnon_padded_lengthsoutput_lengthsrds r9"_get_feature_vector_attention_maskz:Wav2Vec2PreTrainedModel._get_feature_vector_attention_masks ,22r2:1b5A>>?Q_j>k'**5::6#))!,  . /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr8c|jjt|jdi}|j D]D\}}t |t s|jD]\}}||dj||g<Ft |tr8|jjD]\}}||djd|g<|S)NzF has no adapter layers. Make sure to define `config.adapter_attn_dim`..lm_head) rrCrNr named_modulesr)rEnamed_parametersjoinWav2Vec2ForCTCr)radapter_weightsrr param_namers r9 _get_adaptersz%Wav2Vec2PreTrainedModel._get_adapterss ;; ' ' ///uvw w ..0 JLD&&":;)/)@)@)BJ%JDIOCHHdJ-?$@AJ J dN +#||<<> E e?D)T): ;< Er8c|jD]$}t|ts|j|&t|tr|j|j yy)zc (Re-)initialize attention adapter layers and lm head for adapter-only fine-tuning N)modulesr)rErrr)rrs r9init_adapter_layersz+Wav2Vec2PreTrainedModel.init_adapter_layerssU lln +F&":;""6* + dN +   t|| , ,r8 target_langc |jjtd|d||jk(r|stj d|dy|j dd}|j dd}|j d d}|j d d}|j d d}|j d d} |j d d} |jj} d} | dur4tj|} t| | ||||| |}t|} | Ktj|} t| | ||||| |}tt!j"|dd} |j%}t'| j)t'|j)z }t'|j)t'| j)z }t+|dkDr!tdddj-|dt+|dkDr!tdddj-|d| dj.d}||jj0k7rWt3j4|jj6||j8|j:|_||j_| j?Dcic]\}}||jA||} }}|jC| d||_y#t$r| rYt$r| rtd| d| d| dY$wxYw#t$rt$rt$rtd| d| d| dwxYwcc}}w)a Load a language adapter model from a pre-trained adapter model. Parameters: target_lang (`str`): Has to be a language id of an existing adapter weight. Adapter weights are stored in the format adapter..safetensors or adapter..bin force_load (`bool`, defaults to `True`): Whether the weights shall be loaded even if `target_lang` matches `self.target_lang`. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (i.e., do not try to download the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. mirror (`str`, *optional*): Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. Activate the special ["offline-mode"](https://huggingface.co/transformers/installation.html#offline-mode) to use this method in a firewalled environment. Examples: ```python >>> from transformers import Wav2Vec2ForCTC, AutoProcessor >>> ckpt = "facebook/mms-1b-all" >>> processor = AutoProcessor.from_pretrained(ckpt) >>> model = Wav2Vec2ForCTC.from_pretrained(ckpt, target_lang="eng") >>> # set specific language >>> processor.tokenizer.set_target_lang("spa") >>> model.load_adapter("spa") ``` NzCannot load_adapter for - if `config.adapter_attn_dim` is not defined.z#Adapter weights are already set to r cache_dirforce_downloadFproxieslocal_files_onlytokenrevisionuse_safetensors)filenamerrrrrrzCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named cpuT) map_location weights_onlyrzThe adapter weights z has unexpected keys: z, z has missing keys: zlm_head.weightrcrL)strict)"rrCrNrloggerwarningpop _name_or_pathWAV2VEC2_ADAPTER_SAFE_FILEformatrsafe_load_fileOSError ExceptionWAV2VEC2_ADAPTER_PT_FILErr3loadrsetkeysr[rr: vocab_sizerrrrcrLritemsrbload_state_dict)rr force_loadrrrrrrrrmodel_path_or_id state_dictfilepath weight_pathrunexpected_keys missing_keystarget_vocab_sizerrZs r9 load_adapterz$Wav2Vec2PreTrainedModel.load_adapter+sx ;; ' ' /7 }Dqrs s $** *: NN@ QO P JJ{D1 $4e<**Y-!::&8%@ 7D)::j$/ **%6=;;44  % '188EH )$%#1#%5%'  ,K8 &  /66{CH" )$%#1#%5%'  )*"ZZ!&!% .,,.joo/037K7K7M3NN?//12S9J5KK  ! #3K=@VW[W`W`apWqVrrstu u   "3K=@STXT]T]^jTkSllmno o''78>>qA  6 6 699 ..0A$++]a]g]gDL&7DKK "?I>N>N>PQdaaoa011Q Q Z6'] "#  "!45E4FG==M>FZqJ# B   01A0BC99I8JK::B1F 6Rs*L75MM; M!MM1M8r)T)r/r0r1r"r5base_model_prefixmain_input_nameinput_modalitiessupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr3no_gradr LongTensorrCrXrrrrr+rrrs@r9rrs "$O&*#NU]]_66<e>N>NQT>Tcgjncn,Y]%(:?:J:J( -m'm'r8rc,eZdZdeffd ZdZ ddejdejdzdejdzfdZ e dd ejdzdejdzdejdzd e dzd e dzd e dzd e ezfdZxZS) Wav2Vec2Modelrct||||_t||_t ||_|jdkDs|jdkDrEtjtj|jj|_|j rt#||_nt'||_|j(r t+|nd|_|j/yNr)rrrrfeature_extractorrfeature_projectionmask_time_probmask_feature_probrr|r3r!rrmasked_spec_embeddo_stable_layer_normrqencoderrIrradapter post_initrs r9rzWav2Vec2Model.__init__s   !7!?";F"C  3 &&*B*BS*H%'\\%,,v?Q?Q2R2[2[2]%^D "  & &9&ADL*62DL282D2Dv.$  r8c8|jjyz Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)r$rrs r9freeze_feature_encoderz$Wav2Vec2Model.freeze_feature_encoders 113r8Nr+rpr=ct|jdds|S|j\}}}|)|jj |j ||<n|jj dkDr|jrt||f|jj |jj||jj}tj||jtj}|jj |j ||<|jjdkDr|jrt||f|jj|jj |jj"}tj||jtj}|dddfj%d|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTNr)r;r<r=r>r)r;r<r>rJ)rDrrrr(rbrLr&rrmmask_time_lengthmask_time_min_masksr3rorcrXr'mask_feature_lengthmask_feature_min_masksr)rr+rpr=rdrHrmask_feature_indicess r9_mask_hidden_statesz!Wav2Vec2Model._mask_hidden_statesst{{$8$? 4A3E3E3G0 O[  (/3/E/E/H/HI\I\/]M+ , [[ ' '! + 5_-++44 KK88-++99 ! !& ->}G[G[chcmcm n /3/E/E/H/HI\I\/]M+ , ;; ( (1 ,#8[)++77 KK;;++<< $ $)<<0D]MaMainisis#t #74#@#G#GO]_#` 23M. /r8rrrQrRr?c H||n|jj}||n|jj}||n|jj}|j |}|j dd}|!|j |jd|d}|j|\} }|j| ||} |j| ||||} | d} |j|j| } |s | |f| ddzSt| || j| j S) a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Nr!r#Fr)rpr=r=rrQrRr)r_extract_featuresr+r,)rrrQrRr$rrr:r%r8r*r+rr+r,) rrr=rprrQrRrr;r+encoder_outputss r9rzWav2Vec2Model.forward&sb 2C1N-TXT_T_TqTq$8$D $++JjJj &1%uEN+/*A*ABR*S' '00 ->~1 ,, )/!5# ' (* << # LL7M!#34qr7JJ J&+-)77&11   r8rVNNNNN)r/r0r1r"rr0r3r4rr8rr!rXr6rrrrs@r9r!r!s~(47;26 ,((,!,,t3,((4/ ,\/36:)-,0#'8 llT)8  t+8 !,,t3 8  $; 8 #Tk 8 D[8  ( (8 8 r8r!z? Wav2Vec2 Model with a quantizer and `VQ` head on top. cXeZdZdeffd ZdefdZdZe dde jde jde jde fd Z e dd e jd zd e jd zd e jd zde jd zded zded zded zdeezfdZxZS)rrct||t||_t j |j |_t||_ t j|j|j|_ t j|j|j|_|j!yr)rrr!rrrfeat_quantizer_dropoutdropout_featuresrv quantizerrrproj_codevector_dimrr{rr,rs r9rzWav2Vec2ForPreTraining.__init__hs  %f- " 6+H+H I6v>99V%7%79S9ST6#8#8&:T:TU r8rc&||j_y)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rBr)rrs r9set_gumbel_temperaturez-Wav2Vec2ForPreTraining.set_gumbel_temperatureus&1"r8cL|jjjyr.rr$rr/s r9r0z-Wav2Vec2ForPreTraining.freeze_feature_encoder{ ''::>> import torch >>> from transformers import AutoFeatureExtractor, Wav2Vec2ForPreTraining >>> from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices >>> from datasets import load_dataset >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base") >>> model = Wav2Vec2ForPreTraining.from_pretrained("facebook/wav2vec2-base") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values # Batch size 1 >>> # compute masked indices >>> batch_size, raw_sequence_length = input_values.shape >>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item() >>> mask_time_indices = _compute_mask_indices( ... shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2 ... ) >>> sampled_negative_indices = _sample_negative_indices( ... features_shape=(batch_size, sequence_length), ... num_negatives=model.config.num_negatives, ... mask_time_indices=mask_time_indices, ... ) >>> mask_time_indices = torch.tensor(data=mask_time_indices, device=input_values.device, dtype=torch.long) >>> sampled_negative_indices = torch.tensor( ... data=sampled_negative_indices, device=input_values.device, dtype=torch.long ... ) >>> with torch.no_grad(): ... outputs = model(input_values, mask_time_indices=mask_time_indices) >>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states) >>> cosine_sim = torch.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, dim=-1) >>> # show that cosine similarity is much higher than random >>> cosine_sim[mask_time_indices.to(torch.bool)].mean() > 0.5 tensor(True) >>> # for contrastive loss training model should be put into train mode >>> model = model.train() >>> loss = model( ... input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices ... ).loss ```N)r=rrQrprRrr!Fr)rprJr#rz-infirT) reduction)r'r(r)r*r+r,r-r.)$rrRrbr3rXrrrArr:rBrrrLrrpermuterPcontrastive_logits_temperatureallanyr rrbrrrrr cross_entropyryrxrTdiversity_loss_weightr&r+r,)rrr=rprvrrQrRrr?transformer_featuresr;quantized_featuresr*r'r-r.rdrHrnegative_quantized_featuresrO neg_is_postargetnum_codevectorss r9rzWav2Vec2ForPreTraining.forwardsIF&1%uEN59NN 0A5C5 110224>>3H3H3N3NO!^^,>?3777. # /7I7O7O 4J +=*A*A"k*R(--/44R8+ '+F*J*JOR+gaAq! ( 44"47++$ :: F-0KKPPQSTJ~~).vqr :&%%a+33B AGF,1133t;FFq!LTTVF!}}::6<<>6]b:c "kkCCdkkFgFggO.1FF/Y]n]r]r]ttN$dkk&G&G.&XXD24FH]^ahijikalll(*<>STW^_`_aWbb b+1'9"7!//))-)  r8)g?)NNNNNN)r/r0r1r"rrCrEr0rr3r4r rPrr! BoolTensorrXr6r&rrrs@r9rrbs) ~ 1#1 = ! ** ,,"-- (/359<@)-,0#'] llT)]  t+] !++d2 ] #("2"2T"9 ]  $; ] #Tk] D[]  - -] ] r8rzp Wav2Vec2 Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). ceZdZddedzffd ZdZdZdZe dde jdzde jdzd e dzd e dzd e dzd e jdzd e e zfdZxZS)rNrct||t||_t j |j |_||_|jtd|jdt|dr|jr |jn |j}t j ||j|_|j%y)a2 target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`Wav2Vec2ForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.r)rrr!rrr final_dropoutrrr rNrrrrrrrr,)rrrrrs r9rzWav2Vec2ForCTC.__init__>s  %f- zz&"6"67 &    $00@AHH *1)GFL^L^F % %djdvdv yy!3V5F5FG  r8c 8ttjdk(ry|j}|&t |j ddt d|d|-t |j ddtjdy||j|dyy) a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. metaNrCzCannot pass `target_lang`: rz)By default `target_lang` is set to 'eng'.T)r) rr3rcrrDrrNrinfor)rrrs r9 tie_weightszWav2Vec2ForCTC.tie_weights[s 6 75<<;O O  &&  "wt{{ 1` a classification loss is computed (Cross-Entropy). NTr:r!rrJrr#rrr)rrRrrr~r3stackrrrrrrTrrrr:r`rarrrrr+r,)rrr=rrQrRrmrr?r+ norm_weights pooled_output padding_maskexpand_padding_maskrOr'loss_fctrs r9rz)Wav2Vec2ForSequenceClassification.forwards0&1%->r1a-HHMMRSMTM#AJM}5  !)..1.5MBB=CVCVWXCY[ijL"."8"8"<"C"CAq-J]J]^_J`"a 25M.. /)--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UDY)F)G!HHF)-)9TGf$ Ev E'!//))   r8r=)r/r0r1rr0rkrr3r!rXr6rrrrs@r9rrs"=(/3)-,0#'&*C llT)C  t+C  $; C #Tk C D[ C  t#C  ) )C C r8rceZdZfdZdZdZe d dejdzdejdzdejdzde dzd e dzd e dzd e e zfd Z xZ S)#Wav2Vec2ForAudioFrameClassificationct||t|dr|jr t dt ||_|jdz}|jr0tjtj||z |_ tj|j|j |_|j |_|j%y)NrzbAudio frame classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r!)rrrrrNr!rrMrrr|r3r]rrrrrr,rs r9rz,Wav2Vec2ForAudioFrameClassification.__init__;s  6= )f.@.@t &f- --1  ( (!#ejj.Dz.Q!RD ))F$6$68I8IJ ++ r8cL|jjjyr.rGr/s r9r0z:Wav2Vec2ForAudioFrameClassification.freeze_feature_encoderKrHr8cP|jjD] }d|_ yrirjrs r9rkz5Wav2Vec2ForAudioFrameClassification.freeze_base_modelRrlr8Nrr=rmrrQrRr?c ||n|jj}|jjrdn|}|j|||||}|jjrr|t} t j | d} tjj|jd} | | jdddzjd} n|d} |j| } d} |\t} | | jd|jt j |jd|jd} |s| f|tdz}|St#| | |j$|j& S) rNTr:r!rrJr)axisrr)rrRrrr~r3rrrrrrrTrrrrrr+r,)rrr=rmrrQrRrr?r+rrOr'rrs r9rz+Wav2Vec2ForAudioFrameClassification.forwardZsh0&1%->r1a-HHMMRSMTM#AJM/  ')HFKKDOOtuDY)F)G!HHFM$!//))   r8r=)r/r0r1rr0rkrr3r!rXr6rrrrs@r9rr9s =(/3&*)-,0#': llT):  t+:  t# :  $; : #Tk : D[:  & &: : r8rc&eZdZdfd ZdZxZS) AMSoftmaxLossct|||_||_||_t j tj||d|_ t j|_ y)NT)r) rrscalemarginrrr|r3randnrrr')rrrrrrs r9rzAMSoftmaxLoss.__init__sQ   $ll5;;y*#EUYZ '') r8c|j}tjj|jd}tjj|d}t j ||}||jz }tjj||j}|jt j|j||z}|j||}|S)Nrrr!)rrr normalizerr3mmrone_hotrrrrXr') rr+rmr cos_thetapsionehotrOr's r9rzAMSoftmaxLoss.forwards!((!(< // 1/E HH]F3 $++%&&vt?ekk&++-iHHyy( r8)g>@g?rrs@r9rrs * r8rcXeZdZdfd Zdej dej fdZxZS) TDNNLayerct||dkDr|j|dz n|j||_|j||_|j ||_|j||_tj|j|j z|j|_ tj|_ y)Nrr!)rrtdnn_dimrr tdnn_kernelr tdnn_dilationdilationrrkernelrrrs r9rzTDNNLayer.__init__s +BFD]D]!^))F$=$=v?X?XY&v'@'@&BSBST Rs>FcL|jjjyr.rGr/s r9r0z)Wav2Vec2ForXVector.freeze_feature_encoderrHr8cP|jjD] }d|_ yrirjrs r9rkz$Wav2Vec2ForXVector.freeze_base_modelrlr8rfcVd}|jjD] }|||d}|S)z? Computes the output length of the TDNN layers c||z |zdzS)Nr!r7rs r9rzEWav2Vec2ForXVector._get_tdnn_output_lengths.._conv_out_lengths!;.69A= =r8r!)rr)rrfrrs r9_get_tdnn_output_lengthsz+Wav2Vec2ForXVector._get_tdnn_output_lengthss:  >  ;;22 LK,]KKM Lr8Nrr=rrQrRrmr?c ||n|jj}|jjrdn|}|j|||||}|jjrr|t} t j | d} tjj|jd} | | jdddzjd} n|d} |j| } |jD] } | | } |%| jd} | j!d} n|j#|jd}|j%|}g} g} t'|D]U\}}| j)| |d|fjd| j)| |d|fj!dWt j | } t j | } t j*| | gd}|j-|}|j/|}d}||j1||}|s||f|tdz}||f|zS|St3||||j4|j6S) rNTr:r!rrJr)r'rO embeddingsr+r,)rrRrrr~r3rrrrrrrTrrrrrr enumerater_rMr$rrrr+r,)rrr=rrQrRrmrr?r+r tdnn_layer mean_features std_featuresfeat_extract_output_lengthstdnn_output_lengthsrlengthstatistic_poolingoutput_embeddingsrOr'rs r9rzWav2Vec2ForXVector.forwards0&1%->r1a-HHMMRSMTM#AJM}5 )) 6J&}5M 6  !)..1.5M(,,,3L*.*O*OP^PbPbghPbPi*j '"&"?"?@["\ ML&':; J 6$$]1gvg:%>%C%C%C%JK##M!WfW*$=$A$Aa$A$HI J"KK 6M ;;|4L!II}l&CL 223DE!23  >>&&1D/07;X;Y3ZZF)-)9TGf$ Ev E(!//))   r8r=)r/r0r1rr0rkr3rrCrrr!rXr6rrrrs@r9rrs &=( e6F6F6L /3)-,0#'&*P llT)P  t+P  $; P #Tk P D[ P  t#P   P P r8r)rrrrrr!rrrr#)br2rrcollections.abcr dataclassesrnumpyrOr3safetensors.torchrrrtorch.nnrr r activationsr integrations.deepspeedr integrations.fsdpr masking_utilsr modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrmodeling_utilsrrrprocessing_utilsrrrrrrrrr configuration_wav2vec2r"rr get_loggerr/rr~r&r6rCr rndarrayrmr|r~rrModulerrrrr!rrr#r2rArIrqrvrrrErr!rrrrrrr__all__r7r8r9rs $! 9%&!@76B9sr&3,5   H %!"  4;4  4B/3 t c?ttt$$t+ t  t ZZ tn$U$3$[][e[ehl[l$D#=*!;6!;0*bii*Z299%RYY%P 1 1,!% II% <<% % << % LL4' % T\ %%'(%8R/ R/j"))0!5!H+*D+\E biiE PI RYYI XI'BIII'Xbii>299$ryy2\'o\'\'~C +C C L N 4N  N b K ,K  K \ e (?e  e P[ *A[ [ |BII. @ C 0C  C L r8