^jdZddlmZddlZddlZddlmZddlmZm Z m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZddlm Z m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+m,Z,ddl-m.Z.ddl/m0Z0e)jbe2Z3de4de4dejjfdZ6de4de4dejjfdZ7GddejpZ9 d@dejpdejjdejjd ejjd!ejjdzd"e:dzd#e:d$e#e'fd%Z;Gd&d'ejpZ<Gd(d)ejpZ=Gd*d+eZ>Gd,d-ejpZ?e(Gd.d/e!Z@e(Gd0d1e@ZAe(d23Gd4d5e@ZBe(d63Gd7d8e@ZCe(Gd9d:e@ZDe(Gd;de@ZFgd?ZGy)Az PyTorch DistilBERT model adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM) and in part from HuggingFace PyTorch version of Google AI Bert model (https://github.com/google-research/bert) )CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)get_activation)PreTrainedConfig)is_deepspeed_zero3_enabled)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs)DistilBertConfign_posdimoutctrdddl}|jj|d5tj j dk(rt|||cdddS dddyt|||S#1swYyxYw)Nr) modifier_rankr!r"r#)r deepspeedzeroGatheredParameterstorch distributedget_rank_create_sinusoidal_embeddings)r!r"r#r's }/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/distilbert/modeling_distilbert.pycreate_sinusoidal_embeddingsr/>s!# ^^ . .s! . D T  ))+q045csS T T0 T T-5csKK  T Ts /A==Bctjt|Dcgc];}t|Dcgc]$}|tjdd|dzz|z z &c}=c}}}d|_t j tj|dddddf|dddddf<t j tj|dddddf|dddddf<|j|Scc}wcc}}w)Ni'Frr) nparrayrangepower requires_gradr* FloatTensorsincosdetach_)r!r"r#posj position_encs r.r-r-Is88hmnshtuadQVWZQ[\AcBHHUAaL34F$GG\uvLC$$RVVLADqD,A%BCC14a4L$$RVVLADqD,A%BCC14a4LKKM J ]usC< )C7C< 7C< c eZdZdeffd Z d dej dej dzdejdzdej fdZxZ S) Embeddingsconfigc t|tj|j|j |j |_tj|j|j |_ tj|j d|_ tj|j|_ |jdtj|jj!ddy)N) padding_idx-q=eps position_idsrF) persistent)super__init__r Embedding vocab_sizer" pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormDropoutdropoutregister_bufferr*arangeexpandselfr@ __class__s r.rKzEmbeddings.__init__Ss !||F,=,=vzzW]WjWjk#%<<0N0NPVPZPZ#[ fjje<zz&..1   ELL)G)GHOOPWXej  N input_ids inputs_embedsrFreturnc||j|}|jd}|rt|dr|jddd|f}nPt j |tj |j}|jdj|}|j|}||z}|j|}|j|}|S)NrrF)dtypedevicer) rOsizehasattrrFr*rVlongra unsqueeze expand_asrQrRrT)rYr\r]rF seq_lengthrQ embeddingss r.forwardzEmbeddings.forward^s   00;M"''*  t^,#00KZK@ $||JejjQZQaQab +55a8BB9M "66|D"%88 ^^J/ \\*- r[)NN) __name__ __module__ __qualname__r rKr*Tensor LongTensorri __classcell__rZs@r.r?r?Rs_  /  .204 <<||d*&&-   r[r?modulequerykeyvalueattention_maskscalingrTkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)NrHr1rr")ptrainingr) rbr*matmul transposer functionalsoftmaxrTr| contiguous) rqrrrsrtrurvrTrw attn_weights attn_outputs r.eager_attention_forwardr|s**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r[c eZdZdeffd Z d dej dejdzdee de ej fdZ xZ S) DistilBertSelfAttentionr@c4t|||_|j|_|j|_|j|jz|_|j dz|_|j|jzdk7r&td|jd|jdtj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|_d|_y) Nryrzself.n_heads: z must divide self.dim:  evenly in_features out_featuresr{F)rJrKr@n_headsr"attention_head_sizerv ValueErrorrLinearq_link_linv_linout_linrSattention_dropoutrT is_causalrXs r.rKz DistilBertSelfAttention.__init__s  ~~ ::#'88t||#; //5  88dll "a '~dll^;RSWS[S[R\\cde eYY6::FJJO YY6::FJJO YY6::FJJO yyVZZfjjQ zzF$<$<= r[N hidden_statesrurwr^c |jdd}g|d|j}|j|j|j dd}|j |j|j dd}|j |j|j dd}tj|jjt} | |||||f|jsdn|jj|jd|\} } | j g|dj#} |j%| } | | fS)NrHrr1)rTrv)shaperrviewr~rrr get_interfacer@_attn_implementationrr|rTr{rvreshaperr) rYrrurw input_shape hidden_shape query_layer key_layer value_layerattention_interfacerrs r.rizDistilBertSelfAttention.forwardsV $))#2. CCbC$*B*BC 5djj/44lCMMaQRS 2DJJ}-22LAKKAqQ 4djj/44lCMMaQRS (?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH ll;/ L((r[N) rjrkrlr rKr*rmr7rrtuplerirorps@r.rrs^/248)||)))D0)+, ) u||  )r[rceZdZdeffd Zdej dej fdZdej dej fdZxZ S)FFNr@ct|tj|j|_|j |_d|_tj|j|j|_ tj|j|j|_ t|j|_ y)Nrrr)rJrKrrSrTchunk_size_feed_forward seq_len_dimrr" hidden_dimlin1lin2r activationrXs r.rKz FFN.__init__s zzFNN3 '-'E'E$II&**6CTCTU II&*;*;&**U ():):;r[inputr^cZt|j|j|j|Sr)rff_chunkrr)rYrs r.riz FFN.forwards%(8T8TVZVfVfhmnnr[c|j|}|j|}|j|}|j|}|Sr)rrrrT)rYrxs r.rz FFN.ff_chunks= IIe  OOA  IIaL LLOr[) rjrkrlr rKr*rmrirrorps@r.rrsI</<oU\\oelloellu||r[rc eZdZdeffd Z d dej dej dzdeede ej dffd Z xZ S) TransformerBlockr@c~t||j|jzdk7r&t d|jd|jdt ||_tj|jd|_ t||_ tj|jd|_ y)Nrzconfig.n_heads z must divide config.dim rrC)normalized_shaperE) rJrKr"rrr attentionrrR sa_layer_normrffnoutput_layer_normrXs r.rKzTransformerBlock.__init__s  :: &! +v~~.>>VW]WaWaVbbijk k08\\6::5Qv;!#vzzu!Ur[Nrrurwr^.c |j|fd|i|\}}|j||z}|j|}|j||z}|S)Nru)rrrr)rYrrurwattention_output_ ffn_outputs r.rizTransformerBlock.forwardsu-dnn  )  !  --.>.NOXX./ ++J9I,IJ r[r) rjrkrlr rKr*rmrrrrirorps@r.rrsc V/ V /3|| t++,  u||S ! r[rc teZdZdeffd Z d dej dej dzdeede fdZ xZ S) Transformerr@ct||j|_tjt |jDcgc] }t |c}|_d|_ycc}w)NF) rJrKn_layersr ModuleListr4rlayergradient_checkpointing)rYr@rrZs r.rKzTransformer.__init__ sQ  ]]eFOOF\#]$4V$<#]^ &+#$^sA-Nrrurwr^c P|jD] }|||fi|}t|S)N)last_hidden_state)rr)rYrrurw layer_modules r.rizTransformer.forwards> !JJ L(M ??r[r) rjrkrlr rKr*rmrrrrirorps@r.rrsX,/,/3 @|| @ t+ @+, @  @r[rceZdZUeed<dZdZdZdZdZ dZ e e dZ ejdej"ffd ZxZS)DistilBertPreTrainedModelr@ distilbertT)r attentionsrqc ,t||t|tr|jj rt j|jjt|jj|jjtj|jjt j|jtj |jj"dj%dyy)zInitialize the weights.rHrGN)rJ _init_weights isinstancer?r@sinusoidal_pos_embdsinitcopy_rQweightr/rPr"r* empty_likerFrVrrW)rYrqrZs r.rz'DistilBertPreTrainedModel._init_weights.s f% fj ){{// ..550 ;; (()C)C)J)JK JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h *r[)rjrkrlr __annotations__base_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrr_can_record_outputsr*no_gradrModulerrorps@r.rr sd $&*#N"&)- U]]_ iBII i ir[rcjeZdZdeffd Zdej fdZdefdZ dej fdZ dej fd Z e e e dd ej d zd ej d zd ej d zdej d zdeedeeej dfzf dZxZS)DistilBertModelr@ct||t||_t ||_|j yr)rJrKr?rhr transformer post_initrXs r.rKzDistilBertModel.__init__As5  $V,&v. r[r^c.|jjSz1 Returns the position embeddings )rhrQrYs r.get_position_embeddingsz'DistilBertModel.get_position_embeddingsJs222r[new_num_position_embeddingsc||jjz }|dk(rytjd|d||j_|jj j j}tj|jj|jj|j_|jjrKt|jj|jj|j j ntj5|dkDr8tj||jj j d| n1tj|d||jj _ddd|jj j!|j"y#1swY9xYw) Resizes position embeddings of the model if `new_num_position_embeddings != config.max_position_embeddings`. Arguments: new_num_position_embeddings (`int`): The number of new position embedding matrix. If position embeddings are learned, increasing the size will add newly initialized vectors at the end, whereas reducing the size will remove vectors from the end. If position embeddings are not learned (*e.g.* sinusoidal position embeddings), increasing the size will add correct vectors at the end following the position encoding algorithm, whereas reducing the size will remove vectors from the end. rNz(Setting `config.max_position_embeddings=z`...r&)r@rPloggerinforhrQrclonerrLr"rr/r*r Parametertora)rYrnum_position_embeds_diffold_position_embeddings_weights r.resize_position_embeddingsz*DistilBertModel.resize_position_embeddingsPsu$?AdAd#d  $q (  >?Z>[[_`a.I +)-)L)L)S)S)Y)Y)[&.0ll4;;;^;^`d`k`k`o`o.p+ ;; + + (kk99t{{TXTlTlTsTs  +a/]_]i]i6^DOO77>>?YAY@YZBD67P8PQBDOO77>   ++..t{{;  s :A/G!!G*c.|jjSrrhrOrs r.get_input_embeddingsz$DistilBertModel.get_input_embeddingszs...r[new_embeddingsc&||j_yrrrYrs r.set_input_embeddingsz$DistilBertModel.set_input_embeddings}s*8'r[Nr\rur]rFrw.c |du|duz r td|j|||}t|j||}|jd||d|S) input_ids (`torch.LongTensor` of shape `(batch_size, num_choices)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nz:You must specify exactly one of input_ids or inputs_embeds)r@r]ru)rru)rrhr r@r)rYr\rur]rFrwrhs r.rizDistilBertModel.forwardsw0  -t"; <YZ Z__Y |L 2;;$)   t $)   r[)NNNN)rjrkrlr rKrrLrintrrrrrrr*rmrrrrrirorps@r.rr?s/33 ( 112CD 001BC  (():)?)?DUDZDZ[]D^)_agalalmoapqH$&44#..   r[NNNNN)rjrkrl_tied_weights_keysr rKrrLrrrrrrrrr*rmrnrrrrrirorps@r.rrs 34bc 2/ 299 Pc P$ryy$.BII.*..2-1*.,0 1 <<$&1  t+1 ||d* 1   4' 1 llT) 1 +,1  % c 12 21 1 r[rz DistilBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c@eZdZdeffd Zdej fdZdefdZ e e dde jdzd e jdzd e jdzd e jdzd e jdzd eedeee jdfzfdZxZS)#DistilBertForSequenceClassificationr@ct|||j|_||_t ||_t j|j|j|_ t j|j|j|_ t j|j|_ |jyr)rJrK num_labelsr@rrrrr"pre_classifier classifierrSseq_classif_dropoutrTrrXs r.rKz,DistilBertForSequenceClassification.__init__s   ++ )&1 ii FJJ?))FJJ0A0ABzz&"<"<=  r[r^c6|jjSrr rs r.rz;DistilBertForSequenceClassification.get_position_embeddings%r r[rc:|jj|yr r rs r.rz>DistilBertForSequenceClassification.resize_position_embeddings+rr[Nr\rur]rrFrw.c F|jd ||||dd|}|d}|dddf} |j| } tj| } |j | } |j | } d} ||j j|jdk(rd|j _nl|jdkDrL|jtjk(s|jtjk(rd|j _nd|j _|j jdk(rIt} |jdk(r&| | j|j} n| | |} n|j jdk(r=t} | | j!d |j|j!d } n,|j jdk(rt#} | | |} t%| | |j&|j( S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). TrrNr regressionsingle_label_classificationmulti_label_classificationrHrr)rr%rReLUrTr&r@ problem_typer$r`r*rdrrsqueezerrrrrr) rYr\rur]rrFrwdistilbert_output hidden_state pooled_outputrrloss_fcts r.riz+DistilBertForSequenceClassification.forward9s",DOO )'%    )+ $QT* ++M: ! -0  ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'+99(33   r[r)rjrkrlr rKrrLrrrrrr*rmrnrrrrrirorps@r.r"r"s / 99 Pc P*..2-1*.,0 : <<$&:  t+: ||d* :   4' : llT) : +,:  "E%,,*;$< <: : r[r"c`eZdZdeffd Zdej fdZdefdZ e e dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzdeedeee jdfzfdZxZS)DistilBertForQuestionAnsweringr@cXt||t||_t j |j |j|_|jdk7rtd|jt j|j|_ |jy)Nr1z)config.num_labels should be 2, but it is )rJrKrrrrr"r$ qa_outputsrrS qa_dropoutrTrrXs r.rKz'DistilBertForQuestionAnswering.__init__zs  )&1))FJJ0A0AB    !HIZIZH[\] ]zz&"3"34  r[r^c6|jjSrr rs r.rz6DistilBertForQuestionAnswering.get_position_embeddingsr r[rc:|jj|yr r rs r.rz9DistilBertForQuestionAnswering.resize_position_embeddingsrr[Nr\rur]start_positions end_positionsrFrw.c |jd ||||dd|}|d} |j| } |j| } | jdd\} } | j dj } | j dj } d} ||t |jdkDr|j d}t |jdkDr|j d}| jd}|jd|}|jd|}tj|}|| |}|| |}||zd z } t| | | |j|j S) rTrrrrHrzN) ignore_indexr1)r start_logits end_logitsrrr)rrTr8splitr0rlenrbclamprrrrr)rYr\rur]r<r=rFrwr1rrr@rA total_loss ignored_indexr4 start_lossend_losss r.riz&DistilBertForQuestionAnswering.forwards2,DOO )'%    *!,  ]3 /#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM** FH!,@J M:H$x/14J+%!+99(33   r[)NNNNNN)rjrkrlr rKrrLrrrrrr*rmrrrrrirorps@r.r6r6xs / 99 Pc P*..2-1/3-1,0> <<$&>  t+> ||d* > , > ||d* > llT)> +,>  &ellC.?(@ @> > r[r6c@eZdZdeffd Zdej fdZdefdZ e e dde jdzd e jdzd e jdzd e jdzd e jdzd eedeee jdfzfdZxZS) DistilBertForTokenClassificationr@c,t|||j|_t||_t j |j|_t j|j|j|_ |jyr) rJrKr$rrrrSrTr hidden_sizer&rrXs r.rKz)DistilBertForTokenClassification.__init__sg   ++)&1zz&..1 ))F$6$68I8IJ r[r^c6|jjSrr rs r.rz8DistilBertForTokenClassification.get_position_embeddingsr r[rc:|jj|yr r rs r.rz;DistilBertForTokenClassification.resize_position_embeddingsrr[Nr\rur]rrFrw.c F|j|f|||dd|}|d}|j|}|j|} d} |>> from transformers import AutoTokenizer, DistilBertForMultipleChoice >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased") >>> model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> choice0 = "It is eaten with a fork and a knife." >>> choice1 = "It is eaten while held in the hand." >>> labels = torch.tensor(0).unsqueeze(0) # choice0 is correct (according to Wikipedia ;)), batch size 1 >>> encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True) >>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels) # batch size is 1 >>> # the linear classifier still needs to be trained >>> loss = outputs.loss >>> logits = outputs.logits ```NrrHTrPrr) rrrbrr%rr.rTr&rrrr)rYr\rur]rrFrw num_choicesrQr2r3rreshaped_logitsrr4s r.riz#DistilBertForMultipleChoice.forwardJsb-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImq(   r=#5#5b#9=;M;Mb;Q R  "$//  )'%    qz $QT* ++M: ! -0  ]3 / ++b+6  ')HOV4D("!//))   r[r)rjrkrlr rKrrLrrrrrr*rmrnrrrrrirorps@r.rTrT)s / 99 Pc P*..2-1*.,0 U <<$&U  t+U ||d* U   4' U llT) U +,U  #U5<<+<%= =U U r[rT)rrTr6r"rJrr)Nr)H__doc__collections.abcrnumpyr2r*rtorch.nnrrrr r activationsr configuration_utilsr integrations.deepspeedr masking_utilsr modeling_layersrmodeling_outputsrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrr utils.genericrrutils.output_capturingrconfiguration_distilbertr get_loggerrjrrrmr/r-rr?floatrrrrrrrrr"r6rJrT__all__rr[r.rqsy % AA&)3@69G& J56   H % LL#LELLL3U\\&&`!% II% <<% % << % LL4' % T\ %%'(%85)bii5)p"))* 1 F@"))@0iii<g /g g T _ 5_  _ D ^ *C^  ^ Bb %>b b JG '@G G Tw ";w w t r[