^jBdZddlmZddlmZddlZddlmZddlmZm Z m Z ddl m Z dd lmZdd lmZdd lmZmZmZmZmZmZmZdd lmZmZdd lmZddlm Z ddl!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*ddl+m,Z,e%jZe.Z/Gddej`Z1 dBdej`dejddejddejddejddzde3dzde3dee#fdZ4Gdd ej`Z5Gd!d"ej`Z6Gd#d$ej`Z7Gd%d&ej`Z8e$Gd'd(eZ9e$d)*eGd+d,e"Z:e$Gd-d.e9Z;e$d/*Gd0d1e9Z<Gd2d3ej`Z=Gd4d5ej`Z>e$Gd6d7e9Z?e$d8*Gd9d:e9Z@e$Gd;de9ZBe$Gd?d@e9ZCgdAZDy)CzPyTorch ALBERT model.)Callable) dataclassN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)create_bidirectional_mask)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) AlbertConfigc eZdZdZdeffd Z d dejdzdejdzdejdzdejdzd ejf d Z xZ S) AlbertEmbeddingszQ Construct the embeddings from word, position and token_type embeddings. configct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j#dt%j*|j,j/t$j0dy) N) padding_idxeps position_idsrF) persistenttoken_type_ids)dtype)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandzerosr(sizelongselfr# __class__s u/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/albert/modeling_albert.pyr/zAlbertEmbeddings.__init__6s !||F,=,=v?T?Tbhbubuv#%<<0N0NPVPePe#f %'\\&2H2H&J_J_%`"f&;&;AVAVWzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  N input_idsr,r( inputs_embedsreturnc||j}n|jdd}|\}}||jddd|f}|t|drT|jj |j dd}t j|d|}|j ||}n:t j|t j|jj}||j|}|j|} || z} |j|} | | z} |j| } |j| } | S)Nr*r,rr)dimindex)r-device)rCr(hasattrr,rAshaper?gatherrBrDrPr4r8r6r9r=) rFrJr,r(rK input_shape batch_size seq_lengthbuffered_token_type_idsr8 embeddingsr6s rHforwardzAlbertEmbeddings.forwardGsG  #..*K',,.s3K!, J  ,,Q  ^ J"%:: "66|D"55 ^^J/ \\*- rI)NNNN) __name__ __module__ __qualname____doc__r r/r? LongTensor FloatTensorTensorrY __classcell__rGs@rHr"r"1s | &.2260426 '##d*'((4/'&&- ' ((4/ '  'rIr"modulequerykeyvalueattention_maskscalingr=kwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)Nr*r rN)ptrainingr) rCr?matmul transposer functionalsoftmaxr=ro contiguous) rcrdrerfrgrhr=ri attn_weights attn_outputs rHeager_attention_forwardrwrs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$rIc eZdZdeffd Z d dej dejdzdee de ej ej ffdZ xZ S) AlbertAttentionr#c$t||j|jzdk7r1t |ds%t d|jd|j||_|j|_|j|_|j|jz|_|j|jz|_|jdz|_ tj|j|_ tj|j|_tj |j|j|_tj |j|j|_tj |j|j|_tj |j|j|_tj*|j|j,|_d|_y)Nrr2zThe hidden size (z6) is not a multiple of the number of attention heads (rkr&F)r.r/ hidden_sizenum_attention_headsrQ ValueErrorr#attention_head_size all_head_sizerhrr;attention_probs_dropout_probattention_dropoutr<output_dropoutLinearrdrerfdenser9r: is_causalrEs rHr/zAlbertAttention.__init__s     : : :a ?PVXhHi#F$6$6#78 4457  #)#=#= !--#)#5#59S9S#S !558P8PP//5 !#F,O,O!P jj)C)CDYYv1143E3EF 99V//1C1CDYYv1143E3EF YYv1163E3EF f&8&8f>S>STrIN hidden_statesrgrirLc |jdd}g|d|j}|j|j|j dd}|j |j|j dd}|j |j|j dd}tj|jjt} | |||||f|jsdn|jj|jd|\} } | j g|dj#} |j%| } |j'| } |j)|| z} | | fS)Nr*rrl)r=rh)rRr~rdviewrqrerfr get_interfacer#_attn_implementationrwrorrnrhreshapertrrr9) rFrrgrirT hidden_shape query_layer key_layer value_layerattention_interfacervrus rHrYzAlbertAttention.forwards} $))#2. CCbC$*B*BC 5djj/44lCMMaQRS 0DHH]+00,?II!QO 4djj/44lCMMaQRS (?(M(M KK , ,.E) %8      % $}}C$2H2H2J2JLL %  % ! \*k));;;;FFH jj- ))+6 nn][%@A L((rIN rZr[r\r r/r?r`r_rrtuplerYrarbs@rHryrysf|<48")||")))D0")+, ") u||U\\) * ")rIryc eZdZdeffd Z d dej dejdzdee de ej ej ffdZ d ej dej fd Z xZ S) AlbertLayerr#ct|||_|j|_d|_t j |j|j|_ t||_ t j|j|j|_t j|j|j|_t |j"|_t j&|j(|_y)Nrr&)r.r/r#chunk_size_feed_forward seq_len_dimrr9r{r:full_layer_layer_normry attentionrintermediate_sizeffn ffn_outputr hidden_act activationr;r<r=rEs rHr/zAlbertLayer.__init__s  '-'E'E$%'\\&2D2D&J_J_%`"(099V//1I1IJ))F$<$P>PQ !2!23zz&"<"<= rINrrgrirLc |j||fi|\}}t|j|j|j|}|j ||z}|Sr)rrff_chunkrrr)rFrrgriattention_output_rs rHrYzAlbertLayer.forwardse -dnn]NUfU!. MM  ( (      22:@P3PQ rIrcl|j|}|j|}|j|}|Sr)rrr)rFrrs rHrzAlbertLayer.ff_chunks3XX./ __Z0 __Z0 rIr)rZr[r\r r/r?r`r_rrrrYrrarbs@rHrrs >| > 48||))D0+,  u||U\\) * "%,,rIrc eZdZdeffd Z d dej dejdzdee de ej e ej zdffd Z xZ S) AlbertLayerGroupr#ct|tjt |j Dcgc] }t |c}|_ycc}wr)r.r/r ModuleListrangeinner_group_numr albert_layersrFr#rrGs rHr/zAlbertLayerGroup.__init__s= ]]vOeOeIf+gAK,?+gh+gsANrrgrirL.c Tt|jD]\}}|||fi|}|Sr) enumerater)rFrrgri layer_index albert_layers rHrYzAlbertLayerGroup.forwards< *343E3E)F R %K(Q&QM RrIrrrbs@rHrrsri|i48||))D0+,  u||eELL1136 7 rIrc zeZdZdeffd Z d dej dejdzdee de e zfdZ xZ S) AlbertTransformerr#c t|||_tj|j |j |_tjt|jDcgc] }t|c}|_ ycc}wr) r.r/r#rrr2r{embedding_hidden_mapping_inrrnum_hidden_groupsralbert_layer_groupsrs rHr/zAlbertTransformer.__init__sf  +-99V5J5JFL^L^+_(#%==TYZ`ZrZrTs1tq2B62J1t#u 1ts,B NrrgrirLc $|j|}t|jjD]R}t ||jj|jj z z }|j |||fi|}Tt|S)N)last_hidden_state)rrr#num_hidden_layersintrrr )rFrrgrii group_idxs rHrYzAlbertTransformer.forwards 88G t{{445 AA!>!>A^A^!^_`I?D44Y?M  ??rIr)rZr[r\r r/r?r`r_rrr rrYrarbs@rHrrs`v|v48@||@))D0@+, @ 5 @rIrcheZdZeZdZdZdZdZdZ e e dZ e jfdZxZS)AlbertPreTrainedModelalbertT)r attentionsct||t|tr t j |j yt|tryt j|jtj|jjdjdt j |jyy)zInitialize the weights.r*r)N)r. _init_weights isinstance AlbertMLMHeadinitzeros_biasr"copy_r(r?r@rRrAr,)rFrcrGs rHrz#AlbertPreTrainedModel._init_weights0s f% fm , KK $  0 1 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .2rI)rZr[r\r config_classbase_model_prefix_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrry_can_record_outputsr?no_gradrrarbs@rHrr#sOL N"&$% U]]_//rIrz2 Output type of [`AlbertForPreTraining`]. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ e ej dzed<dZ e ej dzed<y)AlbertForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). sop_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logits sop_logitsrr) rZr[r\r]rr?r___annotations__rrrrrrIrHrr;s} &*D%  d ")26u((4/6+/J!!D(/59M5**+d2926Je''(4/6rIrcReZdZeZdZddedeffd ZdejfdZ dejddfd Z e e e dd ej dzd ej"dzd ej dzd ej dzdej"dzdeedeezfdZxZS) AlbertModelrr#add_pooling_layercft||||_t||_t ||_|rItj|j|j|_ tj|_ nd|_ d|_ |j|_|jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N)r.r/r#r"rXrencoderrrr{poolerTanhpooler_activationrattn_implementation post_init)rFr#rrGs rHr/zAlbertModel.__init__Ys   *62(0 ))F$6$68J8JKDK%'WWYD "DK%)D "#)#>#>  rIrLc.|jjSrrXr4rFs rHget_input_embeddingsz AlbertModel.get_input_embeddingsos...rIrfNc&||j_yrr)rFrfs rHset_input_embeddingsz AlbertModel.set_input_embeddingsrs*/'rIrJrgr,r(rKric >|du|duz r td|j||||}t|j||}|j||fd|i|}|d} |j '|j |j | dddfnd} t| | S)Nz:You must specify exactly one of input_ids or inputs_embeds)r(r,rK)r#rKrgr(r)r pooler_output)r}rXr r#rrrr) rFrJrgr,r(rKriembedding_outputencoder_outputssequence_output pooled_outputs rHrYzAlbertModel.forwardus  -t"; <YZ Z?? L_l+ 3;;*)  '$,,   &   *!,VZVaVaVm..t{{?1a4;P/QRsw )-'  rI)T)NNNNN)rZr[r\r rrboolr/rr0rrrrrr?r^r_rrrrrYrarbs@rHrrTsL |,/bll/0",,040 .237260426 $ ##d*$ ))D0$ ((4/ $ &&- $ ((4/ $ +,$  $e +$  $ rIrz Albert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `sentence order prediction (classification)` head. ceZdZdddZdeffd ZdejfdZdejdd fd Z dejfd Z e e dd ejd zd ej d zdejd zdejd zdej d zdejd zdejd zdeedeezfdZxZS)AlbertForPreTraining(albert.embeddings.word_embeddings.weightpredictions.biaszpredictions.decoder.weightzpredictions.decoder.biasr#ct||t||_t ||_t ||_|jyr) r.r/rrr predictions AlbertSOPHeadsop_classifierrrEs rHr/zAlbertForPreTraining.__init__sB  !&) (0+F3 rIrLc.|jjSrrdecoderrs rHget_output_embeddingsz*AlbertForPreTraining.get_output_embeddings'''rInew_embeddingsNc&||j_yrrrFrs rHset_output_embeddingsz*AlbertForPreTraining.set_output_embeddingss#1 rIcB|jjjSrrrXr4rs rHrz)AlbertForPreTraining.get_input_embeddings{{%%555rIrJrgr,r(rKlabelssentence_order_labelric |j|f||||dd|} | dd\} } |j| } |j| } d}|u|st}|| j d|j j |j d}|| j dd|j d}||z}t|| | | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` sentence_order_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`. `0` indicates original order (sequence A, then sequence B), `1` indicates switched order (sequence B, then sequence A). Example: ```python >>> from transformers import AutoTokenizer, AlbertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2") >>> model = AlbertForPreTraining.from_pretrained("albert/albert-base-v2") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> sop_logits = outputs.sop_logits ```Trgr,r(rK return_dictNrlr*)rrrrr) rrrrrr#r1rrr)rFrJrgr,r(rKrrrioutputsrrprediction_scores sop_scores total_lossloss_fctmasked_lm_losssentence_order_losss rHrYzAlbertForPreTraining.forwards N$++  ))%'   *1!& ,,_=((7   "6"B')H%&7&<&->!?@ YYv1163H3HI yy!6!68I8IJ  !2!23rIrrLc|j|}|j|}|j|}|j|}|}|Sr)rrr9r)rFrr s rHrYzAlbertMLMHead.forwardsF =1  6 }5  ]3 )  rI rZr[r\r r/r?r`rYrarbs@rHrrs*4|4!U\\!ell!rIrc\eZdZdeffd Zdej dej fdZxZS)rr#ct|tj|j|_tj |j|j|_ yr) r.r/rr;classifier_dropout_probr=rr{ num_labels classifierrEs rHr/zAlbertSOPHead.__init__sB zz&"@"@A ))F$6$68I8IJrIrrLcJ|j|}|j|}|Sr)r=r)rFrdropout_pooled_outputlogitss rHrYzAlbertSOPHead.forward s% $ ] ;!67 rIrrbs@rHrrs,K|K U\\ellrIrc~eZdZdddZfdZdej fdZdej ddfd Zdejfd Z e e dd e jdzd e jdzd e jdzde jdzde jdzde jdzdeedeezfdZxZS)AlbertForMaskedLMrrrct||t|d|_t ||_|j yNF)r)r.r/rrrrrrEs rHr/zAlbertForMaskedLM.__init__-s7  !&EB (0 rIrLc.|jjSrrrs rHrz'AlbertForMaskedLM.get_output_embeddings6rrIrNc\||j_|j|j_yr)rrrrs rHrz'AlbertForMaskedLM.set_output_embeddings9s$#1 . 3 3rIcB|jjjSrrrs rHrz&AlbertForMaskedLM.get_input_embeddings=rrIrJrgr,r(rKrric :|jd|||||dd|}|d} |j| } d} |Ft} | | jd|jj |jd} t | | |j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, AlbertForMaskedLM >>> tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2") >>> model = AlbertForMaskedLM.from_pretrained("albert/albert-base-v2") >>> # add mask_token >>> inputs = tokenizer("The capital of [MASK] is Paris.", return_tensors="pt") >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> # retrieve index of [MASK] >>> mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0] >>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1) >>> tokenizer.decode(predicted_token_id) 'france' ``` ```python >>> labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"] >>> labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100) >>> outputs = model(**inputs, labels=labels) >>> round(outputs.loss.item(), 2) 0.81 ``` TrJrgr,r(rKr rNr*rr rrr) rrrrr#r1rrr) rFrJrgr,r(rKrrir sequence_outputsr rrs rHrYzAlbertForMaskedLM.forward@s^$++ ))%'   #1: ,,-=>  ')H%&7&<& 1` a classification loss is computed (Cross-Entropy). Tr)rN regressionsingle_label_classificationmulti_label_classificationr*r*r)rr=rr# problem_typerr-r?rDrrsqueezerrrrrr) rFrJrgr,r(rKrrir rr rrs rHrYz'AlbertForSequenceClassification.forwards$$++ ))%'      ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   rIr,)rZr[r\r r/rrr?r^r_rrrrrYrarbs@rHr.r.s | .237260426*.; ##d*; ))D0; ((4/ ; &&- ; ((4/ ;   4'; +,;  "E ); ; rIr.ceZdZdeffd Zee d dejdzdejdzdejdzdejdzdejdzd ejdzd e e d e e zfd ZxZS)AlbertForTokenClassificationr#cxt|||j|_t|d|_|j |j n |j }tj||_ tj|j|jj|_ |jyr$)r.r/rrrrr<rr;r=rr{r#rr)rFr#rrGs rHr/z%AlbertForTokenClassification.__init__s   ++!&EB --9  * *++ zz"9: ))F$6$6 8N8NO rINrJrgr,r(rKrrirLc H|j|f||||dd|}|d} |j| } |j| } d} |r?rir rr rBrCr ignored_indexr start_lossend_losss rHrYz"AlbertForQuestionAnswering.forward&s$++ ))%'   "!*#?#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   rIrrZr[r\r r/rrr?r^r_rrrrrYrarbs@rHr;r;s|.23726042637153 ##d*3 ))D03 ((4/ 3 &&- 3 ((4/ 3 ))D03 ''$.3 +,3  $e +3 3 rIr;ceZdZdeffd Zee d dejdzdejdzdejdzdejdzdejdzd ejdzd e e d e e zfd ZxZS)AlbertForMultipleChoicer#ct||t||_t j |j |_t j|jd|_ |jy)Nr) r.r/rrrr;rr=rr{rrrEs rHr/z AlbertForMultipleChoice.__init__`sV  !&) zz&"@"@A ))F$6$6: rINrJrgr,r(rKrrirLc ||jdn|jd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|j|f||||dd|} | d} |j | } |j | } | jd|} d} |t }|| |} t| | | j| jS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where *num_choices* is the size of the second dimension of the input tensors. (see *input_ids* above) Nrr*Tr r*) rRrrCrr=rrrrr)rFrJrgr,r(rKrri num_choicesr rr reshaped_logitsrrs rHrYzAlbertForMultipleChoice.forwardjsT-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  $++  ))%'      ]3 #}= ++b+6  ')HOV4D("!//))   rIr,rJrbs@rHrLrL^s|.237260426*.M ##d*M ))D0M ((4/ M &&- M ((4/ M   4'M +,M  $e +M M rIrL)rrrr"r.r7r;rL)Nr)Er]collections.abcr dataclassesrr?rtorch.nnrrrr r activationsr masking_utilsr modeling_outputsr rrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrrr utils.genericrrutils.output_capturingrconfiguration_albertr get_loggerrZloggerModuler"r`floatrwryrrrrrrrrrr"r.r7r;rL__all__rrIrHres$! AA&!6G&NMI5.   H %=ryy=N!% II% <<% % << % LL4' % T\ %%'(%8>)bii>)B#"))#Lryy"@ @:/O//.  77  7&G 'G G T \ 0\  \ ~!BII!* BII _ -_ _ D J &;J  J Z: #8: : z@ !6@ @ FZ 3Z Z z rI