^ju{dZddlZddlmZddlmZmZmZddlmZ ddl m Z ddl m Z ddlmZmZmZmZmZmZmZdd lmZdd lmZdd lmZmZmZdd lmZd dl m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'ejPe)Z*Gdde"Z+Gdde%Z,Gdde!Z-Gdde#Z.eGddeZ/Gdde$Z0edGdd e/e Z1eGd!d"e/Z2Gd#d$ejfZ4ed%Gd&d'e/Z5eGd(d)e/Z6eGd*d+e/Z7Gd,d-ejfZ8eGd.d/e/Z9gd0Z:y)1zPyTorch RoBERTa model.N)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)gelu)GenerationMixin),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuple)BertCrossAttentionBertEmbeddings BertLayer BertModelBertSelfAttention) RobertaConfigc eZdZfdZ d dej dzdej dzdej dzdej dzdef dZe d Z e d d Z xZ S) RobertaEmbeddingsct|||`|`|j|_t j |j|j|j|_y)N) padding_idx) super__init__ pad_token_idposition_embeddingsr"nn Embeddingmax_position_embeddings hidden_sizeselfconfig __class__s v/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/roberta/modular_roberta.pyr$zRobertaEmbeddings.__init__-sT     $!..#%<<  * *F,>,>DL\L\$  N input_idstoken_type_ids position_ids inputs_embedspast_key_values_lengthc|<||j||j|}n|j||j}||j}n|jdd}|\}}|t |drm|j j |jj|jdd} tj| d|} | j||}n:tj|tj|jj}||j|}|j!|} || z} |j#|} | | z} |j%| } |j'| } | S)Nr2rr)dimindexdtypedevice)"create_position_ids_from_input_idsr"&create_position_ids_from_inputs_embedssizehasattrr2tor<expandshapetorchgatherzeroslongr3word_embeddingstoken_type_embeddingsr& LayerNormdropout) r,r1r2r3r4r5 input_shape batch_size seq_lengthbuffered_token_type_idsrI embeddingsr&s r/forwardzRobertaEmbeddings.forward8s  $#FFt//1G   $JJ=Z^ZjZjk  #..*K',,.s3K!, J  !t-.*.*=*=*@*@ATAT*U*\*\]i]o]opq]rtv*w'*/,,7NTU]i*j'!8!?!? J!W!&[ SWSdSdSkSk!l   00;M $ : :> J"%:: "66|D"55 ^^J/ \\*- r0c|jdd}|d}tj|dz||zdztj|j}|j dj |S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr7rr:r)r?rDarangerGr< unsqueezerB)r4r"rLsequence_lengthr3s r/r>z8RobertaEmbeddings.create_position_ids_from_inputs_embedshsp$((*3B/ %a.|| !O_{:Q>ejjYfYmYm %%a(// <r= __classcell__r.s@r/r r ,s  .2260426&' .##d*.((4/.&&- . ((4/ . !$ .`==" 8 8r0r c eZdZy)RobertaSelfAttentionNr^r_r`r0r/rgrgr0rgc eZdZy)RobertaCrossAttentionNrhrir0r/rlrlrjr0rlc eZdZy) RobertaLayerNrhrir0r/rnrnrjr0rncneZdZeZdZdZdZdZdZ dZ e e e dZej fdZxZS)RobertaPreTrainedModelrobertaT) hidden_states attentionscross_attentionsct||t|tr t j |j yt|tryt j|jtj|jjdjdt j |jyy)zInitialize the weightsr7)rr7N)r# _init_weights isinstance RobertaLMHeadinitzeros_biasr copy_r3rDrSrCrBr2)r,moduler.s r/rvz$RobertaPreTrainedModel._init_weightss f% fm , KK $  1 2 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .3r0)r^r_r`r config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrnrgrl_can_record_outputsrDno_gradrvrdres@r/rprpsX L!&*#N"&%*1 U]]_//r0rpc eZdZdfd ZxZS) RobertaModelc&t|||yN)r#r$)r,r-add_pooling_layerr.s r/r$zRobertaModel.__init__s v&r0)T)r^r_r`r$rdres@r/rrs ''r0rzS RoBERTa Model with a `language modeling` head on top for CLM fine-tuning. ) custom_introceZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde e e jdzde dzdee jzdeede e jezfdZxZS)RobertaForCausalLM)roberta.embeddings.word_embeddings.weight lm_head.biaszlm_head.decoder.weightzlm_head.decoder.biasct|||jstj dt |d|_t||_|jy)NzOIf you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`Fr r#r$ is_decoderloggerwarningrrqrxlm_head post_initr+s r/r$zRobertaForCausalLM.__init__sL    NNl m#FeD $V,  r0c.|jjSrrdecoderr,s r/get_output_embeddingsz(RobertaForCausalLM.get_output_embeddings||###r0c&||j_yrrr,new_embeddingss r/set_output_embeddingsz(RobertaForCausalLM.set_output_embeddings- r0Nr1attention_maskr2r3r4encoder_hidden_statesencoder_attention_masklabelspast_key_values use_cachelogits_to_keepkwargsreturnc |d} |j|f||||||| | dd | } | j}t| tr t | dn| }|j |dd|ddf}d}|*|j d|||jjd| }t||| j| j| j| jS)am token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base") >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base") >>> config.is_decoder = True >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) rr2r3r4rrrr return_dict)logitsr vocab_size)lossrrrrrsrtri)rqlast_hidden_staterwrYslicer loss_functionr-rr rrrrsrt)r,r1rr2r3r4rrrrrrroutputsrr slice_indicesrrs r/rQzRobertaForCausalLM.forwards`  I@L  A ))%'"7#9+ A  A  11 8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD0#33!//))$55   r0) NNNNNNNNNNr)r^r_r`_tied_weights_keysr$rrrrrDrarbtupleboolrYTensorrrr rQrdres@r/rrs#N . $..237260426:>;?*.BF!%-.O ##d*O ))D0O ((4/ O &&- O ((4/ O  %0047O !& 1 1D 8O   4'O uU%6%6784?O $;O ell*O +,O  u|| @ @O O r0rceZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde e dee jezfdZxZS)RobertaForMaskedLMrrrct|||jrtj dt |d|_t||_|jy)NznIf you want to use `RobertaForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Frrr+s r/r$zRobertaForMaskedLM.__init__.sR     NN1  $FeD $V,  r0c.|jjSrrrs r/rz(RobertaForMaskedLM.get_output_embeddings=rr0c&||j_yrrrs r/rz(RobertaForMaskedLM.set_output_embeddings@rr0Nr1rr2r3r4rrrrrc t|j|f||||||dd| } | d} |j| } d} |a|j| j}t }|| j d|j j|j d} t| | | j| jS)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rr2r3r4rrrrNr7rrrrrs) rqrrAr<rviewr-rr rrrs)r,r1rr2r3r4rrrrrsequence_outputprediction_scoresmasked_lm_lossloss_fcts r/rQzRobertaForMaskedLM.forwardCs:$,,   ))%'"7#9    "!* LL9  YY0778F')H%&7&<&;?*.5 ##d*5 ))D05 ((4/ 5 &&- 5 ((4/ 5  %00475 !& 1 1D 85   4'5 +,5  u|| ~ -5 5 r0rc(eZdZdZfdZdZxZS)rxz*Roberta Head for masked language modeling.ct|tj|j|j|_tj |j|j|_tj|j|j|_ tjtj|j|_y)N)eps)r#r$r'Linearr*denserJlayer_norm_eps layer_normrr ParameterrDrFr{r+s r/r$zRobertaLMHead.__init__s YYv1163E3EF ,,v'9'9v?T?TUyy!3!3V5F5FG LLV->->!?@ r0c |j|}t|}|j|}|j|}|Sr)rrrrr,featuresrxs r/rQzRobertaLMHead.forwards; JJx  G OOA  LLOr0r^r_r`__doc__r$rQrdres@r/rxrx}s4Ar0rxz RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) RobertaForSequenceClassificationct|||j|_||_t |d|_t ||_|jyNFr) r#r$ num_labelsr-rrqRobertaClassificationHead classifierrr+s r/r$z)RobertaForSequenceClassification.__init__sJ   ++ #FeD 3F; r0Nr1rr2r3r4rrrc |j|f||||dd|}|d} |j| } d} ||j| j}|jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt} |j dk(r&| | j|j} n| | |} n|jj dk(r=t} | | jd |j |jd } n,|jj dk(rt} | | |} t!| | |j"|j$ S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trr2r3r4rrNr regressionsingle_label_classificationmulti_label_classificationr7r)rqrrAr<r- problem_typerr;rDrGrYrsqueezerrrrrrrs r,r1rr2r3r4rrrrrrrs r/rQz(RobertaForSequenceClassification.forwards6$,,  ))%'   "!*1  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   r0NNNNNN)r^r_r`r$rrrDrarbrrrrrrQrdres@r/rrs .237260426*.C ##d*C ))D0C ((4/ C &&- C ((4/ C   4'C +,C  u|| 7 7C C r0rc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) RobertaForMultipleChoicect||t||_t j |j |_t j|jd|_ |jy)Nr) r#r$rrqr'Dropouthidden_dropout_probrKrr*rrr+s r/r$z!RobertaForMultipleChoice.__init__sV  #F+ zz&"<"<= ))F$6$6: r0Nr1r2rrr3r4rrc "||jdn|jd}|!|jd|jdnd} |!|jd|jdnd} |!|jd|jdnd} |!|jd|jdnd} |1|jd|jd|jdnd} |j| f| | | | dd|}|d}|j |}|j |}|jd|}d}|.|j |j}t}|||}t|||j|jS)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nrr7T)r3r2rr4rr) rCrr?rqrKrrAr<rr rrrs)r,r1r2rrr3r4r num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsr pooled_outputrreshaped_logitsrrs r/rQz RobertaForMultipleChoice.forwardsV-6,Aiooa(}GZGZ[\G] CLCXINN2,>?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  $,,  *..,      ]3 / ++b+6  YY556F')HOV4D("!//))   r0r)r^r_r`r$rrrDrarbrrrrr rQrdres@r/rrs.22637*.0426P ##d*P ((4/P ))D0 P   4' P &&- P ((4/P +,P  u|| 8 8P P r0rc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) RobertaForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyr)r#r$rrrqclassifier_dropoutrr'rrKrr*rrr,r-rr.s r/r$z&RobertaForTokenClassification.__init__Ps   ++#FeD )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ r0Nr1rr2r3r4rrrc ~|j|f||||dd|}|d} |j| } |j| } d} |W|j| j}t } | | j d|j|j d} t| | |j|jS)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrrNr7r) rqrKrrAr<rrrrrrrsrs r/rQz%RobertaForTokenClassification.forward^s2$,,  ))%'   "!*,,71  YYv}}-F')HFKKDOO= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) Trrrr7rWN) ignore_indexr)r start_logits end_logitsrrrs) rqrsplitr contiguouslenr?clamprrrrrs)r,r1rr2r3r4rrrrrrrr  total_loss ignored_indexr start_lossend_losss r/rQz#RobertaForQuestionAnswering.forwards0$,,  ))%'   "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   r0)NNNNNNN)r^r_r`r$rrrDrarbrrrrrrQrdres@r/rrs.2372604263715> ##d*> ))D0> ((4/ > &&- > ((4/ > ))D0> ''$.> +,>  u|| ; ;> > r0r)rrrrrrrrp);rrDtorch.nnr'rrrrry activationsr generationr modeling_outputsr r r r rrrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrbert.modeling_bertrrrrrconfiguration_robertar get_loggerr^rr rgrlrnrprrrModulerxrrrrr__all__rir0r/r s AA&).&@@-ll0   H %\8\8~ ,  .  9 /_//2'9'  i /i  i XR /R R jBII, Q '=Q  Q h] 5] ] @C $:C C L ,K "8K K \ r0