^jXPdZddlZddlmZmZmZddlmZmZm Z m Z m Z m Z m Z ddlmZddlmZmZddlmZd d lmZmZmZmZmZmZmZmZGd d eZGd deZGddeZ GddeZ!GddeZ"GddeZ#GddeZ$GddeZ%gdZ&y)zPyTorch CamemBERT model.N)BCEWithLogitsLossCrossEntropyLossMSELoss),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)Unpack)TransformersKwargsauto_docstring)can_return_tuple)RobertaForCausalLMRobertaForMaskedLMRobertaForMultipleChoiceRobertaForQuestionAnswering RobertaForSequenceClassificationRobertaForTokenClassification RobertaModelRobertaPreTrainedModelceZdZdZy)CamembertPreTrainedModelrobertaN)__name__ __module__ __qualname__base_model_prefixz/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/camembert/modular_camembert.pyrr,s!r#rc eZdZy)CamembertModelN)rrr r"r#r$r&r&0sr#r&cteZdZdddZfdZee ddejdzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd ejdzde e de ejezfdZxZS)CamembertForMaskedLM)roberta.embeddings.word_embeddings.weight lm_head.biaszlm_head.decoder.weightzlm_head.decoder.biascLt|||`t|d|_yNFadd_pooling_layersuper__init__ camembertr&rselfconfig __class__s r$r2zCamembertForMaskedLM.__init__:#  N%fF r#N input_idsattention_masktoken_type_ids position_ids inputs_embedsencoder_hidden_statesencoder_attention_masklabelskwargsreturnc t|j|f||||||dd| } | d} |j| } d} |a|j| j}t }|| j d|j j|j d} t| | | j| jS)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)r:r;r<r=r>r? return_dictrNlosslogits hidden_states attentions) rlm_headtodevicerviewr6 vocab_sizer rIrJ)r5r9r:r;r<r=r>r?r@rAoutputssequence_outputprediction_scoresmasked_lm_lossloss_fcts r$forwardzCamembertForMaskedLM.forward@s:$,,   ))%'"7#9    "!* LL9  YY0778F')H%&7&<&;?*.5 ##d*5 ))D05 ((4/ 5 &&- 5 ((4/ 5  %00475 !& 1 1D 85   4'5 +,5  u|| ~ -5 5 r#r(c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) "CamembertForSequenceClassificationcLt|||`t|d|_yr-r0r4s r$r2z+CamembertForSequenceClassification.__init__{r8r#Nr9r:r;r<r=r@rArBc |j|f||||dd|}|d} |j| } d} ||j| j}|jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt} |j dk(r&| | j|j} n| | |} n|jj dk(r=t} | | jd |j |jd } n,|jj dk(rt} | | |} t!| | |j"|j$ S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Tr:r;r<r=rDrN regressionsingle_label_classificationmulti_label_classificationrErF)r classifierrLrMr6 problem_type num_labelsdtyperWlongintrsqueezerrNrr rIrJ r5r9r:r;r<r=r@rArPrQrHrGrTs r$rUz*CamembertForSequenceClassification.forwards6$,,  ))%'   "!*1  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   r#NNNNNN)rrr r2rrrWrXrYrrrZr[r rUr\r]s@r$r_r_zsG .237260426*.C ##d*C ))D0C ((4/ C &&- C ((4/ C   4'C +,C  u|| 7 7C C r#r_c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) CamembertForMultipleChoicecLt|||`t|d|_y)NTr.r0r4s r$r2z#CamembertForMultipleChoice.__init__s#  N%fE r#Nr9r;r:r@r<r=rArBc "||jdn|jd}|!|jd|jdnd} |!|jd|jdnd} |!|jd|jdnd} |!|jd|jdnd} |1|jd|jd|jdnd} |j| f| | | | dd|}|d}|j |}|j |}|jd|}d}|.|j |j}t}|||}t|||j|jS)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. NrcrET)r<r;r:r=rDrF) shaperNsizerdropoutrgrLrMrr rIrJ)r5r9r;r:r@r<r=rA num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsrP pooled_outputrHreshaped_logitsrGrTs r$rUz"CamembertForMultipleChoice.forwardsV-6,Aiooa(}GZGZ[\G] CLCXINN2,>?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  $,,  *..,      ]3 / ++b+6  YY556F')HOV4D("!//))   r#ro)rrr r2rrrWrXrYrrrZr[r rUr\r]s@r$rqrqsF .22637*.0426P ##d*P ((4/P ))D0 P   4' P &&- P ((4/P +,P  u|| 8 8P P r#rqc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) CamembertForTokenClassificationcLt|||`t|d|_yr-r0r4s r$r2z(CamembertForTokenClassification.__init__&r8r#Nr9r:r;r<r=r@rArBc ~|j|f||||dd|}|d} |j| } |j| } d} |W|j| j}t } | | j d|j|j d} t| | |j|jS)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrbrNrErF) rrwrgrLrMrrNrir rIrJrns r$rUz'CamembertForTokenClassification.forward,s2$,,  ))%'   "!*,,71  YYv}}-F')HFKKDOO= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) TrbrrcrE)dimN) ignore_indexr)rG start_logits end_logitsrIrJ) r qa_outputssplitrm contiguouslenrvclamprr rIrJ)r5r9r:r;r<r=rrrArPrQrHrr total_loss ignored_indexrT start_lossend_losss r$rUz%CamembertForQuestionAnswering.forwardjs0$,,  ))%'   "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   r#)NNNNNNN)rrr r2rrrWrXrYrrrZr[r rUr\r]s@r$rrcsG .2372604263715> ##d*> ))D0> ((4/ > &&- > ((4/ > ))D0> ''$.> +,>  u|| ; ;> > r#rceZdZdddZfdZee ddejdzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd ejdzde e ejdzde dzde ejzdeede ejezfdZxZS)CamembertForCausalLMr)r*r+cLt|||`t|d|_yr-r0r4s r$r2zCamembertForCausalLM.__init__r8r#Nr9r:r;r<r=r>r?r@past_key_values use_cachelogits_to_keeprArBc |d} |j|f||||||| | dd | } | j}t| tr t | dn| }|j |dd|ddf}d}|*|j d|||jjd| }t||| j| j| j| jS)aq token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, CamembertForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("almanach/camembert-base") >>> config = AutoConfig.from_pretrained("almanach/camembert-base") >>> config.is_decoder = True >>> model = CamembertForCausalLM.from_pretrained("almanach/camembert-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) r:r;r<r=r>r?rrrD)rHr@rO)rGrHrrIrJcross_attentionsr")rlast_hidden_state isinstancerlslicerK loss_functionr6rOrrrIrJr)r5r9r:r;r<r=r>r?r@rrrrArPrI slice_indicesrHrGs r$rUzCamembertForCausalLM.forwards`  I@L  A ))%'"7#9+ A  A  11 8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD0#33!//))$55   r#) NNNNNNNNNNr)rrr rVr2rrrWrXrYrZboolrlr[rrrrUr\r]s@r$rrst"M . G .237260426:>;?*.BF!%-.O ##d*O ))D0O ((4/ O &&- O ((4/ O  %0047O !& 1 1D 8O   4'O uU%6%6784?O $;O ell*O +,O  u|| @ @O O r#r)rr(rqrr_rr&r)'__doc__rWtorch.nnrrrmodeling_outputsrrr r r r r processing_utilsrutilsrr utils.genericrroberta.modeling_robertarrrrrrrrrr&r(r_rqrrr__all__r"r#r$rs AA'7-   "5" \ C -C LL )IL ^Y !9Y x; &C; |G $?G T] -] @ r#