^j2dZddlZddlmZddlmZddlZddlmZddlm Z m Z m Z ddl m Zdd lmZmZdd lmZmZmZdd lmZdd lmZmZdd lmZddlmZmZm Z m!Z!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/ddl0m1Z1m2Z2ddl3m4Z4e,jje6Z7d5dZ8GddejrZ:GddejrZ;GddeZ<GddejrZ=e*Gdd e%Z>e*d!"eGd#d$e)Z?e*Gd%d&e>Z@e*d'"Gd(d)e>eZAe*d*"Gd+d,e>eZBe*d-"Gd.d/e>ZCe*Gd0d1e>ZDe*Gd2d3e>ZEgd4ZFy)6zPyTorch OpenAI GPT-2 model.N)Callable) dataclass)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FNget_activation)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Conv1D) ModelOutputauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)OutputRecordercapture_outputs) GPT2Configc ||jddz}tj||jdd|z}|||z}tj j |d}|j|j}tj j|||j}tj||} | jdd} | |fS)Ndim)ptrainingr$) sizetorchmatmul transposer functionalsoftmaxtypedtypedropoutr-) modulequerykeyvalueattention_maskscalingr7kwargs attn_weights attn_outputs q/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/gpt2/modeling_gpt2.pyeager_attention_forwardrB6s**R.D(<<s}}R'<=GL!#n4 ==((2(>L $$U[[1L==((6??([L,,|U3K''1-K  $$ceZdZd fd ZddZ ddeejdzdedzdejdzdejdzdejdzd e dzd eejeejzd ffd Z xZ S) GPT2AttentionNct|||_|j|_|j |_|j|j z|_|j|_|j|j z|jk7r&td|jd|j d|j|_ |j|_ |j|_ ||_ ||_d|_|jr|jdz|_|jr+|xjt!|jdzzc_|jrNt#d|jz|j|_t#|j|j|_n(t#d|jz|j|_t#|j|j|_t+j,|j.|_t+j,|j2|_| |_y) Nz=`embed_dim` must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g?r(r$r.r )super__init__config hidden_size embed_dimnum_attention_heads num_headshead_dim split_size ValueErrorscale_attn_weightsscale_attn_by_inverse_layer_idxreorder_and_upcast_attnis_cross_attention layer_idxr=floatrc_attnq_attnc_projrDropout attn_pdrop attn_dropout resid_pdrop resid_dropout is_causal)selfrIrTrU __class__s rArHzGPT2Attention.__init__Ls  ++33$..8 .. ==4>> )T^^ ;OPTP^P^O_`NN#2'  #)";";/5/U/U,'-'E'E$"4"  " "==$.DL  / / LLE$..1"45 5L  " " T^^!3T^^DDK @DK T^^!3T^^DDKT^^T^^< JJv'8'89ZZ(:(:;//rCc|j\}}}}|j\} } } } tj||z|| tj|j} t |jj d5|jd|||jddjd|| } } tj| | j| jd|j} | j|||| } ddd|| |z} tjj| d} | jtjk7r t!d | j |j} |j#| } tj$| |}|jd d }|| fS#1swYxYw) N)r6deviceF)enabledr'r)r)betaalphar*zDError with upcasting, attn_weights does not have dtype torch.float32r$r.)r/r0emptyfloat32rcr r5reshaper2baddbmmrVr=rr3r4r6 RuntimeErrorr\r1)r`r9r:r;r<bszrM q_seq_lendk_ k_seq_lenr?qkr@s rA_upcast_and_reordered_attnz(GPT2Attention._upcast_and_reordered_attnqs(- %Y 2 XXZ1i{{3?IyPUP]P]fkfrfrs ELL--u = V==Y3S]]2r5J5R5RSUWY[d5eqA ==qwwy!'')RS[_[g[ghL'//Y 9UL V  %'.8L}},,\r,B     .ef f#((5 ((6 ll<7 !++Aq1 L(() V Vs B F<<G hidden_statespast_key_valuesr<encoder_hidden_statesencoder_attention_maskoutput_attentionsreturn.c |du}|St|trA|jj|j} |r |j } n|j } n|} |rt|ds td|j|} |}|J rH j|jj} | j|jj} n$|j|j|jd\} } g| j ddd|j"}| j%|j'dd} | j%|j'dd} n|j|j|jd\} } } g| j ddd|j"}| j%|j'dd} | j%|j'dd} g| j ddd|j"}| j%|j'dd} ||r|?|r= s; j)| | |j\} } |rd|j|j<|j*j,dk(}t/j0|j*j,t2}|r$|j4r|j7| | | |\}}n?||| | | |f|j8r|j:j<nd |j>d |\}}|j@g|j dd djC}|jE|}|jG|}||fS) NrXzIf class is used as cross attention, the weights `q_attn` have to be defined. Please make sure to instantiate class with `GPT2Attention(..., is_cross_attention=True)`.r.r*r'r$Teager)r7r=r))$ isinstancer is_updatedgetrUcross_attention_cacheself_attention_cachehasattrrPrXlayerskeysvaluesrWsplitrOshaperNviewr2updaterI_attn_implementationr get_interfacerBrSrsr-r\r,r=ri contiguousrYr^)r`rtrur<rvrwrxr>rTr~curr_past_key_values query_states key_states value_statesshape_kvshape_q using_eagerattention_interfacer@r?s rAforwardzGPT2Attention.forwardsx3$>  &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$ 4* p ;;}5L3N*z188HMM 3::4>>JQQ +/;;7L+M+S+STXTcTcij+S+k( LFZ--cr2FBF F'__X6@@AF +00:DDQJ 59[[5O5U5UVZVeVekl5U5m 2L*lB))#2.BBDMMBH#2<z';'B'B:|]a]k]k'l $J !=A**4>>:kk66'A (?(M(M KK , ,.E)  477(,(G(Gj,) %K)< ) 04}}))++#  ) ) %K*k))F;+<+residualr@rocross_attn_outputfeed_forward_hidden_statess rArzGPT2Block.forwards! -0 "  +)     Q$h. ,4!12 =dVDZZ%H ..}=M#'#6#6 /-&;'= $7$ q%'88M  -0 %)XXm%<" #== rCrr) rrrrHrr0rr rrrrrs@rArrs .$)-3759;?!&/U../$6//))D0 / %||d2 / !& 1 1D 8 /$;/ /rCrceZdZdZdeffd Z d dejdejdzdejfdZ xZ S) GPT2SequenceSummarya Compute a single vector summary of a sequence hidden states. Args: config ([`GPT2Config`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. rIcft|t|dd|_|jdk(rtt j |_t|drq|jret|dr(|jr|jdkDr |j}n |j}t j|j||_t|dd}|r t|nt j |_t j |_t|dr3|j"dkDr$t j$|j"|_t j |_t|d r5|j(dkDr%t j$|j(|_yyy) N summary_typelastrsummary_use_projsummary_proj_to_labelsrsummary_activationsummary_first_dropoutsummary_last_dropout)rGrHgetattrrNotImplementedErrorrIdentitysummaryrrr num_labelsrJLinearr activation first_dropoutrrZ last_dropoutr)r`rI num_classesactivation_stringras rArHzGPT2SequenceSummary.__init__SsU #FNFC    && %{{} 6- .63J3Jv78V=Z=Z_e_p_pst_t$// $00 99V%7%7EDL#F,@$GIZN3D$E`b`k`k`m[[] 62 38T8TWX8X!#F,H,H!ID KKM 61 2v7R7RUV7V " 6+F+F GD 8W 2rCNrt cls_indexryc|jdk(r |dddf}n|jdk(r |dddf}n|jdk(r|jd}n|jd k(r|Atj|d ddddf|jd dz tj }nX|j dj d}|jd |jdz z|jdfz}|jd |jd }n|jdk(rt|j}|j|}|j|}|j!|}|S)ak Compute a single vector summary of a sequence hidden states. Args: hidden_states (`torch.FloatTensor` of shape `[batch_size, seq_len, hidden_size]`): The hidden states of the last layer. cls_index (`torch.LongTensor` of shape `[batch_size]` or `[batch_size, ...]` where ... are optional leading dimensions of `hidden_states`, *optional*): Used if `summary_type == "cls_index"` and takes the last token of the sequence as classification token. Returns: `torch.FloatTensor`: The summary of the sequence hidden states. rNr'firstrmeanr$r*r.r))r6r'r)rrr0 full_likerlong unsqueezeexpandr+r/gathersqueezerrrrr)r`rtroutputs rArzGPT2SequenceSummary.forwardpsn    &"1b5)F   ' )"1a4(F   & ("''A'.F   + - !OO!#rr1*-!''+a/** &//3==bA %,,Uimmo6I-JmN`N`acNdMf-fg "))"i8@@DF   & (% %##F+f%(""6* rCr) rrr__doc__r%rHr0r LongTensorrrrs@rArr9sQ2HzH<VZ)"..);@;K;Kd;R)   )rCrceZdZUeed<dZdZdgZdgZdZ dZ dZ dZ e eeddeed dd Zd d gZej&fd ZxZS)GPT2PreTrainedModelrI transformerTrruz.attnr$) layer_nameindexz.crossattention)rt attentionscross_attentionsz attn.biaszcrossattention.biasc t||t|trat j |j d|jj|jt j|jt|trx|jD]d\}}|dk(s t j |d|jjtjd|jjzz fyy)zInitialize the weights.r|)rstdNz c_proj.weightr.)rG _init_weightsr}rinitnormal_weightrIinitializer_rangebiaszeros_rnamed_parametersmathsqrtn_layer)r`r8namer,ras rArz!GPT2PreTrainedModel._init_weightss f% ff % LLSdkk6S6S T{{& FKK( fo .!224 va?*LL$++2O2ORVR[R[\]`d`k`k`s`s\sRt2tu v /rC)rrrr%__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_attention_backend_can_compile_fullgraphrr"rE_can_record_outputs"_keys_to_ignore_on_load_unexpectedr0no_gradrrrs@rArrs %&*#$ #4"5N"&!"$]waP*=EV^_`+67L)M&U]]_vvrCrz^ Base class for outputs of models predicting if two sentences are consecutive or not. ) custom_introc"eZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ e dzed<dZ eej dzed<dZeej dzed <y) GPT2DoubleHeadsModelOutputa\ loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss. mc_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mc_labels` is provided): Multiple choice classification loss. logits (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). mc_logits (`torch.FloatTensor` of shape `(batch_size, num_choices)`): Prediction scores of the multiple choice classification head (scores for each choice before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. Nlossmc_losslogits mc_logitsrurtr)rrrrrr0rrrrr rur rtrrrCrArrs &*D%  d ")(,GU   %,'+FE   $+*.Iu  4'.$(OUT\(59M5**+d2926Je''(4/6rCrcNeZdZfdZdZdZeee dde jdzde dzde jdzde jdzd e jdzd e jdzd e jdzd e jdzd edzdefdZxZS) GPT2Modelc t|||j|_t j |j |j|_t j |j|j|_ t j|j|_ t jt|jDcgc]}t!||c}|_t j$|j|j&|_d|_|j,|_|j/ycc}w)N)rUrF)rGrHrJrKr Embedding vocab_sizewtemax_position_embeddingswperZ embd_pdropdrop ModuleListrangenum_hidden_layersrhrrln_fgradient_checkpointingr post_init)r`rIiras rArHzGPT2Model.__init__s  ++<< 1 14>>B<< > >OJJv001 fNfNfHgh1 &A >hiLLV5N5NO &+#$*$?$?!  is Ec|jSrr)r`s rAget_input_embeddingszGPT2Model.get_input_embeddingss xxrCc||_yrr)r`new_embeddingss rAset_input_embeddingszGPT2Model.set_input_embeddingss !rCN input_idsrur<token_type_ids position_ids inputs_embedsrvrwrryc  | jdd| jdd| | td|G|j|||j} |j d| d}|j d} n0|#|jdd} |j d} n td||j d| d}| r^|t |j}|jjr0t|ts t|t |j}||j|}|V||jnd} tj|j d |j | z}|j!d}|j#|}||j%|jz}|!|j&d kr|j | d}t)|j|||| }d}|t+|j||| }||j|}||z}|j-|}d| d dz|jdfz}t/|j0D]/\}}|||j2r |j4s|nd||f|| |d| }1|j7|}|j |}| r|nd}t9||S) input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) rxNoutput_hidden_stateszDYou cannot specify both input_ids and inputs_embeds at the same timer'rz5You have to specify either input_ids or inputs_embeds)rIr$rcr)rIr&r<rur%)rIr&r<rvr)rwrr%)last_hidden_stateru)poprP%warn_if_padding_and_no_attention_maskr/rrrrIrr}rrget_seq_lengthr0arangercrrtondimrrr enumeraterrr-rr)r`r#rur<r$r%r&rvrwrr> input_shape batch_sizepast_seen_tokensposition_embedsrt causal_masktoken_type_embeds output_shaperblocks rArzGPT2Model.forwards<  &- )40  ]%>cd d  "  6 6y. Q#..*K!r;r?;I"+J  &',,.s3K&,,Q/JTU U  %+00[_EN &".dkk"B{{..z/Sf7g"5o|[_[f[fGg"h   HHY/M  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L((<0%(:(:=;O;O(PP   %.*=*=*A+00R@N(;;')+%  "& ,%>{{+5&; & "  % $ 8 ),==M -0 {12.-2D2DR2H1JJ !$&&) HAu!(,(C(C \`%  (>#)  M  -0 %**<8 -6/D8++  rC) NNNNNNNNN)rrrrHrr"r!r#rr0rr rrrrrrrs@rAr r s$" .2(,3726042659;?!%r ##d*r r ))D0 r ((4/ r &&- r ((4/r  %||d2r !& 1 1D 8r $;r  3r  r rCr z The GPT2 Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ceZdZddiZfdZee ddejdzde dzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd ejdzde dzde ejzdefdZxZS)GPT2LMHeadModellm_head.weighttransformer.wte.weightct||t||_t j |j |jd|_|jyNFr) rGrHr rrrn_embdrlm_headrr`rIras rArHzGPT2LMHeadModel.__init__sG  $V,yy0A0AN  rCNr#rur<r$r%r&rvrwlabelsrlogits_to_keepryc |j|f|||||||| d| } | j}t| tr t | dn| }|j |dd|ddf}d}| *|j || fd|jji| }t||| j| j| j| jS)ai input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, input_ids_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` )rur<r$r%r&rvrwrNr)rrrurtrr)rr+r}intslicerC loss_functionrIrrrurtrr)r`r#rur<r$r%r&rvrwrErrFr>transformer_outputsrt slice_indicesrrs rArzGPT2LMHeadModel.forwardsFJZIYIY  J +))%'"7#9 J  J ,== 8B>SV8W~ot4]k mA}a,?@A  %4%% ;;11 D1/??-;;*550AA   rC) NNNNNNNNNNr)rrr_tied_weights_keysrHrrr0rr rrrrHrrrrs@rAr<r<xs@+,DE.2(,3726042659;?*.!%-.C ##d*C C ))D0 C ((4/ C &&- C ((4/C  %||d2C !& 1 1D 8C   4'C $;C ell*C  +C C rCr<a The GPT2 Model transformer with a language modeling and a multiple-choice classification head on top e.g. for RocStories/SWAG tasks. The two heads are two linear layers. The language modeling head has its weights tied to the input embeddings, the classification head takes as input the input of a specified classification token index in the input sequence). c`eZdZddiZfdZee ddejdzde dzdejdzdejdzd ejdzd ejdzd ejdzd ejdzd ejdzde dzde fdZ xZS)GPT2DoubleHeadsModelr=r>ct||d|_t||_t j |j|jd|_ t||_ |jy)Nr$FrA) rGrHrr rrrrBrrCrmultiple_choice_headrrDs rArHzGPT2DoubleHeadsModel.__init__s\  $V,yy0A0AN $7$?! rCNr#rur<r$r%r& mc_token_idsrE mc_labelsrryc |j|f|||||| d| } | j} |j| }|j| |j d}d}| At }||j d|jd| j d}d}||j|j}|dddddfj}|dddfj}t }||j d|jd|j d}t||||| j| j| jS)af input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) mc_token_ids (`torch.LongTensor` of shape `(batch_size, num_choices)`, *optional*, default to index of the last token of the input): Index of the classification token in each input sequence. Selected in the range `[0, input_ids.size(-1) - 1]`. labels (`torch.LongTensor` of shape `(batch_size, input_ids_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids`. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]` mc_labels (`torch.LongTensor` of shape `(batch_size)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices]` where *num_choices* is the size of the second dimension of the input tensors. (see *input_ids* above) Example: ```python >>> import torch >>> from transformers import AutoTokenizer, GPT2DoubleHeadsModel >>> tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2") >>> model = GPT2DoubleHeadsModel.from_pretrained("openai-community/gpt2") >>> # Add a [CLS] to the vocabulary (we should train it also!) >>> num_added_tokens = tokenizer.add_special_tokens({"cls_token": "[CLS]"}) >>> # Update the model embeddings with the new vocabulary size >>> embedding_layer = model.resize_token_embeddings(len(tokenizer)) >>> choices = ["Hello, my dog is cute [CLS]", "Hello, my cat is cute [CLS]"] >>> encoded_choices = [tokenizer.encode(s) for s in choices] >>> cls_token_location = [tokens.index(tokenizer.cls_token_id) for tokens in encoded_choices] >>> input_ids = torch.tensor(encoded_choices).unsqueeze(0) # Batch size: 1, number of choices: 2 >>> mc_token_ids = torch.tensor([cls_token_location]) # Batch size: 1 >>> outputs = model(input_ids, mc_token_ids=mc_token_ids) >>> lm_logits = outputs.logits >>> mc_logits = outputs.mc_logits ```rur<r$r%r&rr'N.r$)rrrr rurtr)rr+rCrQrrrr/r0rcrrrurtr)r`r#rur<r$r%r&rRrErSrr>rKrt lm_logitsr rloss_fctlm_loss shift_logits shift_labelss rArzGPT2DoubleHeadsModel.forwardswBJZIYIY  J +))%' J  J ,== LL/ --m\JRRSUV   ')Hy~~b)..2DEy~~VXGYZG  YYy//0F$S#2#q[1<<>L!#qr'?557L')H|00\5F5Fr5JK\M^M^_aMbcG)/??-;;*55  rC) NNNNNNNNNN)rrrrMrHrrr0rr rrrrrrs@rArOrOs++,DE.2(,3726042604*.-1!%c ##d*c c ))D0 c ((4/ c &&- c ((4/c &&-c   4'c ##d*c $;c  $c c rCrOa The GPT2 Model transformer with a sequence classification head on top (linear layer). [`GPT2ForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZfdZee d dejdzdedzdejdzdejdzdejdzdejdzd ejdzd e dzd e fd Z xZ S)GPT2ForSequenceClassificationct|||j|_t||_t j |j|jd|_|jyr@) rGrHrr rrrrBscorerrDs rArHz&GPT2ForSequenceClassification.__init__]sR   ++$V,YYv}}dooEJ  rCNr#rur<r$r%r&rErryc |j|f||||||d| } | j} |j| } ||jdd\} }n|jdd\} }|jj | dk7r t d|jj d}n|||jj k7j| jtj}tj|jd| jtj}||zjd}n.d}tj|jj d| tj| | j |f}d}||jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}t7||| j8| j:| j< S)aB input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rUNr.r$z=Cannot handle batch sizes > 1 if no padding token is defined.r')rcr6z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r* regressionsingle_label_classificationmulti_label_classification)rrrurtr)rr+r^rrI pad_token_idrPr0rcr0int32r/argmaxlogger warning_oncerar problem_typerr6rrHrrrrrrrurtr)r`r#rur<r$r%r&rErr>rKrtrr4sequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsrrWs rArz%GPT2ForSequenceClassification.forwardfs@JZIYIY  J +))%' J  J ,== M*  *3//"1*= 'J*7*=*=bq*A 'J ;; # # + a\] ] ;; # # +!#   "%)A)AAEEfmmUZU`U`aL!LL)!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6/ /??-;;*55   rCNNNNNNNN)rrrrHrrr0rr rrrrrrs@rAr\r\Ns.2(,37260426*.!%^ ##d*^ ^ ))D0 ^ ((4/ ^ &&- ^ ((4/^   4'^ $;^  *^ ^ rCr\ceZdZfdZee d dejdzdedzdejdzdejdzdejdzdejdzd ejdzd e dzd e fd Z xZ S)GPT2ForTokenClassificationct|||j|_t||_t |dr|j |j }n't |dr|j |j}nd}tj||_ tj|j|j|_ |jy)Nclassifier_dropouthidden_dropoutg?)rGrHrr rrrrrsrrZr7rrJ classifierr)r`rIrrras rArHz#GPT2ForTokenClassification.__init__s   ++$V, 6/ 0V5N5N5Z!'!:!:  V- .63H3H3T!'!6!6 !$ zz"45 ))F$6$68I8IJ rCNr#rur<r$r%r&rErryc |j|f||||||d| } | j} |j| } |j| } d} |W|j | j }t }|| jd|j|jd} t| | | j| jS)aR input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). rUNr')rrrtr) rr+r7rtr0rcrrrrrtr)r`r#rur<r$r%r&rErr>rKrtrrrWs rArz"GPT2ForTokenClassification.forwards@JZIYIY  J +))%' J  J ,==  ]3 /  YYv}}-F')HFKKDOOoutputssequence_outputrr~r total_loss ignored_indexrW start_lossend_losss rArz GPT2ForQuestionAnswering.forward$s6>NT=M=M > ))%' >  > "331#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""="@"@ATAT"U=%%'(1, - 5 5b 9 < rs" $! AA&1CC)J9G# HE*   H %%*W)BIIW)tbii"?*?F`"))`F&v/&v&vR  77  74N #N N b P )?P  P fr .r r j  j $7j  j ZL !4L L ^L 2L L ^ rC