^j]dZddlmZddlZddlZddlmZddlmZm Z m Z ddl m Z ddlmZmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(e"jRe*Z+dZ,dZ- d/dej\dej^dej^dej^dej^dzde0dzde0deefdZ1Gddej\Z2d Z3Gd!d"ej\Z4e Gd#d$eZ5e Gd%d&e5Z6e d'(Gd)d*e5eZ7e d+(Gd,d-e5Z8gd.Z9y)0zPyTorch CTRL model.)CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPastSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)capture_outputs) CTRLConfigcPdtjdd|dzz|z z }||zS)Nri')torchpow)posi d_model_size angle_ratess q/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/ctrl/modeling_ctrl.py angle_defnr%-s/eiiQ!V 'DEEK  cttj|tjj |j dtj|tjj |j d|}tj |dddddf}tj|dddddf}tj||gd}|S)N)dtyperrrdim) r%rarangeint64to unsqueezesincoscat)positionr"r( angle_radssinescosines pos_encodings r$positional_encodingr82s XU[[144U;EEaH \588?II!LJ IIjADqD) *Eii 1add7+,G99eW-26L r&modulequerykeyvalueattention_maskscalingdropoutkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)Nr)rrr*)ptrainingr) sizermatmul transposer functionalsoftmaxr?rD contiguous) r9r:r;r<r=r>r?r@ attn_weights attn_outputs r$eager_attention_forwardrMBs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$r&c8eZdZdfd Z ddeefdZxZS)MultiHeadAttentionc~t|||_|j|_|j |_||_d|_t|j |jz |_ |jdz|_ tj|j |j |_tj|j |j |_tj|j |j |_tj|j |j |_y)NTrB)super__init__confign_head num_headsn_embdr" layer_idx is_causalinthead_dimr>rLinearWqWkWvdenseselfrSrW __class__s r$rRzMultiHeadAttention.__init___s  "MM"D-->? }}d* ))D--t/@/@A))D--t/@/@A))D--t/@/@AYYt00$2C2CD r&r@c |jdd}g|d|j}|j|j|j dd} |j |j|j dd} |j |j|j dd} | |j| | |j\} } tj|jjt} | || | | |fd|jd|\} }| jg|dj!} |j#| } | |fS)Nr)rr)r?r>)shaperZr\viewrGr]r^updaterWr get_interfacerS_attn_implementationrMr>reshaperJr_)ravkq layer_pastr=r@ input_shape hidden_shape query_states key_states value_statesattention_interfacerLrKs r$forwardzMultiHeadAttention.forwardpsTggcrl 88b8$--8 wwqz|4>>q!D WWQZ__\2<>q!D  !'1'8'8\SWSaSa'b $J (?(M(M KK , ,.E) %8      % LL %  % ! \*k));;;;FFH jj- L((r&NNN__name__ __module__ __qualname__rRrrru __classcell__rbs@r$rOrO^s&E, #)+,#)r&rOctjtj||tjtj||Srv)r Sequentialr[ReLU)r"dffs r$point_wise_feed_forward_networkrs2 ==<5rwwy"))CQ]B^ __r&c8eZdZdfd Z ddeefdZxZS) EncoderLayerct|t|||_t |j |j |_tj|j d|_ tj|j d|_ tj|j|_tj|j|_y)NrWgư>eps)rQrRrOmulti_head_attentionrrVrffnr LayerNorm layernorm1 layernorm2Dropout resid_pdropdropout1dropout2r`s r$rRzEncoderLayer.__init__s $6v$S!26==&**M,,v}}$?,,v}}$? 6#5#56  6#5#56 r&r@c |j|}|j|||f||d|\}}|j|}||z}|j|} |j | } |j | } || z} | S)Nrnr=)rrrrrr) raxrnr=r@normedrL_out1out2 ffn_outputs r$ruzEncoderLayer.forwards#222    ")     QmmK0 ;t$XXd^ ]]:. j  r&rvrwrxr}s@r$rrs% 7  +, r&rcHeZdZUeed<dZdZdZdZdZ e e dZ fdZ xZS)CTRLPreTrainedModelrS transformerT) hidden_states attentionsct||t|trXt j |j t|jj|jtjyyrv) rQ _init_weights isinstance CTRLModelinitcopy_r7r8rS n_positionsr"rfloat)rar9rbs r$rz!CTRLPreTrainedModel._init_weightssX f% fi ( JJ##%89R9RTZTgTginitit%u  )r&)ryrzr{r__annotations__base_model_prefix_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrO_can_record_outputsrr|r}s@r$rrs> %N"&%( r&rceZdZfdZdZdZeee dde jdzde dzde jdzde jdzd e jdzd e jdzd e dzd eed efdZxZS)rc t|||j|_|j|_t j|j|j|_ t j|j|_ t jt|jDcgc]}t||c}|_t j"|j|j$|_|j)dt+|j,|jt.j0d|j3ycc}w)Nrrr7F) persistent)rQrRrVr"n_layer num_layersr Embedding vocab_sizewr embd_pdropr? ModuleListrangerhrlayer_norm_epsilon layernormregister_bufferr8rrr post_init)rarSr!rbs r$rRzCTRLModel.__init__s  "MM ..f//?zz&"3"34 5QWQ_Q_K`aa Vq Aabfmm9R9RS  /0B0BDDUDUW\WbWbcpu   bs,Ec|jSrvr)ras r$get_input_embeddingszCTRLModel.get_input_embeddingss vv r&c||_yrvr)ranew_embeddingss r$set_input_embeddingszCTRLModel.set_input_embeddingss r&N input_idspast_key_valuesr=token_type_ids position_ids inputs_embeds use_cacher@returnc ,||n|jj}| | td||j|}n | td|jdd} |jd} |j } |r|t |j}||jnd} |>tj| | d| ztj| }|jd}|4|j|} | tj|jz} nd} |!|jdkr|j!| d}t#|j|||| }|tj|jz}|j$j'| |j( |_|j$|ddf}||z| z}|j+|}|j,D]}||f||d |}|j/|}t1||r| Sd S) a Example: ```python >>> from transformers import AutoTokenizer, CTRLModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 5, 1280] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedsr)r)rS)r(device)rSrr=rrrr(r)last_hidden_stater)rSr ValueErrorrrerr get_seq_lengthrr,longr/npsqrtr"ndimrfr r7r.r(r?rrr)rarrr=rrrrr@ro batch_sizer past_lengthtoken_type_embeds causal_mask pos_embedsrrs r$ruzCTRLModel.forwards=D"+!6IDKKcd d   FF9-M  "TU U#))#2. "((+ %% 0*$++>O:I:Uo446[\   << [_{5RZ_ZdZdmstL'11!4L  % $~ 6  ):):!; ;  !   %.*=*=*A+00R@N(;;')+%   !2!233 !--00mFYFY0Z&&|Q7 % 25FF  ]3  A** M }5 &+/8O  >B  r&)NNNNNNN)ryrzr{rRrrrrrr LongTensorr FloatTensorboolrrrrur|r}s@r$rrs&  .2(,37260426!%\ ##d*\ \ ))D0 \ ((4/ \ &&- \ ((4/\ $;\ +,\  !\  \ r&rz The CTRL Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introcXeZdZddiZfdZee ddejdzde dzdejdzdejdzd ejdzd ejdzd ejdzd e dzd e ejzdeedefdZ dfd ZxZS)CTRLLMHeadModelzlm_head.weightztransformer.w.weightct||t||_t j |j |jd|_|jy)NTbias) rQrRrrrr[rVrlm_headrrarSrbs r$rRzCTRLLMHeadModel.__init__]sG  $V,yy0A0AM  r&Nrrr=rrrlabelsrlogits_to_keepr@rc l|j|f||||||d| } | d} t| tr t| dn| } |j | dd| ddf}d}|*|j ||fd|j ji| }t||| j| j| jS)ag labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLLMHeadModel >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> sequence_ids = model.generate(inputs["input_ids"]) >>> sequences = tokenizer.batch_decode(sequence_ids) >>> sequences ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,'] >>> outputs = model(**inputs, labels=inputs["input_ids"]) >>> round(outputs.loss.item(), 2) 9.21 >>> list(outputs.logits.shape) [1, 5, 246534] ```rr=rrrrrNr)losslogitsrrr) rrrYslicer loss_functionrSrrrrr)rarrr=rrrrrrr@transformer_outputsr slice_indicesrrs r$ruzCTRLLMHeadModel.forwardes\/d..   +))%'    ,A. 8B>SV8W~ot4]k mA}a,?@A  %4%% ;;11 D&/??-;;*55   r&c Tt||f|||d|}|jdd|S)N)rris_first_iterationr)rQprepare_inputs_for_generationpop)rarrrrr@ model_inputsrbs r$rz-CTRLLMHeadModel.prepare_inputs_for_generationsH w<  +1     )40r&) NNNNNNNNr)NNF)ryrzr{_tied_weights_keysrRrrrrr rrrYTensorrrrrurr|r}s@r$rrTs3+,BC.2(,37260426*.!%-.K ##d*K K ))D0 K ((4/ K &&- K ((4/K   4'K $;K ell*K +,K  K K \SXr&ra The CTRL Model transformer with a sequence classification head on top (linear layer). [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). c"eZdZfdZee ddejdzdedzdejdzdejdzdejdzdejdzd ejdzd e dzd e e d e fd ZxZS)CTRLForSequenceClassificationct|||j|_t||_t j |j|jd|_|jy)NFr) rQrR num_labelsrrrr[rV classifierrrs r$rRz&CTRLForSequenceClassification.__init__sR   ++$V,))FMM4??O r&Nrrr=rrrrrr@rc ^|j|f||||||d| } | d} |j| } ||jdd\} }n|jdd\} }|jj| dk7r t d|jjd}n|||jjk7j | jtj}tj|jd| jtj}||zjd}n.d}tj|jjd | tj| | j |f}d}||jj |j"dk(rd |j_nl|j"dkDrL|j$tj&k(s|j$tj(k(rd |j_nd |j_|jj d k(rIt+}|j"dk(r&||j-|j-}n|||}n|jj d k(r=t/}||j1d|j"|j1d}n,|jj d k(rt3}|||}t5||| j6| j8S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> import torch >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> labels = torch.tensor(1) >>> loss = model(**inputs, labels=labels).loss >>> round(loss.item(), 2) 0.93 ``` Example of multi-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained( ... "Salesforce/ctrl", problem_type="multi_label_classification" ... ) >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> num_labels = len(model.config.id2label) >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to( ... torch.float ... ) >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() # doctest: +IGNORE_RESULT ```rrNrrz=Cannot handle batch sizes > 1 if no padding token is defined.r)rz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)rrrr)rrrerS pad_token_idrr.rrint32r,argmaxlogger warning_oncerbry problem_typerr(rrYrsqueezerrfrrrr)rarrr=rrrrrr@rrrrsequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fcts r$ruz%CTRLForSequenceClassification.forwardsv/d..   +))%'    ,A. /  *3//"1*= 'J*7*=*=bq*A 'J ;; # # + a\] ] ;; # # +!#   "%)A)AAEEfmmUZU`U`aL!LL)!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6' -;;*55   r&)NNNNNNNN)ryrzr{rRrrrrr rrrrrrur|r}s@r$rrs.2(,37260426*.!%Y ##d*Y Y ))D0 Y ((4/ Y &&- Y ((4/Y   4'Y $;Y +,Y  "Y Y r&r)rrrr)Nrd):__doc__collections.abcrnumpyrrrtorch.nnrrrr r cache_utilsr r generationr masking_utilsr modeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrconfiguration_ctrlr get_loggerryrr%r8ModulerrrMrOrrrrrr__all__r&r$r s$ AA&.)/iiF& 85*   H % ,!% II% <<% % << % LL4' % T\ %%'(%85)5)p`%299%P/(y #y y x j)?j  jZ e $7e  e P cr&