^jpudZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z m Z dd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZddlmZmZddlmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*e$jVe,Z-Gddej\Z/ d2dej`dejbdejbdejbdejbdzde2de2fdZ3Gdd ej`Z4Gd!d"eZ5e"Gd#d$eZ6Gd%d&e6Z7e"Gd'd(e6Z8Gd)d*e6eZ9e"d+,Gd-d.e6Z:e"Gd/d0e6Z;gd1ZL(>9A=CCEL'+A+B(BCLw|dkk9::r,)rN) __name__ __module__ __qualname____doc__intr(r3 LongTensorr6 __classcell__r*s@r+r r -s]FsF3F'(04 ;((;!$;&&- ;;r,r modulequerykeyvaluer-scalingdropoutc tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)N)r2dtypeptrainingrr%) r3matmul transposer functionalsoftmaxfloat32torHrDrK contiguous) r?r@rArBr-rCrDkwargs attn_weights attn_outputs r+eager_attention_forwardrVJs<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r,ceZdZdZ d dededzffd Z d dejde dzdejdzd e d e ejejdze dzff d Z xZ S) OPTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNconfig layer_idxc t|||_|j|_|j |_|j|_|j|_ ||_ |-tjd|jjd|j|j z|_d|_|j|j z|jk7r&t#d|jd|j d|jdz|_t'j(|j|j|j|_t'j(|j|j|j|_t'j(|j|j|j|_t'j(|j|j|j|_y) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.Tz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩bias)r'r(rY hidden_size embed_dimnum_attention_heads num_headsattention_dropoutrD enable_biasrZlogger warning_oncer*r7head_dim is_causal ValueErrorrCrLineark_projv_projq_projout_proj)r)rYrZrSr*s r+r(zOPTAttention.__init__ds  ++33// !--"      !8!8 9:,,  $..8  MMDNN *t~~ =MdnnM]$T^^$4B8 }}d* iiTEUEUV iiTEUEUV iiTEUEUV  $..$..tGWGWX r, hidden_statespast_key_valuesr-output_attentionsreturnc \|j\}}}|j||jz} | j|d|j|j j dd} |j|} |j|} | j|d|j|j j dd} | j|d|j|j j dd} | |j| | |j\} } tj|jjt} | || | | |f|j sdn |j"dd|\} }| j%||dj'} |j)| } | |fS)z#Input shape: Batch x Time x ChannelrFrr%g?)rDrC)sizerlrCviewrarfrMrjrkupdaterZr get_interfacerY_attn_implementationrVrKrDreshaperRrm)r)rnror-rprSbsztgt_len_ query_states key_states value_statesattention_interfacerUrTs r+r6zOPTAttention.forwards(,,.Wa{{=1DLL@ #((b$..$--PZZ[\^_` [[/ {{=1 __S"dnndmmLVVWXZ[\ #((b$..$--PZZ[\^_`  &'6'='=j,X\XfXf'g $J (?(M(M KK , ,.E) %8      % $}}C$,, %  % ! \"))#w;FFH mmK0 L((r,N)NNF)r7r8r9r:rr;r(r3Tensorr booltupler6r=r>s@r+rXrXasG !%!Y!Y:!YL)-.2"' .)||.).) t+ .)  .) u||U\\D0%$,> ?.)r,rXceZdZd dededzffd Z ddejdejdzdedzde dzd ejdzd e e d ejfd Z xZS)OPTDecoderLayerNrYrZct||j|_t |||_|j |_|j|_t|j|_ tj|j|j|_tj|j|j |j"|_tj|j |j|j"|_tj|j|j|_y)N)rYrZelementwise_affiner\)r'r(r^r_rX self_attndo_layer_norm_beforerDr activation_function activation_fnr LayerNormlayer_norm_elementwise_affineself_attn_layer_normriffn_dimrcfc1fc2final_layer_norm)r)rYrZr*s r+r(zOPTDecoderLayer.__init__s ++%VyI$*$?$?!~~ #F$>$>?$&LL NNv/S/S% !99T^^V^^&BTBTU99V^^T^^&BTBTU " T^^PVPtPt ur,rnr-ro use_cacher/rSrqc |}|jr|j|}|jd||||d|\}}tjj ||j |j }||z}|js|j|}|j} |jd|jd}|}|jr|j|}|j|}|j|}|j|}tjj ||j |j }||zj| }|js|j|}|S)N)rnror/r-rIrF)rrrrrNrDrKshaperyrtrrrrru) r)rnr-rorr/rSresidualr|hidden_states_shapes r+r6zOPTDecoderLayer.forwardsw!  $ $ 55mDM*4>> '+%)     q --mt||VZVcVc-d  =0 (( 55mDM,11%--b-2D2DR2HI    $ $ 11-@M/ **=9 /  --mt||VZVcVc-d !M1778KL (( 11-@Mr,r)NNFN)r7r8r9rr;r(r3rr rr<rrr6r=r>s@r+rrsvyvS4Zv(/3(,!&04 3||3 t+3 3 $; 3 &&- 3-.3 3r,rcDeZdZUeed<dZdZdgZdZdZ dZ dZ dZ e edZy)OPTPreTrainedModelrYmodelTr)rn attentionsN)r7r8r9r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphrrX_can_record_outputsrr,r+rrsH &*#*+"&N!("r,rceZdZdZdeffd Zeee dde jdzde jdzde dzde jdzd edzd e jdzd eed efd ZxZS) OPTDecoderz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`OPTDecoderLayer`] Args: config: OPTConfig rYc 8t|||j|_|j|_|j|_|j |_|j|_tj|j|j|j |_ t|j |j|_|j|jk7r2tj |j|jd|_nd|_|j|jk7r2tj |j|jd|_nd|_|j&r=|j(s1tj*|j|j,|_nd|_tj0t3|j4Dcgc]}t7||c}|_d|_|j=ycc}w)NFr\r)rZ)r'r(rD layerdrop pad_token_id padding_idxmax_position_embeddingsmax_target_positions vocab_sizer Embeddingword_embed_proj_dim embed_tokensr r^embed_positionsri project_out project_inr_remove_final_layer_normrrr ModuleListrangenum_hidden_layersrlayersgradient_checkpointing post_init)r)rYir*s r+r(zOPTDecoder.__init__s  ~~ ))!..$*$B$B! ++LL):):Fs$  -t"; <YZ Z  !r9??2+>?I   --i8M 0*$++>O?N?Z?99;`a  !)M,?,?,BBJ"ZZ (;(;A(> S`SgSghN   <<A>L(>9A=CCEL'+;+<(<=L(;;')+  )).:JYe)f ?? & OOM:M% m6J6J(KK #,DKK"8  C}}&+jjn#&7)*) /#   M   , 11-@M    ' ,,];M&++  r,NNNNNN)r7r8r9r:rr(rrrr3r<rr FloatTensorrrrrr6r=r>s@r+rrs#y#J .2.2(,26!%04K ##d*K  t+K  K ((4/ K $; K &&-K +,K  !K  K r,rceZdZdeffd ZdZdZee dde jdzde jdzde dzd e jdzd edzd e jdzd eed efdZxZS)OPTModelrYcdt||t||_|j yr)r'r(rdecoderrr)rYr*s r+r(zOPTModel.__init__s&  !&)  r,c.|jjSrrrr)s r+get_input_embeddingszOPTModel.get_input_embeddingss||(((r,c&||j_yrrr)rBs r+set_input_embeddingszOPTModel.set_input_embeddingss$) !r,Nrr-rorrr/rSrqc |jd||||||d|}t|j|j|j|j S)Nrr-r/rorr)rrornrr)rrrrornr) r)rr-rorrr/rSdecoder_outputss r+r6zOPTModel.forwardsi4@4<<4 )%+' 4 4 '-??+;;)77&11   r,r)r7r8r9rr(rrrrr3r<rr rrrrrr6r=r>s@r+rrsy )*.2.2(,26!%04 ##d*  t+   ((4/  $;  &&- +,  !  r,rc<eZdZddiZfdZdZdZee dde jdzde jdzd e dzd e jdzd e jdzd edzd e jdzdee jzdeedeezfdZxZS)OPTForCausalLMzlm_head.weightz!model.decoder.embed_tokens.weightct||t||_t j |j |jd|_|jyNFr\) r'r(rrrrirrlm_headrrs r+r(zOPTForCausalLM.__init__sK  f% yy!;!;V=N=NUZ[  r,cB|jjjSrrrrrs r+rz#OPTForCausalLM.get_input_embeddingszz!!...r,c:||jj_yrrrs r+rz#OPTForCausalLM.set_input_embeddings*/ 'r,Nrr-rorlabelsrr/logits_to_keeprSrqc |jjd||||||d| } | j} t|tr t | dn|} |j | dd| ddfj} d}|*|jd| ||jjd| }t|| | j| j| jS)an labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, OPTForCausalLM >>> model = OPTForCausalLM.from_pretrained("facebook/opt-350m") >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious. I'm just a little bit of a weirdo." ```rN)logitsrrlossrrornrr)rrr isinstancer;slicerrR loss_functionrYrrrornr)r)rr-rorrrr/rrSoutputsrn slice_indicesrrs r+r6zOPTForCausalLM.forwardsJ,>4::+=+=, )%+' , ,  11 8B>SV8W~ot4]k mA}a,?@ALLN  %4%%pVFt{{OeOepiopD%#33!//))   r,)NNNNNNNr)r7r8r9_tied_weights_keysr(rrrrr3r<rr rrr;rrrrr6r=r>s@r+rrs*,OP/0.2.2(,26*.!%04-.< ##d*<  t+<  < ((4/ <   4' < $;< &&-< ell*< +,<  ' '< < r,ra The OPT Model transformer with a sequence classification head on top (linear layer). [`OPTForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introceZdZdeffd Zee ddejdzdejdzde dzdejdzdejdzd e dzd ejdzd e e d eezfd ZdZdZxZS)OPTForSequenceClassificationrYct|||j|_t||_t j |j|jd|_|jyr) r'r( num_labelsrrrrirscorerrs r+r(z%OPTForSequenceClassification.__init__sT   ++f% YYv994??QVW  r,Nrr-rorrrr/rSrqc |j|f|||||d|} | j} |j| } ||jdd\} } n|jdd\} } |jj | dk7r t d|jj d}n|||jj k7j| jtj}tj|jd| jtj}||zjd}n.d}tj|jj d| tj| | j |f}d}||jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}t7||| j8| j:| j< S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). ror-r/rrNr%rz=Cannot handle batch sizes > 1 if no padding token is defined.rF)rrHz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)rrrrrYrrhrQrr3int32arangeargmaxrdrer*r7 problem_typerrHr5r;rsqueezerrurrrornr)r)rr-rorrrr/rStransformer_outputsrnr batch_sizesequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fcts r+r6z$OPTForSequenceClassification.forward's&8Btzz 8 +)%' 8 8 ,== M*  *3//"1*= 'J*7*=*=bq*A 'J ;; # # + a\] ] ;; # # +!#   "%)A)AAEEfmmUZU`U`aL!LL)!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6/ /??-;;*55   r,cB|jjjSrrrs r+rz1OPTForSequenceClassification.get_input_embeddings|rr,c:||jj_yrrrs r+rz1OPTForSequenceClassification.set_input_embeddingsrr,)NNNNNNN)r7r8r9rr(rrr3r<rr rrrrrr6rrr=r>s@r+rrsy.237(,26*.!%04Q ##d*Q ))D0Q  Q ((4/ Q   4' Q $;Q &&-Q +,Q  1 1Q Q f/0r,rc:eZdZdeffd Zee ddejdzdejdzde dzdejdzdejdzd ejdzd e dzd ejdzd e e d eezfdZdZdZxZS)OPTForQuestionAnsweringrYct||t||_t j |j d|_|jyr$) r'r(rrrrir qa_outputsrrs r+r(z OPTForQuestionAnswering.__init__s@  f% ))F$>$>B r,Nrr-rorstart_positions end_positionsrr/rSrqc 8|j|f|||||d| } | j} |j| } | jdd\} }| j dj } |j dj }d}||t |jdkDr|j d}t |jdkDr|j d}| jd}|jd|j| j}|jd|j| j}t|}|| |}|||}||zdz }t|| || j| j S) a Example: ```python >>> from transformers import AutoTokenizer, OPTForQuestionAnswering >>> import torch >>> torch.manual_seed(4) # doctest: +IGNORE_RESULT >>> tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m") >>> # note: we are loading a OPTForQuestionAnswering from the hub here, >>> # so the head will be randomly initialized, hence the predictions will be random >>> model = OPTForQuestionAnswering.from_pretrained("facebook/opt-350m") >>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" >>> inputs = tokenizer(question, text, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> answer_offset = len(tokenizer(question)[0]) >>> predict_answer_tokens = inputs.input_ids[ ... 0, answer_offset + answer_start_index : answer_offset + answer_end_index + 1 ... ] >>> predicted = tokenizer.decode(predict_answer_tokens) >>> predicted ' a nice puppet' ```rrrFr1Nr) ignore_indexr%)r start_logits end_logitsrnr)rrrsplitrrRlenrtclamprQrrrrnr)r)rr-rorrrrr/rSrrnrrr total_loss ignored_indexr start_lossend_losss r+r6zOPTForQuestionAnswering.forwards\8Btzz 8 +)%' 8 8 ,== /#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EHHWO)//=ADDV]]SM']CH!,@J M:H$x/14J+%!-;;*55   r,cB|jjjSrrrs r+rz,OPTForQuestionAnswering.get_input_embeddingsrr,c:||jj_yrrrs r+rz,OPTForQuestionAnswering.set_input_embeddingsrr,)NNNNNNNN)r7r8r9rr(rrr3r<rr rrrrrr6rrr=r>s@r+rrsy.237(,263715!%04Q ##d*Q ))D0Q  Q ((4/ Q ))D0 Q ''$.Q $;Q &&-Q +,Q  - -Q Q f/0r,r)rrrrr)rs)=r:collections.abcrr3rtorch.nnrrr activationsr cache_utilsr r generationr masking_utilsr modeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrconfiguration_optr get_loggerr7rdrr ModulerfloatrVrXrrrrrrr__all__rr,r+r8s$ AA!.)/B9 G&RR75(   H %;BLL;H% II% <<% % << % LL4' %  %%.T)299T)nE0EP    { #{ |( !( ( VQ 'Q h  c0#5c0 c0Lb00b0b0J r,