^j .dZddlZddlZddlmZddlmZddlmZddlm Z m Z m Z ddl m Z dd lmZmZdd lmZdd lmZmZdd lmZdd lmZmZddlmZej:eZGddej@Z!Gddej@Z"GddejFZ$GddejFZ%GddeZ&eGddeZ'Gdde'Z(edGd d!e'Z)ed"Gd#d$e'e Z*d$dgZ+y)%z/PyTorch TrOCR decoder model (based on RoBERTa).N)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)auto_docstringlogging) TrOCRConfigcveZdZdZdedeffd Z d dejdedejdzffd ZxZ S) TrOCRLearnedPositionalEmbeddingzN This module learns positional embeddings up to a fixed maximum size. num_embeddings embedding_dimcNd|_t| ||jz|yN)offsetsuper__init__)selfrr __class__s s/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/trocr/modeling_trocr.pyrz(TrOCRLearnedPositionalEmbedding.__init__*s$  $++5}EN input_idspast_key_values_length position_idsc$|a|jdd\}}tj|||ztj|jj j |d}n|jd}t|%||jzS)z3`input_ids' shape is expected to be [bsz x seqlen].Nr)dtypedevicer) shapetorcharangelongweightr(expand unsqueezerforwardr)rr#r$r%bszseq_lenr s r!r1z'TrOCRLearnedPositionalEmbedding.forward0s  $??2A.LC <<&(>(HPUPZPZcgcncncucufS"o (11!4Lw|dkk9::r")rN) __name__ __module__ __qualname____doc__intrr+Tensorr1 __classcell__r s@r!rr%sWFsF3Fmq ; ;?B ;V[VbVbeiVi ; ;r"rc `eZdZdZd dededededzffd Zdejffd Z xZ S) TrOCRScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. rr padding_idx embed_scaleNc6t||||||_yN)rrr?)rrrr>r?r s r!rz!TrOCRScaledWordEmbedding.__init__Fs  D&r"r#c<t|||jzSrA)rr1r?)rr#r s r!r1z TrOCRScaledWordEmbedding.forwardJswy)D,<,<<c|t|d|_||_||_|j ||||_yr)rrrrr> get_embeddingweights)rrGrr>r s r!rz+TrOCRSinusoidalPositionalEmbedding.__init__Qs;  *&))- T r"rc|dz}tjd|dz z }tjtj|tj j | z}tj|tj j jd|jdz}tjtj|tj|gdj|d}|dzdk(r-tj|tj|dgd}| d||ddf<|jtjS) z Build sinusoidal embeddings. This matches the implementation in tensor2tensor, but differs slightly from the description in Section 3.5 of "Attention Is All You Need". ri'r)r'rdimr)N)mathlogr+expr,int64rDr0catsincosviewzerostoget_default_dtype)rrr>half_dimembs r!rIz0TrOCRSinusoidalPositionalEmbedding.get_embeddingXs !A%hhuoA.ii XU[[AGGISDPQll>=CCEOOPQRUXUbUbcdUeeii338a@EEnVXY 1  !))S%++na"@AqIC  ""#C Q vve--/00r"r#r$c|j\}}|j||j|j|j}|jdz|z}|j ||j jdkDr,|j ||j|j|_|j jd|jdj||dj}|S)Nrrr)) size"create_position_ids_from_input_idsr>rWr(rJrIr index_selectrUdetach)rr#r$r2r3r%max_posxs r!r1z*TrOCRSinusoidalPositionalEmbedding.forwardks ~~' W>>y$JZJZ\rsvv    ""Q&0 << 7T\\->->q-A#A--gt7I7I4K[K[\DL LL % %a):):2)> ? D DS'SU V ] ] _r"c|j|j}tj|dj ||z|z}|j |zS)z Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. rrL)ner8r+cumsumtype_asr-)rr#r>r$maskincremental_indicess r!r]zETrOCRSinusoidalPositionalEmbedding.create_position_ids_from_input_ids}sW||K(,,.$||Da8@@FI__cgg"'')K77r"rA)r)r4r5r6r7r8r staticmethodrIr+no_gradr9r1r]r:r;s@r!rFrFNsNUcU#UCRVJU1c1#1CRVJ11$U]]_s$_` 8 847 8QTW[Q[ 8r"rFcZeZdZdZ ddedededzdedzdedzdedzd edzd edzd edzffd Z dd ejdejdzde dzdejdzdedzde ejejdze ejdzff dZ xZ S)TrOCRAttentionz>Multi-headed attention from 'Attention Is All You Need' paper.N embed_dim num_headskdimvdimdropout is_decoderbiasis_cross_attention layer_idxc Pt |||_||n||_||n||_||_||_||z|_|j|z|jk(std|jd|d|jdz|_ ||_ | |_ tj|j|||_tj|j|||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rr)rrrlrnrormrphead_dim ValueErrorscalingrqrtrLineark_projv_projq_projout_proj) rconfigrlrmrnrorprqrrrsrtr s r!rzTrOCRAttention.__init__s " ,D)  ,D) " !Y.  )T^^;MdnnM]^;b" }}d* $"ii 94@ ii 94@ ii 94@  )YTB r" hidden_stateskey_value_statespast_key_valuesattention_maskoutput_attentionsreturnc z |du}|j\}} } |j||jz} d} |St|trA|j j |j} |r |j} n|j} n|} |r|n|}|rK|I| rG j|jj}| j|jj}n|j|}|j|}|j|d|j |j"j%dd}|j|d|j |j"j%dd}|K j'|||j\}}|r)t|trd|j |j<||j zd|j"f}| j|| |j |j"j%dd} | j(|} |j(|}|j(|}|jd}t+j,| |j%dd}|j||j z| |fk7r/t/d||j z| |fd|j|{|j|d| |fk7r#t/d |d| |fd|j|j||j | ||z}|j||j z| |}t0j2j5|d }|r?|j||j | |}|j||j z| |}nd}t0j2j7||j6|j8 }t+j,||}|j||j z| |j"fk7r7t/d ||j | |j"fd|j|j||j | |j"}|j%dd}|j)|| | }|j;|}||fS) z#Input shape: Batch x Time x ChannelNFr)rrTz$Attention weights should be of size z , but is z!Attention mask should be of size rLptrainingz `attn_output` should be of size )r\r}ry isinstancer is_updatedgetrtcross_attention_cacheself_attention_cachelayerskeysvaluesr{r|rUrmrw transposeupdatereshaper+bmmrxr functionalsoftmaxrprr~)rrrrrrkwargsrsr2tgt_lenrl query_statesrcurr_past_key_valuescurrent_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r!r1zTrOCRAttention.forwards.T9"/"4"4"6Wi{{=1DLL@   &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML^4J;;~6L#b$..$--PZZ[\^_`J',,S"dnndmmT^^_`bcdL*+?+F+FzS_aeaoao+p( L%*_FY*ZAEO..t~~>DNN*B > #((gt~~t}}U__`acde +|++Z8 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<$>?"(";";$&LL$@!    ... 88777700#'# !D ,.<<+GD (99T^^V-C-CD99V33T^^D " T^^ <r"rrencoder_hidden_statesencoder_attention_maskrr use_cachec *|} |j||||\}} tjj||j|j}| |z}|j |}d} |g|} |j |||||\}} tjj||j|j}| |z}|j|}|} |j|j|}tjj||j|j}|j|}tjj||j|j}| |z}|j|}|f} |r| | | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrrrN)rrrrr) rrrrprrrrrrrrr) rrrrrrrrrresidualself_attn_weightscross_attn_weightsoutputss r!r1zTrOCRDecoderLayer.forward<s2!,0>>'+)/ ,:, ( ( --mt||VZVcVc-d  =0 11-@ " ,$H040A0A+!65 /"3 1B1 -M-MM11-4< >Gr"rA)NNNNFT) r4r5r6rrr+r9rrr1r:r;s@r!rrs"={"=N/3596:(,).!%G||G t+G %||d2 G !& t 3 G  G $;G$;Gr"rc&eZdZUeed<dZdZdgZy)TrOCRPreTrainedModelrmodelTrN)r4r5r6r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modulesr"r!rrs &*#,-r"rcDeZdZdZdeffd Z ddZxZS) TrOCRDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`TrOCRDecoderLayer`] Args: config: TrOCRConfig rc `t|||j|_|j|_|j |_|jrtj|jnd}t|j|j|j ||_ |jr&t|j |j|_n@t%|j |j zdz|j|j |_|j&r%t)j*|j|_nd|_t)j,t/|j0Dcgc]}t3||c}|_d|_|j9ycc}w)NrC)r?r)rtF)rrrpdecoder_layerdrop layerdrop pad_token_idr>scale_embeddingrNsqrtrr= vocab_size embed_tokensuse_learned_position_embeddingsrmax_position_embeddingsembed_positionsrFlayernorm_embeddingrr ModuleListrangedecoder_layersrrgradient_checkpointing post_init)rrr?ir s r!rzTrOCRDecoder.__init__sP  ~~ 11!..7=7M7Mdii 2 23SV 4   v1143C3CQ\   1 1#B6CaCacicucu#vD #E..1A1AAAE""  $D  % %')||F4F4F'GD $'+D $mmUZ[a[p[pUq$rPQ%6v%K$rs &+#  %ss3F+c B||n|jj}| | n|jj} ||n|jj}| | n|jj} | | t d|"|} |j d| jd}n| |dddddf} n t d|jr%|jr|rtjdd}|rd|b||jjr4tt|jt|jnt|j}||jnd} ||j!|}|jj"r|j%| | }n|j%|| }|j'|j(}||z}|j*|j+|}t,j.j1||j0|j }t3|j||| }||t5|j||| }| rd nd}|rd nd}|r|d nd}t7|j8D]j\}}| r||fz }|jr%t;j<g}||j>kr?||||||||}|d}|sV||dfz }|b||dfz }l| r||fz }| stAd|||||fDStC|||||S)aU Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer)zEYou have to specify either decoder_input_ids or decoder_inputs_embedsz^`use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`...F)rr)r$r)r inputs_embedsrr)rrrrr)rrrrrrc3$K|]}|| ywrAr).0vs r! z'TrOCRDecoder.forward..[s=s)last_hidden_staterr attentionscross_attentions)"rroutput_hidden_statesr return_dictrxrUr*rrlogger warning_onceis_encoder_decoderr rget_seq_lengthrrrrWr(rrrrpr r enumeraterr+randrrr)rr#rrrrrrrrrrinputr$ embed_posrall_hidden_statesall_self_attnsall_cross_attentionsidx decoder_layerdropout_probability layer_outputss r!r1zTrOCRDecoder.forwardsz@2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKst t  "E!r5;;r?;I  &!!Q(+Ede e  & &4==##t" 0)4 8V8V$L $DlZ^ZeZeFfg!5  FUE`!?!?!Afg   --i8M ;; 6 6,,UKa,bI,,YOe,fILL!5!56 % 1  # # / 44]CM --mt||VZVcVc-d +;;')+   ! ,1G1S%>{{+5&; & "#7BD0d&7>> from transformers import ( ... TrOCRConfig, ... TrOCRProcessor, ... TrOCRForCausalLM, ... ViTConfig, ... ViTModel, ... VisionEncoderDecoderModel, ... ) >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> # TrOCR is a decoder model and should be used within a VisionEncoderDecoderModel >>> # init vision2text model with random weights >>> encoder = ViTModel(ViTConfig()) >>> decoder = TrOCRForCausalLM(TrOCRConfig()) >>> model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder) >>> # If you want to start from the pretrained model, load the checkpoint with `VisionEncoderDecoderModel` >>> processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-handwritten") >>> model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-handwritten") >>> # load image from the IAM dataset >>> url = "https://fki.tic.heia-fr.ch/static/img/a01-122-02.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())).convert("RGB") >>> pixel_values = processor(image, return_tensors="pt").pixel_values >>> text = "industry, ' Mr. Brown commented icily. ' Let us have a" >>> # training >>> model.config.decoder_start_token_id = processor.tokenizer.eos_token_id >>> model.config.pad_token_id = processor.tokenizer.pad_token_id >>> model.config.vocab_size = model.config.decoder.vocab_size >>> labels = processor.tokenizer(text, return_tensors="pt").input_ids >>> outputs = model(pixel_values, labels=labels) >>> loss = outputs.loss >>> round(loss.item(), 2) 5.30 >>> # inference >>> generated_ids = model.generate(pixel_values) >>> generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> generated_text 'industry, " Mr. Brown commented icily. " Let us have a' ```N) r#rrrrrrrrrrr)r)losslogitsrrrr)rrrrrrrrrUrrrrrr)rr#rrrrrrrrrrrrrrloss_fctoutputs r!r1zTrOCRForCausalLM.forwardsBP2C1N-TXT_T_TqTq$8$D $++JjJj &1%:>(,26*.!%)-,0#'m ##d*m  t+m  %0047 m !& 0 04 7 m  m ((4/m   4'm $;m  $;m #Tkm D[m  2 2m m r"r),r7rNr+rtorch.nnr activationsr cache_utilsrrr generationr masking_utilsr r modeling_layersr modeling_outputsrrmodeling_utilsrutilsrrconfiguration_trocrr get_loggerr4r Embeddingrr=ModulerFrkrrrrr__all__rr"r!r(s&6 %!CC)J9l-,,   H %;bll;8 =r|| =9898xJ2RYYJ2Zl2l^.?..X 'X v-.-- H +_H  H V 5 6r"