^jLdZddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl mZdd lmZdd lmZmZmZmZmZdd lmZdd lmZdd lmZmZddlmZej@e!Z"GddejFZ$GddejFZ%GddejFZ&GddejFZ'GddejFZ(GddejFZ)GddejFZ*GddeZ+Gd d!ejFZ,Gd"d#ejFZ-eGd$d%eZ.eGd&d'e.Z/ed()Gd*d+e.Z0eGd,d-e.Z1Gd.d/ejFZ2eGd0d1e.Z3gd2Z4y)3zPyTorch LiLT model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)create_bidirectional_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)auto_docstringlogging) LiltConfigc:eZdZfdZ ddZdZdZxZS)LiltTextEmbeddingsct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j |_tj|j|j |j*|_ y)N padding_idxeps position_idsrF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandrselfconfig __class__s q/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/lilt/modeling_lilt.pyr#zLiltTextEmbeddings.__init__+s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  "..#%<<  * *F,>,>DL\L\$  c&|I|6|j||jj|j}n|j |}||j }n|j dd}|:t j|t j|jj}||j|}|j|}||z}|j|}||z }|j|}|j|}||fS)Nr dtypedevice)"create_position_ids_from_input_idsrtor?&create_position_ids_from_inputs_embedssizer3zeroslongrr(r,r*r-r1) r7 input_idstoken_type_idsr inputs_embeds input_shaper, embeddingsr*s r:forwardzLiltTextEmbeddings.forward?s  $#FFyRVRbRbcff$$   $JJ=Y  #..*K',,.s3K  !"[[EJJtO`O`OgOghN   00;M $ : :> J"%:: "66|D)) ^^J/ \\*- <''r;c|j|j}tj|dj ||z}|j |zS)a Args: Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. x: torch.Tensor x: Returns: torch.Tensor rdim)neintr3cumsumtype_asrE)r7rFrmaskincremental_indicess r:r@z5LiltTextEmbeddings.create_position_ids_from_input_idscsP||K(,,.$||Da8@@F$N"'')K77r;c|jdd}|d}tj|jdz||jzdztj|j }|j dj|S)z Args: We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids.: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr rr=r)rCr3r4rrEr? unsqueezer5)r7rHrIsequence_lengthrs r:rBz9LiltTextEmbeddings.create_position_ids_from_inputs_embedsps$((*3B/ %a.||   q /D4D4D"Dq"HPUPZPZcpcwcw %%a(// <">|"L&AD[&[#&*nn5P&Q#&*ll3N&O#**3 cZ[ab b csA,D&& E/ D;;E)NN)rXrYrZr#rKr[r\s@r:r^r^s >*+r;r^c4eZdZdfd ZddZ ddZxZS)LiltSelfAttentionct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j|j|_ tj|j|jz|j|jz|_tj|j|jz|j|jz|_tj|j|jz|j|jz|_tj$|j&|_|j|_||_y)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ())r"r#r&num_attention_headshasattr ValueErrorrPattention_head_size all_head_sizerrjquerykeyvaluerh layout_query layout_key layout_valuer/attention_probs_dropout_probr1 layer_idx)r7r8rr9s r:r#zLiltSelfAttention.__init__s     : : :a ?PVXhHi#F$6$6#78 445Q8  $*#=#= #&v'9'9F,,7 _kB %--aAq9DDF "/"4"4"6s";t?Q?Q>S"S* **,CD  "67  22Gr;N)rNF)rXrYrZr#rrKr[r\s@r:rxrxs#>% Ar;rxcneZdZfdZdej dej dej fdZxZS)LiltSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr) r"r#rrjr&denser-r.r/r0r1r6s r:r#zLiltSelfOutput.__init__#s` YYv1163E3EF f&8&8f>S>STzz&"<"<= r;r input_tensorreturncr|j|}|j|}|j||z}|Srrr1r-r7rrs r:rKzLiltSelfOutput.forward)7 =1  ]3 }|'CD r;rXrYrZr#r3TensorrKr[r\s@r:rr"1> U\\RWR^R^r;rc eZdZd fd Z d dej dej dej dzdedzdeej f dZ xZ S) LiltAttentionNct|t|||_t ||_|j }|j |jz|_t ||_||_y)Nr) r"r#rxr7routputr&rh layout_output)r7r8rori_hidden_sizer9s r:r#zLiltAttention.__init__1sa %f B $V,  ,,#//63N3NN+F3,r;rrrrrc|j||||}|j|d|}|j|d|}||f|ddz}|S)Nrrrm)r7rr) r7rrrr self_outputsattention_outputlayout_attention_outputrs r:rKzLiltAttention.forward;shyy       ;;|A F"&"4"4\!_m"T#%<= QR@PPr;rr) rXrYrZr#r3r FloatTensorbooltuplerKr[r\s@r:rr0se-48). ||||))D0   $;  u||  r;rcVeZdZfdZdej dej fdZxZS)LiltIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r"r#rrjr&intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr6s r:r#zLiltIntermediate.__init__Ps] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r;rrcJ|j|}|j|}|Sr)rr)r7rs r:rKzLiltIntermediate.forwardXs& =1 00? r;rr\s@r:rrOs#9U\\ellr;rcneZdZfdZdej dej dej fdZxZS) LiltOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r"r#rrjrr&rr-r.r/r0r1r6s r:r#zLiltOutput.__init__`s` YYv779K9KL f&8&8f>S>STzz&"<"<= r;rrrcr|j|}|j|}|j||z}|Srrrs r:rKzLiltOutput.forwardfrr;rr\s@r:rr_rr;rc eZdZd fd Z d dej dej dej dzdedzdeej f dZ d Z d Z xZ S) LiltLayerNct||j|_d|_t |||_t ||_t||_ |j}|j}|j|jz|_ |j|jz|_ t ||_ t||_||_ ||_ y)Nrr)r"r#chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr&rrhlayout_intermediater)r7r8rrori_intermediate_sizer9s r:r#zLiltLayer.__init__ns '-'E'E$&vC,V4 (  ,, & 8 8#//63N3NN#)#;#;v?Z?Z#Z #3F#; '/,#8 r;rrrrrc |j||||}|d}|d}|dd}t|j|j|j|} t|j |j|j|} | | f|z}|S)N)rrrrm)rrfeed_forward_chunkrrlayout_feed_forward_chunk) r7rrrrself_attention_outputsrrr layer_outputlayout_layer_outputs r:rKzLiltLayer.forwards"&   / "0"  2!4"8";(,0  # #T%A%A4CSCSUe 8  * *D,H,H$JZJZ\s  !45?r;cL|j|}|j||}|Sr)rrr7rintermediate_outputrs r:rzLiltLayer.feed_forward_chunks,"//0@A{{#68HI r;cL|j|}|j||}|Sr)rrrs r:rz#LiltLayer.layout_feed_forward_chunks."667GH))*=?OP r;rr) rXrYrZr#r3rrrrrKrrr[r\s@r:rrmso9*48). ||||))D0   $;  u||  : r;rceZdZfdZ d dej dej dej dzdedzdedzdedzd eej e zfd Z xZ S) LiltEncoderct|||_tjt |j Dcgc] }t|c}|_ycc}wr) r"r#r8r ModuleListrangenum_hidden_layersrlayer)r7r8_r9s r:r#zLiltEncoder.__init__sC  ]]uVE]E]?^#_!If$5#_` #_sANrrrroutput_hidden_states return_dictrc|rdnd}|rdnd}t|jD].\} } |r||fz}| ||||} | d}| d}|s&|| dfz}0|r||fz}|std|||fDSt|||S)Nrrrmc3$K|]}|| ywrr).0vs r: z&LiltEncoder.forward..s = s)last_hidden_stater attentions) enumeraterrr ) r7rrrrrrall_hidden_statesall_self_attentionsi layer_module layer_outputss r:rKzLiltEncoder.forwards#7BD$5b4(4 POA|#$58H$H!(! M*!,M)!,M &9]1=M>> from transformers import AutoTokenizer, AutoModel >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModel.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer z5You have to specify either input_ids or inputs_embeds)r=)r?rG)rFrrGrH)r8rHr)rpr)rrrrrr)r pooler_outputrr)r8rrrr~%warn_if_padding_and_no_attention_maskrCr?r3rDrEonesr}rJrGr5r rrrr rr)r7rFrprrGrrHrrrkwargsrI batch_size seq_lengthr?buffered_token_type_ids buffered_token_type_ids_expandedembedding_outputlayout_embedding_outputencoder_outputssequence_outputr s r:rKzLiltModel.forwardsTP2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T <;;{T1FSD  !"ZZ*j)A6RN  !t(89*.//*H*HKZK*X'3J3Q3QR\^h3i0!A!&[ SY!Z)-%)' *9* &,3;;*)  #'"8"8dQ]"8"^,,  #)/!5# ' *!,8< 8O O4UY #]3oab6II I)-')77&11   r;)T) NNNNNNNNN)rXrYrZr#r#r%rr3rrrr rKr[r\s@r:rrs"/0*.$(.2.2,0-1)-,0#'j <<$&j llT!j  t+ j  t+ j llT) j ||d*j  $;j #Tkj D[j  u|| 9 9j j r;rz LiLT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introcZeZdZfdZe ddej dzdejdzdejdzdej dzdej dzdejdzd ej dzd e dzd e dzd e dzd e eje zfdZ xZ S)LiltForSequenceClassificationct|||j|_||_t |d|_t ||_|jyNF)r ) r"r# num_labelsr8rrLiltClassificationHead classifierrr6s r:r#z&LiltForSequenceClassification.__init__sJ   ++ f> 08 r;NrFrprrGrrHlabelsrrrrc T| | n|jj} |j|||||||| |  } | d} |j| }d}||j |j }|jj |jdk(rd|j_nl|jdkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt}|jdk(r&||j|j}n|||}n|jj dk(r=t}||jd|j|jd}n,|jj dk(rt!}|||}| s|f| d dz}||f|zS|St#||| j$| j& S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForSequenceClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForSequenceClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_idx = outputs.logits.argmax(-1).item() >>> predicted_class = model.config.id2label[predicted_class_idx] ```NrprrGrrHrrrrr regressionsingle_label_classificationmulti_label_classificationr rmlosslogitsrr)r8rrr;rAr? problem_typer9r>r3rErPrsqueezerrrrrrr7rFrprrGrrHr<rrrr+rr3rDrCloss_fctrs r:rKz%LiltForSequenceClassification.forwards\&1% )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ r;NrFrprrGrrHr<rrrrc | | n|jj} |j|||||||| |  } | d} |j| } |j | }d}|W|j |j }t}||jd|j|jd}| s|f| ddz}||f|zS|St||| j| jS)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForTokenClassification >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForTokenClassification.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> predicted_class_indices = outputs.logits.argmax(-1) ```Nr>rr rmrB) r8rrr1r;rAr?rrr9rrrrGs r:rKz"LiltForTokenClassification.forward s V&1% ))F$6$68I8IJ r;NrFrprrGrrHstart_positions end_positionsrrrrc (| | n|jj} |j||||||| | |  } | d}|j|}|j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|}|jd|}t|}|||}|||}||zdz }| s||f| ddz}||f|zS|St|||| j| j S) a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, AutoModelForQuestionAnswering >>> from datasets import load_dataset >>> tokenizer = AutoTokenizer.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> model = AutoModelForQuestionAnswering.from_pretrained("SCUT-DLVCLab/lilt-roberta-en-base") >>> dataset = load_dataset("nielsr/funsd-layoutlmv3", split="train") >>> example = dataset[0] >>> words = example["tokens"] >>> boxes = example["bboxes"] >>> encoding = tokenizer(words, boxes=boxes, return_tensors="pt") >>> outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> predicted_answer = tokenizer.decode(predict_answer_tokens) ```Nr>rrr rM) ignore_indexrm)rC start_logits end_logitsrr)r8rrr[splitrFrlenrCclamprrrr)r7rFrprrGrrHr\r]rrrr+rr3rDr`ra total_loss ignored_indexrH start_lossend_lossrs r:rKz LiltForQuestionAnswering.forward~s^&1%rws AA&!69.6,*   H %S=S=l5+2995+pf fT RYY BII> ryy   7*7t4 "))4 p   i/ i iC #C C L m $7m  m `^ !4^ ^ DRYY,k 2k k \ r;