^jkWfddlmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z mZddlmZmZdd lmZdd lmZmZmZdd lmZdd lmZd dlmZGddej<ZGddej<Z Gddej<Z! d4dej<dejDdejDdejDdejDdzde#de#deefdZ$dZ%dejDd e&d!ejDfd"Z'd#ejDd$ejDd%ejDd&ejDd!e(ejDejDff d'Z)Gd(d)ej<Z*Gd*d+e Z+Gd,d-ej<Z,eGd.d/eZ-ed01Gd2d3e-Z.d/d3gZ/y)5)CallableN)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstring torch_int)merge_with_config_defaults)capture_outputs)MLCDVisionConfigcVeZdZfdZdej dej fdZxZS)MLCDMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yN) super__init__configr hidden_act activation_fnnnLinear hidden_sizeintermediate_sizefc1fc2selfr __class__s q/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/mlcd/modeling_mlcd.pyrzMLCDMLP.__init__&sd  #F$5$5699V//1I1IJ99V55v7I7IJ hidden_statesreturncl|j|}|j|}|j|}|Sr)r!rr")r$r(s r&forwardzMLCDMLP.forward-s4/ **=9 / r')__name__ __module__ __qualname__rtorchTensorr+ __classcell__r%s@r&rr%s$KU\\ellr'rceZdZUejed<d dededdffd ZdejdejfdZ xZ S) MLCDRotaryEmbeddinginv_freqdimthetar)Nct|||_||_d|t j d|dtj |z zz }|jd|dy)N?rdtyper5F persistent)rrr6r7r/arangefloatregister_buffer)r$r6r7r5r%s r&rzMLCDRotaryEmbedding.__init__7sY  %ELLC%++$NQT$TUV ZeDr' position_idsc\|jd|jzjdS)Nr) unsqueezer5flatten)r$rBs r&r+zMLCDRotaryEmbedding.forward>s'&&r*T]]:CCAFFr')g@) r,r-r.r/r0__annotations__intr@rr+r1r2s@r&r4r44sIllECEEDEGELLGU\\Gr'r4ceZdZdeffd Zdej dededej fdZdejdej fd Z xZ S) MLCDVisionEmbeddingsrc|t|||_|j|_|j |_|j |_tjtj|j|_ tj|j|j|j |j d|_|j |j zdz|_|jdz|_|j#dtj$|j j'ddy)NF) in_channels out_channels kernel_sizestridebiasr:rrBrrDr=)rrrr embed_dim image_size patch_sizer Parameterr/randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positionsrAr?expandr#s r&rzMLCDVisionEmbeddings.__init__Cs  ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1 ^U\\$:L:L-M-T-TU\-]jopr' embeddingsheightwidthr)c|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrDg?rr:bicubicF)sizemode align_cornersr6)shapeposition_embeddingweightrEr/jit is_tracingrBrTrreshapepermuter functional interpolateviewcat) r$r^r_r`r[rhr\class_pos_embedpatch_pos_embedr6 new_height new_widthsqrt_num_positionss r&interpolate_pos_encodingz-MLCDVisionEmbeddings.interpolate_pos_encodingXsv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr' pixel_valuescT|jd}|jjj}|j|j |}|j dj dd}|jj|dd}tj||gd}|S)Nrr;r:rrDrf) rgrZrir<torF transposerWr]r/rq)r$rx batch_size target_dtype patch_embeds class_embedsr^s r&r+zMLCDVisionEmbeddings.forwards!''* ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY l;C r') r,r-r.rrr/r0rHrw FloatTensorr+r1r2s@r&rJrJBsbq/q*'D5<<'D'DUX'D]b]i]i'DR E$5$5 %,, r'rJmodulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||j dd|z} || |z} t j j| dtjj|j} t j j| ||j} tj| | } | j ddj} | | fS)Nr:rrD)r6r<)ptrainingr) repeat_kvnum_key_value_groupsr/matmulr{rrnsoftmaxfloat32rzr<rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r&eager_attention_forwardrs3 ; ;