^jdZddlZddlZddlmZddlZddlmZddlm Z ddl m Z ddl mZmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZddlmZej<eZ edeGddeZ!edeGddeZ"edeGddeZ#edeGddeZ$GddejJZ&Gdd ejJZ'Gd!d"ejJZ(Gd#d$ejJZ)Gd%d&ejJZ*Gd'd(ejJZ+Gd)d*eZ,Gd+d,ejJZ-eGd-d.eZ.eGd/d0e.Z/ed1Gd2d3e.Z0ed4Gd5d6e.Z1ed7Gd8d9ee.Z2gd:Z3y);zPyTorch FocalNet model.N) dataclass)nn)initialization)ACT2FN) BackboneMixinfilter_output_hidden_states)GradientCheckpointingLayer)BackboneOutput)PreTrainedModel) ModelOutputauto_docstringlogging)can_return_tuple)FocalNetConfigzC FocalNet encoder's outputs, with potential hidden states. ) custom_introceZdZUdZdZej dzed<dZe ej dzed<dZ e ej dzed<y)FocalNetEncoderOutputa reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nlast_hidden_state hidden_statesreshaped_hidden_states) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rtuplery/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/focalnet/modeling_focalnet.pyrr%sT37u((4/659M5**+d29>BE%"3"34t;Br"rzZ FocalNet model's outputs that also contains a pooling of the last hidden states. ceZdZUdZdZej dzed<dZej dzed<dZ e ej dzed<dZ e ej dzed<y)FocalNetModelOutputa pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`, *optional*, returned when `add_pooling_layer=True` is passed): Average pooling of the last layer hidden-state. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nr pooler_outputrr) rrrrrrrrr&rr rr!r"r#r%r%:si 37u((4/6.2M5$$t+259M5**+d29>BE%"3"34t;Br"r%z. FocalNet masked image model outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e ej dzed<dZ e ej dzed<y)!FocalNetMaskedImageModelingOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `bool_masked_pos` is provided): Masked image modeling (MLM) loss. reconstruction (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Reconstructed pixel values. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nlossreconstructionrr) rrrrr)rrrr*rr rr!r"r#r(r(Rsh &*D%  d ")/3NE%%,359M5**+d29>BE%"3"34t;Br"r(z4 FocalNet outputs for image classification. ceZdZUdZdZej dzed<dZej dzed<dZ e ej dzed<dZ e ej dzed<y)FocalNetImageClassifierOutputa7 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nr)logitsrr) rrrrr)rrrr-rr rr!r"r#r,r,lsh &*D%  d ")'+FE   $+59M5**+d29>BE%"3"34t;Br"r,ceZdZdZdfd Z d dej dzdejdzdeejfdZ xZ S) FocalNetEmbeddingszX Construct the patch embeddings and layernorm. Optionally, also the mask token. c  t|t||j|j|j |j |jd|_|jj|_ |r4tjtjdd|j nd|_tj |j |j"|_tj&|j(|_y)NT)config image_size patch_size num_channels embed_dimuse_conv_embedis_stemreps)super__init__FocalNetPatchEmbeddingsr2r3r4r5r6patch_embeddings grid_size patch_gridr Parameterrzeros mask_token LayerNormlayer_norm_epsnormDropouthidden_dropout_probdropout)selfr1use_mask_token __class__s r#r;zFocalNetEmbeddings.__init__s  7((((,,&&!00! //99O]",,u{{1a9I9I'JKcgLL!1!1v7L7LM zz&"<"<= r"N pixel_valuesbool_masked_posreturnc8|j|\}}|j|}|j\}}}|K|jj ||d}|j dj |} |d| z z|| zz}|j|}||fS)N?)r=rEsizerBexpand unsqueezetype_asrH) rIrLrM embeddingsoutput_dimensions batch_sizeseq_len_ mask_tokensmasks r#forwardzFocalNetEmbeddings.forwards)-(=(=l(K% %YYz* !+!2 GQ  &//00WbIK",,R088ED#sTz2[45GGJ\\*- ,,,r")FN) rrrrr;rr BoolTensorr Tensorr] __classcell__rKs@r#r/r/sQ>&bf-!--4-GLGWGWZ^G^- u|| -r"r/czeZdZ dfd ZdZdej dzdeejee ffdZ xZ S)r<c dt |t|tjj r|n||f}t|tjj r|n||f}|d|dz|d|dzz} ||_||_||_| |_ |d|dz|d|dzf|_ |r/|rd} d} d} nd} d} d} tj||| | | |_ ntj|||||_ |r'tj||j |_yd|_y) Nrrr) kernel_sizestridepadding)rhrir8)r:r; isinstance collectionsabcIterabler2r3r4 num_patchesr>rConv2d projectionrCrDrE)rIr1r2r3r4r5add_normr6r7rorhrjrirKs r#r;z FocalNetPatchEmbeddings.__init__s7 #-j+//:R:R#SZZdfpYq #-j+//:R:R#SZZdfpYq !!} 1 5*Q-:VW=:XY $$(&$Q-:a=8*Q-:VW=:XY    iii[Y`DO!ii iZ`jkDO  YF4I4IJDIDIr"cn||jdzdk7rDd|jd||jdzz f}tjj||}||jdzdk7rFddd|jd||jdzz f}tjj||}|S)Nrr)r3r functionalpad)rIrLheightwidth pad_valuess r# maybe_padz!FocalNetPatchEmbeddings.maybe_pads 4??1% % *T__Q/%$//!:L2LLMJ==,,\:FL DOOA& &! +Q4??1#5QRAS8S#STJ==,,\:FLr"rLNrNcN|j\}}}}||jk7r td|j|||}|j |}|j\}}}}||f}|j dj dd}|j|j|}||fS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rfr)shaper4 ValueErrorryrqflatten transposerE)rIrLrZr4rvrwrVrWs r#r]zFocalNetPatchEmbeddings.forwards)5););&< 4,, ,w ~~lFEB __\2 (..1fe#UO''*44Q: 99 :.J,,,r")FFF) rrrr;ryrrr r`intr]rarbs@r#r<r<sL(T-E$5$5$<-u||UZ[^U_G_A`-r"r<c&eZdZdfd ZdZxZS)FocalNetModulationct |||_|j||_|j ||_||_|j|_|j|_ tj|d|z|j dzz||_ tj||dd||_tj|_tj|||_tj$||_tj(|_g|_t/|j D]}|j|z|jz}|j*j1tj2tj|||d||dzdtj|j,j1||jr'tj4||j6|_yy)Nrfr)bias)rhrirF)rhrigroupsrjrr8)r:r;dim focal_windows focal_window focal_levels focal_level focal_factor use_post_layernorm_in_modulationnormalize_modulatorrLinear projection_inrpprojection_contextGELU activationprojection_outrFprojection_dropout ModuleList focal_layers kernel_sizesrangeappend SequentialrCrD layernorm) rIr1indexrrrrkrhrKs r#r;zFocalNetModulation.__init__s "007!..u5(060W0W-#)#=#= YYsAGt7G7G!7K,LSWX"$))C!ATX"Y'') iiS1"$**-?"@MMOt''( 2A++a/$2C2CCK    $ $ IISk!CYdhiYipuGGI      $ $[ 1 2  0 0\\#63H3HIDN 1r"c,|jd}|j|jddddj}t j ||||j dzfd\}}}d}t|j D]+}|j||}|||dd||dzfzz}-|j|jddjdd} || |dd|j dfzz}|jr||j dzz }|j|} || z} | jddddj} |jr|j| } |j| } |j!| } | S) z Args: hidden_state: Input features with shape of (batch_size, height, width, num_channels) rPrrrrfNT)keepdim)r{rpermute contiguousrsplitrrrrmeanrrrrrr) rI hidden_stater4xqctxgatesctx_alllevel ctx_global modulatorx_outs r#r]zFocalNetModulation.forwards $))"-    | , 4 4Q1a @ K K M A lDDTDTWXDX'Y[\] 34++, BE*$##E*3/CeAuuqy/@,@&A AAG B__SXXaX%>%C%CAt%C%TU Jq$2B2B2D/D)EEE  # #!1!1A!56G++G4 I  aAq)446  0 0NN5)E##E*''. r")rfTrrrr;r]rarbs@r#rrsJB"r"rc&eZdZdfd ZdZxZS) FocalNetMlpc t||xs|}|xs|}tj|||_t |j |_tj|||_tj||_ yr^) r:r;rrfc1r hidden_actrfc2rFdrop)rIr1 in_featureshidden_features out_featuresrrKs r#r;zFocalNetMlp.__init__=sh #2{ )8[99[/: !2!2399_l;JJt$ r"c|j|}|j|}|j|}|j|}|j|}|Sr^)rrrr)rIrs r#r]zFocalNetMlp.forwardFsNxx - |4 yy. xx - yy. r")NNrrrbs@r#rr<s %r"rcreZdZdZd deddffd ZdejdejfdZde fdZ xZ S) FocalNetDropPathzStochastic depth (DropPath) per sample, for residual blocks. Identity when ``drop_prob`` is 0 or outside training. See `Deep Networks with Stochastic Depth `_. drop_probrNNc0t|||_yr^)r:r;r)rIrrKs r#r;zFocalNetDropPath.__init__Ws "r"rcP|jdk(s |js|Sd|jz }|jdfd|jdz zz}t j ||j |j}t j||z}|j||zS)Nrrr)r)dtypedevice) rtrainingr{ndimrrandrrfloordiv)rIr keep_probr{ random_tensors r#r]zFocalNetDropPath.forward[s >>S   & $$Q')DM4F4F4J,KK 5 0C0CML`L`a  MI$=>   +m;;r"c d|jS)Nzp=)rrIs r# extra_reprzFocalNetDropPath.extra_reprdsDNN#$$r"r) rrrrfloatr;rr`r]strrrarbs@r#rrPsB #%#$#'>|DJJ|D\ #((VULQ |499*funVbc +/+B+B| S_H`  $.. 1L"MM #dnn LL595L5Ltzz$((<01RVRZRZ[_[e[efr[sRt v'  r"r)rrrrr;r]rarbs@r#rrhs g@r"rcjeZdZfdZdej deeefdeej fdZxZ S) FocalNetStagect |||_t|j|_t |j Dcgc]}|jd|zz}}||}||j dz kr||dznd}||j dz krtnd}tjd|jt|jdD cgc]} | j} } | t|jd|t|jd|dz} tjt |j|Dcgc]'}t!||||t#| t$r| |n| )c}|_|'|||d||d|j(d |_d|_yd|_d|_ycc}wcc} wcc}w) Nrfrrcpu)r)r1rrrrTF)r1r2r3r4r5rrr6r7)r:r;r1lendepths num_stagesrr5r<rlinspacedrop_path_ratesumitemrrrrklistlayersr6 downsamplepointing) rIr1rrir5rout_dimrrdprrrKs r#r;zFocalNetStage.__init__s  fmm,8=doo8NO1V%%A.O O+04??Q3F+F)EAI&T1619L1L,SW "'63H3H#fmmJ\ej!klAqvvxllFMM&512S{QR9S5TU mmv}}U34 !%5.8D.Iily       !(+ !%44 DO #DO IP m sF<G ,GrrrNc |\}}|jD] }|||} |}|jK|\}}|jddj|jdd||}|j|\}}n||||f}|||f}|S)NrrfrrP)rrr~reshaper{) rIrrrvrw layer_module!hidden_states_before_downsamplingrW stage_outputss r#r]zFocalNetStage.forwards(  KK JL(8HIM J-:) ?? &,MFE)33Aq9AA177:BM04}/M ,M,"( > &(IK\] r") rrrr;rr`r rr]rarbs@r#rrs=*XU\\U3PS8_Y^_d_k_kYlr"rc|eZdZfdZ d dej deeefdedzdedzdedzdee zf d Z xZ S) FocalNetEncoderc 2t|t|j|_||_t jt|jDcgc]$}t|||dd|zz|dd|zzf&c}|_ d|_ ycc}w)Nrrfr)r1rrF) r:r;rrrr1rrrrstagesgradient_checkpointing)rIr1r>i_layerrKs r#r;zFocalNetEncoder.__init__s fmm, mm %T__5    !!&/lq'z&BIaLUVX_U_D`%a    ',# s)Brroutput_hidden_statesN(output_hidden_states_before_downsampling return_dictrNc|rdnd}|rdnd}|rE|j\}} } |j|g|| } | jdddd} ||fz }|| fz }t|jD]\} } | ||}|d}|d}|d}|d|df}|rP|rN|j\}} } |j|g|d|df| } | jdddd} ||fz }|| fz }z|s}|r|j\}} } |j|g|| } | jdddd} ||fz }|| fz }|st d||fDSt ||| S) Nr!rrrrfrPc3&K|] }|| ywr^r!).0vs r# z*FocalNetEncoder.forward..1sXq!-Xs)rrr)r{rr enumeraterr r)rIrrrrrall_hidden_statesall_reshaped_hidden_statesrXrZ hidden_sizereshaped_hidden_stater stage_modulerrrWs r#r]zFocalNetEncoder.forwards#7BD+?RT" )6)<)< &J;$6M$6$6z$bDT$bVa$b !$9$A$A!Q1$M ! -!1 1  &+@*B B &(5 GOA|(8HIM)!,M0=a0@ - -a 0  1" 57H7LM #(P-N-T-T* A{)O(I(N(N)"3A"68I!8L!M)OZ)%)>(E(EaAq(Q%!&G%II!*/D.FF*%.V-:-@-@* A{(: (:(::(fHX(fZe(f%(=(E(EaAq(Q%!m%55!*/D.FF*3 G6X]4E$FXX X$++#=  r")FFT) rrrr;rr`r rboolrr]rarbs@r#rrsp,,-2@E#' 5 ||5  S/5 #Tk 5 37+ 5 D[ 5  & &5 r"rcdeZdZUeed<dZdZdZdgZe jfdZ xZ S)FocalNetPreTrainedModelr1focalnetrLTrct||t|tr-|j t j |jyyt|tr|jjrit j|j|jjt j|j|jjyyy)zInitialize the weightsN)r: _init_weightsrkr/rBinitzeros_rr1r constant_rrr)rImodulerKs r#rz%FocalNetPreTrainedModel._init_weightsBs f% f0 1  , F--.-  .{{))v~~t{{/K/KLv~~t{{/K/KL*/r") rrrrrbase_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modulesrno_gradrrarbs@r#rr:s? "$O&*#()U]]_ M Mr"rc eZdZd fd ZdZe d dejdzdejdzde dzde dzde e zf d Z xZ S) FocalNetModelct||||_t|j|_t |jd|j dz zz|_t|||_ t||jj|_ tj|j|j |_|rtj$dnd|_|j)y)z add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer use_mask_token (`bool`, *optional*, defaults to `False`): Whether to use a mask token for masked image modeling. rfr)rJr8N)r:r;r1rrrrr5 num_featuresr/rVrr?encoderrrCrDrAdaptiveAvgPool1dpooler post_init)rIr1add_pooling_layerrJrKs r#r;zFocalNetModel.__init__Qs   fmm, 0 0119L3M MN,VNS&vt/I/IJ d&7&7V=R=RS1Bb**1-  r"c.|jjSr^)rVr=rs r#get_input_embeddingsz"FocalNetModel.get_input_embeddingsfs///r"NrLrMrrrNc ||n|jj}||n|jj}| td|j ||\}}|j ||||}|d} |j | } d} |j7|j| jdd} tj| d} |s | | f|ddz} | St| | |j|jS) z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rMrrrrrf)rr&rr)r1rrr|rVr$rr&r~rr}r%rr) rIrLrMrrkwargsembedding_outputrencoder_outputssequence_output pooled_outputoutputs r#r]zFocalNetModel.forwardis%9$D $++JjJj &1% Note that we provide a script to pre-train this model on custom data in our [examples directory](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining). c eZdZfdZe d dej dzdejdzdedzdedzde e zf dZ xZ S) FocalNetForMaskedImageModelingct||t|dd|_t |j |_t|jd|j dz zz}tjtj||jdz|jzdtj|j|_|j!y)NFT)r(rJrfr) in_channels out_channelsrh)r:r;r!rrrrrr5rrrpencoder_strider4 PixelShuffledecoderr')rIr1r#rKs r#r;z'FocalNetForMaskedImageModeling.__init__s  %fVZ[ fmm,6++aDOOa4G.HHI }} II(v7L7La7ORXReRe7est  OOF11 2    r"NrLrMrrrNc ||n|jj}|j||||}|d}|jdd}|j\}} } t j | dzx} } |j|| | | }|j|} d}||jj|jjz}|jd||}|j|jjdj|jjdjdj}tjj!|| d }||zj#|j#d zz |jj$z }|s| f|ddz}||f|zS|St'|| |j(|j* S) a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Examples: ```python >>> from transformers import AutoImageProcessor, FocalNetConfig, FocalNetForMaskedImageModeling >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> image_processor = AutoImageProcessor.from_pretrained("microsoft/focalnet-base-simmim-window6-192") >>> config = FocalNetConfig() >>> model = FocalNetForMaskedImageModeling(config) >>> num_patches = (model.config.image_size // model.config.patch_size) ** 2 >>> pixel_values = image_processor(images=image, return_tensors="pt").pixel_values >>> # create random boolean mask of shape (batch_size, num_patches) >>> bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool() >>> outputs = model(pixel_values, bool_masked_pos=bool_masked_pos) >>> loss, reconstructed_pixel_values = outputs.loss, outputs.logits >>> list(reconstructed_pixel_values.shape) [1, 3, 192, 192] ```N)rMrrrrrfg?rPnone) reductiongh㈵>)r)r*rr)r1rrr~r{mathrrr;r2r3repeat_interleaverTrrrtl1_lossrr4r(rr)rIrLrMrrr-outputsr0rXr4sequence_lengthrvrwreconstructed_pixel_valuesmasked_im_lossrRr\reconstruction_lossr2s r#r]z&FocalNetForMaskedImageModeling.forwardsN&1%t|||j|_t||_|jdkDr4t j |jj|jnt j|_ |jy)Nr) r:r; num_labelsr!rrrr#r classifierr'rIr1rKs r#r;z'FocalNetForImageClassification.__init__sx   ++%f- IOHYHY\]H]BIIdmm00&2C2C Dcecncncp  r"NrLlabelsrrrNc <||n|jj}|j|||}|d}|j|}d} ||j |||j} |s|f|ddz} | | f| zS| St | ||j |jS)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr,rrf)r)r-rr)r1rrrK loss_functionr,rr) rIrLrMrrr-rBr1r-r)r2s r#r]z&FocalNetForImageClassification.forward%s&1%>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> processor = AutoImageProcessor.from_pretrained("microsoft/focalnet-tiny-lrf") >>> model = AutoBackbone.from_pretrained("microsoft/focalnet-tiny-lrf") >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) ```NTr,r!) feature_mapsr attentions) r1rrrrr  stage_namesrrr ) rIrLrrr-rBrrVidxstager2s r#r]zFocalNetBackbone.forwardas<&1%rks` ! &!H9.-99-2   H %   CK C  C  C+C  C$  C C  C(  CKC  C(%-%-PD-biiD-NDDN"))(%ryy%0BBIIBJ?.?DH biiH VMoMM(I +I I X  b %<b  b J 8 %<8  8 v B }&=B  B J r"