^jIdZddlmZddlmZddlmZddlZddlmZm Z ddl m Z dd l mZdd lmZdd lmZdd lmZmZdd lmZmZddlmZddlmZmZmZmZm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*erddl+m,Z,e jZe.Z/dejdejfdZ0dejdejfdZ1eeGddeZ2dedefdZ3dedefdZ4d Z5d!Z6ed"#eGd$d%eZ7ed&#eGd'd(eZ8Gd)d*e jrZ:Gd+d,e jrZ; dQd-e jrd.ejd/ejd0ejd1ejdzd2eGd8d9e jrZ?Gd:d;eZ@eGd<d=eZAGd>d?e jrZBGd@dAeAZCGdBdCeAZDGdDdEeAZEGdFdGeAZFeGdHdIeAZGGdJdKe jrZHGdLdMe jrZIGdNdOeAZJgdPZKy)RzPyTorch OWL-ViT model.)Callable) dataclass)AnyN)Tensornn)initialization)ACT2FN)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringis_vision_availablelogging torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) OwlViTConfigOwlViTTextConfigOwlViTVisionConfig)center_to_corners_formatlogitsreturnctjj|tjt ||j S)Ndevice)r functional cross_entropytorcharangelenr$)r s u/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/owlvit/modeling_owlvit.pycontrastive_lossr+6s/ == & &vu||CKPVP]P]/^ __ similaritycRt|}t|j}||zdz S)Ng@)r+T)r- caption_loss image_losss r*image_text_contrastive_lossr2;s*#J/L!*,,/J : % ,,r,ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ e ed<dZe ed <d eefd Zy) OwlViTOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size * num_max_text_queries, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`OwlViTTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`OwlViTVisionModel`]. text_model_output (tuple[`BaseModelOutputWithPooling`]): The output of the [`OwlViTTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`OwlViTVisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputr!cHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw)r:r;Ngetattrto_tuple.0kselfs r* z(OwlViTOutput.to_tuple..a=  LLDGRYZ^`aRbRkRkRm m -0tuplekeysrEs`r*rAzOwlViTOutput.to_tuple`# YY[   r,)__name__ __module__ __qualname____doc__r5r' FloatTensor__annotations__r6r7r8r9r:rr;rJrrAr,r*r4r4As(&*D%  d ")15e''$.504OU&&-4,0K""T)0-1L%##d*148186:3: %* r,r4tc|jr>|jtjtjfvr|S|j S|jtj tjfvr|S|jSN) is_floating_pointdtyper'float32float64floatint32int64int)rUs r*_upcastr`hs`GG u}}==qL1779LGG U[[99qFquuwFr,boxescft|}|dddf|dddfz |dddf|dddfz zS)a Computes the area of a set of bounding boxes, which are specified by its (x1, y1, x2, y2) coordinates. Args: boxes (`torch.FloatTensor` of shape `(number_of_boxes, 4)`): Boxes for which the area will be computed. They are expected to be in (x1, y1, x2, y2) format with `0 <= x1 < x2` and `0 <= y1 < y2`. Returns: `torch.FloatTensor`: a tensor containing the area for each box. Nrrr)r`)ras r*box_areardqsB ENE !Q$K%1+ %%1+ad *C DDr,c^t|}t|}tj|dddddf|ddddf}tj|dddddf|ddddf}||z j d}|dddddf|dddddfz}|dddf|z|z }||z } | |fS)Nrcrminr)rdr'maxrgclamp) boxes1boxes2area1area2left_top right_bottom width_heightinterunionious r*box_iourts V E V Eyy4! ,fQUmr?rBs r*rFz7OwlViTObjectDetectionOutput.to_tuple..rGrHrIrLs`r*rAz$OwlViTObjectDetectionOutput.to_tuplerMr,)rNrOrPrQr5r'rRrSrdictr rr8r9rr:rr;rJrrArTr,r*r~r~s8&*D%  d ")!Itd{!'+FE   $++/J!!D(/,0K""T)0-1L%##d*1-1L%##d*148186:3: %* r,r~zM Output type of [`OwlViTForObjectDetection.image_guided_detection`]. c0eZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ eed <dZeed <d eefd Zy) &OwlViTImageGuidedObjectDetectionOutputa logits (`torch.FloatTensor` of shape `(batch_size, num_patches, num_queries)`): Classification logits (including no-object) for all queries. image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`OwlViTVisionModel`]. OWL-ViT represents images as a set of image patches and computes image embeddings for each patch. query_image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`OwlViTVisionModel`]. OWL-ViT represents images as a set of image patches and computes image embeddings for each patch. target_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual target image in the batch (disregarding possible padding). You can use [`~OwlViTImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. query_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual query image in the batch (disregarding possible padding). You can use [`~OwlViTImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. class_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Class embeddings of all image patches. OWL-ViT represents images as a set of image patches where the total number of patches is (image_size / patch_size)**2. text_model_output (tuple[`BaseModelOutputWithPooling`]): The output of the [`OwlViTTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`OwlViTVisionModel`]. Nr r9query_image_embedstarget_pred_boxesquery_pred_boxesrr:r;r!cHtfdjDS)Nc3dK|]'}|dvr|nt|j)ywr>r?rBs r*rFzBOwlViTImageGuidedObjectDetectionOutput.to_tuple.. rGrHrIrLs`r*rAz/OwlViTImageGuidedObjectDetectionOutput.to_tuple rMr,)rNrOrPrQr r'rRrSr9rrrrr:rr;rJrrArTr,r*rrs8(,FE   $+-1L%##d*137))D0726u((4/615e''$.5-1L%##d*148186:3: %* r,rceZdZdeffd Zdej dededej fdZd dejd e dej fd Z xZ S) OwlViTVisionEmbeddingsconfigct||j|_||_|j|_t jtj|j|_ t j|j|j |j|jd|_ |j|jzdz|_|jdz|_t j"|j |j |_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestridebiasrcr position_idsr persistent)super__init__ patch_sizer hidden_size embed_dimr Parameterr'randnclass_embeddingConv2d num_channelspatch_embedding image_size num_patches num_positions Embeddingposition_embeddingregister_bufferr(expandrEr __class__s r*rzOwlViTVisionEmbeddings.__init__s  ++ ++!||EKK8J8J,KL!yy++))$$  #--1B1BBqH!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr, embeddingsheightwidthr!c|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrg?rrcbicubicF)sizemode align_cornersdim)shaperweight unsqueezer'jit is_tracingrrrreshapepermuterr% interpolateviewcat) rErrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r*interpolate_pos_encodingz/OwlViTVisionEmbeddings.interpolate_pos_encoding(sv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4  *11!Q1=BB1b#Nyy/?;CCr, pixel_valuesrch|j\}}}}|j|}|jdjdd}|jj |dd}t j||gd} |r| |j| ||z} | S| |j|jz} | S)Nrcrrr) rrflatten transposerrr'rrrr) rErr batch_size_rr patch_embedsrrs r*forwardzOwlViTVisionEmbeddings.forwardNs'3'9'9$ Avu++L9 #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr,F) rNrOrPrrr'rr_rrRboolr __classcell__rs@r*rrsmq1q*$D5<<$D$DUX$D]b]i]i$DL E$5$5 QU bgbnbn r,rc eZdZdeffd Z d dej dzdej dzdejdzdejfdZ xZ S) OwlViTTextEmbeddingsrc^t|tj|j|j |_tj|j|j |_|jdtj|jjddy)NrrFr) rrrr vocab_sizertoken_embeddingmax_position_embeddingsrrr'r(rrs r*rzOwlViTTextEmbeddings.__init__\s !||F,=,=v?Q?QR"$,,v/M/MvOaOa"b  ELL)G)GHOOPWXej  r,N input_idsr inputs_embedsr!c||jdn|jd}||jddd|f}||j|}|j|}||z}|S)Nr)rrrr)rErrr seq_lengthposition_embeddingsrs r*rzOwlViTTextEmbeddings.forwardfs{ -6,AY__R(}GZGZ[]G^  ,,Q  ^L==((6??([L,,|U3K''1-88:K  $$r,c eZdZdZfdZ d dej dej dzdeede ej ej dzffdZ xZ S) OwlViTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ d|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rF)rrrrrnum_attention_heads num_headshead_dimrwscaleattention_dropoutr is_causalrLineark_projv_projq_projout_projrs r*rzOwlViTAttention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r,N hidden_statesrrr!c |jdd}g|d|j}|j|j|j dd}|j |j|j dd}|j |j|j dd}tj|jjt} | |||||f|j|jsdn |jd|\} } | jg|dj!} |j#| } | | fS)Nrrrc)rr)rrrrrrrr get_interfacer_attn_implementationrrrrrrr) rErrr input_shape hidden_shape query_states key_states value_statesattention_interfacerrs r*rzOwlViTAttention.forwardsQ $))#2. 88b8$--8 6t{{=166 EOOPQSTU 4T[[/44lCMMaQRS 6t{{=166 EOOPQSTU (?(M(M KK , ,.E) %8      % JJ#}}C$,, %  % ! \*k));;;;FFH mmK0 L((r,rW) rNrOrPrQrr'rrrrJrrrs@r*rrsfGB./3)||) t+)+, ) u||U\\D00 1 )r,rcVeZdZfdZdej dej fdZxZS) OwlViTMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yrW) rrrr hidden_act activation_fnrrrintermediate_sizefc1fc2rs r*rzOwlViTMLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr,rr!cl|j|}|j|}|j|}|SrW)rrr)rErs r*rzOwlViTMLP.forwards4/ **=9 / r,)rNrOrPrr'rrrrs@r*rrs$KU\\ellr,rceZdZdeezffd Zdejdejdee dejfdZ xZ S)OwlViTEncoderLayerrcDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yNeps) rrrrr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rs r*rzOwlViTEncoderLayer.__init__sm ++(0<<F> ')   q !=0  ((7 /  =0 r,) rNrOrPrrrr'rrrrRrrrs@r*rrsWS14DDS|| +,     r,rceZdZUeed<dZdZdZdZdZ dZ dZ dgZ e edZej"dej&ffd ZxZS) OwlViTPreTrainedModelrowlvit)imagetextTr)r attentionsrc t|||jj}t |t rt j|jjd|dzt j|jjd|dzt j|jtj|jjdj!dy t |t"rt j|j$d|j&dz|zt j|j(j|jj*|zt j|jj|jj*|zt j|jtj|jjdj!dy t |t,r|j&dzd|jj.zdzz|z}|j&dz|z}t j|j0j|t j|j2j|t j|j4j|t j|j6j|y t |t8r|jj:dzd|jj.zdzz|z}d|jj:zdz|z}t j|j<j|t j|j>j|y t |t@rt j|jBj|jDdz|zt j|jFj|jHdz|zt jJ|jL|jjNy t |tPrEt j|jR|jU|jV|jXy y ) zInitialize the weightsrg{Gz?)meanstdrrr)r,rcN)-r _init_weightsrinitializer_factor isinstancerinitnormal_rrrcopy_rr'r(rrrrrrinitializer_rangernum_hidden_layersrrrrrrrr OwlViTModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dim constant_ logit_scalelogit_scale_init_valueOwlViTForObjectDetectionbox_biascompute_box_biasnum_patches_heightnum_patches_width)rErfactor in_proj_std out_proj_stdfc_stdrs r*r-z#OwlViTPreTrainedModel._init_weightssi f%// f2 3 LL//66Sftm T LL2299RV W JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h  6 7 LL//cv?O?OQU?UX^?^ _ LL//66FMM<[<[^d>> from transformers import AutoProcessor, OwlViTTextModel >>> model = OwlViTTextModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrT)rp)rErrrs r*rzOwlViTTextModel.forwards,6t )   r,NN)rNrOrPrrSrGrrrPrsrurr'rrrrJrrrrs@r*rnrns  / :bii:;*..2 <<$&  t+ +,  + +   r,rnc eZdZdeffd Zeede d dejde dzde e d e ezfd ZxZS) OwlViTVisionTransformerrcDt||t||_t j |j |j|_t||_ t j |j |j|_ |jyr) rrrrrrrr pre_layernormrRr_post_layernormrars r*rz OwlViTVisionTransformer.__init__sr  08\\&*<*<&BWBWX$V,  ll6+=+=6CXCXY r,FrbrrNrr!c T|jjjj}|j |}|j||}|j |}|j dd|i|}|j}|dddddf}|j|}t||S)N)rrrrfrT) rrrrYrir|r_rZr}r) rErrrexpected_input_dtyperrkrZrls r*rzOwlViTVisionTransformer.forwards $>>EEKK#';<  Ogh **=9 +74<<, ', ,  ,==)!Q'2 ++M: )/'  r,r)rNrOrPrrrrrr'rRrrrrJrrrrs@r*rzrzsz 1  E216 '' #'+ +,  + +  3  r,rzc eZdZUeed<dZdZdeffd ZdejfdZ e d de jdzded eedefd ZxZS) OwlViTVisionModelrr)r'cdt||t||_|j yrW)rrrz vision_modelrars r*rzOwlViTVisionModel.__init__s'  3F; r,r!cB|jjjSrW)rrrrLs r*rsz&OwlViTVisionModel.get_input_embeddingss  ++;;;r,Nrrc ,|jd||d|S)a' Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, OwlViTVisionModel >>> model = OwlViTVisionModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrT)r)rErrrs r*rzOwlViTVisionModel.forwards.8!t   %%=   r,rT)rNrOrPrrSmain_input_namerGrrrPrsrr'rRrrrrrrrs@r*rrs $O!1 Q>QX]^<< V5R5R(ST r,Nrrrr!c t|jd||d|}|j}|j||_|S)a input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> import torch >>> from transformers import AutoProcessor, OwlViTModel >>> model = OwlViTModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ```rwrT)rprgr6)rErrr text_outputsrls r*get_text_featureszOwlViTModel.get_text_featuresBsP64C4??4 )4 4 %22 %)%9%9-%H "r,rrc p|jd||d|}|j|j|_|S)a Examples: ```python >>> import torch >>> from transformers.image_utils import load_image >>> from transformers import AutoProcessor, OwlViTModel >>> model = OwlViTModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ```rrT)rr8rg)rErrrvision_outputss r*get_image_featureszOwlViTModel.get_image_featuresgsM26GT5F5F6 %%=6 6  (,'='=n>Z>Z'[$r, return_lossreturn_base_image_embedsc ||jd ||d|}|jd ||d|} | j} |j| } |j} |j | } | t j j| dddz } | t j j| dddz } |jjj| j} t j| | j| z}|j}d}|r t|}| } t|||| | | |S) aw return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. return_base_image_embeds (`bool`, *optional*): Whether or not to return the base image embeddings. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, OwlViTModel >>> model = OwlViTModel.from_pretrained("google/owlvit-base-patch32") >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(text=[["a photo of a cat", "a photo of a dog"]], images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```rrwrcrT)ordrkeepdimN)r5r6r7r8r9r:r;rT)rrprgr6r8r'linalgnormr;exprir$rrUr2r4)rErrrrrrrrrr8r9text_embeds_normr;r7r6r5s r*rzOwlViTModel.forwardsdF6GT5F5F6 %%=6 6 4C4??4 )4 4 #00 **;7 %33 --l; $ell&7&7 !QS]a&7&bb &):):;ASU_c):)dd&&**,// 0C0CD ,,'79IJ[X*,,. .?D& -+#%* .  r,rWr)NNNNFN)rNrOrPrrSrrrr'rrrrJrrrrrrRr4rrrs@r*r5r5*s |(/3!<<! t+!+, ! + + !!F*/ll#'+,  + + @.215.2#').04K ##d*K ''$.K  t+ K D[ K #' K #'+K +,K   K K r,r5cbeZdZddedeffd ZdejdejfdZ xZ S)OwlViTBoxPredictionHeadrout_dimc"t||jj}t j |||_t j |||_t j|_ t j |||_ yrW) rrrrrrdense0dense1GELUgeludense2)rErrrrs r*rz OwlViTBoxPredictionHead.__init__sb $$00iiu- iiu- GGI iiw/ r,image_featuresr!c|j|}|j|}|j|}|j|}|j|}|SrW)rrrr)rEroutputs r*rzOwlViTBoxPredictionHead.forwardsM^,6"V$6"V$ r,)) rNrOrPrr_rr'rrRrrrs@r*rrs30|0c0ellu7H7Hr,rc eZdZdeffd Zdej dej dzdejdzdeej fdZ xZ S) OwlViTClassPredictionHeadrct||jj}|jj|_t j|j ||_t j|j d|_ t j|j d|_ t j|_ y)Nr) rrrrr query_dimrrr logit_shiftr;ELUelu)rErrrs r*rz"OwlViTClassPredictionHead.__init__s $$00--99ii8 99T^^Q799T^^Q7668r,r9 query_embedsN query_maskr!c0|j|}|S|j}|jdd\}}tj|||j fj |}||fS|tjj|dddzz }|tjj|dddzz }tjd||}|j|} |j|} |j| dz} || z| z}||jdkDrtj|d }tj|d k(tj |j"j$|}|j tj&}||fS) NrcrT)rrgư>z...pd,...qd->...pqrrrr)rr$rr'zerosrrirreinsumrr;rndimrwherefinforYrgrZ) rEr9rrimage_class_embedsr$rr pred_logitsrr;s r*rz!OwlViTClassPredictionHead.forwards "[[6  '..F&8&>&>r&B #J ++z;&OPSSTZ[K!34 405<<3D3DEW]_im3D3nqu3uv#u||'8'82W['8'\_c'cd ll#79K\Z &&|4 &&|4 hh{+a/ "[0K?  !""__ZR@ ++jAou{{;CTCT7U7Y7Y[fgK%..7K/00r,) rNrOrPrrr'rRrrJrrrs@r*rrs_ | !1''!1''$.!1LL4' !1 u  ! !1r,rcveZdZUeed<deffd ZedededejfdZ dededejfdZ ddejd ejd e dejfd Z ddejd ejd zdejd zdeejfdZ ddejdejdejd e deedeejf dZ ddejd e deedeejfdZ ddejdejd e dejfdZee ddejdejd zd e deedef dZee ddejdejdejd zd e deedef dZxZS)r=rct||t||_t ||_t ||_tj|jj|jj|_ tj|_||_|j jj"|j jj$z|_|j jj"|j jj$z|_|j+d|j-|j&|j(d|j/y)Nrr>Fr)rrr5r&r class_headrbox_headrrrrr layer_normSigmoidsigmoidrrrr@rArr?rars r*rz!OwlViTForObjectDetection.__init__s   !&) 3F;/7 ,,v';';'G'GVMaMaMpMpqzz|  "&++";";"F"F$++JcJcJnJn"n!%!:!:!E!EIbIbImIm!m  --d.E.EtG]G]^kp   r,r@rAr!cjtjd|dztj}tjd|dztj}tj||d\}}tj||fd}|dxx|zcc<|dxx|zcc<|j dd }|S) Nr)rYxy)indexingrr.r.rrc)r'r(rZmeshgridstackr)r@rA x_coordinates y_coordinatesxxyybox_coordinatess r*!normalize_grid_corner_coordinatesz:OwlViTForObjectDetection.normalize_grid_corner_coordinates1s Q(9A(=U]]S  Q(:Q(>emmT  }tLB ++r2hB7#44#55*..r15r,c|j||}tj|dd}tj|dztj| dzz }tj |d}|dxx|zcc<|dxx|zcc<tj|dztj| dzz }tj ||gd}|S)Nrg?g-C6?rrrr)rr'cliploglog1p full_liker)rEr@rArbox_coord_biasbox_size box_size_biasr>s r*r?z)OwlViTForObjectDetection.compute_box_biasBs@@ASUfg**_c3??T#9:U[[/IY\`I`=aa??>37--.. (T/2U[[(TAQ5RR 99nm<"Er, image_feats feature_maprc|j|}|r$|j\}}}}|j||}n |j}|j |j }||z }|j |}|S)a Args: image_feats: Features extracted from the image, returned by the `image_text_embedder` method. feature_map: A spatial re-arrangement of image_features, also returned by the `image_text_embedder` method. interpolate_pos_encoding: Whether to interpolate the pre-trained position encodings. Returns: pred_boxes: List of predicted boxes (cxcywh normalized to 0, 1) nested within a dictionary. )rrr?r>rir$r) rErrrrrr@rAr>s r* box_predictorz&OwlViTForObjectDetection.box_predictorTs|&]];/  $:E:K:K 7A!#4a,,-?ARSH}}H;;{112h \\*- r,Nrrc6|j|||\}}||fS)a8 Args: image_feats: Features extracted from the `image_text_embedder`. query_embeds: Text query embeddings. query_mask: Must be provided with query_embeddings. A mask indicating which query embeddings are valid. )r)rErrrrrs r*class_predictorz(OwlViTForObjectDetection.class_predictorus)-1OOKWa,b)(/00r,rrrrc |jd||||d|}|rX|j\}}}} ||jjjz} | |jjjz} n|j } |j } |jd} |jjj| } tj| ddddddf| ddddfj}| ddddddf|z} |j| } | jd| | | jdf}| j|} |d}|| |fS)N)rrrrrrrrT)r&rrrrr@rAr;rr}r' broadcast_torr)rErrrrroutputsrrrr@rArZr9class_token_outnew_sizer8s r*image_text_embedderz,OwlViTForObjectDetection.image_text_embeddersy$++ %)%=     $"."4"4 Aq&%!'4;;+D+D+O+O!O  %)B)B)M)M M !%!8!8  $ 6 6 $77:{{//>>?PQ  ,,\!RaR(-C\RSUXVXUXRXEYE_E_`$Aqr1H-? |4    q !     r "   $++H5 bk \733r,c |jjd||d|}|rX|j\}}}}||jjj z}||jjj z} n|j }|j} |d} |jjj| } tj| ddddddf| ddddfj} | ddddddf| z} |j| } | jd|| | jdf} | j| } | |fS)NrrrrrT) r&rrrrrr@rAr}r'rrr)rErrrrrrrr@rArZr9rrs r*image_embedderz'OwlViTForObjectDetection.image_embeddersg6NT[[5M5M6 %@X6 \b6  $"."4"4 Aq&%!'4;;+D+D+O+O!O  %)B)B)M)M M !%!8!8  $ 6 6 +1-{{//>>?PQ  ,,\!RaR(-C\RSUXVXUXRXEYE_E_`$Aqr1H-? |4    q !     r "   $++H5 n--r,query_image_featuresquery_feature_mapcj|j|\}}|j|||}t|}g}g} |j} t |j dD]} t jgdg| } || } t| | \}}t j|ddk(r t| | }t j|dz}|d|k\j}|js|| |jd}t j|| d}t j d||}|t j"|}|j%|| || j%|"|r+t j&|}t j&| }nd \}}|||fS) Nr)rrrrr#rg?r)axiszd,id->irx)rrrr$rVrr'rrtrvr{rhnonzeronumelsqueezer+rargminappendr)rErrrrrrpred_boxes_as_cornersbest_class_embedsbest_box_indicespred_boxes_deviceieach_query_boxeach_query_pred_boxesious iou_threshold selected_indsselected_embeddings mean_embedsmean_sim best_box_indr box_indicess r*embed_image_queryz*OwlViTForObjectDetection.embed_image_querys ../CD<''(<>OQij 8 D188+11!45 6A"\\<.ARSN$9!$< !n.CDGD!yyaC(*>;PQ"IIdOc1M!!W 5>>@M""$&21om6K6KA6N&O##jjaqA  << ;@ST,U\\(-CD !((a)FG '' 5' 6*  ;;'89L++&67K(2 %L+[*44r,query_pixel_valuesc |j||d}|jd||d|\}}|j\}} } } tj||| | z| f} |j\}} } } tj||| | z| f} |j | ||\}}}|j | |\}}|j | ||}t||||||d|S)a query_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values of query image(s) to be detected. Pass in one query image per target image. Examples: ```python >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import torch >>> from transformers import AutoProcessor, OwlViTForObjectDetection >>> processor = AutoProcessor.from_pretrained("google/owlvit-base-patch16") >>> model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch16") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> query_url = "http://images.cocodataset.org/val2017/000000001675.jpg" >>> with httpx.stream("GET", query_url) as response: ... query_image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, query_images=query_image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model.image_guided_detection(**inputs) >>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2] >>> target_sizes = torch.Tensor([image.size[::-1]]) >>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax) >>> results = processor.post_process_image_guided_detection( ... outputs=outputs, threshold=0.6, nms_threshold=0.3, target_sizes=target_sizes ... ) >>> i = 0 # Retrieve predictions for the first image >>> boxes, scores = results[i]["boxes"], results[i]["scores"] >>> for box, score in zip(boxes, scores): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected similar object with confidence {round(score.item(), 3)} at location {box}") Detected similar object with confidence 0.856 at location [10.94, 50.4, 315.8, 471.39] Detected similar object with confidence 1.0 at location [334.84, 25.33, 636.16, 374.71] ```rr)rrN)r9rrrr rr:r;rT)rrr'rrrrr)rErrrrrrrrr@rA hidden_dimrquery_image_featsrrrrrrs r*image_guided_detectionz/OwlViTForObjectDetection.image_guided_detection sH^!//+F^0  ':d&9&9' %%=' ' # ^ ITHYHYE &(9:mmK*>PSd>dfp1qr HYH_H_E &(9:!MM  ,>AR,RT^_ <@;Q;Q 02J< 8 &(8 '+&:&:{am&:&n#l!..{KIab5$0/-%" .  r,c |jd||||d|\}}}|j} |j} |j\} } } }t j || | | z|f}|jd| z}|j | ||jd}|j | ||jd}|ddkD}|j |||\}}|j|||}t|||||| | S)a input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids). Examples: ```python >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import torch >>> from transformers import OwlViTProcessor, OwlViTForObjectDetection >>> processor = OwlViTProcessor.from_pretrained("google/owlvit-base-patch32") >>> model = OwlViTForObjectDetection.from_pretrained("google/owlvit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text_labels = [["a photo of a cat", "a photo of a dog"]] >>> inputs = processor(text=text_labels, images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2] >>> target_sizes = torch.tensor([(image.height, image.width)]) >>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax) >>> results = processor.post_process_grounded_object_detection( ... outputs=outputs, target_sizes=target_sizes, threshold=0.1, text_labels=text_labels ... ) >>> # Retrieve predictions for the first image for the corresponding text queries >>> result = results[0] >>> boxes, scores, text_labels = result["boxes"], result["scores"], result["text_labels"] >>> for box, score, text_label in zip(boxes, scores, text_labels): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected {text_label} with confidence {round(score.item(), 3)} at location {box}") Detected a photo of a cat with confidence 0.707 at location [324.97, 20.44, 640.58, 373.29] Detected a photo of a cat with confidence 0.717 at location [1.46, 55.26, 315.55, 472.17] ```)rrrrrrr)r9r8rr rr:r;rT) rr:r;rr'rrrr~)rErrrrrrrrrrrr@rArrmax_text_queriesrrrrs r*rz OwlViTForObjectDetection.forwardasCf.FT-E-E. %)%= .  . * k700  44HSHYHYE &(9:mmK*>PSd>dfp1qr %??1-;#++J8H,J\J\]_J`a %%j2BIOOTVDWX v&* '+&:&:; V`&a#l'' [BZ[ *$$!%* .  r,rrxrT)rNrOrPrrSr staticmethodr_r'rrr?rRrrrJrrrrrrrrrrr~rrrs@r*r=r=s |$cVY^c^j^j 33SXS_S_,*/ &&&&#'     H26*. 1&&1''$.1LL4' 1 u  ! 10*/ -4<<-4''-4 -4 #' -4 +, -4 u  !-4d*/'.'''.#''.+, '. u  ! '.Z*/ *5#//*5!,,*5#' *5    *5X8<). Q ''Q "--4Q #' Q +, Q 0 Q Q f /3). V <<V ''V  t+ V #' V +, V  %V V r,r=)r5r%rnrr=)Nr)LrQcollections.abcr dataclassesrtypingrr'rrr r0 activationsr masking_utilsr modeling_layersr modeling_outputsr rmodeling_utilsrrprocessing_utilsrutilsrrrrrr utils.genericrrutils.output_capturingrconfiguration_owlvitrrrtransformers.image_transformsr get_loggerrNloggerr+r2r4r`rdrtr{r~rrPrrr\rrrrr%rRr]rnrzrr5rrr=__all__rTr,r*rs$! &!/9KF&J5TTF   H %`U\\`ell` -ELL-U\\-  ! ;!  ! JGvG&G EF Ev E" "'0  + ++   + \  * [*   * ZFRYYFR299L!% II% <<% % << % LL4' % T\ %%'(%86)bii6)t   3B4vO4v4vp BII D< 1< ~/ +/ d( 3( V. -. bk 'k k \bii&-1 -1`] 4] @ wr,