^j{dZddlZddlmZddlmZddlmZddlm Z m Z m Z ddl Z ddlZddlmZddlmZddlmZd d lmZd d lmZmZd d lmZmZd d lmZm Z m!Z!m"Z"m#Z#m$Z$d dl%m&Z&m'Z'd dl(m)Z)m*Z*m+Z+e+rddl,Z,Gdde&dZ-e*GddeZ.ddde/de0fdZ1d3dZ2d4dZ3de4e0de0de0fd Z5 d5d!e0d"e0d#e/d$e0dzd%e4e0dzd&e6e4e4e0e4e0ff d'Z7d(Z8d)e0d&ejrfd*Z: d6d+Z; d7d!e0d,ejrd-e6e0e0fd&ejrfd.Ze fd&ejrfd0Z?d8d1Z@d9d2ZAdgZBy):zImage processor class for SAM.N)Iterable)deepcopy)product)AnyOptionalUnion) functional) batched_nms)TorchvisionBackend) BatchFeature get_size_dict)group_images_by_shapereorder_images)IMAGENET_DEFAULT_MEANIMAGENET_DEFAULT_STDChannelDimension ImageInputPILImageResamplingSizeDict) ImagesKwargsUnpack) TensorTypeauto_docstringis_vision_availablec:eZdZUdZeeefed<eeefed<y)SamImageProcessorKwargsaV mask_size (`dict[str, int]`, *optional*): The size `{"longest_edge": int}` to resize the segmentation maps to. mask_pad_size (`dict[str, int]`, *optional*): The size `{"height": int, "width": int}` to pad the segmentation maps to. Must be larger than any segmentation map size provided for preprocessing. mask_size mask_pad_sizeN)__name__ __module__ __qualname____doc__dictstrint__annotations__w/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/sam/image_processing_sam.pyrr/s%CH~S>!r)rF)totalc#eZdZeZej ZeZ e Z ddiZ ddiZ dZdZdZdZdZdddZdddZdeeffd Ze d3d ed edzdeed effd Z d4d eeezeeefzezdzdeeezeeefzezdzd effd Z de!eefdefdZ" d3dddeddd dffd Z# d3d ed edzde$de%dee&zdzde'edfdzd efdZ(d e)dde$deddde$d ed!e$d"e*d#e$d$e*e)e*zdzd%e*e)e*zdzd&e$dzd'edzd(e$dzd e!e)de)e!eefffd)Z+ d5dd*d+ed,e*d-edzd.e)edzde,df d/Z- d6d0Z. d7d1Z/d2Z0xZ1S)8SamImageProcessor longest_edgeiTheightwidthkwargsc $t|di|y)Nr()super__init__)selfr3 __class__s r*r6zSamImageProcessor.__init__Ls "6"r)Nimagessegmentation_mapsreturnc &t|||fi|S)zp segmentation_maps (`ImageInput`, *optional*): The segmentation maps to preprocess. )r5 preprocess)r7r9r:r3r8s r*r=zSamImageProcessor.preprocessOsw!&*;FvFFr)rrc t|di|}|&t|tstdit |d}|&t|tstdit |d}||d<||d<|S)z Update kwargs that need further processing before being validated Can be overridden by subclasses to customize the processing of kwargs. r) param_namerr()r5_standardize_kwargs isinstancerr)r7rrr3r8s r*r@z%SamImageProcessor._standardize_kwargs\sv,6v6  Ix)H T={#STI  $Z x-P$`}]'_`M'{"/ r) old_shapec|\}}|dzt||z }||z||z}}t|dz}t|dz}||fS)zW Compute the output size given input size and target long side length. ??)maxr&)r7rBr.oldholdwscalenewhnewws r*_get_preprocess_shapez'SamImageProcessor._get_preprocess_shapeqsW ds"St_4E\4%T-=-=.2.6N. * 'E``qaiilooekk:`DN>BBA@<as D915D> do_resizedo_center_crop crop_sizererescale_factorrd image_mean image_stddo_padrfdisable_groupingc t||\}}i}|jD] \}}|r|j|||}|||<"t||}|Dcgc]}|jdd}}t||\}}i}|jD]4\}}|r|j ||}|j |||| | | }|||<6t||}| r|j|| |}||fScc}w)N)r)rMrOrPrR)rfr)ritemsrVrrU center_croprescale_and_normalizepad)r7r9r{rOrPr|r}rer~rdrrrrfrr3grouped_imagesgrouped_images_indexresized_images_groupedrUstacked_imagesresized_imagesrMraprocessed_images_groupedprocessed_imagess r*rlzSamImageProcessor._preprocesssC&0EV^n/o,,!#%3%9%9%; ; !E>!%>W_!`,: "5 ) ;((>@TU>LMU BC 0MM0E^fv/w,,#% %3%9%9%; = !E>!%!1!1.)!L!77 NL*V_N/= $U + =**BDXY #xx(88^nxo !555' NsC-z+np.ndarray | PIL.Image.Image | torch.Tensor crop_n_layers overlap_ratiopoints_per_cropcrop_n_points_downscale_factorc|j|}t||||||\}}} } |tjd}|j |}|j |}| j |} ||| | fS)a Generates a list of crop boxes of different sizes. Each layer has (2**i)**2 boxes for the ith layer. Args: image (`torch.Tensor`): Input original image target_size (`int`): Target size of the resized image crop_n_layers (`int`, *optional*, defaults to 0): If >0, mask prediction will be run again on crops of the image. Sets the number of layers to run, where each layer has 2**i_layer number of image crops. overlap_ratio (`float`, *optional*, defaults to 512/1500): Sets the degree to which crops overlap. In the first crop layer, crops will overlap by this fraction of the image length. Later layers with more crops scale down this overlap. points_per_crop (`int`, *optional*, defaults to 32): Number of points to sample from each crop. crop_n_points_downscale_factor (`list[int]`, *optional*, defaults to 1): The number of points-per-side sampled in layer n is scaled down by crop_n_points_downscale_factor**n. device (`torch.device`, *optional*, defaults to None): Device to use for the computation. If None, cpu will be used. cpu) process_image_generate_crop_boxesrsr]rr) r7rM target_sizerrrrr] crop_boxescropped_images input_labelss r*generate_crop_boxesz%SamImageProcessor.generate_crop_boxess>""5)DX      * E A O^\ >\\%(F]]6* ),,V4#v. ?NLHHr)c |\} } |jdd}|jdd}|jd|jdk7r td|j|jk7r|j |j}|jd} t j | t j|j} |dkDr| ||kDz} |dkDrt|||} | | |kDz} || }|| }||kD}t|}t||dd| | g} || }|| }|| }t||| | }t|}|||fS)a  Filters the predicted masks by selecting only the ones that meets several criteria. The first criterion being that the iou scores needs to be greater than `pred_iou_thresh`. The second criterion is that the stability score needs to be greater than `stability_score_thresh`. The method also converts the predicted masks to bounding boxes and pad the predicted masks if necessary. Args: masks (`torch.Tensor`): Input masks. iou_scores (`torch.Tensor`): List of IoU scores. original_size (`tuple[int,int]`): Size of the original image. cropped_box_image (`torch.Tensor`): The cropped image. pred_iou_thresh (`float`, *optional*, defaults to 0.88): The threshold for the iou scores. stability_score_thresh (`float`, *optional*, defaults to 0.95): The threshold for the stability score. mask_threshold (`float`, *optional*, defaults to 0): The threshold for the predicted masks. stability_score_offset (`float`, *optional*, defaults to 1): The offset for the stability score used in the `_compute_stability_score` method. rz3masks and iou_scores must have the same batch size.dtyper]) flattenrUrSr]rrrsonesbool_compute_stability_score_batched_mask_to_box_is_box_near_crop_edge _pad_masks _mask_to_rle)r7masks iou_scores original_sizecropped_box_imagepred_iou_threshstability_score_threshmask_thresholdstability_score_offsetoriginal_heightoriginal_width batch_size keep_maskstability_scoresscoresconverted_boxess r* filter_maskszSamImageProcessor.filter_masks-swF+8'''1-  a# ;;q>Z--a0 0RS S <<:,, ,#u||4J[[^ JJzELLQ S !Z/%ABI "C '7~Oef !%58N%NOII&i &.u5+ .A~0W   "i ))45"3_nUU#fo--r)c| |jn|}|d|df}t|tjtj fr|j }t|tjtj fr|j }g}t|D]\} } t|| tj rtj|| || <n(t|| tjs tdtj|| |dd} | dd|| d d|| d f} tj| | dd} |r| |kD} |j| |S) aF Remove padding and upscale masks to the original image size. Args: masks (`Union[List[torch.Tensor], List[np.ndarray]]`): Batched masks from the mask_decoder in (batch_size, num_channels, height, width) format. original_sizes (`Union[torch.Tensor, List[Tuple[int,int]]]`): The original sizes of each image before it was resized to the model's expected input shape, in (height, width) format. reshaped_input_sizes (`Union[torch.Tensor, List[Tuple[int,int]]]`): The size of each image as it is fed to the model, in (height, width) format. Used to remove padding. mask_threshold (`float`, *optional*, defaults to 0.0): The threshold to use for binarizing the masks. binarize (`bool`, *optional*, defaults to `True`): Whether to binarize the masks. pad_size (`int`, *optional*, defaults to `self.pad_size`): The target size the images were padded to before being passed to the model. If None, the target size is assumed to be the processor's `pad_size`. Returns: (`torch.Tensor`): Batched masks in batch_size, num_channels, height, width) format, where (height, width) is given by original_size. Nr1r2zIInput masks should be a list of `torch.tensors` or a list of `np.ndarray`bilinearF)mode align_corners.rr) rfrArsTensornpndarraytolist enumerate from_numpy TypeErrorF interpolateappend) r7rr`rarbinarizerftarget_image_size output_masksirinterpolated_masks r*post_process_masksz$SamImageProcessor.post_process_masks{sk>%-$44==(%h/'1BC nu||RZZ&@ A+224N *U\\2::,F G#7#>#>#@  ). 9 3 A}%(BJJ/ ++E!H5aa%,,7 kll ! eAh8IPZjo p  1#7S9Ma9PQR9S7SUqWklmWnopWqUq2q r  ! .?U_ot u $5$F!    1 2 3r)ct||||S)a$ Post processes mask that are generated by calling the Non Maximum Suppression algorithm on the predicted masks. Args: all_masks (`torch.Tensor`): List of all predicted segmentation masks all_scores (`torch.Tensor`): List of all predicted iou scores all_boxes (`torch.Tensor`): List of all bounding boxes of the predicted masks crops_nms_thresh (`float`): Threshold for NMS (Non Maximum Suppression) algorithm. )!_post_process_for_mask_generation)r7 all_masks all_scores all_boxescrops_nms_threshs r* post_process_for_mask_generationz2SamImageProcessor.post_process_for_mask_generations1J Scddr)N)NN)rg? rN)g)\(?gffffff?rr)rTN)2r r!r"r valid_kwargsrBILINEARrPrrrrrOrr{rerdrZrrfrrr6rrr r=r&rr$r%rr@tuplerLrVrrrrrzlistfloatrlrrrrr __classcell__)r8s@r*r-r-<si*L!**H&J$I D !D%IIJLN F.H"S1M#(?!@#04 G G&, G01 G  G GMQPT#&c3h7(BTIXc]*T#s(^;hFM  * uS#X c OS    L    L590C0C&,0C 0C , 0C j(4/ 0Cc>)*T10C 0Cd-6^$-6-6 -6 L -6  -6-6-6-6-6DK'$.-64;&--6t -6T/-6+-6" tN#T%S/%:: ;#-6f)&(;<+//I</I /I  /I t /I)-S D(8/I(/In# L.f3jer)r-rrNrrc(|||zkDjdtjjdtj}|||z kDjdtjjdtj}||z }|S)Nr)sumrsint16int32)rrr intersectionsunionsrs r*rrs .#99 :??%++?VZZ[]ejepepZq~(>>? D DRu{{ D [ _ _`bjojuju _ vF$v- r)cltj|dk(r1tjg|jdddd|jiS|j}|dd\}}tj |d\}}|tj ||jdddfz}tj |d\}}|||zz}tj|d\}}tj |d\} }| tj || jdddfz} tj | d\} }| || zz} tj| d\} }| | k||kz} tj| || |gd}|| jdz}|jg|ddd}|S) aL Computes the bounding boxes around the given input masks. The bounding boxes are in the XYXY format which corresponds the following required indices: - LEFT: left hand side of the bounding box - TOP: top of the bounding box - RIGHT: right of the bounding box - BOTTOM: bottom of the bounding box Return [0,0,0,0] for an empty mask. For input shape channel_1 x channel_2 x ... x height x width, the output shape is channel_1 x channel_2 x ... x 4. Args: - masks (`torch.Tensor` of shape `(batch, nb_mask, height, width)`) rNrRr]rdimr]) rsnumelzerosrUr]rFarangeminstack unsqueezereshape)rrUr1r2 in_heightrxin_height_coords bottom_edges top_edgesin_widthin_width_coords right_edges left_edges empty_filterouts r*rrs" {{5Q{{EEKK,EaE EE KKE"#JMFE99U+LIq 5<<y?O?O#PQUWXQX#YYii 0b9OL!'&YJ*??99-26LIq))Er*KHaeHOO!LTSTW!UUOYYB7NK%((;;OIIo26MJ *, 1IJL ++z9k<Hb QC ,))"- -C #++ %uSbz %1 %C Jr)cptj|tj|j}tj|tj|j}|\}}}}tj||||gg|j} t |j dk(r| jd} || zj}tj||dddf|d} tj||dddf|d} tj| | } tj| dS) zNFilter masks at the edge of a crop, but not at the edge of the original image.rrr rNr)atolrtolr) rs as_tensorrr]tensorlenrUrisclose logical_andany) boxescrop_boxorig_boxrcrop_box_torchorig_box_torchlefttoprxoffsetnear_crop_edgenear_image_edges r*rrs__XU[[VN__XU[[VNOD#q! \\D#tS125<< HF 5;;1!!!$ V^ " " $E]]5.q*ASTUNmmE>$'+BTUVO&&~7GHN 99^ ++r)r orig_height orig_widthc|\}}}}|dk(r|dk(r ||k(r||k(r|S|||z z |||z z } }|||z || |z f} tjjj|| dS)Nr)value)rsnnr r) rrrrr r rightbottompad_xpad_yrs r*rrs'D#uf qySAX%:"5&K:O . v|0L5E sECK 0C 88   " "5#Q " 77r)rrrrrr;cpt|tr td|jdd}g}t |dzD]-}t |||zz } |j t| /t|||\} } t| ||| ||\} } tj| } | j} tj| }|jdjdddd}tj| } tj |dddddddftj"}| || |fS) a Generates a list of crop boxes of different sizes. Each layer has (2**i)**2 boxes for the ith layer. Args: image (`torch.Tensor`): Image to generate crops for. target_size (`int`): Size of the smallest crop. crop_n_layers (`int`, *optional*): If `crops_n_layers>0`, mask prediction will be run again on crops of the image. Sets the number of layers to run, where each layer has 2**i_layer number of image crops. overlap_ratio (`int`, *optional*): Sets the degree to which crops overlap. In the first crop layer, crops will overlap by this fraction of the image length. Later layers with more crops scale down this overlap. points_per_crop (`int`, *optional*): Number of points to sample per crop. crop_n_points_downscale_factor (`int`, *optional*): The number of points-per-side sampled in layer n is scaled down by crop_n_points_downscale_factor**n. z.Only one image is allowed for crop generation.rRNrrrbr r)rArrSrUranger&r_build_point_grid_generate_per_layer_crops_generate_crop_imagesrsrrrrpermute ones_likert)rMrrrrrr points_gridrn_pointsr layer_idxsrpoint_grid_per_croprs r*rrs08%IJJKK$MK =1$ %8*H!*KLM,X6787}mUbcJ *?E; K+'N'j)J!!#Jkk"56O%//2::1aAFO[[0N???1aA:#>ekkRL  DDr)c gg}}|\}}t||}|jdd||g|jdt|D]}d|dzz} t||zd| z z} tt j | | dz z|z| z } tt j | | dz z|z| z } t| D cgc]} t| | z | z}} t| D cgc]} t| | z | z}} t ||D]J\}}||t|| z|t|| z|g}|j||j|dzL||fScc} wcc} w)aq Generates 2 ** (layers idx + 1) crops for each crop_n_layers. Crops are in the XYWH format : The XYWH format consists of the following required indices: - X: X coordinate of the top left of the bounding box - Y: Y coordinate of the top left of the bounding box - W: width of the bounding box - H: height of the bounding box rrbr)rrrr&mathceilr)rrrrr  im_heightim_width short_sidei_layern_crops_per_sideoverlap crop_width crop_heightr crop_box_x0 crop_box_y0r r boxs r*rrNs  J'IxY)Jq!Xy12a' +1-mj0A8H4HIJG/?!/C$Dx$OSc#cde $))W0@10D%E %QUe$efg @EFV@WX1sJ0A56X XAFGWAXYAsK'1Q67Y Y k: +ID#c$"3X>C+DUW`@abC   c "   gk * + + z !!YYs E)/E. n_per_sidecdd|zz }tj|d|z |}tj|dddf|df}tj|dddfd|f}tj||gdj dd}|S)z;Generates a 2D grid of points evenly spaced in [0,1]x[0,1].rrbNrr)rslinspacetilerr)r0r points_one_sidepoints_xpoints_ypointss r*rrps !j. !FnnVQZDOzz/$'2ZODHzz/!T'2Q ODH [[(H-2 6 > >r1 EF Mr)c\g}g}t|D]\} } | \} } } }|dd| || | f}|j||jdd}tj|j dj d}||| |z}t|||}|j|||fS)z Takes as an input bounding boxes that are used to crop the image. Based in the crops, the corresponding points are also passed. NrR)r)dimsr)rrrUrsrflipr_normalize_coordinates)rrMrr rrr[rtotal_points_per_croprrr r rr cropped_imcropped_im_size points_scaler7normalized_pointss r*rrzsN , 8 8#+ c5&1c&j$u*45 j)$**23/||O499t9DNNqQ Z]+l:2; V$$%67 8 0 00r)coordsrc<|\}}|dzt||z }||z||z}}t|dz}t|dz}t|j}|r|j ddd}|d||z z|d<|d||z z|d<|r|j dd}|S)z Expects a numpy array of length 2 in the final dimension. Requires the original image size in (height, width) format. rDrErrb).r).rr)rFr&rrr) rrAris_bounding_box old_height old_widthrI new_height new_widths r*r;r;s*J # J : :E&. E0A JIO$IZ#%&J f  # # %FAq)F^y9'<=F6NF^zJ'>?F6NA& Mr)rlec|d\}}tj||zt}d}d}|dD]}|||||z||z }| }|j||}|j ddS)z/Compute a binary mask from an uncompressed RLE.rOrrFcountsr)rsemptyrr transpose)rHr1r2maskidxparitycounts r* _rle_to_maskrQsKMFE ;;v~T 2D C FX"(S3; u  <<v &D >>!Q r)ct|j|tj|jd|}||}|Dcgc]}|| }}||}|Dcgc] }t |}}||||fScc}wcc}w)a Perform NMS (Non Maximum Suppression) on the outputs. Args: rle_masks (`torch.Tensor`): binary masks in the RLE format iou_scores (`torch.Tensor` of shape (nb_masks, 1)): iou_scores predicted by the model mask_boxes (`torch.Tensor`): The bounding boxes corresponding to segmentation masks amg_crops_nms_thresh (`float`, *optional*, defaults to 0.7): NMS threshold. r)rridxs iou_threshold)r rrsrrUrQ) rle_masksr mask_boxesamg_crops_nms_thresh keep_by_nmsrrHrs r*rrs  [[))!, -* KK(J'23!13I3K(J*3 43\#  4E 4 *i 33 4 4s  A8A=c|j\}}}|jdddjd}|ddddf|ddddfz }|j}g}t |D]}||dddf|k(dfdz}t |dk(rA||dfdk(r|j ||g||zgdn|j ||gd||zgdf|dd|ddz } ||dfdk(rgndg} | |djg| jz||z|djz gzz } |j ||g| d|S)z^ Encodes masks the run-length encoding (RLE), in the format expected by pycoco tools. rrbrNr)rOrJ) rUrrnonzerorrritemr) input_maskrr1r2diffchange_indicesrrcur_idxsbtw_idxsrJs r*rrs !+ 0 0J##Aq!,44Q7J ae z!SbS&1 1D\\^N C :  @!.A"6!";Q">?!C x=A !Q$1$ VUO?OPQ VUO6E>?RST AB<(3B-/!!Q$'1,1#8A;##%&)::funxXZ|O`O`Ob>b=ccc VUOv>? @ Jr))rrN)g4@)rrrrr)F)gffffff?)r\rN)Cr#r#collections.abcrrkr itertoolsrtypingrrrnumpyrrstorch.nnr rtorchvision.ops.boxesr torchvision.transforms.v2tvFimage_processing_backendsr image_processing_utilsr rimage_transformsrr image_utilsrrrrrrprocessing_utilsrrutilsrrrPILrr-rr&rrrrrrrrrrrr;r$r%rQrr__all__r(r)r*rqs% $'' $-7;AE5DD "l% "Ae*AeAeH NEcf.b,$8S 888%"$78 2E2E2E 2E 4Z 2E %)I$4 2E 4S ?DI %&2Ej"D#%,,_c14]b#ll;@c? \\8  d38n    4:>  r)