^j5dZddlmZddlZddlmcmZd d dZ d d dZ d ddZ d ddZ d ddZ d dd Z y)uVision utility functions for pre-computing very dynamic and data-dependent tensors that can break model graph capturing. All functions are standalone (no model weights) and compute tensors from `grid_thw` + config scalars. They are used by vision encoders and can be precomputed before `torch.compile` / `torch.export` tracing since they use untraceable ops (`repeat_interleave`, `.tolist()`, `nonzero()`, loops). Each `get_*` accepts an optional `kwargs` dict; if it contains the precomputed tensor under the natural key (`"cu_seqlens"`, `"position_ids"`, …), the function pops and returns it instead of computing. Vision encoders write `x = get_vision_x(..., kwargs=kwargs)` and the matching key is removed from the caller's kwargs as a side-effect of the pop. ) annotationsNcP||jddx}|Stj|dddf|dddfz|dddfjdtjj r |j ntj}tj|ddS) uGet cumulative sequence lengths from vision grid info, or pop from `kwargs` if precomputed. Args: grid_thw: `(num_images_or_videos, 3)` — temporal, height, width per entry. kwargs: optional caller kwargs — if it contains `"cu_seqlens"` it is popped and returned. Returns: `cu_seqlens`: `(total_patches + 1,)` int32 cumulative sequence boundaries. N cu_seqlensr)dimdtype)rr)value) poptorchrepeat_interleavecumsumjit is_tracingr int32Fpad)grid_thwkwargsrs d/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/vision_utils.pyget_vision_cu_seqlensr#sVZZ d-KKzX((!Q$(1a4.)H(STVWSW.Y`` uyy';';'=X^^5;;aJ 55V1 --c ||jddx}|S|j}t|tr0t j |g|j t|}g}t|j|jD]e\\}}}} t jt j||t j||d\} } || z| || z| f} | j| jddj} | j| jddj} |rrt j||j||z} |j!t j"| | j%|| j%|gd .|j!t j"| | gd j%|dht j&|d S) uGet position IDs for vision rotary embeddings, or pop from `kwargs` if precomputed. Args: grid_thw: `(num_images_or_videos, 3)` spatial_merge_size: merge block size — either a single `int` (same for all images) or a `(num_images_or_videos,)` tensor (per-image). kwargs: optional caller kwargs — if it contains `"position_ids"` it is popped and returned. include_temporal: when ``True``, prepend a temporal-index column and return `(total_tokens, 3)` — for encoders whose rotary embedding rotates T/H/W axes (minimax_m3_vl). When ``False`` (default), return `(total_tokens, 2)` for the 2-axis (h, w) case (qwen2_5_vl / qwen3_vl / glm4v / paddleocr_vl); the h/w indices are still repeated ``t`` times for video inputs. Returns: `position_ids`: `(total_tokens, 3)` long if ``include_temporal`` else `(total_tokens, 2)`, with the spatial indices laid out block-major over ``m×m`` spatial-merge blocks. N position_idsdeviceij)indexingrrrr)r r isinstanceintr tensorexpandlenziptolistmeshgridarangereshape transposeflattenr appendstackrepeatcat)rspatial_merge_sizeinclude_temporalrrrthw merge_sizehpos_idswpos_ids block_shapetpos_idss rget_vision_position_idsr;5s.vzz.$/OO|\ __F$c*"\\+=*>vNUUVYZbVcdL!$X__%68J8Q8Q8S!T X Aq:"^^ LL6 * LL6 * ( J AOZP ##K0::1a@HHJ##K0::1a@HHJ ||Af5GGANH    Xxq7I8??[\K],^df g h    Xx,@b I P PQRTU V W X 99\q ))rcL|,|jdd}|jdd}||||fSg}dg}d}||z|z}|dz} |jD]o\} } } t| t| t| } } } | |z} | |z}tj| | z|zj | | |}|| |zz }|||zz }| |z|z}||z|z}t j|d|d|fdd}|j | ||||}|jddd dd j | ||z||}|dk7jdd gj d }|j d }||dk7}|j||z|jd| z|d z}|j|j|| | z|zz }rtj|d }tj||jtj }tj"|}||fS)uGet window attention indices, or pop `"window_index"`/`"cu_window_seqlens"` from `kwargs` if both precomputed. Args: grid_thw: `(num_images_or_videos, 3)` spatial_merge_size: merge block size from vision config. window_size: window size from vision config. patch_size: patch size from vision config. kwargs: optional caller kwargs — if it contains both `"window_index"` and `"cu_window_seqlens"` they are popped and returned. Returns: `window_index`: `(total_tokens,)` long — reorder indices for windowed attention. `cu_window_seqlens`: `(num_windows + 1,)` int32 — cumulative window boundaries. N window_indexcu_window_seqlensrrconstantirrr )rr )r r'r"r r)r*rrpermutesumr-rextendr0r#rrunique_consecutive)rr1 window_size patch_sizerr=r>window_index_idvit_merger_window_sizespatial_merge_unitgrid_tgrid_hgrid_w llm_grid_h llm_grid_windexpad_hpad_w num_windows_h num_windows_w index_paddedseqlens index_newcu_seqlens_tmps rget_vision_window_indexrYfs(zz.$7 "JJ':DA  #(9(E!22 2L cO(,>>*L+Q."*//"3<!$Vc&k3v;11 11  Vj0:=>FFvz[ef&6L)LL&6L)LL#e+0FF #e+0FF uuUQq%$8*dK #++ M#9=J` $++Aq!Q:BB MM13IKa  4',,aV4<4!8Qv>**,**,A+$$$2A+$$$2'!'! 4 AtG $wtQw'7 7 @ @ B AtG $vdAg 6 ? ? A 1d7 #gdAg&6 6 ? ? A 1d7 #fT1Wo 5 > > @  &j!T' "a&j$'%: : C C E&j!T' "VD!G_ 4 = = ? AtG_F D!G4 4 = = ? AtG_vdAg . 7 7 9   Qv.33AOZP Qv.33AOZPAtT)*Q.tT1a7G1HHSSTUWXYaacjjklmq ?A aL  q 1' : ; O " ">!#4W#= > ?E$?L{{)#DQEIIaL#DE{{,#GQEIIaL#GH - --W+A-CP$E#Gs N*$ N/ N4N9c2||jddx}|S|j}tjd|z dd|z |}g}|j D]\}}t |t |}}tj|||z } tj|||z } tj | |d} tj | |d} |j| dddf|z| zjtj|S)uGet nearest-neighbor position IDs into a `num_patches_per_side**2` 2-D table, or pop from `kwargs` if precomputed. For each image of size `(h, w)`, maps fractional grid coordinates `i/h` to the nearest bucket on a `num_patches_per_side` grid (via `bucketize`) and flattens to 1-D embedding indices, concatenated across all images. Used by NaViT-style packers (e.g. MiniCPM-V). Args: target_sizes: `(num_images, 2)` int — `(h, w)` per image. num_patches_per_side: side length of the learned 2-D position-embedding grid. kwargs: optional caller kwargs — if it contains `"position_ids"` it is popped and returned. Returns: `position_ids`: `(sum(h_i * w_i),)` long — flat indices into a `num_patches_per_side**2` table. Nrrg?rT)right) r rr r)r'r" bucketizer-r,r0) target_sizesnum_patches_per_siderpos_idsr boundaries pos_ids_listheightwidthh_coordsw_coordsbucket_hbucket_ws rget_vision_nearest_position_idsrs$&**^T*JJwW  Fa"66QAU=U^deJL%,,.] F SZ<<v6?<<f5=??8ZtD??8ZtDXag.1EEPYY[\ ] 99\ ""rc||jddx}|S|\}}t|dj|zt|dj|zfS)uGet post-window-merge `(merged_h, merged_w)` Python ints, or pop from `kwargs` if precomputed. `.view()` needs Python ints, but `target_sizes[0].item()` is non-traceable. Callers must pop the precomputed value from `kwargs` when running under `torch.export`. Assumes uniform `target_sizes` across the batch (standard NaViT preprocessing output). Args: target_sizes: `(num_images, 2)` int — `(h, w)` per image. window_kernel_size: `(window_h, window_w)` window-attention kernel. kwargs: optional caller kwargs — if it contains `"merged_shape"` it is popped and returned. Returns: `(merged_h, merged_w)`: per-image grid size after window merging, as Python ints. N merged_shape)rr)rr)r r"item)r|window_kernel_sizermergedwindow_hwindow_ws rget_vision_merged_shaper sj"ND)IIvV +Hh |D!&&( )X 5s<;M;R;R;T7UYa7a aar)N)r torch.Tensorr dict | Nonereturnr)FN) rrr1zint | torch.Tensorr2boolrrrr) rrr1r"rFr"rGr"rrr!tuple[torch.Tensor, torch.Tensor]) rrrbr"r1r"rrrr)r|rr}r"rrrr)r|rrtuple[int, int]rrrr)__doc__ __future__rr torch.nn.functionalnn functionalrrr;rYrxrrrrrs9 # .*# .*.**.*.*  .*  .*l :+:+:+:+ :+  :+ ' :+B F.F.F.F.  F. ' F.TRV##69#CN##D\`bb4CbMXbbr