^j=dZddlZddlmZmZddlmZddlmZddl m Z m Z ddl Z ddl mZmZmZmZmZmZmZmZmZmZmZmZed Gd d Zed Gd d eeZy)zj Definition of CuTe inspired Layouts for DeviceMesh internal bookkeeping and functions to manipulate them N)IteratorSequence) dataclass)product)NoReturnoverload) as_tuplecoalesce complement compositionflattenIntTupleis_intis_tupleLayout make_layoutmatch_structuresuffix_productT)frozenceZdZUdZeedfed<eedfed<ddededzddfdZde fdZ d ede fd Z de fd Z defd Zdd ZdeddfdZdeefdZdedeeefdZdefdZdefdZedeeeeffdZy) _FlatLayouta A canonical CuTe layout for a single dimension of a DeviceMesh Utility class for representing an integer layout by borrowing ideas from CuTe Layout Algebra. See https://docs.nvidia.com/cutlass/media/docs/cpp/cute/02_layout_algebra.html for more details. Each layout is represented as a list of sizes and strides. We use it as a way for mechanical bookkeeping of the integers such as ranks in a SPMD mesh, and the transformation on top of it. Lots of methods of layout like coalesce, composition, complement, etc. are borrowed from pycute. https://github.com/NVIDIA/cutlass/blob/6dd13d42784ee5bfa232d2441e6b9a021c5c6290/python/pycute/layout.py#L137,L257 Note this is a CuTe-inspired layout, because CuTe uses co-lexicographic way in linearization while PyTorch is using lexicographic. So even though the CuTe documentation can still be referenced, the implementation will be different from that of PyCute's. This layout is _not_ itself subdivided into multiple dimensions. It might internally sometimes use multidimensional tuple to represent "irregular" layouts (e.g., flattening non-adjacent dims), but this should be considered an opaque implementation detail. This class guarantees that all equivalent layouts are encoded as the same normalized representation, and thus compare equal. This is achieved by flattening and coalescing compatible adjacent dimensions (which includes removing all dimensions of size 1). .shapestrideNreturnct|s"t|stdt|||n t |}t|s"t|stdt|t ||st d|d|dtt||}t|j}t|j}|dk(r |dk(rd}d}tj|d |tj|d |y) Nz"shape must be a tuple or int, got z#stride must be a tuple or int, got zsizes z and strides z don't match))rrr)rr TypeErrortyperr ValueErrorr rr rrobject __setattr__)selfrrcoalesced_layout flat_shape flat_strides i/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/torch/distributed/_mesh_layout.py__init__z_FlatLayout.__init__=sve}@e NO O!->%3HvA$v,PQ Quf-veWM&NO O#F5&$9:-334 .556   +"5JK 4*54;7ctd)NzWYou should never need to know the length of the internal representation of a FlatLayout RuntimeErrorr#s r'__len__z_FlatLayout.__len__Ts e  r)ictd)NzOYou should never need to index into the internal representation of a FlatLayoutr+r#r/s r' __getitem__z_FlatLayout.__getitem__Ys ]  r)cr|js tddSt|j|jS)Nrr)rrrr-s r' to_pycutez_FlatLayout.to_pycute^s*zz!Q< djj$++..r)c@tj|jSN)mathprodrr-s r'numelz_FlatLayout.numelcsyy$$r)ct|j|j}tt|jt|j Dcgc]\}}t ||}}}t|Scc}}w)u By-dimension composition allows one layout to "select from" or "filter through" another layout. Think of it as function composition: (self ∘ layout)(input) = self(layout(input)) between two layouts. This function is a wrapper of pycute's composition. Mental model about how to understand the composition logic: - The LEFT layout (self) defines the "output space" - what indices are possible - The RIGHT layout (layout parameter) acts as a "selector" - which specific indices to pick - The composition only generates indices that the left layout could originally produce, but the right layout determines which indices to be picked. - The stride of the composition layout will not be smaller than the stride of the right layout, because when picking the indices the composition will at least follow the right layout's stride to move forward. Example: self = (6,2):(2,1) # sizes=(6,2), strides=(2,1) layout = (3:2) # sizes=(3,), stride=(2,) self o layout = (3:2) Returns: Layout being composed. )r r4zipr rrr _MeshLayout)r#layoutresultrr result_axess r'r z_FlatLayout.compositionfsu.T^^-v/?/?/AB"%Xfll%;Xfmm=T!U v v &  ;''  sB world_sizecvt|j|}t|j|jS)uG Compute the "complement layout" relative to a given world_size. A complement layout fills in the "missing" factor so that: self repeat a layout of complement(self, world_size) will get a complete world_size. We use ⊗ to denote the repeat operation. Example: self = (4:1) # size=4, stride=1 world_size = 8 Then: complete needed factor = 8 / 4 = 2 complement(self, 8) = (2:1) Together they form: (4:1) ⊗ (2:1) = (4,2):(2,1) which has world_size = 4 * 2 = 8, as required. In distributed terms, complement() is often used to derive the "other" rank grouping when splitting processes into 2D meshes. For a visualized explanation, see https://x.com/ezyang/status/1962364978393981433/ )r r4rrr)r#r@r>s r'r z_FlatLayout.complements,,DNN,j96<<77r)c td|jDDcgc](}tdt||jD*c}Scc}w)a This function computes the all ranks specified by the layout staring from zero. How it works: 1. we enumerates every possible coordinate (like a nested for-loop). If sizes = (2, 3), we get the following coordinates: (0,0), (0,1), (0,2), (1,0), (1,1), (1,2) 2. For each coordinate, we compute a linear rank index as: all_ranks_from_zero = sum(coord[i] * strides[i] for i in range(ndim)) Example A: sizes = (2, 3) # 2 rows, 3 cols strides = (3, 1) # row-major layout coords = (0,0) -> 0*3 + 0*1 = 0 (0,1) -> 0*3 + 1*1 = 1 (0,2) -> 0*3 + 2*1 = 2 (1,0) -> 1*3 + 0*1 = 3 (1,1) -> 1*3 + 1*1 = 4 (1,2) -> 1*3 + 2*1 = 5 result = [0, 1, 2, 3, 4, 5] Example B: sizes = (2, 3) strides = (1, 2) # non-standard / strided layout coords = (0,0) -> 0*1 + 0*2 = 0 (0,1) -> 0*1 + 1*2 = 2 (0,2) -> 0*1 + 2*2 = 4 (1,0) -> 1*1 + 0*2 = 1 (1,1) -> 1*1 + 1*2 = 3 (1,2) -> 1*1 + 2*2 = 5 result = [0, 2, 4, 1, 3, 5] c32K|]}t|ywr6)range).0ss r' z2_FlatLayout.all_ranks_from_zero..s"@58"@sc3,K|] \}}||zywr6r)rEcrFs r'rGz2_FlatLayout.all_ranks_from_zero..s:$!QA:s)rrsumr;r)r#coords r'all_ranks_from_zeroz_FlatLayout.all_ranks_from_zerosJH!"@TZZ"@A  :#eT[["9: :   s-A c |j|jDcgc]#}|jDcgc]}||z c}%c}}Scc}wcc}}w)us Build global ranks specified by the layout via two-level ranks composition. The nested list forms the Cartesian product of all ranks for one layout and offset regarding filling up the world_size with the layout. The final global ranks are the addition of these two. The result is a list of lists: one sublist per layout. This rank list will be used to build the communicator underlying the layout and the given `world_size`. Example: world_size = 16 self.size = 4 self.stride = 1 ranks = [0, 1, 2, 3] offsets = [0, 4, 8, 12] result = [ [0+0, 0+1, 0+2, 0+3], # → [0, 1, 2, 3] [4+0, 4+1, 4+2, 4+3], # → [4, 5, 6, 7] [8+0, 8+1, 8+2, 8+3], # → [8, 9, 10,11] [12+0, 12+1, 12+2, 12+3], # → [12,13,14,15] ] )r rL)r#r@offsetranks r' global_ranksz_FlatLayout.global_rankssQ2//*5IIK (,'?'?'A BtVd] B  B sA A AAc\tt|jd|jk(S)NTreverse)tuplesortedrr-s r' check_sortedz_FlatLayout.check_sorteds!VDKK674;;FFr)ct|jdkryttt|j|jd\t fdt tdz DS)u Check if the layout has any overlap between the ranks it generates. If there is overlap, we return False, otherwise True. The layout is supposed to be injective i.e, aside from indice 0, indices from each dim of the layout must be non-overlapping. Example 1 - Valid (no overlap): Layout: sizes=(2,3), strides=(6,1) - Dim 1: stride=1, span=3*1=3, covers indices [0,1,2] - Dim 0: stride=6, span=2*6=12, covers indices [0,6] → No overlap since 6 > 3 Example 2 - Invalid (overlap): Layout: sizes=(2,3), strides=(2,1) - Dim 1: stride=1, span=3*1=3, covers indices [0,1,2] - Dim 0: stride=2, span=2*2=4, covers indices [0,2] → Overlap! stride=2 < span=3, so indices [0,2] are duplicated Example 3 - Invalid (overlap): Layout: sizes=(4,2), strides=(1,1) - Dim 1: stride=1, span=4, covers indices [0,1,2,3] - Dim 0: stride=1, span=2, covers indices [0,1] → Overlap! stride is same for two dims, so indices [0,2] are duplicated Returns: bool: True if no overlap, False if overlap detected TrRc3RK|]}||dz|dzzzdk( yw)rrNr)rEr/rrs r'rGz/_FlatLayout.check_orthogonal..s<  1IAq1u5 6! ; s$'r)lenrr;rUrallrD)r#rrs @@r'check_orthogonalz_FlatLayout.check_orthogonalsc: tzz?Q VC TZZ$@$OP  3v;?+   r)cBt|j|jS)z5Iterate over (size, stride) pairs for each dimension.)r;rrr-s r'sizes_and_stridesz_FlatLayout.sizes_and_stridess4::t{{++r)r6)r=r<rr<)__name__ __module__ __qualname____doc__rTint__annotations__rr(rr.r2rr4r9r r listrLrPboolrVr\propertyrr^rr)r'rrs8 c? #s(O8h84848.   S X /6/ %s%(<8S8]82% T#Y% N s tDI 8GdG# $# J,8E#s(O#<,,r)rceZdZUdZeedfed<deeddfdZe ddee dfdee dfdzddfd Z de fd Z e d e defd Ze d eddfd Zd e ezddfdZdeefdZdefdZedee dffdZde fdZde fdZdefdZde de ddddfdZdej6dej6fdZy)r<a% A multi-dimensional structure consisting of a series of dimension-less layouts This class represents the layout of a full DeviceMesh, where the overall top-level ndim and "logical" shape are well defined, but each individual mesh axis is squashed and normalized into a canonical _FlatLayout. It only contains methods that need to make use of this multi-dimensional structure (i.e., which access the ndim or the top-level sizes). Everything else should go on _FlatLayout and accessed by first calling .collapse(). .axesrNcDtj|dt|y)Nri)r!r"rT)r#ris r'r(z_MeshLayout.__init__s4t5r)sizesstridesc|tt|}t|t|k7r#tdt|dt|t dt ||D}||S)Nz1sizes and strides must have the same length, got z and c3>K|]\}}t|f|fywr6)r)rErFds r'rGz1_MeshLayout.from_sizes_strides..,sMA[!t,Ms)r rrZr rTr;)clsrkrlris r'from_sizes_stridesz_MeshLayout.from_sizes_strides"st ?nU34G u:W %CCJ.CsD$T^^-D)rZrirrr-s r'r4z_MeshLayout.to_pycute@s3 tyy>Q !Q< D$))DEEr)c:td|jDS)Nc3<K|]}|jywr6)r9r}s r'rGz._MeshLayout.top_level_sizes..Gs8dTZZ\8r)rTrir-s r'top_level_sizesz_MeshLayout.top_level_sizesEs8dii888r)c@tj|jSr6)r7r8rr-s r'r9z_MeshLayout.numelIsyy--..r)c>|jjSr6)r4cosizer-s r'rz_MeshLayout.cosizeLs~~&&((r)ctd|jD}td|jD}t||S)z Merge all axes into a single _FlatLayout. This is used to "forget" the multi-dimensional structure of this object and recover a "flat" (and coalesced) representation. c34K|]}|jywr6)rr}s r'rGz'_MeshLayout.collapse..Vs8dtzz8c34K|]}|jywr6)rr}s r'rGz'_MeshLayout.collapse..Ws: :r)rTrir)r#shapesrls r'collapsez_MeshLayout.collapseOs98dii88: ::67++r)startendr=crt|j}t|j|||t|S)z Replace (out-of-place) the start:end slice with the given list of layouts Returns the concatenation of self[:start] + layout + self[end:]. )rerir<)r#rrr=new_axess r'splicez_MeshLayout.spliceZs1  ?"6;;/s8$$r)rank_mapc|jdk7rt|jst|j|j krt|j }|j |j}|j|j|jz|j|jzjdg|jS)a Leverage layout as an index for mesh tensor that re-maps the indexes after layout transformation to actual device ranks. With this method, the cute layout serves as the backend of indices bookkeeping for the mesh tensor when it comes to flatten, unflatten and slicing operations. The actual mesh tensor still represents the actual device assignment and ranks. We need this function to specify device allocation and create backend for a mesh. Although any transform of mesh tensors can be treated as a view or subset of mesh tensor, we do need to use the actual view or sub-tensor for DeviceMesh and its backend creation. The shape of the `rank_map` must be 1D and contiguous. Examples: Case 1 - Consecutive ranks, full world: original_mesh_tensor = [[0,1],[2,3]] # 2x2 mesh, ranks 0-3 world_size = 4 layout = Layout(2:2) Return: [[0,2],[1,3]] Case 2 - Non-consecutive ranks: original_mesh_tensor = [[10,20],[30,40]] # custom rank assignment world_size = 4 layout = Layout(2:2) Return: [[[10,30],[20,40]]] Args: rank_map: The concrete mesh tensor with actual device ranks Returns: torch.Tensor: A tensor representing the actual device allocation from rank_map r) ndimAssertionError is_contiguousr9rrr as_stridedrrreshaper)r#r self_layoutcomplement_layouts r'remap_to_tensorz_MeshLayout.remap_to_tensordsD ==A  %%' >> dkkm + mmo '228>>3CDx""  # #k&7&7 7  $ ${'9'9 9  '"-++- -r)r6)r_r`rarbrTrrdrr( classmethodrcrqr.rr2rwrrzrr4rgrr9rrrtorchTensorrrr)r'r<r<sm   S !!6Xk26t6GK #s(O .3CHo.D    5S5[55 9U9}99S5[-H (;/F6F 9sCx99/s/)) ,+ ,%C%c%=%]%/- /-/-r)r<)rbr7collections.abcrr dataclassesr itertoolsrtypingrrrtorch.distributed._pycuter r r r r rrrrrrrrr<rr)r'rs| .!%       $n,n,n,b $D-(;'D-D-r)