^j{UddlmZmZddlmZddlZddlmZddl m Z ddl m Z ddl mZmZmZddlmZmZGdd eZdaedzed <d ed dfd Z ddej0dedzd eej0edzffdZ ddej0dededzd ej0fdZ ddej0dedededej:dedzd ej0fdZ ddej0dedededzd ej0f dZ ddej0dedzd eej0e e ffdZ! ddededzdedzd ej0fdZ"y))ABCabstractmethod)AnyN) ShardedTensor)Shard)_all_gather_dtensor_create_chunk_dtensor_create_chunk_sharded_tensor) DeviceMeshDTensorceZdZdZedej deej edzffdZ edej dedej fdZ e ddej de d e d e d e jd ejdzdej fd Zedej de dedej fdZedej deej eeffdZedededzdej fdZy)FSDPExtensionsz This enables some customizable hooks to enable composability with tensor parallelism. To activate these hooks, use :func:`_set_fsdp_extensions` to set a custom :class:`FSDPExtensions` that implements the hooks. tensorreturnNcy)z6E.g. converting ``DistributedTensor`` to local tensor.Nselfrs r/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/torch/distributed/fsdp/_fsdp_extensions.pypre_flatten_transformz$FSDPExtensions.pre_flatten_transforms param_extensioncy)z6E.g. converting local tensor to ``DistributedTensor``.Nr)rrrs rpost_unflatten_transformz'FSDPExtensions.post_unflatten_transforms rrank world_sizenum_devices_per_nodepgdevicecy)z6Shards a tensor to chunks and returns the local chunk.Nr)rrrrrrrs r chunk_tensorzFSDPExtensions.chunk_tensor( r device_meshcy)zAShards a tensor/DTensor to DTensor and returns the local DTensor.Nr)rrrr#s r chunk_dtensorzFSDPExtensions.chunk_dtensor5s rcy)z This is to be called before loading a *sharded* model state dict and should return the tensor and list of shards from which to load data. Nrrs rpre_load_state_dict_transformz,FSDPExtensions.pre_load_state_dict_transform?s r parent_meshcy)z This is to be called before loading a *sharded* DTensor state dict. This gathers tensor in FSDP dimension and returns local tensor of TP DTensor. Nr)rrr(s rall_gather_dtensorz!FSDPExtensions.all_gather_dtensorJr"rN)__name__ __module__ __qualname____doc__rtorchTensortuplerrrintdist ProcessGrouprr!r r%listrr'r r*rrrrrs     u||S4Z' (          '+         "       t#                    u||T%[( )       $&       rr _extensions flattenerrc|ayr+)r7)r8s r_set_fsdp_extensionsr:[sKrrfsdp_extensioncB||j|\}}|||fS|dfSr+)r)rr; new_tensorrs r_ext_pre_flatten_transformr>`s:!&4&J&J6&R# O  &. . 4<rrc2|||j||S|Sr+)r)rrr;s r_ext_post_unflatten_transformr@ks' !o&A66vOO MrrrrrcB| |jnt}||||||Sr+)r!r )rrrrrr;chunk_tensor_fns r_ext_chunk_tensorrCus;  % ## )    rr#c>| |jnt}||||Sr+)r%r )rrr#r;chunk_dtensor_fns r_ext_chunk_dtensorrFs5  % $$ "   rc||j|St|turtdt||j }||fS)NzExpected ShardedTensor, got )r'typerAssertionError local_shards)rr;shardss r"_ext_pre_load_state_dict_transformrLsW!;;FCC F|=(;DL>JKK  "F F rr(c<| |jnt}|||Sr+)r*r)rr(r;all_gather_dtensor_fns r_ext_all_gather_dtensorrOs,  % ))  ! 55rr+)#abcrrtypingrr0torch.distributed distributedr4+torch.distributed._shard.sharded_tensor.apir-torch.distributed._shard.sharded_tensor.shardr#torch.distributed.fsdp._shard_utilsrr r torch.distributed.tensorr r rr7__annotations__r:r1r2r>r@r3r5rCrFr6rLrOrrrrYs## E? 9E SE P&* ^d ")Nt-1 LL"T) 5<<t #$-1 LL#T) \\  -1  LL     #T)  \\4-1  LL #T)   \\ (-1  LL "T)  5<<e $%  -1 6  6d" 6#T) 6 \\ 6r