^jY| UddlZddlZddlZddlZddlmZmZmZddlm Z m Z m Z ddl m Z ddlmZmZmZddlZddlmZddlmZddlmZddlmZmZmZmZmZm Z ddl!m"Z"dd l#m$Z$m%Z%m&Z'm(Z(m)Z)m*Z*m+Z+m,Z,dd l-m.Z.m/Z/dd l0m1Z1dd l2m3Z3dd l4m5Z5ddl6m7Z8ddl9m:Z:gdZ;dZdZ?e@eAZBe3ezejzeDzeEzeAzZFeFeGeFzeHeFzeIeAdfzZJeIeAeJfZKeGeKZLeIeAeKeLzfZMe@ZNe@eeOd<ejdZQe GddZRe GddeRZS dKdejdeAdeAdeUd eUd!eBf d"ZVGd#d$ZWdLd%ZXddd&dejd'eHejjd(fd)eUd*e@ejdzd+eRdzd!eSf d,Z[d-eIeAeJfd.eMd/eSd!dfd0Z\d1ejejjzd2eAd!efd3Z]dejd/eSd!ejdzfd4Z_d5eIeAefd/eSd!eIeAeffd6Z`ejdejd/eSd!eIeAeJffd7Zbejdejd5eIeAeJfd/eSd!e5fd8Zcd9ejjd!dfd:Zdd5eMd!eIeAeJffd;Zed9ejjd5eIeAeJfd/eSd!eMfd<Zfejdejd=eHejjd(fd/eSd!eMfd>Zgdejd9ejjd.eMd/eSd!eMf d?Zhejdejd=eHejjd(fd5eMd/eSd!df d@Ziddd&dejd*e@ejdzd+eRdzd!eIeAeJffdAZjddd&dejd=ejjeejjzd*e@ejdzd+eRdzd!eMf dBZkddd&dejd=ejjeejjzd*e@ejdzd+eRdzd!eHeIeAeJfeMff dCZldejd5eIejeIeAeJffeIeAeJfzd!eIeAeJffdDZmddEdejd-eIeAeJfd+eRdzd!e5fdFZnddEdejd=ejjeejjzd.eMd+eRdzd!df dGZoddEdejd=ejjeejjzd-eIeAeJfd.eMd+eRdzd!e5f dHZpeddEdejd+eRdzd!dfdIZqeddEdejd=eHejjd(fd+eRdzd!dfdJZry)MN)Callable GeneratorIterable)asdict dataclassfield)chain)Anycast no_type_check) ShardedTensor)_broadcast_state_dict_distribute_state_dict_flatten_state_dict_gather_state_dict_offload_state_dict_to_cpu_unflatten_state_dict)_CHECKPOINT_PREFIX)FullOptimStateDictConfigFullStateDictConfigFullyShardedDataParallelOptimStateDictConfigShardedOptimStateDictConfigShardedStateDictConfigStateDictConfig StateDictType)._get_module_fsdp_state_if_fully_sharded_moduleFSDP_WRAPPED_MODULE)HYBRID_SHARDING_STRATEGIES)DTensor)_IncompatibleKeys)DistributedDataParallel) tree_map_only) FQNS_T PrimitiveType ValueType DictValueTypeListDictValueTypeOptimizerStateTypeStateDictOptionsget_model_state_dictget_optimizer_state_dictget_state_dictset_model_state_dictset_optimizer_state_dictset_state_dict _flat_param param_groupsparamsstater&_patched_state_dictc#Ktj}tj d|rtjyy#|rtjwwxYwwN)gc isenableddisableenable) is_enableds r/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/torch/distributed/checkpoint/state_dict.py _gc_contextr>RsDJJJL  IIK : IIK s)A$AA$A!!A$ceZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed<dZ eed <dZ eed <d Z eed <y )r*ap This dataclass specifies how get_state_dict/set_state_dict will work. - ``full_state_dict``: if this is set to True, all the tensors in the returned state_dict will be gathered. No ShardedTensor and DTensor will be in the returned state_dict. - ``cpu_offload``: offload all the tensors to cpu. To prevent CPU OOM, if ``full_state_dict`` is also true, then only the rank0 will get the state_dict and all other ranks will get empty state_dict. - ``ignore_frozen_params``: if the value is True, the returned state_dict won't contain any frozen parameters -- the ``requires_grad`` is False. The default value is False. - ``keep_submodule_prefixes`` (deprecated): when ``submodules`` is not None, this option indicates whether to keep the submodule prefixes from the state_dict keys. or example, if the submodule is ``module.pretrain`` and the full FQN of the parameter is ``pretrain.layer1.weight`` of the param. When this option is True, the parameter's key in the returned state_dict will be ``pretrain.layer1.weight``. If the options is False, the key will be ``layer1.weight``. Note that if ``keep_submodule_prefixes`` is False, there may be conflicted FQNs, hence there should be only one submodule in ``submodules``. - ``strict``: the ``strict`` option when ``set_state_dict`` calls model.load_state_dict(). - ``broadcast_from_rank0``: when the option is True, rank0 should receive a full state_dict and will broadcast the tensors in the state_dict/ optim_state_dict one by one to other ranks. Other ranks will receive the tensors and shard according to the local shards in the model and optimizer. ``full_state_dict`` must be set to True when using this option. This option currently only supports DTensor, not the legacy ShardedTensor. Ffull_state_dict cpu_offloadignore_frozen_paramsTkeep_submodule_prefixesstrictbroadcast_from_rank0flatten_optimizer_state_dict_fqn_modifiersdsd_fqn_modifiersN)__name__ __module__ __qualname____doc__r@bool__annotations__rArBrCrDrErFrHstrr=r*r*]s_"H"OT!K!&$&$(T(FD!&$&). $.-s-rQr*cXeZdZUeeZeeejze ejzfe d<eeZ eeejze ejzfe d<ee Z e ee d<dZee d<dZee d<ej$Zee d<eeZeej0e d <y ) _StateDictInfo)default_factoryfqn_param_mappingshared_params_mappingsubmodule_prefixesT handle_model handle_optim fsdp_context fsdp_modulesN)rIrJrKrdictrUrOtorchTensorr$rNrVsetrWrXrMrY contextlib nullcontextrZrlistr[nnModulerPrQr=rSrSs d#t ell $ d#4 ell $$)#=C=L$L$'33L(3$)$$?L$ryy/?rQrSmodelnamerHskip_ddp_prefixskip_compiler_prefixreturnc|jtd}d|vr|hS|jd}g}|}t|D]\}} t |t r5| dk7rt d| d|j}|r:|j| Lt |tr|t|dz krW||dztk(rHdj|} t|t} | r| d} | jD chc]} | |  c} cSt|t}| tk7s|j| t|| }t |t j"j$j&r7| dk7rt d| d|j(}|rN|j| at+||r:t||j-| x} rt+|| r t|| }|j| | t.j0jj2k(r|t|dz k7st5d t|| } dj|jtdhScc} w) a This API is used to convert the name of a parameter to the FQNs. For FSDP without `use_orig_params`, the name of FlatParameter can be mapped to multiple original parameters. As a result, the return type of this function is `set[str]`. Args: module (nn.Module): the root model. name (str): the name skip_ddp_prefix (bool): whether to skip DDP's `module` prefix Returns: The canonical FQNs based on the model traversal. .modulezExpected 'module', got '' _orig_modzExpected '_orig_mod', got 'z-Expect `_extra_state` to be the last obj name)replacersplit enumerate isinstanceDDPAssertionErrorrmappendFSDPlen _FLAT_PARAMjoingetattr_fqnsrr]_dynamo eval_frameOptimizedModulerphasattrgetrcmodules_EXTRA_STATE_KEY_SUFFIX RuntimeError)rerfrHrgrh obj_names fqn_obj_namescurr_obji curr_obj_nameprefix flat_paramfqn removed_fqns r= _get_fqnsrsS. <<*B /D $v  3IMH%i0&<= h $($'? a%PQQH"$$]3 $ '3y>A%%)AE*:k*I-0$X{;  &xq\F4>4D4DES6(3%(EEx)<=H 33$$]3"8]; %--":":"J"J K +$'B=/QR%STT))H'$$]3x!23"F'(4E"F"H"L"L!#;x5#*8[#A   / 1 1 I III**&'VWW"8];M& C DD7Fs0 I0c eZdZy) _EXTRA_STATEN)rIrJrKrPrQr=rrsrQrc#Ktdtjdtdtffd |dEd{y7w)Nrmcurr_fqnric3Kj||r|dnd}|jD]T\}}|vr t|r'|t|j vr|dd}n||}||Ed{Vt |j d|jdD] \}}||jvr||}||f"t|jdtjjtjjk7r7|tjjj}|t!fyy7׭w)NrlrkF)recurseget_extra_state)addnamed_childrenrr|valuesr named_buffersnamed_parameters_non_persistent_buffers_set __class__rcrdrrrmrr) rmrrf submodulenew_fqnobjrHrvisited_moduless r=rz+_iterate_valid_model_state..recursesrF#%-hZq>2%446 3OD)O+ 12>GF,=>@GGII#3B-%Jtf-y'2 2 2 3   /1H1HQV1H1W  ID#v999! 4&)G3,    F$$&79R9R Syy(( )" 2::#4#4#L#L"MNG<>) )  ) 3sA;E>E?CErk)r_rcrdrOr)rerHrrs `@@r=_iterate_valid_model_staters?&)eO * *S *Y *Dub!!!s;AAA) submodulesoptionsoptims. optim_onlyrrc |rtjdtd|r |s td|xs t }i}i}t |D]\}}t |trt||} |j|} | 2ttt||j| ||||<n| j||<| D]} t |tr||| <t|j!D])\} } | D]} tt"j$| || <!+t} |ret|}|j'D]G\}}||vr t||} t)| dk7r t+d| jd| DI|j,r|j.s t1dt3j4|}|r|j.rat7|j8|j8 }t;|j8|j8xs |j, }t<j>}nz"_verify_options..Ls%@CQi%@sz?full_state_dict must be True when broadcast_from_rank0 is True.)offload_to_cpu rank0_only)rc3Ktj5tjddtt j ||||5dddddddy#1swYxYw#1swYyxYww)NignorezFSDP.state_dict_type)messagecategoryrmstate_dict_typestate_dict_configoptim_state_dict_config)warningscatch_warningsfilterwarnings FutureWarningrxrrs r=$fsdp_state_dict_type_without_warningz=_verify_options..fsdp_state_dict_type_without_warningjsy((* ''&<}))!$3&7,C       s4A;6A/ A#A/ A;#A, (A//A84A;rr)rUrVrWrZr[rXrYrP),rwarnrrr*rrtrrrr r_rOupdatecopyrbitemsr]r^ named_modulesryrvrEr@ ValueErrorrxr[rrArrFULL_STATE_DICTrrSHARDED_STATE_DICTr`contextmanager functoolspartialrarSrrcrd)rerrrrrUrVrfparamfqnsrparam_fqns_rWrmr[rrrrrZs r=_verify_optionsrs#  I   & I  +)+GKMOQ1%8/ e e\ * %##E* ? S,U3 4 ; ;D A+>O  " "  # $!(( 0+/$;  "--   / +3-!$ryy/<8#^&kAo  rQmodel_state_dictoptim_state_dictinfoc |jD]}t|}|td|jrk|si|js]|j sQ|j r |js9|jr-|js!tdtjd|jr4|s2|j r |js|jstd||D]}t|vs t|dtdy)Nz)Expected a fsdp_state with a fsdp module.z}The option indicates that model state_dict is required to save or load, but model state_dict is empty.rank = dist.get_rank()=rlzgThe option indicates that model state_dict is required to save, or load but optim state_dict is empty. z contains z6. This can happen if the model is not the root module.)r[rrvrXrWrBrAr@rDrErdistget_rankrYrz)rrrrm fsdp_statekeys r=_verify_state_dictrs ##NCFK   !LM MN  ''))!!d&:&: KK)) 'mmo'q *    %%$*>*>..::J9KM    # %z+/** rQrapict||}|tvr+tjt|j||}|S)N)self)r|r5rrr)rrcalls r=_state_dict_fnrs9 3 D ""  !<3G KrQct|tr|}n+|jrtt|jd}ny|jt vry|j }t|try|S)Nr)rtrxr[r sharding_strategyr process_grouptuple)rer fsdp_modulers r=_get_fsdp_process_grouprsf%  4!2!21!56 $$(BB--M-' rQ state_dictc|jrF|jrtjj sdnd}t ||j|S|jr t |S|S)NrP)r)rA ranks_only)r@rAr] distributedis_initializedrr)rrrs r=_maybe_full_or_cpu_state_dictrsn $$E,=,=,L,L,N   " D$4$4     )*55rQc f|jsiS|j5t|d}dddtj D]}t ||}t |dk7rtd|dt |d|tt|}||k7sTdtfd}|||std|d ||j|||<|jrYi}|D]P}|jD]?}|j|s|jr ||||<*|t |d} |||| <AR|}|j rI|j#D]6\}} | j$rt ||}|D]}|j|8t'||S#1swYwxYw) NrroExpected 1 FQN for key '', got z: ric t|t|k\ry|jd}|jd}d}t|D]:\}}|||k(r'|dz }|t|k(s"|t|dz k(cS|dvr:yy)NFrlrro)rmrpT)ryrrrs)rr fqn_split key_splitfqn_idxkey_idxkey_names r=verifyz%_get_model_state_dict..verifyss8s3x' IIcN IIcN )29)=%%GX9W#551 "c)n4#*c)nq.@#@@!%<< $%rQzAn unexpected key, z, exists. FQN is )rXrZrrbkeysrryrvnextiterrMrpoprW startswithrCrBr requires_gradr) rerrrrrrnew_state_dictrrrs r=_get_model_state_dictrs      ;8^E<8: ;JOO%&2$ t9> *3%ws4ykD6J 4: #: D "#s#"%8=Nse#TUU(nnS1JsO=2@ /1 >C11 >~~f-//*4S/N3'!#f+-0G.8oN7+ > >$    002 $JC""UC(D $s# $  $ )T ::s;;s F&&F0c |jr|s|js tiiSi}t||jD]\}}t |||j}t |||jdd}t ||D]f\}} |jrtjdk(r9|| k7r4|j|d} | |jrtd|d| || <||| <hd} |js |jrnt} |jD]G}tj |s|j#dkDs-| j%|j&Itj&d| vr&| j)tj&dd} t+| dk(r.| j%tj,j/nt+| dkDr t1d |jr3t3||| j|j|j4 n(|jrt7||| j |j9||j;5t=tt?|d ||j| cdddS#1swYyxYw)NF)rgrhrz Missing key: rlmetaTrozMultiple devices found)devicerDrArload_state_dict)rrDassign) rXrEr!rrHrziprrrrDrr@r_rr] is_tensordimrrremoverydistributed_c10d_get_pg_default_devicerrrArrrZr r) rerrlocal_state_dictrvaluerfqns_with_prefixrfqn_with_prefix load_valuerdevicess r=_load_model_state_dictr 0sq   Z8Q8Q R((08N8NO6 UT%;%;<$    " "!!&  %(.>$? 6 C--A1E('^^C6 %{{*]3%q+ABB2 S!   JJtJ))+ ;  # K + OOO%rQc < dtttfdtdtttff fd d i}t t |t jD]C\}}t d|}|j t tttf||Et t|tD]\}|jt}t tt|D]+}|jD]\}}||td|d|<-^|S)a3 This API flattens the optimizer state_dict to support optimizer resharding for MPMD, e.g., pipeline parallelism. Without the API, the original optimizer state_dict looks like: { "state": { "layer1.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, "layer2.weight": { "step": 10, "exp_avg": SomeTensor, "exp_avg_sq": SomeTensor }, }, "param_groups": [ { "lr": 0.0, "betas": (0.9, 0.95), ..., "params": ["layer1.weight", "layer2.weight"] } ] } With this API, the optimizer state_dict looks like: { "state.layer1.weight.step": 10, "state.layer2.weight.step": 10, "state.layer1.weight.exp_avg": SomeTensor, "state.layer2.weight.exp_avg": SomeTensor, "state.layer1.weight.exp_avg_sq": SomeTensor, "state.layer2.weight.exp_avg_sq": SomeTensor, "param_groups.layer1.weight.lr": 0.1, "param_groups.layer2.weight.lr": 0.1, "param_groups.layer1.weight.betas": (0.9, 0.95), "param_groups.layer2.weight.betas": (0.9, 0.95), } The "state" section supports arbitrary levels of nesting for optimizers like Shampoo. nested_dictrrici}|jD]Q\}}t|}|r|d|n|}t|tr|j ||E||||<S|S)a Recursively flatten a nested dictionary with dot-separated keys. Args: nested_dict: The dictionary to flatten prefix: The prefix to prepend to all keys Returns: Flattened dictionary with dot-separated keys rl)rrOrtr\r) rr flattenedrrstr_keyfull_key_flatten_state_nested_dict_raise_if_type_not_supporteds r=r!z=_flatten_optim_state_dict.._flatten_state_nested_dicts+- %++- ,JC#hG06&7),GH%&  !;E8!LM-U3&+ (# ,rQct|tjttt fst dt|dy)Nz[Flattening optimizer state_dict only supports tensor, int, float, dict states now. Type is rl)rtr]r^intfloatr\NotImplementedErrortype)vs r=r"z?_flatten_optim_state_dict.._raise_if_type_not_supporteds@!ellC=>%7)1& ?rQrl)r\rOr r&r r'_STATErrr(_PGrrrb) rretrr4 state_prefixrrkr(r!r"s @@r=_flatten_optim_state_dictr.s R#s(^-0 c9n <!#C=*V*<=CCE U 3%(  &tDcNE'BL Q  -z#?, w'S 4( ,C#))+ ,1*+se1SE1#&' , ,, JrQc dtdtttfdtttffd}i}g}t|t|i}|j D]}|j tgi|tD]}|j|D]} | |jvr)d} |D]!} | tk(r td| d| } | |vrd} nnd} | s?|dt} t| tstd t| | j | |jsi|| <|j|D]M}td| d|}||vr |||}|t!t"|| |<5||t!t"|| |<O t!tt|dtd }|D]V} | tk(r |td|d| }| |dvr ||d| <.|d| |k7s:t%d |d| d |d |d| d |S)z This API unflattens the state_dict generated by _flatten_optim_state_dict(). Supports arbitrary levels of nesting in the state section through recursive reconstruction. See the docstring of _flatten_optim_state_dict() for more detail. flattened_keyflattened_dictric|d}i}|jD]q\}}|j|s|t|d}|jd}|}|ddD])} | |vri|| <t || t st || }+|||d<s|S)z Reconstructs a potentially nested value from flattened keys. For non-nested values, returns the value directly. For nested values, reconstructs the nested structure with string keys. rlNr)rrryrrrtr\rv) r0r1rrrr remaining_keypartscurrentparts r=_reconstruct_nested_dictz=_unflatten_optim_state_dict.._reconstruct_nested_dict s"?!$&( )..0 'JC>>&) F .M"'',E!Gcr  (w&$&GDM!'$-6((!$- ("'GE"I ; '@rQFrlTrExpected list, got rzaAll the parameters in the same parameter group should have the same saved param_group value. But z is z while other(s) is )rOr\r&r)r*r2rwrrUrVrtrbrvr'rr4r r'r)r rrr7r4pg_state return_osdrrr in_paramsr- flatten_keyr3 state_nameflattened_state_keyreconstructed_valuefirst_param_fqnrs r=_unflatten_optim_state_dictrAs00,0i,@0 c9n 0dE"$H&,eS(%CJ))A " & )1 E--e40 $444 %I(<$),Qse1QC&8 &*4(,I !%I !"g.!&$/(+>tF|n)MNN c"** c "'++e"4J-3HAcU!J<*H'**<.F//+ 0]E#J7 C GQ/G]E#J7 C?0 1 ftCy(2,w*?@C AG|#a'8!=>E $"' Q"aE)"==L.s-UQaj-Usrorr)$rYr)r*rrr[rZrxrrrbrrrqrr from_iterabler2r\rrangeryrrrvrrr r'rr(extendrFr)r.r)rerBrrr osdr-rGr3param_pid_mappingfqn_pid_mappingrrrrpidrEs r=_get_optim_state_dictrOs    ,2BR+@9H% 1nUL13   ""$ ++1%>  #f+**,- R!#?B6{q?QCK , ;< RX $?@zJ!!))L"5JJ#'  $%---U%BTBT-UUVF $Ss6{1C%D E  O#446 + U ,t9>(23%ws4ykJ4:& 11'.'*$'*$ + CK,,./ 8%c*$'v;??3#7F C  8 S RBG.!Q3/#"6!Qg R  ],V45<=L  L>L c Vi}g}t|t|i}i}tdtt|tDr|S|j D]t}|j tgi|tD]o} |j| D]Y} | |jvrCd} tt|tD]&} | ttt| tvs$d} nnd} | sZ|dt} t| tstdt| | j | | j rR| tt|tvrtt|t| || <n|j"rt%d| dtt|tD]D} | ttt| tvs$t'|tdz |t)| <F\rt'|td k(sg}tt|tD]>} t'ttt| td k(s.|j | @t'|dk7r t+d t'|tt'|j k7r t+d t'|tdz |t) <wtt|tD]M}|j-t)|d}|dk(r$|j/D]\}}|tk(r||||<O|S) a Extract the corresponding optim state_dict from ``optim_state_dict`` for ``optim`` and return the result optim state_dict. Args: model (nn.Module): the root model. optim (torch.optim.Optimizer): the optimizer. optim_state_dict (Dict[str, ValueType]): the superset optim state_dict that contains the optim state_dict of ``optim``. info (_StateDictInfo): state dict information. Returns: The optim state_dict of ``optim``. c3<K|]}t|tywr7)rtr$)rr-s r=rz*_split_optim_state_dict..s U!:a  UsFTrr8z'Missing optimizer state for parameter 'z' in checkpoint. The parameter requires gradients but has no saved optimizer state. To load anyway, use StateDictOptions(strict=False).rorzThere are param groups that have zero parameters. In such a case, DSD only support exactly one param group with zero parameters.But the loaded state_dict has zero or more than one param groups that have zero parameters.z`When there is a parameter group that has zero parameters, multiple optimizers are not supported.)r)r*allr r'r2rwrrUrVr(rbrOrtrvr'rrDrryidrrr)rer rrr4r9r: pg_mappingrrrr;loaded_param_groupr3r+pg_idxrrs r=_split_optim_state_dictrWsI*E"$H&,eS(%CJ!#J UtM;KF;S'T UU))7J " & ) VE--e4 V$444 %I.2)+;C+@/"*$tCy2DW2M"NN(,I! "!%I !"g.!&$/(+>tF|n)MNN c"&&d=2B62JKK%)-9I&9Q%RSV%Wc *EcUKRR +/%'7'<+V&d49.@.IJJ=@C=QTU=U 2&8#9: V7 V VD {7# $ )C&*+<>Ns>S&T 3"tDI'9''BCDIJJ12 33x1} 1#C()S1C1C-DD =25Z_1E1IJr,- .o7Jr-/?/DE * ;4 R< %++- *JCg~$)HV S !  * * rQc H|jsy|D]}t||r@t|vrt||||}n+t |t t ttf||}ni}|jrx|jD]&\}}t||}t||d} || k(r't|dk7rtd|dt||j} | j} |tD]N} t t tt f| } | t"Dcgc]}|j%| | }}|| t"<Pt t&|t}t)|j+D]+}| |vs|j|||j%| | <-)|j-5t/j0|||t3||}dddn|j4rd|_t7||f|}d|_dfd}t9t:j<||} td t?|\}}t?|\}}|j@rtC|| ntE|| |D]*}||vs||vrtd |d ||||<||||<,tG||}|tD]/}t"|vs gt t ttf|t"<1tI|d |ycc}w#1swY%xYw)NF)rhrozExpected 1 FQN for 'rrDTc|jdkDr*|j|S|jk7r td|S)NrzDevice mismatch)rrr)trs r=_devicez'_load_optim_state_dict.._devicezsD557Q;~!" 188+():;;rQzExpected device to be setrzExpected key 'z' in osd_mappingr)r)%rYrr)rWrAr r\rOr&r[rrryrvrr*r rrqr'rbrrZrxoptim_state_dict_to_loadrr@rOr#r]r^rrErrrr)rerBrrr r original_fqn_rfqns_with_compilerrfqn_with_compilerrGvalrr3 osd_stater-rr[ flatten_osd osd_mappingflatten_local_osdlocal_osd_mapping optim_keypgrs @r=_load_optim_state_dictri:s   ^N% ##:5*d$ $?4S)^ 4jA4$  "    $)#9#9#; X a  5%.(.|nGCI;Ohhj$6$:$:$<!)#.*AtCH~q1CGJ7|@C C):;F$*CL * !0@0HI inn./XAaxGP}}UVGW !))C1B"CDX+ X2""$ #'#@#@$1%> $    ! !#(D 4UUHdK #'D F ellG5EFA~$%@AA':;K'L $K3FGW3X 0 0((%k3DVT&{4EfU ) J $55 3,,YK7GH4?y3I%i03>y3I%i0 J 5!#4  's+ A"$>@Dc9n-r27; A 1u/0rrr)rerrrrs r=r+r+sV0     !  1=+R6    s +AA ct5t|tjjr|fn t |}t ||d||}t|||}ti|||cdddS#1swYyxYw)a Return the combined state_dict for optimizers. See ``get_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: The state_dict for ``optimizers``. :rtype: OptimizerStateType TrkN) r>rtr]r  OptimizerrrrOr)rerBrrrrs r=r,r,s6  *ekk&;&;<Mz"    !  1 DI2/6   s AA33A<ct5t|tjjr|fn t |}t ||d||}t||}t|||}t|||||fcdddS#1swYyxYw)a Return the model state_dict and optimizers state_dict. ``get_state_dict`` can process any module that is parallelized by PyTorch FSDP/fully_shard, DDP/replicate, tensor_parallel/parallelize_module, and any combination of these parallelisms. The main functions of ``get_state_dict`` are: 1.) returning a model and optimizer state_dict that can be resharded with a different number of trainers and/or different parallelisms. 2.) hiding the parallelism-specific state_dict APIs. Users don't have to call these APIs. 3.) sanity checking the result state_dict. The keys of the result state dictionary are the canonical FQNs (Fully Qualified Names). A canonical FQN refers to the FQN based on a parameter's position in an nn.Module hierarchy. More specifically, a canonical FQN to a parameter is the FQN returned by ``module.named_parameters()`` or ``module.named_buffers()`` when the module is not distributed by any parallelisms. Since the optimizer internally uses parameter IDs to represent a parameter, there will be a conversion from the parameter IDs to the canonical FQNs when calling this API. ``get_state_dict`` can also process a module that is not parallelized. In such a case, ``get_state_dict`` only performs one function -- converting the optimizer parameter IDs to the canonical FQNs. Example: >>> # xdoctest: +SKIP >>> import torch >>> from torch.distributed.fsdp import FullyShardedDataParallel as FSDP >>> from torch.nn.parallel import DistributedDataParallel as DDP >>> from torch.distributed.checkpoint.state_dict import get_state_dict >>> fsdp_model = FSDP(copy.deepcopy(model)) >>> fsdp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_model = DDP(copy.deepcopy(model)) >>> ddp_optim = torch.optim.Adam(model.parameters(), lr=1e-3) >>> ddp_state_dict, ddp_optim_state_dict = get_state_dict(ddp_model, ddp_optim) >>> fsdp_state_dict, fsdp_optim_state_dict = get_state_dict( ... fsdp_model, fsdp_optim ... ) >>> # if we simply call ddp_model.state_dict() and fsdp_model.state_dict(), >>> # the asserts will fail. >>> assert ddp_state_dict == fsdp_state_dict >>> assert ddp_optim_state == fsdp_optim_state_dict Args: model (nn.Module): the nn.Module to the model. optimizers (Union[None, Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. submodules (deprecated): Optional[set[nn.Module]]: only return the model parameters that belong to the submodules. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be returned. See `StateDictOptions` for the details. Returns: ``Tuple`` that contain model state_dict and optimizer state_dict. :rtype: typing.Tuple[typing.Dict[str, ValueType], OptimizerStateType] FrkN) r>rtr]r rmrrrrOr)rerBrrrrrs r=r-r-sP 2*ekk&;&;<Mz"    !  1=0 DI+-=tD!11!222s A,BB c |siSttt|jtj rt jdtdtttj tttff|}i}|jD]\}}|jD]}\}}||k7r t||}t!|dk7r t#dtt|d} |j%|jD cic] \} } | | z|  c} } |Sttttf|Scc} } w)NzPassing model_state_dict as a ``Dict[nn.Module, Dict[str, Any]]``is deprecated and will be removed in 2.5. If you need this feature, please preprocess the model_state_dict to achieve the same functionality.rrroz/FQNs for a submodule should only have 1 elementrl)rtrrrrcrdrrrr r\rOr&rrrryrvr) rercast_state_dictrrsub_state_dictrfmrrsubfqnrs r=_unflatten_model_state_dictrtRsF  $tJOO-./;  "   tBIItCN/C$CDjQ/1)8)>)>)@  %I~ ..0 a > -t9>(I!d,-Q/%%AOAUAUAWX Vf_e+X  Di(*55 YsE)rct||}t5t|dd|}t|i|t |||cdddS#1swYyxYw)a=Load the model state_dict. The counterpart of ``get_model_state_dict`` to set the state_dict to the model. See ``set_state_dict`` for the detail usage. Args: model (nn.Module): the nn.Module to the model. model_state_dict: (Dict[str, ValueType]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys * **unexpected_keys** is a list of str containing the unexpected keys :type model_state_dict: typing.Dict[str, ValueType] rPFrrN)rtr>rrr )rerrrs r=r.r.ws`:.I . EubUGL+R6%e-=tD EEEs )A  Act5t|tjjr|fn t |}t ||d|}ti||t||||dddy#1swYyxYw)aLoad the optimizers state_dict. The counterpart of ``get_optimizer_state_dict`` to set the state_dict to the optimizers. See ``set_state_dict`` for the detail usage. WARN: ``set_optimizer_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None :type optim_state_dict: typing.OptimizerStateType TrvN) r>rtr]r rmrrrri)rerBrrrs r=r/r/sz>  J*ekk&;&;<Mz"  ujT7S2/6uj2BDI J J Js AA11A:c.t||}t5t|tjj r|fn t |}t||| |}t|||t||||t|||cdddS#1swYyxYw)aLoad the model state_dict and optimizers state_dict. The counterpart of ``get_state_dict`` to set the state_dict to the model and optimizers. The given ``model_state_dict`` and ``optim_state_dict`` do not have to be returned by ``get_state_dict`` but must meet the following requirements: 1) all FQNs are canonical FQNs as defined in ``get_state_dict``, 2) if a tensor is sharded, it must be either a ShardedTensor or DTensor, 3) optimizer state_dict cannot contain the parameter IDs; the keys should be the canonical FQNs. WARN: ``set_state_dict`` can only be called before ``backward()`` or after ``step()`` is called on the optimizers. Otherwise, the optimizer states won't be initialized correctly. Args: model (nn.Module): the nn.Module to the model. optimizers (Union[Optimizer, Iterable[Optimizer]]): The optimizers that are used to optimize ``model``. model_state_dict: (Union[Dict[nn.Module, Dict[str, ValueType]], Dict[str, ValueType]]): the model state_dict to load. If the key of the ``model_state_dict`` is nn.Module, the key is a submodule of ``model`` and the value should be the state_dict of the submodule. When loading the state_dict, the prefix of the submodule will be append to the state_dict. optim_state_dict: OptimizerStateType: the optimizer state_dict to load. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: ``NamedTuple`` with ``missing_keys`` and ``unexpected_keys`` fields: * **missing_keys** is a list of str containing the missing keys of the model state_dict. * **unexpected_keys** is a list of str containing the unexpected keys of the model state_dict. :type model_state_dict: typing.Dict[str, ValueType] :type optim_state_dict: typing.OptimizerStateType rvN) rtr>rtr]r rmrrrrir )rerBrrrrs r=r0r0s\.I .  E*ekk&;&;<Mz"   :.>*>  +-=tDuj2BDI%e-=tD E E Es A*B  Bc$tjt||fd}||_tjt||dt t tfffd }||_tj|tj|y)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``model``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``model`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rercSr7rP_state_dict_callsr=state_dict_callz0_patch_model_state_dict..state_dict_call, !!rQrc|y)N)rrPr_load_state_dict_calls r=load_state_dict_callz5_patch_model_state_dict..load_state_dict_call7 z:rQN) rrr+rr.r\rOr rr5r)rerr}rrr|s @@r=_patch_model_state_dictr s6!(( "'E%-- ;c3h;1EO,01rQctjt|||fd}tjt|||dtt t fffd }tj|tj|t|tjjr|fn t|}|D]}||_||_y)aPatch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers``. Patch the ``state_dict`` and ``load_state_dict`` attributes of ``optimizers`` to be a partial function to call ``get_state_dict`` and ``set_state_dict``. Note that if there are multiple optimizers, all of the optimizers will be patched. So users only need to call one of the state_dict() to get the full result. Example: from torch.distributed.fsdp import FullyShardedDataParallel as FSDP from torch.distributed.checkpoint.state_dict import patch_model_state_dict model = fsdp(model) patch_model_state_dict(model) Args: model (nn.Module): the nn.Module to the model. options (StateDictOptions): the options to control how model state_dict and optimizer state_dict should be loaded. See `StateDictOptions` for the details. Returns: None )rerBrcSr7rPr{sr=r}z4_patch_optimizer_state_dict..state_dict_callhr~rQrc|y)N)rrPrs r=rz9_patch_optimizer_state_dict..load_state_dict_callrrrQN)rrr,r/r\rOr r5rrtr]r rmrrr)rerBrr}rr rr|s @@r=_patch_optimizer_state_dictrBs>!((  "&--  ;c3h;O,01 j%++"7"7 8  :  5* 45rQ)rGTT)rG)sr`rr8rcollections.abcrrr dataclassesrrr itertoolsr typingr r r r]torch.distributedrrtorch.nnrc'torch.distributed._shard.sharded_tensorr #torch.distributed._state_dict_utilsrrrrrr;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrtorch.distributed.fsdprrrrxrrrrr$torch.distributed.fsdp._common_utilsrr"torch.distributed.fsdp._init_utilsrtorch.distributed.tensorr torch.nn.modules.moduler!torch.nn.parallelr"rutorch.utils._pytreer#__all__rzr*rr)r_rOr$r^r$r%r%rbrr\r&r'r(r)r5rNrr>r*rSrdrMrrrr rmrrr ProcessGrouprrno_gradrr rr.rArOrWrir+r,r-rtr.r/r0rrrPrQr=rs 9900++ A   J,5<- "    S-'%,,6>c;FVAWW S)^$ '#}/@@@A&)US]*  ,.,. ,.^  @% @  @&. !% FE 99FE FEFE FE  FE  FER  %"Z)-'+ } 99} %++'', -}} BII% }  $ }}@+3 >*+(+ + +\ EKK$9$99 99* &S#X&4 #s(^$?; 99?;*?; #y.?;?;DA 99A S)^$A  A  A A H'&U[[22'&t'&T_*<_c9nAU_DE ;; ES)^$E E EPIA 99IAekk++S01IA IA IAIAXa 99a ;; a)a  a  aHgN 99gNekk++S01gN#gN  gN  gNgNZ)-'+ " 99" BII%"  $ "  #y. " R)-'+ * 99*  %%1F1F(GG* BII% *  $ *  * b)-'+ X2 99X2 %%1F1F(GGX2BII% X2  $ X2  4Y !3 34 X2v"6 99"6RYYS)^ 445S)^8LL"6 #y."6R(, $E 99$E3 >*$E $ $E  $EX(, (J 99(J %%1F1F(GG(J)(J  $ (J  (Jb(, =E 99=E %%1F1F(GG=E3 >* =E ) =E  $ =E=ED(,12 9912 $12 1212l (, ;5 99;5ekk++S01;5 $ ;5  ;5;5rQ