^j1 DddlZddlmZddlmZmZddlZddlmZGddeZdede eeffd Z d d d ede e e effd Z dede e efdede efdZd d de e efde e efdee eefe effdZej$d d de e efde e effdZy)N) defaultdict)Any NamedTuple)TensorcJeZdZUdZeeefed<eeefed<ee ed<y)_GroupSwapinInfoaPer-group bookkeeping for ``swap_in_optimizer_params_and_state``. Fields: live_group: live ``optimizer.param_groups[i]`` dict, mutated in place. swapin_group: per-group hyperparameter dict from the input state_dict (e.g. lr, betas, ...) plus packed ``params`` ids. swapin_params: replacement parameter tensors for ``live_group["params"]``. Same order as ``swapin_group["params"]`` (i.e. the i-th tensor corresponds to the i-th packed id), so the two are zippable. live_group swapin_group swapin_paramsN) __name__ __module__ __qualname____doc__dictstrr__annotations__listra/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/torch/optim/_stateless.pyrr s/ S#XsCx. <rrstatereturnct|ts"tdt|jdt d|Dr tdt d|j Dr td|S)zPCheck ``state`` is a dict mapping packed parameter ids to per-param state dicts.zswap_in_optimizer_params_and_state requires swapin_optim_state['state'] to be a dict mapping packed parameter ids to per-param state dicts, got .c3PK|]}t|tj ywN) isinstancetorchr).0ks r z(_validate_state_field..#s 61:a & 6s$&zmswap_in_optimizer_params_and_state requires optimizer.state_dict()-style state keyed by packed parameter ids.c3hK|]*\}}t|txrt|t ,ywr)rintr)rr vs rr!z(_validate_state_field..(s- Tda:a  9jD&9"9 9 Ts02zeswap_in_optimizer_params_and_state requires per-parameter optimizer state entries to be dictionaries.)rr RuntimeErrortyper anyitems)rs r_validate_state_fieldr)s eT " ;''( +   6 66 3   Tekkm TT 0   Lr optimizerztorch.optim.Optimizer param_groupsct|ts"tdt|jdt |j t |k7r td|S)zW``param_groups`` must be a list whose length matches the live optimizer's param groups.zvswap_in_optimizer_params_and_state requires swapin_optim_state['param_groups'] to be a list of param-group dicts, got rzVswapin_optim_state has a different number of parameter groups than the live optimizer.)rrr%r&r lenr+)r*r+s r_validate_param_groups_fieldr.0sn lD ) 66:<6H6Q6Q5RRS U   9 ! !"c,&77 "   ridxgroupr c t|ts td|jd}t|trt d|Ds tdt |dt |k7rtd|d|Dcgc] }||vs| }}|Dcgc] }||vs| }}|s|rtd|d|d |d |Scc}wcc}w) aVValidate a single swap-in param group against its live counterpart and return its packed parameter ids. Asserts: 1. swap-in group is a dict. 2. ``group['params']`` is a list of ints, and its length matches the params of the live optim group. 3. Keys inside swapin group should match the keys in the live optim group. zZswap_in_optimizer_params_and_state requires each optimizer param group to be a dictionary.paramsc3<K|]}t|tywr)rr#)rpids rr!z/_validate_group_against_live..Ts9!$ 39szswap_in_optimizer_params_and_state requires optimizer.state_dict()-style param_groups[*]['params'] entries keyed by packed parameter ids.zswapin_optim_state param group zF has a different number of params than the live optimizer param group.zswap_in_optimizer_params_and_state requires optimizer.state_dict()-style param group keys to exactly match the live optimizer group keys for group z. Keys only in swap-in: z. Keys only in live: r)rrr%getrallr-)r/r0r swapin_param_idsr swapin_only live_onlys r_validate_group_against_liver:As lD ) "  $''1 & -S9(896 O   5?s#344-cU3: :  +=aun1=K=!;qQl%:;I;i E%%0M1FykQR T  >;s CC C %C swapin_parametersswapin_optim_statec|js tdt|ts tdt |j d}t ||j d}t|j}d}t}g}tt|j|dD]q\} \} } t| | | } |j| |t| z} | t|kDr td||| }| }|j!t#| | | s|Dcgc] }||vs| }}|rtd |d ||fScc}w) a Validate and normalize optimizer state for ``swap_in_optimizer_params_and_state``. This follows the same structural assumptions as DCP-compatible optimizers, but consumes the raw ``optimizer.state_dict()`` format: ``state`` is keyed by packed parameter ids and each param group contains the live optimizer group fields plus a packed ``params`` list whose order matches ``optimizer.param_groups``. zHswap_in_optimizer_params_and_state requires initialized optimizer state.zMswap_in_optimizer_params_and_state requires a DCP-style optimizer state_dict.rr+rTstrictzrswap_in_optimizer_params_and_state requires the explicit parameter state to match optimizer.param_groups ordering.)r r r zswap_in_optimizer_params_and_state requires swapin_optim_state['state'] to be keyed only by packed parameter ids from param_groups[*]['params']; got extra keys r)rr%rrr)r5r.rvaluesset enumeratezipr+r:updater-appendr)r*r;r< swapin_stateswapin_param_groupsflat_parametersflat_param_offsetseen_param_idsgroup_swapin_infosr/r0r r7 next_offsetr r extra_keyss r_prepare_swap_inrNls ?? V   ($ / [  ));)?)?)HIL6%)).9,3356O"uN&/ I " "$7E'# " "e\8ULQ./'#.>*?? _- -9 ((9+F ' !!  )+  ;# J*EQn-D!EJE 99Ca I  + ++Fs ( E 2E c#6Kt|||\}}|j}|jDcgc] }t|}} t t}|D]} | j | j d<| jjD]\} } | dk(r | | j | <t| j | jddD]#\} } t|j| i|| <%||_dt|j|dD]&\}}|j|j|(||_ycc}w#t|j|dD]&\}}|j|j|(||_wxYww)u Temporarily replace an optimizer's parameters and state with the supplied params and optim states, then restore them on exit. For the duration of the context, all optimizer APIs (including user hooks) see the swap-in values; the live optimizer is restored on exit. The difference between this API and ``optimizer.load_state_dict`` is that ``optimizer.load_state_dict`` only updates the optimizer's state and leaves the parameters in ``param_groups`` untouched. This API also swaps in the parameters, so that ``optimizer.step()`` acts on the swap-in parameter tensors you supply. Args: optimizer: the live optimizer; its state must already be initialized. swapin_parameters: tensors to use as parameters during the context, provided in the same order as the existing input parameters to the optimizer (most commonly in ``model.named_parameters()`` order). swapin_optim_state: an ``optimizer.state_dict()``-shaped dict (``{"state": ..., "param_groups": ...}``) holding the state to install. ``"state"`` is keyed by packed integer parameter ids and ``"param_groups"`` mirrors ``optimizer.param_groups``, with each ``"params"`` entry as a list of those packed ids and the remaining keys carrying per-group hyperparameters (``lr``, ``betas``, ``foreach``, ``capturable``, ...). Only in-place tensor edits propagate back to the user supplied ``swapin_optim_state``; all other side-effects (e.g., assigning a new tensor to the optim state) are ignored. Example: One use of this API is to run ``optimizer.step()`` against ``FakeTensor`` versions of the parameters and state for nonstrict tracing — capturing an FX graph of the step without touching the live optimizer:: from torch.fx.experimental.proxy_tensor import make_fx from torch._subclasses import FakeTensorMode from torch.utils import _pytree as pytree fake_mode = FakeTensorMode(allow_non_fake_inputs=True) with fake_mode: fake_params = { n: fake_mode.from_tensor(p) for n, p in model.named_parameters() } fake_osd = pytree.tree_map_only( torch.Tensor, fake_mode.from_tensor, optimizer.state_dict() ) def step_fn(params, osd): with swap_in_optimizer_params_and_state(optimizer, params, osd): optimizer.step() return params, osd gm = make_fx(step_fn)(fake_params, fake_osd) r2Tr>N) rNrr+rrr r r r(rCr5clearrD)r*r;r<rrKoriginal_stategoriginal_group_snapshotsrFinfokeyvalue swapin_paramparam_idr0snapshots r"swap_in_optimizer_params_and_staterZsF!1$&8!E __N 2;1G1GHAQHH)1AF A FF) contextlib collectionsrtypingrrrrrrr)rrr.r#r:tuplerNcontextmanagerrZrrrr`s=#"  z "c3h*&69 $sCx."( ( S>(( #Y (VQ,&Q,CK(Q,S#XQ, 4S>4 01 12 Q,h k)&k)CK(k)S#Xk)k)r