^j)dZddlZddlZddlZddlmZmZmZmZm Z m Z ddl Z ddl m Z dZddlmZmZmZmZmZdd lmZdd lmZdd lmZej4eZd Zd ZdggdgdgddZe e e e!e!e!fee e!e!e!ffZ"dFde jFde$de jFfdZ%de!de e&dfde!fdZ'ede jPdfde jFde&dee e!e!e!fde!de!d e jRd!e$de jFfd"Z* dGd#e jVd$e de!fd%Z, dGd#e jVd$e de e!e$ffd&Z- dHde jFd'e&d(e!d)e$de e$e e$e fff d*Z. dId+e jFd,e de e jFe jVffd-Z/dd.d/ee jFd0ee jFd1ee jFd2e!d3e!d4e!d5e$d6e&d7e"de!de!d$ee d8e d9e$d!e$de$ddf"d:Z0dd.d/ee jFd0ee jFd1ee jFd;ee jFdZ1dd.d/ee jFd0ee jFd1ee jFd2e!d3e!d4e!d5e$d6e&d7e"de!de!d$ee d8e d9e$d!e$de$ddf"d?Z2dd.d/ee jFd0ee jFd1ee jFd;ee jFd)guV @ggn@@))gͪV@gg"~j@)gv@ggj+6g F%u@)ga4 @gH}]g\Cm@)g2%@g?$ g/L F?)g6>W[@gQkgH}8?))g8y @gWr"b(7g=90@)g3kT@g)6$g}U"?)g")i@gV}gT"?)g߸%~b @g_O"Dveg)E?)gT`@gU4N/g ?)gͶ?g!̲gG߆?)g_?g ]sig??N!?)g9+_?gWg-|?))gh('P @g־{6g ]!/@)g=.+@gixgےl%?)g""@gP+ .gPYJخ1?)g.69I @gG4gH5?)gSO|g@gVg@ę?)gΐX?gxDgPh?)gkei?gB?Ggd݈ؼ?)g1?gjgZ?)originalquintic polar_expresspolar_express_saferparam capturablereturnc@t|r|jSdS)N)device)rr)rrs Z/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/optim/muon.py _init_steprZs   EE EEepsshape.c,|d|d}}|||z dzzS)uScale epsilon for Newton-Schulz based on matrix dimensions (μP-style). For μP compatibility, epsilon should scale as eps * sqrt(din/dout) to maintain consistent damping behavior across different model widths. Reference: https://arxiv.org/abs/2512.05620 Args: eps: Base epsilon value shape: Shape of the matrix (out, in) or (batch, out, in) Returns: Scaled epsilon value ?)r r!doutdins rscale_eps_for_nsr)^s)(rE"I#D #*$ $$r?Gsteps coefficients safety_factordtype scale_epsc 2|jdvsJd|jdt|}|dk\rt|ddk(sJ||kr|d|n||dg||z zz}|rt||j}|j |d } | j d | j dkD} | r | j } |rB| j| jd d d j|j|nB| j| jd d d j|j|txrt|t} | r4|D].\} } }| | j z}| |z|||zzz}| | z|| zz} 0n| jd kDrtj ntj"}| j%} tj&g| jdd| j d | j(| j*}tj,|}tj,| }|D]>\} } }||| | j dd|||||| |||| || | d||| }} @| r | j } | S)uNewton-Schulz quintic iteration to compute the zeroth power / orthogonalization of gradient. Supports batched operation over leading dimensions. See - https://github.com/KellerJordan/Muon/blob/master/muon.py - https://github.com/NoahAmsel/PolarExpress/blob/main/polar_express.py - https://github.com/KellerJordan/modded-nanogpt/blob/master/train_gpt.py Args: G: Input gradient tensor of shape (m, n) or (batch, m, n) steps: Number of Newton-Schulz iterations coefficients: Coefficients (a, b, c) for the iteration eps: Numerical stability epsilon for norm safety_factor: Multiplicative safety factor for norm (1.01 is common safety value in 'polar express' variants) dtype: Computation dtype scale_eps: If True, scale epsilon by sqrt(din/dout) for μP compatibility Returns: Orthogonalized tensor of same shape as G )zInput must be 2D or 3D, got zD. Flatten batch dims first.r rr3Nr$T)r/copyr#r2)r#r$)dimkeepdim)min)rr/gr*)betaalphaout)ndimlenr)r!tosizemTdiv_normmuladd_clamp_ has_dtensor isinstancer torchbaddbmmaddmm contiguousemptyrr/ empty_like)r+r,r-r r.r/r0num_cscoeff_sequenceX transposed is_dtensorabcABmm_fnCs rzeropower_via_newtonschulzrYvs< 66V `;AFF8C_``   F Q;3|A/144 4-2f_\&5) R()UV^<<sAGG, 5t$AaffRj(J DD qvvaXtv488GLLSQR qvvaXtv488GNNSVNWX7Aw!7J% GAq!ADDAAQU #AAQA "#!  LLN KK3!''#2,3r 3AHHAGG T   Q    Q & GAq! !Q3cq 9 !Q 2 !Q! 4aqA    DD Hr param_shape adjust_lr_fnct|dkDr |d|dfnd\}}|dk(rtd||z dzS|dk(rdt||dzzS|d k(r||z dzSJd |d ) avAdjust learning rate based on parameter shape for Muon. Args: param_shape: Shape of the parameter tensor adjust_lr_fn: Scaling function name - "original": sqrt(max(1, out/in)) - Original Muon impl - "match_rms_adamw": 0.2 * sqrt(max(out, in)) - Kimi scaling - "rms_to_rms": sqrt(out/in) - Scion/Bernstein scaling r r#r$r*r*rr%match_rms_adamw皙? rms_to_rmsInvalid scaling function "z " for Muon)r<maxrZr[out_chsin_chss r get_lr_scalerfs=@ True (or (True, "ok") if return_reason=True) (96, 3, 4, 4) -> True - will be flattened to (96, 48) (4, 2048) -> False - extreme aspect ratio (64,) -> False - insufficient dims (1, 196, 768) -> False - leading unit dims NOTE: these rules were created to balance complexity with covering common timm model cases Please let me know if there are non-optimal cases that you run into. r2c3,K|] }|dkDs dyw)r Nr&).0dim_sizes r z(_is_suitable_for_muon..$sA8HqLQAs )Finsufficient_dimsFrr )Fleading_unit_dimsr3Nzmin_dim_too_small:zextreme_aspect_ratio:z.1f)TokT)r!r;sumr7rb) rrkrlrmsout_chin_ch_with_spatiald check_dimsmin_sizemax_size aspect_ratios r_is_suitable_for_muonrs:  A zzA~AqAAAE/<+G%G  tqyAaDAI/<+G%G zzQ112 $A ! #  $01  :H, .xj99 9:Hh&L&& 1,s1CDD D(<2d2rtensormodec^|j}|jdk(r||fS|jdkrtd|j|jdd\}}|dk(r|j|d|fS|dk(r*|j||d}|j ddd}||fStd |) aReshape high-dimensional tensor for Muon processing. Args: tensor: Input tensor of shape (out, in, *spatial) mode: How to handle spatial dimensions - "flatten": Flatten spatial into output dimension (out, in*H*W) - "batched": Batch over spatial positions (spatial_prod, out, in) for per-position orthogonalization Returns: Reshaped tensor and original shape for restoration r2z,Tensor must have at least 2 dimensions, got Nflattenr$batchedrr zUnknown mode: )r!r; ValueErrorreshapepermute)rroriginal_shaperxin_chreshapeds rreshape_for_muonrMs\\N {{a~%% {{QG }UVVLL!$MFE y~~fb)>99  >>&%4##Aq!,''>$011r)rparamsgrads momentum_bufslr weight_decaymomentumnesterovns_stepsns_coefficients conv_modenormalize_spatialc :t|||||||||| | | | | ||y)z8Functional API that performs Muon algorithm computation. rrrrrrr r.r[rrr0rN)_single_tensor_muon)rrrrrrrrrr r.r[rrr0rs rmuonrps<(  !' #!+!r exp_avg_sqs state_stepsbeta2c>t|||||f||||| | | | | |||||dy)a<Functional API that performs AdaMuon algorithm computation. AdaMuon extends Muon with element-wise second moment estimation applied to orthogonalized update directions, providing Adam-like adaptive scaling while preserving Muon's geometric benefits. Reference: https://arxiv.org/abs/2507.11005 rrrrrrrr r.r[rrr0rN)_single_tensor_adamuon)rrrrrrrrrrrrr r.r[rrr0rs radamuonrsL<   !' #!+'rc vt|t}t|D]\}}||}||}|jd||zz |j |d|z |r|j ||n|j }|j dk\rt|| \}}n|}|j}t|||| | |}| rt|j| }nd}| dk(r9|j dk\r*| r||jddzz}|jdd d}|j|}t||| |zy ) zSingle tensor Muon update.r r*r3rr r.r0rrrir2N)resolve_ns_coefficients _COEFFICIENTS enumeratemul_lerp_cloner;rr!rYrfrrr )rrrrrrrrrr r.r[rrr0rirgrad momentum_bufupdateupdate_reshapedr update_orthoscales rrrs\(.o}MOf%-75Qx$Q'   1rL(() 4h/7?L(3\EWEWEY ;;! .>vI.V +O^$O#\\N2   '    !3!3\BEE  !l&7&71&< ++A.$66'//1a8L$++N;  UL2#+6[-7rc t| t} |r t||t|D],\}}||}||}||}||}|j d|r|n t |}|j d||zz |j|d|z |r|j||n|j}|jdk\rt||\}}n|}|j}t|| | | | |}|dk(r"|jdk\r|jddd}|j|}|j | j||d| z |rt!|j|\}}nd \}}|r |j#j | } nXt%j&|rdt%j(| |z nd| |zz }!||!z j#j | } || z }"|r$|"j+j | }#|"|#z }"|dk(r)t-|dk\r|rd}$|dd D]}%|$|%z}$ ||$d zz}t/||"| |z/y ) u)Single tensor AdaMuon update. AdaMuon applies second-moment estimation to the orthogonalized directions, then rescales using RMS-alignment to maintain stable step sizes. Algorithm: 1. Update momentum buffer: M = β₁·M + (1-β₁)·G 2. Orthogonalize: O = Newton-Schulz(M) or Newton-Schulz(nesterov_update) 3. Update second moment: v = β₂·v + (1-β₂)·O² 4. Bias correct: v̂ = v/(1-β₂^t) 5. Adaptive scaling: Ô = O / (√v̂ + ε) 6. RMS-aligned rescaling and apply update r r*r3rrrr2r)value)r*FNri)rrr rrCr rrrr;rr!rYrraddcmul_rjsqrtrG is_tensorpowrAr<r )&rrrrrrrrrrrrr r.r[rrr0rrrrr exp_avg_sqstep_tsteprrrrr use_rms_normdenombias_correction2update_adaptive update_norm spatial_prodrzs& rrrsF.o}MO!&+6f%M:5Qx$Q'  ^ Q  A#vF);  1rL(() 4h/7?L(3\EWEWEY ;;! .>vI.V +O^$O#\\N2   '    !l&7&71&<'//1a8L#++N;  '' l#PU+'V "6|7I7I<"X E<", E< OO%**3/E@Et?TsUYYud%;;Z]`eim`mZm "2288:??DE'. )..055c:K- ;O  !c.&9Q&>  '+&A A%L&-- UObS5[9[M:rc'eZdZdZddddededdd d d dd d dddfdededededede de dedede e de dede edede eefde deded ef&fd! Zfd"Zej"d$d#ZxZS)%MuonaMuon - MomentUm Orthogonalized by Newton-schulz Combines Muon for 2D+ parameters (weight matrices) with AdamW for 1D parameters (biases, norms) and parameter groups with 'use_fallback=True' set (or 'use_muon=False' for compatibility). Supports two algorithms: - "muon": Standard Muon algorithm with momentum + orthogonalization - "adamuon": AdaMuon algorithm that adds element-wise second moment estimation to orthogonalized directions for Adam-like adaptive scaling g{Gz?rffffff?Frr*r^rTN)g?rrrrrrrrrr r.r[rradamw_lrfallback_lr_scalebetasalgor0rverbosectd|td|td|dtd|| dvrtd| |d vrtd |d | Qtjd tdt j |r td|dk(r td| |z }td"id|d|d|d|d|d|d|d| d| d| d| d|d|d|d|d |d!|}t|%||y )#u0 Create Muon optimizer. Args: params: Iterable of parameters or dicts defining parameter groups lr: Learning rate (default: 0.02 for Muon parameters) weight_decay: Weight decay coefficient momentum: Momentum factor for Muon nesterov: Whether to use Nesterov momentum ns_steps: Number of Newton-Schulz iterations ns_coefficients: Coefficients for NS iteration eps: Numerical stability epsilon safety_factor: Multiplicative safety factor for NS norm adjust_lr_fn: LR adjustment function - "original", "match_rms_adamw", or "rms_to_rms". For adamuon mode, can set to None to disable (RMS rescaling handles scaling). conv_mode: How to handle convolutions - "flatten" or "batched" normalize_spatial: Whether to normalize by sqrt(spatial_size) in batched mode fallback_lr_scale: Scale factor applied to lr for AdamW fallback parameters. The effective fallback LR is lr * fallback_lr_scale. This ensures the LR scheduler properly schedules fallback parameters. adamw_lr: Deprecated. Use fallback_lr_scale instead. betas: Beta coefficients - (beta1, beta2) where beta1 is used for AdamW fallback and beta2 is used for both AdamW fallback and AdaMuon second moment algo: Algorithm - "muon" for standard Muon, "adamuon" for AdaMuon with adaptive second moment estimation (https://arxiv.org/abs/2507.11005) scale_eps: If True, scale epsilon by sqrt(din/dout) in Newton-Schulz for μP compatibility (https://arxiv.org/abs/2512.05620) capturable: Whether this instance is safe to capture in a CUDA graph. Capturable mode supports tensor learning rates and requires optimizer state tensors to live on the parameter device. verbose: Log parameter routing decisions (Muon vs AdamW) Example: ```python # Simple usage - automatically uses Muon for 2D+ params, AdamW for 1D optimizer = Muon(model.parameters(), lr=0.02) # Use AdaMuon algorithm for adaptive scaling optimizer = Muon(model.parameters(), lr=6e-4, algo="adamuon") # Manual control over parameter groups optimizer = Muon([ {'params': weight_matrices, 'lr': 0.02}, {'params': biases, 'use_fallback': True, 'lr': 3e-4}, # use AdamW if use_fallback=True ]) ``` z learning raterrr*) max_valueepsilon)rrzInvalid conv_mode: )rrzInvalid algo: z. Must be 'muon' or 'adamuon'Nzpadamw_lr is deprecated, use fallback_lr_scale=adamw_lr/lr instead. adamw_lr will be removed in a future release.r2) stacklevelzHadamw_lr is not supported with tensor lr; use fallback_lr_scale instead.rz8Cannot compute fallback_lr_scale from adamw_lr when lr=0rrrrr r.r[rrrrrr0rrr&) rrwarningswarn FutureWarningrGrdictsuper__init__)selfrrrrrrrr r.r[rrrrrrr0rrdefaults __class__s rrz Muon.__init__sD "-6X=C( 2 229+>? ? * *~dV3PQR R   MM@   r" !kllQw ![\\ (2   %      ,   ( &   0 0     "! "# & *rct|||jD]g}|jdd|jdd|jddd|vs>|j d|d}|dd k7r||dz nd |d<iy) Nrrr0Frrrrrr*)r __setstate__ param_groups setdefaultpop)rstategrouprrs rrzMuon.__setstate__s U#&& aE   VV ,   [% 0   \5 1"%/ 99Zt=GLT{VWGWXd -C]`)* arcF |jjddr9t|dr|jnt|dr|j d}|$t j 5|}ddd|jjdd}d}d}|rind}|jD]}|jdd }g} g} g} g} g} g}g}g}g}g}|d D]}|j|jjr td |j|}d |vrd}|jd dr d|d <|r9d}n6d |vr |d |d <|r(d}n%|rt|d\}}n t|d}||d <|A|?djd|jD}||vrg||<||j||d }|r| j|| j|j|dz }d|vr(t j |t j"|d<| j|d|dk(s[d|vr:t%||d|d<t j |t j"|d<| j|d| j|d|j||j|j|dz }d|vrbt%||d|d<t j |t j"|d<t j |t j"|d<|j|d|j|d|j|d| r|dk(rM|d\}}t'| | | | | f|d|d|d|d||d |d!|d"|d#|d$|d%|d&|d'|dd(nBt)| | | |d|d|d|d|d |d!|d"|d#|d$|d%|d&|d'|d)|sb|d\}}|d|d*z}|dr%t+|||||d||||d|d"dd|dd+t-||||g|dd||||d|d"dd|dd,|rt/|dkDrt0j3d-|d.|d/i}t5|j7D])\}}|D]}||vrg||<||j|!+g}t5|j7D]$\}} |j|d0t/| &t0j3d1d2j|t0j9t:j<rt5|j7D]\}} |d3k(rd4nd5}!t0j3d6|d7|!d8| dd9D]}"t0j3d:|"t/| d9kDs\t0j3d;t/| d9z d<|S#1swYxYw)=z$Performs a single optimization step.rF'_accelerator_graph_capture_health_check _cuda_graph_capture_health_checkNrrrrrz&Muon does not support sparse gradientsuse_muon use_fallbackuse_fallback_flag use_muon_flagT)rmxc32K|]}t|ywN)str)rprws rrrzMuon.step..Us,ESV,Esr momentum_buffer) memory_formatrrrexp_avgrrrrrrrr r.r[rrr0rrr) foreachbeta1rrrr cautionmaximizermax_lr) ramsgradrrrrr rrrrzMuon parameter routing: z Muon, z AdamW=z Breakdown: , rurAdamWz z -> : z z ... and z more)rgethasattrrrrG enable_gradrr is_sparse RuntimeErrorrrjoinr!append zeros_likepreserve_formatrrrrrr<_loggerinfosorteditems isEnabledForloggingINFO)#rclosurelossr muon_count adamw_countrouting_reasonsrr muon_params muon_gradsmuon_momentum_bufsmuon_exp_avg_sqsmuon_state_steps adamw_params adamw_gradsadamw_exp_avgsadamw_exp_avg_sqsadamw_state_stepsprreasonsuitable shape_strr_rr fallback_lr reason_groupsreasonsreason_summaryshapesoptimizer_namer!s# rrz Muon.steps7 ==  \5 1tFG<<>AB557  ""$ !y !--##Iu5  '"T&&r E99VV,DKJ!# ! ! LKN "  " 8_H <66>66##&'OPP 1 U*!Fyy7,1j)"%8F#u,,1*,=j)"%4F#/DQVZ/[,Hf'!''*&&qvv.1$KU*(21eL6I(Jf +0+;+;AUMbMb+ci(.3.>.>qPUPePe.fl+"))% *:;%,,U<-@A%,,U6];QH %:!&z!2!&z!2#!&z!2(-.?(@!%L&+O&<%*>%:"' "4*/0C*D"' "4#(#6',#"* ;%*>%:!&z!2!&z!2!&z!2(-.?(@!%L&+O&<%*>%:"' "4*/0C*D"' "4#(#6!($W~ u#DkE2E,FF $$#&)) $##&%*>%:!%L %!&#(#6#$$#&)) $ %##&%*>%:!%L %!&#(#6##Ar j s?3a7 LL3J"? A%%#f+&?@ A LL=>)B(CD E##GLL1&,]-@-@-B&CONFF/5~V7NLL4xtN3C1!EF!'7 veW%5676{R' ~c&kB6F5Gu%MN O w ! !s .XX r)__name__ __module__ __qualname____doc__DEFAULT_NS_STEPSMUON_EPSrfloatboolintNSCoeffrrrrrrGno_gradr __classcell__)rs@rrrsK "#"",'0!#&*;&&*(,'*)4#$!)k+k+k+ k+  k+  k+k+%k+k+!k+#3-k+k+ $k+uok+ %k+ &!k+"#k+$%k+&'k+()k+Z aU]]_eerrrpresetsc d ddttdttttff fd }t|tr}||vr9dj t |j}td|d|||} |rt|d k(rtd |d |Dcgc] }|| c}S |s td t|d k(rtfd|Dr ||gSg}t|D]5\}} |std|d||j||7|s td|Scc}w)NcTt|txrt|ttf Sr)rFrrbytesrs rz)resolve_ns_coefficients..s z!X.Rz!c5\7R3Rrc\t|tjxrt|t Sr)rFnumbersRealr'r0s rr1z)resolve_ns_coefficients..s 1gll3OJq$.as_coeff..s2I!71:2Iz3Coefficient must be length-3 of real numbers, got: )r<allrr&)rrRrSrTr9is_seqs ras_coeffz)resolve_ns_coefficients..as_coeffsZayCFaKs2Iq2I/IRSTRWXY Y1aQxq58++rrzUnknown coefficients preset 'z'. Valid options: rzPreset 'z' is empty or invalidz]Coefficients must be a preset name (str), a 3-sequence (a,b,c), or a sequence of 3-sequences.r3c3.K|] }|ywrr&r7s rrrz*resolve_ns_coefficients..s9awqz9r:zItem z is not a sequence: z Coefficient list cannot be empty)rr&rrFrrrkeysrr< TypeErrorr;rr) rr,r=validseqitemr:rr9r<s @@rrrsj SFOG,HUO,eUE.A(B, %  IIfW\\^45E"# ;<< J)0s0E)F)r^)g`@F)r)8r#rr3rtypingrrrrrrrGtorch.distributed.tensorr rE ImportError_helpersr r r rr_typesrrr getLoggerr rr%r$rrr&r)Tensorr'rr(r)bfloat16r/rYSizerfrjrrrrrroptim Optimizerrrr&rrrPs,BB 0Kkj '  H %  "  5& R U5%./eE5%.OZZOO 5$;O:"&# E3||E3E3 E3 E3  4tSy! !" E3T 2  2 2 5<< #$ 2h!#%U\\"%ELL!%ELL)%  %  %%%%!%%%sm%% % !%"#%$ %%x!)2U\\"2ELL!2ELL)2%,,' 2 %,,' 2 222222!222 sm!2"#2$ %2&'2()2* +2L!#C7U\\"C7ELL!C7ELL)C7  C7  C7C7C7C7!C7C7C7smC7C7 C7 !C7"#C7$ %C7t!)t:U\\"t:ELL!t:ELL)t:%,,' t: %,,' t: t:t:t:t:t:t:!t:t:t: sm!t:"#t:$ %t:&'t:()t:* +t:nj5;; jZ )S(5/8HUO+DDE)hx778) %ue# $%)oKsMM)(M)