^j_dZddlmZddlZddlZddlmZddlmZerddlm Z ddl Z ddl m Z m Z ddl mZdd lmZmZdd lmZdd lmZdd lmZmZeZGd deZGddeZy)zUBest-model checkpointing and early stopping callbacks for RF-DETR Lightning training.) annotationsN)Path) TYPE_CHECKING)Any)LightningModuleTrainer) __version__) EarlyStoppingModelCheckpoint) get_logger) get_version)_make_fit_loop_statestrip_checkpointceZdZdZdZ d dfd Ze d ddZeddZe ddZ e d ddZ eddZ dfd Z dfd Z dd Zdfd Zdd ZxZS)BestModelCallbackaS Track best validation mAP and save best checkpoints during training. Extends :class:`pytorch_lightning.callbacks.ModelCheckpoint` to save stripped ``{model, args, epoch}`` ``.pth`` files (instead of full ``.ckpt`` files) and to track a separate EMA checkpoint in parallel. At the end of training the overall winner (regular vs EMA, strict ``>`` for EMA) is copied to ``checkpoint_best_total.pth`` and optimizer/scheduler state is stripped via :func:`rfdetr.util.misc.strip_checkpoint`. Checkpoints are only updated on validation epochs where the monitor metric is actually logged. On non-eval epochs (when ``eval_interval > 1`` causes COCO evaluation to be skipped) the callback is a no-op. ``state_dict()`` and ``load_state_dict()`` are overridden to persist ``_best_ema`` in the Lightning callback state, ensuring that ``trainer.fit(ckpt_path=...)`` resumes EMA high-water-mark tracking from the correct value. Args: output_dir: Directory where checkpoint files are written. monitor_regular: Metric key for the regular model mAP. monitor_ema: Metric key for the EMA model mAP. ``None`` disables EMA tracking. run_test: If ``True``, run ``trainer.test()`` on the best model at the end of training. skip_best_epochs: Ignore the first N epochs (0..N-1) when tracking best regular and EMA checkpoints. Useful when fine-tuning from ``pretrain_weights``: the pretrained model's epoch-0 mAP can artificially dominate best-checkpoint selection before training adapts to the new dataset. smooth_alpha: Exponential-moving-average smoothing factor in ``[0.0, 1.0)`` applied to the regular monitor metric before checkpoint comparison. ``0.0`` (default) disables smoothing and keeps legacy behaviour: ``trainer.callback_metrics[monitor_regular]`` is consumed as-is by the parent :class:`~pytorch_lightning.callbacks.ModelCheckpoint`. ``smooth_alpha > 0`` maintains an internal EMA state ``self._smoothed_regular = alpha * self._smoothed_regular + (1 - alpha) * raw`` and temporarily substitutes the smoothed value into ``trainer.callback_metrics`` for the duration of the parent's improvement check; the original raw value is always restored before returning so what gets logged to ``metrics.csv`` is unaffected. Useful for noisy metrics (e.g. keypoint mAP under NLL-Cholesky losses) where raw per-epoch swings can lock the best checkpoint to an early local peak. The EMA accumulator is updated on every validation epoch including epochs within the ``skip_best_epochs`` window so the smoothed value is pre-warmed by the first eligible comparison. Examples: Skip the first 3 epochs so pretrained weights do not dominate: >>> import tempfile >>> from rfdetr.training.callbacks.best_model import BestModelCallback >>> with tempfile.TemporaryDirectory() as tmp: ... cb = BestModelCallback(output_dir=tmp, skip_best_epochs=3) ... cb._skip_best_epochs 3 z.pthc t||d|dddddd ||_||_d|_t ||_t|tst|ts td|dkr td ||_ t|tst|ttfs td tj|r |dks|d k\r td t||_d|_d|_d|_y) Ncheckpoint_best_regularmaxF) dirpathfilenamemonitormode save_top_ksave_on_train_epoch_endverboseauto_insert_metric_nameenable_version_counter/skip_best_epochs must be a non-negative integerr3skip_best_epochs must be greater than or equal to 0z*smooth_alpha must be a float in [0.0, 1.0)?z"smooth_alpha must be in [0.0, 1.0))super__init__ _monitor_ema _run_test _best_emar _output_dir isinstanceboolint TypeError ValueError_skip_best_epochsfloatmathisfinite _smooth_alpha_smoothed_regular_best_raw_regular_current_pl_module)self output_dirmonitor_regular monitor_emarun_testskip_best_epochs smooth_alpha __class__s o/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/rfdetr/training/callbacks/best_model.pyr$zBestModelCallback.__init__Os  .#$)$)#(  (! # + & -Z@PRU5VMN N a RS S!1 lD )L3PU,1WHI I}}\*lS.@LTWDWAB B$),$7),),:>c |||j|jDcic] \}}d|| c}}|jtt |jdiidiidggd }|||d<|||d<t }|||d<|Scc}}w)a~Build a PTL-compatible RF-DETR checkpoint payload. Args: model_state_dict: Model weights with raw (non-prefixed) keys. args_dict: Serialized training args/config payload. trainer: Active Lightning trainer providing epoch/step counters. model_name: Name of the model class (e.g. ``"RFDETRLarge"``). model_config_dict: Serialized architecture config needed to reconstruct schema-dependent models. Returns: Checkpoint dictionary that supports ``Trainer.fit(ckpt_path=...)`` while intentionally omitting optimizer/scheduler states. zmodel. state_dict)fit_loop validate_loop test_loop) modelargsepochrA global_stepzpytorch-lightning_versionloopsoptimizer_states lr_schedulers model_name model_configrfdetr_version) current_epochitemsrH ptl_versionrr ) model_state_dict args_dicttrainerrLmodel_config_dictkvpayloadversions r>_build_checkpoint_payloadz+BestModelCallback._build_checkpoint_payload|s,&**7G7M7M7OPtq!VA3<?P"..)401F1FG#/!3*B/ !#'& .  !$.GL !  (&7GN #-  (/G$ %9QsB ct|jdd}t|tjj r|n |j}|j S)a Resolve live model weights from the active Lightning module. Args: pl_module: The ``RFDETRModelModule`` being trained. Returns: State dict from the live, non-EMA model, unwrapped from ``torch.compile`` when needed. _orig_modN)getattrrEr)torchnnModulerA) pl_module_origraws r>_get_live_model_state_dictz,BestModelCallback._get_live_model_state_dicts? d;!%9ey~~r?cN|jD])}t|dd}t|s|}||cSntj dt|j dd}t |tjjr|n |j }|jS)a,Resolve EMA model weights from the active EMA callback. Args: trainer: The Lightning Trainer instance. pl_module: The ``RFDETRModelModule`` being trained. Returns: EMA model state dict when available, otherwise the live model state dict. get_ema_model_state_dictNzhEMA metric improved but EMA callback weights were unavailable; saving current model weights as fallback.r\) callbacksr]callableloggerwarningrEr)r^r_r`rA)rTracallbackgetterrArbrcs r>_get_ema_model_state_dictz+BestModelCallback._get_ema_model_state_dicts )) HX'A4HF#X )%%    v  d;!%9ey~~r?ct|dd}|yt|tr|St|dd}t|sy|}t|tsy|Yt|jdd}t|t j jr|n |j}|j}|jd}t|t jrM|jdk(r>d|vr:|jd jDcgc] }t|c}|d<|jd } t| t jr(| jdk(rd |vr| jd dz |d <|Scc}w) aSerialize the model architecture config when the module exposes one. Schema-critical fields (``num_keypoints_per_class``, ``num_classes``) are synced from live model weights so the saved config reflects what the model actually learned, not a stale constructor default (e.g. COCO ``[0, 17]`` when the model was fine-tuned on ``[0, 33]``). Args: pl_module: The Lightning module whose model config will be serialized. state_dict: Pre-computed model state dict. If ``None``, computed from the live model. Pass the already-retrieved checkpoint state dict to avoid a redundant copy. rMN model_dumpr\_kp_active_masknum_keypoints_per_classr)dimzclass_embed.weight num_classesr)r]r)dictrhrEr^r_r`rAgetTensorndimsumtolistr+shape) rarArMrodumpedrbrc_kp_maskn _ce_weights r>_serialize_model_configz)BestModelCallback._serialize_model_configsOy.$?   lD ) \<>  #&$'  IOO[$?E%eUXX__=%9??C)J>>"34 h -(--12DIbflIlAIRSATA[A[A]0^AQ0^F, -^^$89 j%,, /JOOq4H]^dMd$.$4$4Q$7!$;F= ! 1_sE<c`t|dd}| t|ddnd}t|tr|j}|r|S|t |j nd}|j drtd|d|jdr"|j d r|jd Sy) aResolve checkpoint model_name from model_config or config type. The CLI/PTL path does not call ``RFDETR.train()``, so ``model_config.model_name`` may be unset. In that case, infer the model class from concrete config names like ``RFDETRSmallConfig``. Note: The ``DeprecatedConfig`` ``RuntimeError`` guard is only reachable from the CLI/PTL path. ``RFDETR.train()`` pre-populates ``model_config.model_name`` before saving any checkpoint, so the config type-name branch (and therefore the ``DeprecatedConfig`` guard) is never reached when training is started via ``RFDETR.train()``. rMNrLDeprecatedConfigzDeprecated model config 'zL' is no longer supported. Re-train your model using a current model variant.RFDETRConfig) r]r)strstriptype__name__endswith RuntimeError startswith removesuffix)rarMconfigured_namenormalized_nameconfig_type_names r>_resolve_model_namez%BestModelCallback._resolve_model_name sy.$? GSG_', dCei os +-335O&&:F:R4 -66XZ  $ $%7 8+,<+=>EE   & &x 05E5N5Nx5X#00: :r?c~t|}|j|d<|j|d<|j|d<|S)aWReturn callback state including ``_best_ema``, ``_smoothed_regular``, and ``_best_raw_regular``. Extends the parent :class:`~pytorch_lightning.callbacks.ModelCheckpoint` state dict with three extra keys so that ``trainer.fit(ckpt_path=...)`` resumes the EMA high-water mark, the smoothed-metric accumulator, and the raw metric at the smoothed-best epoch from their correct values rather than resetting to ``0.0``. Returns: State dict with all parent fields plus ``"_best_ema"``, ``"_smoothed_regular"``, and ``"_best_raw_regular"``. r'r3r4)r#rAr'r3r4)r6stater=s r>rAzBestModelCallback.state_dict'sG"$!^^k%)%;%;!"%)%;%;!" r?ct|}t|jdd|_t j |jsd|_t|jdd|_t j |j sd|_t|jdd|_t j |jsd|_t|%|y)aRestore callback state from a Lightning checkpoint. Pops ``"_best_ema"``, ``"_smoothed_regular"``, and ``"_best_raw_regular"`` from a shallow copy of *state_dict* before delegating to the parent so the parent does not receive unexpected keys. Each key defaults to ``0.0`` when absent (e.g. checkpoints saved before these fields were persisted) and is reset to ``0.0`` when the stored value is non-finite. Args: state_dict: Callback state dict as produced by :meth:`state_dict`. r'rr3r4N) rur/popr'r0r1r3r4r#load_state_dict)r6rArr=s r>rz!BestModelCallback.load_state_dict8sZ uyyc:;}}T^^, DN!&uyy1Dc'J!K}}T334%(D "!&uyy1Dc'J!K}}T334%(D " &r?c |jsy|j}|td|d|jdt |}|j j dd|j|}|j}t|jdd}|-t|dr!t|dd|jd|i }t|d r|jn|}|j|} |j||} t!j"|j%|||| | ||j&|_|j*j-|j.} t!j0| r| j3d } n| t5| } nd } t6j9d||j|j.| y)umSave stripped ``.pth`` format instead of a full ``.ckpt``. Skips on non-main processes. Intentionally does NOT call ``trainer.save_checkpoint()`` — we only want ``{model, args, epoch}``. Args: trainer: The Lightning Trainer instance. filepath: Destination path (ends in ``.pth`` via ``FILE_EXTENSION``). Nz8BestModelCallback._save_checkpoint called with filepath=z at epoch=z but pl_module was not set.Tparentsexist_ok class_names model_copyupdaterorLrUz.6gunknownzDBest regular checkpoint saved to %s (epoch %d, monitor=%s, value=%s))is_global_zeror5rrOrparentmkdirrd train_configr] datamodulehasattrrrorrr^saverZrH_last_global_step_savedcallback_metricsrvr is_tensoritemrriinfo) r6rTfilepathrapth_pathrRrdataset_class_namesrSrLrU monitor_valuemonitor_displays r>_save_checkpointz"BestModelCallback._save_checkpointPs%% ++  J8,W#1122MO >dT: ::9E!-- %g&8&8-N  + l3 mT:B'22=J]:^2_L18|1TL++-Zf --i8  88DTU  * * %"3 +   (/':':$0044T\\B ??= )!.!3!3!5c :O  &!-0O'O R   ! ! LL   r?c t||_d}|jdkDrp|j|jvrX|j|jj }|j|j zd|jz |zz|_|j |jkry|j|jvry|jdkDr,||n&|j|jj }|j|jj n td }|j|j}tj|j |j|j|j|j< t|=||||j|j<|j|jj n td }||kDr||_nt|=|||j" |j$sy|jj'|j"tjdj }||j(kDr"||_|j*j-dd|j/||} |j0} t3|j4dd} | -t7| d r!t3| dd| j9d| i } t7| d r| j;n| } |j=|} |j?|| }tj@|jC| | || | |j*d z tDjGd||j yy#||j|j<wxYw)aSave best regular/EMA checkpoints when validation mAP improves. Delegates regular-model checkpoint management to the :class:`~pytorch_lightning.callbacks.ModelCheckpoint` parent (handles improvement detection, fast_dev_run/sanity guards, ``best_model_path`` and ``best_model_score`` bookkeeping). EMA is tracked independently. Args: trainer: The Lightning Trainer instance. pl_module: The ``RFDETRModelModule`` being trained. Nrr"inf)dtypedeviceTrrrrrorcheckpoint_best_ema.pthz(Best EMA mAP improved to %.4f (epoch %d))$r5r2rrrr3rOr.best_model_scorer/r^tensorrrr#on_validation_endr4r%rrvr'r(rrmrr]rrrrorrrrZrir)r6rTrarc current_rawprev_best_scoreoriginalnew_best_scoreema_valema_state_dictema_train_configr ema_args_dictema_model_nameema_model_config_dictr=s r>rz#BestModelCallback.on_validation_endsq#,!    # 8P8P(P**4<<8==?C%)%7%7$:P:P%PTWZ^ZlZlTlpsSs%sD "  4#9#9 9  <B>S>S>_d3388:fklqfrerO// =H5:\\&&hnnX__6G $ $T\\ 2 B)'9=9A((6>B=R=R=^T22779ejkpeqdqN/)4& G %gy 9    $G,B,B **..t/@/@%,,sBSTYY[ T^^ #$DN    " "4$ " ?!;;GYON )55 ")'*<*#>}ViFj#>#k 189I<1X ++-^n "55i@N$($@$@N$[ ! JJ.."!-&; /  #<<  KK:%% ; $:B((6s NN7cD|jsy|jdkDr |j}n(|j|jj nd}|j rt |j nd}|jdz }|jdz }|j|kD}|r|jr|n|}|rW|jrGtj||t|tjd|rdnd||j|jrt!t#|dd} | duxr| t$j&u} | r|jstj)d yt+j,|d d } t!|j.d d} t1| t*j2j4r| n |j.} | j7| ddtjd||j9||j:d yyy)aSelect the overall best model and optionally run test evaluation. Copies the winner (regular vs EMA, strict ``>`` for EMA) to ``checkpoint_best_total.pth``, strips optimizer/scheduler state, then optionally runs ``trainer.test()``. Args: trainer: The Lightning Trainer instance. pl_module: The ``RFDETRModelModule`` being trained. Nrrzcheckpoint_best_total.pthz on_fit_endzBestModelCallback.on_fit_ends%%     #11L;?;P;P;\400557beL595I5ItD001t ##&??%%(CC nn|3 !,1BH ))+ LLJ / Z ( KKN$)   >>#DO[$GM)5h-OhOh:hM!((*NN+ zz*5uU dC)%Aey##DM$#? NPZ[ Y73E3Eu U# r?) val/mAP_50_95NTrr)r7rr8rr9 str | Noner:r*r;r+r<r/returnNone)NN) rRdict[str, torch.Tensor]rSobjectrTrrLrrUz object | Nonerzdict[str, object])rarrr)rTrrarrr)N)rarrAzdict[str, Any] | Nonerzdict[str, object] | None)rarrr)rdict[str, Any])rArrr)rTrrrrrrTrrarrr)r __module__ __qualname____doc__FILE_EXTENSIONr$ staticmethodrZrdrmrrrArrrr __classcell__r=s@r>rrsd,\N /"& !!+?+?+? +?  +?  +?+? +?Z "&+/ 51555 5 ) 5  55n      "  !  6HL("(0E( !((T:"'0= ~Zx@Vr?rcneZdZUdZdZded< d dfd Zd fd ZxZS) RFDETREarlyStoppinguMEarly stopping callback monitoring validation mAP for RF-DETR. Extends :class:`pytorch_lightning.callbacks.EarlyStopping` with dual-metric monitoring: by default it monitors ``max(regular_mAP, ema_mAP)`` (legacy behaviour); set ``use_ema=True`` to monitor the EMA metric exclusively. The effective metric is injected into ``trainer.callback_metrics`` under a synthetic key before delegating to the parent's stopping logic, so all parent features are available for free: ``state_dict``/``load_state_dict`` for checkpoint resumption, NaN/inf guard via ``check_finite``, and ``stopping_threshold``/``divergence_threshold``. Early stopping evaluates only on validation epochs where the monitored metrics are logged; non-eval epochs (``eval_interval > 1``) are skipped automatically. Args: patience: Number of epochs with no improvement before stopping. min_delta: Minimum mAP improvement to reset the patience counter. use_ema: When ``True`` and both regular and EMA metrics are available, monitor only the EMA metric. When ``False``, monitor ``max(regular, ema)``. monitor_regular: Metric key for the regular model mAP. monitor_ema: Metric key for the EMA model mAP. verbose: If ``True``, log early stopping status each epoch. skip_best_epochs: Ignore the first N epochs (0..N-1) when evaluating patience and best-score baselines. Set this when fine-tuning from ``pretrain_weights`` to avoid premature stopping before the model adapts to the new dataset. Examples: Fine-tuning from pretrained weights — skip first 3 epochs: >>> from rfdetr.training.callbacks.best_model import RFDETREarlyStopping >>> cb = RFDETREarlyStopping(patience=10, skip_best_epochs=3) >>> cb._skip_best_epochs 3 __rfdetr_effective_map__r_SYNTHETIC_MONITORc t||jd||d|ddd t|tst|t s t d|dkr td||_||_ ||_ ||_ y)NrFT) rrpatience min_deltacheck_on_train_epoch_endr check_finiterlog_rank_zero_onlyr rr!) r#r$rr)r*r+r,r-_monitor_regularr%_use_emar.) r6rruse_emar8r9rr;r=s r>r$zRFDETREarlyStopping.__init__Ss ++%*#   & -Z@PRU5VMN N a RS S /' !1r?c|j|jkry|j}|j|j}|j|j }||j nd}||j nd}||y|jr||}n|| t||}n||}n|}tj||j|j<t |5||y)aCompute effective mAP and delegate to parent stopping logic. Computes ``ema_mAP`` or ``max(regular_mAP, ema_mAP)`` depending on ``use_ema``, injects the result under the synthetic monitor key, then calls :meth:`EarlyStopping.on_validation_end` which handles patience, ``trainer.should_stop``, logging, and ``state_dict`` persistence. Args: trainer: The Lightning Trainer instance. pl_module: The ``RFDETRModelModule`` being trained. N)rOr.rrvrr%rrrr^rrr#r) r6rTrametricsregular_tensor ema_tensor regular_valr effectiver=s r>rz%RFDETREarlyStopping.on_validation_endrs  4#9#9 9 ** T%:%:;[[!2!23 =K=WN$7$7$9]a 5?5K  1QU  7?  ==W0I  $)<K1I  I#Irr.s D98 .. !222 2  2  222 2>"6"6r?r)r __future__rr0rpathlibrtypingrrr^pytorch_lightningrrr rQpytorch_lightning.callbacksr r rfdetr.utilities.loggerr rfdetr.utilities.packager rfdetr.utilities.state_dictrrrirrr?r>r s[ \"   68F.0N MVMV`f6-f6r?