^jX` dZddlZddlmZddlZddlmZddlmZm Z m Z ddl m Z ddl mZmZmZmZddlmZ dd lmZdd lmZmZmZdd lmZmZmZm Z dd l!m"Z"dd l#m$Z$e$Z%ddZ&e GddeZ'ndZ'GddeZ(de)de*fdZ+de)dzde*fdZ,d de-e)zde)dzde*fdZ.dddedede)dzdedef dZ/y#e$rdZYwxYw)!uATrainer factory — assembles a PTL Trainer from RF-DETR configs.N)Any)Trainer)ModelCheckpointRichProgressBarTQDMProgressBar)RichProgressBarTheme) CSVLogger MLFlowLoggerTensorBoardLogger WandbLogger) DDPStrategy)_MultiProcessingLauncher)KeypointTrainConfig ModelConfig TrainConfig)BestModelCallbackDropPathCallbackRFDETREarlyStoppingRFDETREMACallback)COCOEvalCallback) get_loggerreturncddlm}y)aProbe the full tensorboard import chain to surface numpy/tensorflow incompatibilities early. When tensorboard is installed alongside a numpy-2.0-incompatible tensorflow, importing ``torch.utils.tensorboard`` raises ``AttributeError`` at module level (e.g. ``np.float_`` was removed in NumPy 2.0). Calling this function inside the logger-construction try/except lets ``build_trainer`` degrade gracefully to CSV-only logging instead of crashing mid-training. Raises: ImportError: If the ``tensorboard`` package is absent. AttributeError: If ``torch.utils.tensorboard`` fails to import due to a NumPy 2.0 / tensorflow incompatibility. r SummaryWriterN)torch.utils.tensorboardrrs b/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/rfdetr/training/trainer.py&_try_import_tensorboard_summary_writerr's6c&eZdZdZedefdZy)_InteractiveSpawnLauncherz=Spawn launcher that reports itself as interactive-compatible.rcy)NTselfs ris_interactive_compatiblez3_InteractiveSpawnLauncher.is_interactive_compatibleOsrN)__name__ __module__ __qualname____doc__propertyboolr&r#rrr!r!LsK  t   rr!ceZdZdZddZy)_NotebookSpawnDDPStrategyzFSpawn-based DDP strategy that works inside Jupyter / Kaggle notebooks.Nc|j tdt tdt||j|_y)Nzs_NotebookSpawnDDPStrategy requires a cluster environment; ensure the strategy is initialised through PTL's Trainer.zNotebook spawn strategy requires pytorch_lightning.strategies.launchers.multiprocessing._MultiProcessingLauncher. Your installed PyTorch Lightning version changed this private API; pin/upgrade PTL to a compatible version in the supported >=2.6,<3 range.) start_method)cluster_environment RuntimeErrorr! _start_method _launcherr$s r_configure_launcherz-_NotebookSpawnDDPStrategy._configure_launcherZsR  # # +L  % ,[  34dFXFXYrrN)r'r(r)r*r5r#rrr.r.Ws P Zrr.strategycL|jtfddDS)zLReturn whether a TrainConfig strategy string requests distributed execution.c3&K|]}|v ywNr#).0token strategy_names r z5_is_distributed_strategy_requested..msP%u %Ps)ddpfsdp deepspeed)lowerany)r7r=s @r"_is_distributed_strategy_requestedrDjs!NN$M P3OP PPr acceleratorc|xsdjj}|dvrAtjj xr!tjj dkDSy)zJReturn whether PTL auto/all device resolution can select multiple devices.auto)rGcudagpuF)striprBtorchrH is_available device_count)rEaccelerator_names r&_accelerator_has_multiple_auto_devicesrPpsU#-v446<<>22zz&&(JUZZ-D-D-F-JJ rdevicescnt|tr|dk(r t|S|dkDS|jj }|dvr t|S|j rt|dkDSd|vrPe%++-EPQTUUU Qs B2#B2)rE train_config model_configtrainer_kwargsc | jdtffd }|jdj|jdj}|jdj }t trjjnd}tj}t t|k7r*tdt tdjd ttxs|d kDxs t|} |r| rtd d |d |d|dvr$t!ddt"j%d|nH|dk(s|dk(r>| rj@jB&jDd'kDr%| j5tGjD(| j5tIjJjLjNjPjR)jTd k7r+| j5tWjXd*d d d+d+d+,| j5tWjXd-jTd.d+d+d+,|rd/} d0}njLrd1} d2}nd3} d4}| r|nd}jZ}| j5t]jX| |j^j`|5jbrH| j5tejfjhjj| |j`6tmjXd7d78g}jnr2 tq|j5tsjXd7d78jzr< |j5t}j~jjX:jr@ |j5tjxsd<j~jX=jr td?j}j}|}|rd }d}nj}|}jjj ||||| |r|nd+j2dujXd@d+dA}|j||d<|rWdBD]H}||vs|dCk(rdDndE}|dCk(rdFndG}t-j.dH|dI||dJ|dK|dL|dM t0dJd |dC<d|dN<tdOi|S#tttvf$r!}t"jyd9|Yd}~d}~wwxYw#t$r!}t"jyd;|Yd}~d}~wwxYw#t$r!}t"jyd>|Yd}~d}~wwxYw)PajAssemble a PTL ``Trainer`` with the full RF-DETR callback and logger stack. Resolves training precision from ``model_config.amp`` and device capability, guards EMA against sharded strategies, wires conditional loggers, and applies promoted training knobs (sync_batchnorm, strategy). Args: train_config: Training hyperparameter configuration. model_config: Architecture configuration. Used for precision resolution (``model_config.amp``) and to guard against unsupported distributed configurations for keypoint models. accelerator: PTL accelerator string (e.g. ``"auto"``, ``"cpu"``, ``"gpu"``). Defaults to ``None`` which reads from ``train_config.accelerator`` (itself defaulting to ``"auto"``). Pass ``"cpu"`` to override auto-detection (e.g. when the caller explicitly requests CPU training via ``device="cpu"``). **trainer_kwargs: Extra keyword arguments forwarded to ``pytorch_lightning.Trainer``. Use this to pass PTL-native flags that are not exposed through ``TrainConfig``, for example:: build_trainer(tc, mc, fast_dev_run=2) Most keys present in both ``trainer_kwargs`` and the built config dict are overridden by the value in ``trainer_kwargs``. Detection and segmentation models forward ``accumulate_grad_batches`` from ``train_config.grad_accum_steps`` and ``gradient_clip_val`` from ``train_config.clip_max_norm`` to the Trainer normally. Keypoint models force ``accumulate_grad_batches=1`` and ``gradient_clip_val=None`` because ``RFDETRModelModule`` owns both operations under manual optimization; passing those keys for a keypoint config raises a ``UserWarning`` to make the override explicit. Returns: A configured ``pytorch_lightning.Trainer`` instance. Nrcjs:jdk7r*tjdjdtdydk(ryj}t j jr~|dk(ry |d k(rQt j jry tjd tjd tdy t j jrd Sd St jjjr7|d k(r1tjd tjd tdy y)NrGz amp_dtype=z+ has no effect when model_config.amp=False. stacklevelz32-truecpufp16z16-mixedbf16z bf16-mixedzqamp_dtype='bf16' was requested but this CUDA device does not support bfloat16; falling back to fp16 ('16-mixed').zXamp_dtype='bf16' is not applied on MPS; RF-DETR uses fp16 ('16-mixed') for MPS autocast.) amp amp_dtypewarningswarn UserWarningrLrHrMis_bf16_supported_loggerwarningbackendsmps)rirEr^tcs r_resolve_precisionz)build_trainer.._resolve_precisions&||v%   00[\   %  LL  :: " " $F"!F":://1'9 9  "#(::#?#?#A< Qz Q >>   * * ,F"n n  rr7rQ num_nodesz;Config/model mismatch: isinstance(tc, KeypointTrainConfig)=z* but model_config.use_grouppose_keypoints=zT. Pass KeypointTrainConfig for keypoint models and TrainConfig for detection models.rJzMKeypoint training currently does not support distributed execution (strategy=z , devices=z , num_nodes=z]). Use single-process training for now (for example strategy='auto', devices=1, num_nodes=1).) ddp_notebook ddp_spawnspawnT)r0find_unused_parametersuI%s → spawn-based DDP to avoid OpenMP thread pool corruption after fork.r?rG)rxuXstrategy='auto' with distributed execution → DDPStrategy(find_unused_parameters=True).u<strategy='ddp' → DDPStrategy(find_unused_parameters=True).c3TK|]}|tjv!ywr:)strrB)r;sr7s rr>z build_trainer..*s#L!s8}**,,Ls%()r@rAzTEMA disabled: RFDETREMACallback is not compatible with sharded strategies (strategy=z.). Set use_ema=False to suppress this warning.rbrcrichz.3e)metrics_format) refresh_ratethemetqdm)r)decaytauupdate_interval_stepsg) drop_path)max_dets segmentation eval_intervallog_per_class_metricskeypoint_oks_sigmaslastF)dirpathfilenameevery_n_epochs save_top_kenable_version_counterauto_insert_metric_nameverbosezcheckpoint_{epoch}rSzval/keypoint_map_50_95zval/ema_keypoint_map_50_95zval/segm_mAP_50_95zval/ema_segm_mAP_50_95z val/mAP_50_95zval/ema_mAP_50_95) output_dirmonitor_regular monitor_emarun_testskip_best_epochs smooth_alpha)patience min_deltause_emarrr)save_dirnameversionzTensorBoard logging disabled: %s. If using NumPy 2.x, ensure your TensorBoard installation is NumPy 2.0 compatible (the failure can originate from tensorboard.compat.tensorflow_stub). Install TensorBoard with: pip install tensorboard)rprojectrz;WandB logging disabled: %s. Install with: pip install wandbrfdetr)experiment_namerun_namerz=MLflow logging disabled: %s. Install with: pip install mlflowzKClearML logging is not yet supported. Remove clearml=True from TrainConfig.2) max_epochsrErQrtr7 precisionaccumulate_grad_batchesgradient_clip_valsync_batchnorm callbacksloggerenable_progress_bardefault_root_dirlog_every_n_steps deterministic)rrr1Nonegrad_accum_steps clip_max_normz'build_trainer() ignored trainer_kwargs[z]=z1 for a keypoint model. The model will train with =z regardless of the value passed here because RFDETRModelModule owns gradient accumulation and clipping under manual optimization. To change the effective value, set TrainConfig.z instead.rr#)KrErzgetr7rQrtrUrKrBr,use_grouppose_keypointsr ValueErrorrDr\NotImplementedErrorr.rninfo _DDPStrategyrCrrjrkrl progress_barappendrrrr ema_decayema_tauema_update_intervalrrr eval_max_detssegmentation_headrrrcheckpoint_intervalrrrrrrearly_stoppingrearly_stopping_patienceearly_stopping_min_deltaearly_stopping_use_emar tensorboardrr ImportErrorAttributeErrorrowandbr runrModuleNotFoundErrormlflowr clearmlrsync_bnrepochsupdater)r]r^rEr_rsrQrtr= has_keypointsdistributed_requestedsharded enable_emarrearly_stopping_monitor_emarbest_model_smooth_alphaloggersexcrrmanual_optimizationrrtrainer_configkey effectivealtr7rrs `` @@r build_trainerrsH Bnn ??D!!*bkk:H  BJJ7G""; =I0:8S0IHNN$**,tM==>M")*m;I*UWYlJmIno88D8\8\7]^a a  +3x=9 < q= < %g{ ; ." ! Jwki]Si i  55,'Z^_ W   % MV$;@U t< F " LLj  LLN L6KLLGbjj!1'kJ zzg ! $R T   I &  *%@  F "a89 llJJ&(&<&<   ||c)BLLAB%%%77**"$":": " 6 6   "   ',(-  MM)11#($)  2%A"  ' '.%=")%8"0:,K oo}}+#[[000     335511 /6!#!4!4   " BKLG ~~  2 4 NN!]]   xx ` NNJJ]]  yy b NN$&JJ$:(VV]]  zz!"opp++MJJG('(*."$"5"5)ii"::\\')#:.!$'%!d:MM&N".)!)N: D Cn$#&*C#CC ,/3L,L(Ra =cWB~VYGZF]^88;uAi[ITTWSXXac   5601 /3*+  $^ $$A^,  OOD    $# ` OOY[^ _ _ `# b OO[]` a a bsH1Y>;Z?Z4Z#Y??Z Z1Z,,Z14 [=[[r6r:)0r*rjtypingrrLpytorch_lightningrpytorch_lightning.callbacksrrr2pytorch_lightning.callbacks.progress.rich_progressrpytorch_lightning.loggersr r r r pytorch_lightning.strategiesr r6pytorch_lightning.strategies.launchers.multiprocessingrr rfdetr.configrrrrfdetr.training.callbacksrrrr#rfdetr.training.callbacks.coco_evalrrfdetr.utilities.loggerrrnrr!r.rzr,rDrPrVr\rr#rrrs/ H %YYS]]D $_HG A. , 6F'$<!%Z Z&QQQ d t c d VZ (# J%J%J%t J%  J%  J%c$#$sCC  C