^j-CdZddlmZddlZddlmZddlmZmZddl Z ddl m Z m Z ddl mZddlmZdd lmZdd lmZeZed Gd dZddZ d ddZ d ddZ d ddZddZ d ddZy)a Auto-batch probing: find a safe micro-batch size before training. Probe assumptions (worst-case so training does not OOM): - Resolution: When multi_scale is True we use the maximum of the multi-scale augmentation scales (same as compute_multi_scale_scales). Otherwise we use model resolution. This ensures the step uses the max resolution seen in training. - Targets: Memory grows with number of targets per image. We use auto_batch_max_targets_per_image (config) to synthesize that many targets per image so the probe reflects worst-case matcher and loss memory. - EMA: When use_ema is True, an EMA copy of the model is kept in memory. We apply auto_batch_ema_headroom (e.g. 0.7) to the probed batch size so the effective safe batch leaves room for the EMA model. ) annotationsN) dataclass)Anycast) ModelConfig TrainConfig)compute_multi_scale_scales)build_criterion_from_config) get_logger) NestedTensorT)frozenc:eZdZUdZded<ded<ded<ded<y) AutoBatchResultaResult of auto-batch probing: safe micro-batch size and recommended grad accumulation. Attributes: safe_micro_batch: Per-device batch size that fits in memory for one train step. recommended_grad_accum_steps: Steps to accumulate to reach target effective batch. effective_batch_size: safe_micro_batch * recommended_grad_accum_steps. device_name: Human-readable GPU name used for probing. intsafe_micro_batchrecommended_grad_accum_stepseffective_batch_sizestr device_nameN)__name__ __module__ __qualname____doc____annotations__e/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/rfdetr/training/auto_batch.pyrr%s!"%%rrcHt|j}d|vxsd|vS)Nz out of memoryzcuda error: out of memory)rlower)excmessages r _is_cuda_oomr"6s(#hnnG g % O)D)OOrc tj|||||}tj|||tj|}t ||} t d|dz } t d|} g} t |D]} tjgdgtj|j| d}tjtd| gtj|j| }tj| tj|}tj| fdtj|}||tj| tj|tj||gtj|tj||gtj|||d}|r+tj| ||tj||d <| j|| | fS) aBuild a minimal (samples, targets) batch for probing. Uses max_targets_per_image targets per image so memory reflects worst-case matcher and loss. When segmentation_head is True, each target dict includes "masks" of shape (max_targets_per_image, resolution, resolution). )device)dtyper$r)?r'皙?r(g{Gz?)boxeslabelsimage_id orig_sizesizeiscrowdareamasks)torchrandnzerosboolr maxrangetensorfloat32expandminint64fullappend)micro_batch_size resolutionr$ num_classessegmentation_headmax_targets_per_image num_channelstensorsmasksamples max_labelntargetsidxr*r+r/r0ts r_make_synthetic_batchrM;skk*L*jY_`G ;;'ZuzzZ` aD7D)GA{Q'I A$%A-/G%& 235==QWX__`acdes1i01VT[[\]^++au{{6Bzz1$EMM&I S FKz:&>ekkZ`aLL*j!9U[\&  Q JejjY_`AgJq#$ G rc  |jd|jdt||||||| \} } d|d} | | | d<tjd i| 5|| | }t t t tjf||| }t t t tft|d}|Dcgc]}||vs||||z}}|s td tj|j}dddtjs td tjj||jd|jdycc}w#1swYrxYw#t$r4}t!|r#tj"j%Yd}~y d}~wwxYw) zKRun one forward + loss + backward; return True if successful, False on OOM.T set_to_none)r?r@r$rArBrCrDcuda) device_typeenabledNr% weight_dictznauto-batch probe could not build weighted losses: no overlap between criterion loss_dict and weight_dict keys.z5auto-batch probe produced a non-finite training loss.Fr) zero_gradrMr2autocastrdictrTensorfloatgetattr RuntimeErrorstacksumisfiniteautogradbackwardr"rQ empty_cache)model criterionr?r@r$rAamprBrCrDautocast_dtyperGrJautocast_kwargsoutputs loss_dictrTnameweighted_losseslossr s r _probe_steprles' D)-0-!#/"7% ;AS*Q  %'5OG $ ^^ .o . 6GW-GT#u||"34i6QRItCJ/M1RSKOXpt\`do\oyT1BBpOp""(;;/335D 6~~d#VW W % D)-q 6 6&    JJ " " $ sPAFA#E9? E4 E42E9A+F4E99F>F G)F=<F==Gc |jdk7r td| dkr tdd|cxkrdkstdtd|j} |j} |j |j  d}d}d}|| kr&t |||||||||| | r|}|d z}n|}n|| kr&|dkr td || dz}|dz}t |dz | }||kr0||zd z}t |||||||||| | r|}|dz}n|dz }||kr0tdtj||z}||j | |j | |jd |jd tjjS#|j | |j | |jd |jd tjjwxYw) amFind the largest per-device batch size that fits in memory for one train step. Uses exponential search (1, 2, 4, ...) up to the first failure, then binary search between the last successful size and the first failure to get the exact maximum. The returned value is floor(max_ok * safety_margin), so safety_margin in (0, 1] scales down the result for headroom (e.g. 0.9 keeps 10% margin). Args: model: The model to probe (will be set to train mode). criterion: The loss criterion (must match model output and target format). resolution: Input spatial size (square). device: CUDA device to run on. num_classes: Number of classes (for synthetic targets). amp: Whether to use autocast for the forward. segmentation_head: If True, synthetic targets include "masks" for loss_masks. max_targets_per_image: Number of synthetic targets per image (worst-case memory). safety_margin: Fraction of max batch to return (0 < safety_margin <= 1). max_micro_batch: Cap on batch size to try. num_channels: Number of input image channels (for synthetic probe images). Returns: Safe micro-batch size (>= 1). Raises: RuntimeError: If device is not CUDA or if micro_batch_size=1 already fails (OOM). ValueError: If max_micro_batch < 1 or safety_margin not in (0, 1]. rQz0auto-batch probing currently supports CUDA only.r&zmax_micro_batch must be >= 1.rg?z safety_margin must be in (0, 1].Nzjauto-batch probe failed at micro_batch_size=1. Try lowering resolution or enabling gradient_checkpointing.TrO)typer[ ValueErrortrainingtrainrlr;r6mathfloorrUr2rQra)rbrcr@r$rArdrBrC safety_marginmax_micro_batchrDremodel_trainingcriterion_traininglower_ok candidate upper_faillohimidrs rprobe_max_micro_batchrsEP{{fMNN899 $ $;<< %;<<^^N"++ KKM OO@!  ?*!% %Q & %?*( a<N   (1,J \ a 1Bh7q.C!% 1W1W%Bh*q$**X -E"FG N#*+ D)-   N#*+ D)-  s0F7AF#FA&Hc|dkr td|dkr tdtdtj||z S)aRecommend gradient accumulation steps to reach target effective batch size. Args: safe_micro_batch: Per-step batch size that fits in memory. target_effective_batch: Desired effective batch (micro_batch * accum_steps). Returns: ceil(target_effective_batch / safe_micro_batch), at least 1. Raises: ValueError: If either argument is < 1. r&zsafe_micro_batch must be >= 1.z$target_effective_batch must be >= 1.)rpr6rsceil)rtarget_effective_batchs rrecommend_grad_accum_stepsrsI!9::!?@@ q$))25EEF GGrcd|j}tjjr|jdk7r t dt |dd}t |dd}|r[|sYt |dd}t |dd } t |d d } t|j|| | } | r t| n |j} n |j} t |d d } t||\}}|j|}t|j}t |dd}|rU|dk(rtj}nAtjjrtj ntj}nd}t#|j$|| ||j&||j(| ||t |dd| }t |dd}|rFt |dd}tdt+j,||z}t.j1d||t |dd}t |dd}t3|t4r t3|t4rtd||z}nd}|j6}|dkDr#tdt+j8||z }n|}t;||}||z}||z}tjj=|}t.j1d|||j(| | t.j1d||||t.j1dt.j1dt?|||| S)!aResolve batch_size='auto' into concrete batch_size and grad_accum_steps using a probe. Expects model_context to have attributes: .device (torch.device) and .model (nn.Module). Runs probe_max_micro_batch on the current model/criterion, then recommend_grad_accum_steps using train_config.auto_batch_target_effective. Logs device, segmentation flag, resolution, and the chosen values; also logs that the probe is train-step-only and that eval/test may use more memory. Args: model_context: Object with .device and .model (e.g. RFDETR.model from get_model()). model_config: Architecture config (resolution, num_classes, amp, segmentation_head). train_config: Training config (auto_batch_target_effective); batch_size should be "auto". safety_margin: Fraction of max batch to use (passed to probe_max_micro_batch). max_micro_batch: Upper bound on batch size to try (passed to probe_max_micro_batch). Returns: AutoBatchResult with safe_micro_batch, recommended_grad_accum_steps, effective_batch_size, and device_name. Raises: RuntimeError: If CUDA is not available or model_context.device is not CUDA. rQz;batch_size='auto' requires a CUDA device for probing in v1. multi_scaleFdo_random_resize_via_paddingexpanded_scalesT patch_size num_windowsr) auto_batch_max_targets_per_imaged amp_dtypeautofp16NrD) rbrcr@r$rArdrBrCrurvrDreuse_emaauto_batch_ema_headroomgffffff?r&z=[auto-batch] Applied EMA headroom (%.2f): safe_micro_batch=%sdevices num_nodesza[auto-batch] device=%s world_size=%s segmentation=%s probe_resolution=%s max_targets_per_image=%szl[auto-batch] safe_micro_batch=%s grad_accum_steps=%s effective_batch_per_device=%s global_effective_batch=%szC[auto-batch] This probe estimates train-step-safe micro-batch only.zY[auto-batch] Validation/test (especially segmentation mask eval) may require more memory.)rrrr) r$r2rQ is_availableror[rZr r@r6r tor5rdfloat16is_bf16_supportedbfloat16rrbrArBrsrtloggerinfo isinstancerauto_batch_target_effectiverrget_device_namer) model_context model_config train_configrurvr$rdo_random_resizerrrscalesprobe_resolutionrCrc_ amp_enabled amp_dtype_strprobe_autocast_dtyperrheadroomrr world_sizetarget_effective_globaltarget_effective_per_devicegrad_accum_stepseffective_batch_size_per_deviceglobal_effective_batch_sizers rresolve_auto_batch_configr(s6 ! !F :: " " $ v(=XYY, u=K|-KUS+!,0A4H\<< lM1= +  # #      +13v;l6M6M'22#L2TVYZ.|\JLIq V$I|''(KL+v>M F "7<}} 6;ZZ5Q5Q5S5>>Y^YfYf #,!!# ,, &883#'\>1=+ lIu5G<)BCHq$**-=-H"IJ SU]_oplIt4G k15I'3Jy#$>Gi/0  +FFA~&)!TYY7NQ[7[-\&]#&=#12BD_`&69I&I#"AJ"N**,,V4K KKk&&  KKv'#  KKUV KKkl )%5<  r)r BaseExceptionreturnr5)Fr&r)r?rr@rr$ torch.devicerArrBr5rCrrDrrz2tuple[NestedTensor, list[dict[str, torch.Tensor]]])Fr&rN)rbtorch.nn.Modulercrr?rr@rr$rrArrdr5rBr5rCrrDrretorch.dtype | Nonerr5)Fr&?rN)rbrrcrr@rr$rrArrdr5rBr5rCrrurYrvrrDrrerrr)rrrrrr)rr) rrrrrrrurYrvrrr)r __future__rrs dataclassesrtypingrrr2 rfdetr.configrrrfdetr.datasets.cocor rfdetr.modelsr rfdetr.utilities.loggerr rfdetr.utilities.tensorsr rrr"rMrlrrrrrrrs2 # ! 2;5.1  $    P$!"''' ' '  '  ''8'd$!")-5 555 5  5  5 5555'5 5~$!")-t! t!t!t!  t!  t!  t!t!t!t!t!t!'t! t!nH0 @@@@ @  @  @r