^jha XdZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z mZddlZddlmZmZmZddlmZddlmZdd lmZ dd ed ed e ed ededef dZGddZde e ee eeffde e ede e ede e e eefe eeffdZ GddeZ!y)a Dynamic Sequence Length Datasets for Variable Resolution Image Processing Implements two dataset wrappers: 1. NaFlexMapDatasetWrapper - Map-style dataset that returns batches with variable sequence lengths TODO: 2. NaFlexIterableDatasetWrapper - Iterable dataset that yields batches with variable sequence lengths Both support: - Pre-initialized transforms for efficiency - Distributed training - Multiple workers - Variable batch sizes based on sequence length Hacked together by / Copyright 2025, Ross Wightman, Hugging Face N)partial)AnyIteratorListTupleDictOptionalUnionCallable)DatasetIterableDataset DataLoader)Image)Patchify) to_2tupletokens_per_batchseq_lenmax_sizedivisorroundingreturnc ||z }|dkDrT|dk(rtj||z |z}n>|dk(rtj||z |z}nt||z |z}n t |}t d|}| t ||}|S)aCalculate batch size based on sequence length with divisibility constraints. Args: tokens_per_batch: Target number of tokens per batch. seq_len: Sequence length for this batch. max_size: Optional maximum batch size. divisor: Ensure batch size is divisible by this value. rounding: Rounding method ('floor', 'ceil', 'round'). Returns: Calculated batch size. rfloorceil)mathrrroundintmaxmin)rrrrrraw_batch_size batch_sizes c/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/data/naflex_dataset.pycalculate_naflex_batch_sizer$s(&/N{ w NW$<=GJ  >G#;z*NaFlexCollator.__call__..usO4d9o33A6Osr1 patch_coord patch_valid)r1r8r9r) isinstancetuplelentorchTensorstacktensorint64r(rndimr2zerosfloat32bool enumerater )r+r-r"r4targets patch_dicts max_patchespatches_tensoris_unflattened patch_dimsr1 patch_dimr8r9i patch_dict num_patchesresults r#__call__zNaFlexCollator.__call__Ws+%(E***Z (--t47-- gaj%,, /kk'*Gll7%++>G,114tAw1 1    '**KO;OOK%Q 2',,1 (--ab1Jkk:{"HZ"HPUP]P]^G',,Q/Ikk:{I"Femm\Gkk:{A">ekkR kk:{";5::N '{3 SMAzj399!r rRr%r#r'r'IsO*. .!# .  .Ad5c5<<.?)@%U\\HYBZ)Z#[\Aafgkloqvq}q}l}g~AFAMAMhMbNAr%r' patch_sizepatch_size_choicespatch_size_choice_probsc||d}|du|duk(r td|t|g}dg}d}n|Dcgc] }t|}}|dt|z gt|z}nXt|t|k7r tdtt |}|dkr td|Dcgc]}||z  }}d }|||fScc}wcc}w) aResolve patch size configuration. Args: patch_size: Single patch size to use. patch_size_choices: List of patch sizes to choose from. patch_size_choice_probs: Probabilities for each patch size choice. Returns: Tuple of (sizes, probs, variable) where sizes is list of patch size tuples, probs is list of probabilities, and variable indicates if patch size varies. Nz ##DJJ/r%c t|j}|}|}|jr|jdkDr||jzdk7rA|j||jzz }||z }t d|j d|d|dnd}||jzdk7rt d|d|j||jz}n|jr|jdkr ||_|dk(rg|_d|_ y tj}|j|jg}|}d}|dkDrtjdt|jd | j!} |j| } t#|j$| ||j&d } t)d| } t+| |} | dkrt-j.d| d|dn#|j1| | f|| z}|| z }|dkDr||k7r+t-j.d|j d|d|d|||_t||_ t d|j d|jd|jdy )a% Calculates the canonical batch schedule (seq_len, batch_size pairs) based on the dataset size, padded for distributed training. This schedule is the *same* for all ranks and ensures consistent epoch length. It is calculated once during initialization. rrRank z: Padding dataset with z. samples for distributed training (total size z).z$Internal Error: Padded total length z not divisible by world size N)r generatorr)rrrrrz$Calculated batch size <= 0 (seq_len=z , remaining=z&). Stopping schedule generation early.z": Canonical schedule accounts for z samples, but expected z samples per rank. This might happen if min_batch_size or batch_divisor constraints prevent utilizing all samples. Check parameters. Remaining samples: z": Created canonical schedule with z batches for z samples/rank.)r<rirprrprintrq RuntimeErrorrrrr= Generator manual_seedrnrandintrjr4r$rkrtrr warningswarnr) r+ total_lenpadded_total_lennum_samples_per_rankpad_sizegcurrent_scheduleremaining_samplestotal_scheduled_samplesseq_idxrr"s r#rz2NaFlexMapDatasetWrapper._create_canonical_schedule7s))* $(   ! 34??*a/ OOy4??/JK!X-!tyyk)@ JxzJyKKMNO$//1Q6#&JK[J\\yz~{J{JzK%LMM#3t#F   $//Q"6 (<% 1 $.0T +*+T '  OO  dii 240"#!#mmAs4=='941MRRTGmmG,G5!%!:!:*** JQ +JZ):;JQ!EgYl[lZmnT UV  # #Wj$9 :  +  #z 1 #1 !#6 #&: : MM {"DE\D]^ 45688I7JL *:&%()9%:" dii[ B4C]C]B^^klpmJmJlKKYZ [r%c tj}|j|j|zt |j }|j r&tj||j}ntt|}|}|jrb|jdkDrS|j|jz}||kDr ||z }||d|z}t ||k7rtdt |d||jr,|jdkDr||jd|j}n|}t ||jk7rat!j"d|jdt |d|jd t%t ||j} |d| }n |j} |j rLtj|j&|j} | D cgc]} |j(| } } nt|j(} g|_d } d }| D]x\}}t%|| | z }|d kr0|| kr)t!j"d|jd | d | d n1|| | |z}|j*j-||f| |z } ||z }z|| k7r0t!j"d|jd|d| d| |z d yycc} w)a Prepares the batches for the current epoch by: 1. Shuffling the full dataset indices (using epoch seed). 2. Applying padding if in distributed mode. 3. Selecting indices for the current rank. 4. Shuffling the *order* of the canonical batch schedule (using epoch seed). 5. Assigning the rank's indices to the shuffled batches. rrNz)Internal Error: Padded index list length z does not match expected rz#: Number of indices for this rank (z3) does not match expected padded samples per rank (z*). Epoch generation might be inconsistent.rz: Ran out of samples (/z?) before processing entire schedule. Check schedule generation.z : Assigned z" samples to batches, but expected z2 effective samples this epoch. Indices remaining: .)r=rrrnr<rirorandpermtolistrrangerprrrrrqrrr rrrr)r+rsrrall_indices_shuffledindices_for_ranksrrindices_this_rankeffective_samples_this_rank schedule_permrNshuffled_scheduleidx_posscheduled_samples_countrbs actual_bs batch_indicess r#rz.NaFlexMapDatasetWrapper._prepare_epoch_batchessE OO  dii%'())* <<#(>>)q#I#P#P#R #'i(8#9 1   ! 3#<| j%|| j%|a|j*|j+| | \} } | Dcgc] }| | } }t-t/| | }|s>|j0||Wy#t&$rt!j"d| dYt($r'}t!j"d| d |dYd}~d}~wwxYwcc}ww) zIterates through pre-calculated batches for the current epoch. Yields: Tuple of (input_dict, targets) for each batch. rrNz"Transform returned None for index z. Skipping sample.z!IndexError encountered for index z= (possibly due to padding/repeated indices). Skipping sample.zError processing sample index z . Error: )r=utilsdataget_worker_info num_workersidrr multinomialr@rr4rgetrrirrr IndexError Exceptionrmrzipr)r+ worker_infor worker_idbatches_for_workerrindicesr transform_key transformbatch_patchifier batch_imgs batch_targetsidximglabel processed_imge batch_sampless r#__iter__z NaFlexMapDatasetWrapper.__iter__s( kk&&668 1>_/ ?@"]]%FseLI$JK MM$B3%yQRPSSe"fgGsZC-H09F8)H*"F8 'H3HHH8"H HH $HHH  H)NNN)r*iii@NN*TFrrrT)rSrTrUrVr r r rrrr_r rEr,rrrrwrrrWr=r>r __classcell__)rs@r#rhrhs AE6:=A(B(048+/ %!"+/#b0!b0!sE#s(O';!<=b0!)c 3 b0 &.d5k%: b0 CHo b0#&b0 (1b0x(b0b0b0b0b0b0b0 !b0"%)#b0$ %b0HS[lTCTl /s /t /**<?(5c5<<.?)@%,,)N#OP<?r%rh)Nrr)"rVrrandomr functoolsrtypingrrrrrr r r r=torch.utils.datar r rPILrnaflex_transformsr timm.layersrrrWr$r'r_rErfrhrXr%r#rs   NNN AA'! #' (((3-( (  (  (VOOd)"U3c3h#789)"$T#Y/)""*$u+!6)" 4c3h $u+t 34 )"Xk?ok?r%