^j~;dZddlZddlmZmZmZddlZddlmZddl mcm Z ddl m Z ddlmZddlmZej$eZGdd ej*ZGd d eZy) zJToken-based distillation training task for models with distillation heads.N)DictOptionalUnion) create_model) unwrap_model) TrainingTaskcxeZdZdZ ddeeejfdee de deedee jdee jf fd Z dd ed eed dfd Zd e jd e jfdZ dd e jdee jdee jd e jfdZxZS)TokenDistillationTeacheraKWrapper for a teacher model used in token-based distillation. Creates and manages a pre-trained teacher model for token distillation, handling model creation and normalization differences between teacher and student. Can be created from: - A model name string (creates the model internally) - An existing nn.Module (wraps it with the necessary interface) Args: model_name_or_module: Either a model name string or an nn.Module num_classes: Number of output classes (required if model_name_or_module is a string) in_chans: Number of input channels (used if model_name_or_module is a string) pretrained_path: Optional path to pretrained weights (used if model_name_or_module is a string) device: Device to place the model on dtype: Model dtype (uses float32 if None) model_name_or_module num_classesin_chanspretrained_pathdevicedtypec t|t|trAtj d|dddi}|rt |||d<td|||||d|}n>t|tjr|}n!tdt|j|j||_t|} t!| d r9| j"j%d d } | j"j%d d } nd } d } t'j(| ||j+dddd} t'j(| ||j+dddd} |j-d| d|j-d| dy)Nz,Creating token distillation teacher model: '' pretrainedT)filer pretrained_cfg_overlay) model_namer rrrz8model_name_or_module must be a string or nn.Module, got pretrained_cfgmean)g ףp= ?gv/?gCl?std)gZd;O?gy&1?g?rrrmean_kdF persistentstd_kd)super__init__ isinstancestr_loggerinfodictrnnModule TypeErrortype__name__evalmodelrhasattrrgettorchtensorviewregister_buffer)selfr r rrrrpretrained_kwargsr/model_unwrappedrrrr __class__s g/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/task/token_distillation.pyr#z!TokenDistillationTeacher.__init__$s  *C 0 LLGH\G]]^_ `!-t 4 >B( +?!":; !/'!  $ E,bii 8(EJ4PdKeKnKnJop    'u- ?$4 5"1155f>STD!0044UCCAr1aP YEB Xv%@backendmodereturnc Vtj|jf||d||_|S)z Compile teacher logit inference.r<r=)r2compiler/r6r<r=compile_kwargss r:rAz TokenDistillationTeacher.compileYs)]]4::\wT\^\  r;inputc$|j|S)zForward pass through teacher model. Args: input: Input tensor (should already be normalized for teacher) Returns: Teacher logits )r/)r6rDs r:forwardz TokenDistillationTeacher.forwardcszz%  r; student_mean student_stdc|||Stj||jr"tj||jr|S||z|z|jz |jz S)aoNormalize input to match teacher's expected normalization. Args: input: Input tensor (already normalized for student) student_mean: Student normalization mean buffer [1, 3, 1, 1] student_std: Student normalization std buffer [1, 3, 1, 1] Returns: Input tensor normalized for the teacher model )r2equalrr )r6rDrGrHs r:normalize_inputz(TokenDistillationTeacher.normalize_inputnsa  ;#6L ;;|T\\ 2u{{;PTP[P[7\L #l2T\\AT[[PPr;)NNNNinductorN)NN)r- __module__ __qualname____doc__rr%r)r*rintr2rrr#rATensorrFrK __classcell__r9s@r:r r s!**.-1-1+/3A"'RYY"73A"#3A 3A &c] 3A U\\* 3AEKK(3An&"&3- $  !U\\ !ell !4826 Q<<Q#5<<0Q"%,,/ Q  Qr;r ceZdZdZ ddej deeej efde ej de edede e de e d e d e e jd e e jd effd Z dde eddfdZ ddede edej fdZde j&de j&deee j&ffdZxZS)TokenDistillationTaskaToken-based distillation task for models with distillation heads. For models like DeiT that have a dedicated distillation token/head that returns a tuple (main_logits, dist_logits) when distilled_training is enabled. The main head is trained against ground truth labels while the distillation head matches teacher outputs. Supports two distillation modes: - 'soft': KL divergence with temperature scaling (default) - 'hard': Cross-entropy with teacher's hard predictions (argmax) Loss weighting supports two modes: 1. Independent weights: loss = task_loss_weight * task_loss + distill_loss_weight * distill_loss 2. Complementary mode: loss = task_loss_weight * task_loss + (1 - task_loss_weight) * distill_loss (used when only task_loss_weight is specified) Args: student_model: Student model with set_distilled_training() method teacher_model: Teacher model - can be a model name string, nn.Module, or TokenDistillationTeacher criterion: Task loss function for main head (default: CrossEntropyLoss) teacher_pretrained_path: Path to teacher pretrained weights (used when teacher_model is a string) distill_type: 'soft' for KL-div or 'hard' for CE with teacher argmax distill_loss_weight: Weight for distillation loss task_loss_weight: Weight for task loss temperature: Softmax temperature for soft distillation (ignored for hard) device: Device for task tensors/buffers dtype: Dtype for task tensors/buffers verbose: Enable info logging Example: >>> # With model name string (num_classes/in_chans inferred from student) >>> task = TokenDistillationTask( ... student_model=model, teacher_model='deit_base_patch16_224', ... criterion=nn.CrossEntropyLoss(), ... distill_type='soft', temperature=3.0, task_loss_weight=0.5, ... device=torch.device('cuda'), ... ) >>> # With raw model >>> task = TokenDistillationTask( ... student_model=model, teacher_model=my_teacher_model, ... criterion=nn.CrossEntropyLoss(), ... distill_type='hard', task_loss_weight=0.5, ... ) student_model teacher_model criterionteacher_pretrained_path distill_typedistill_loss_weighttask_loss_weight temperaturerrverbosec t|| | | t|} t| ds#t d| j j d| jdt|tr|} nt|tst|tjr>| j}| j}t|||||j|j } n!t#dt%|j ||_| |_||ntj*|_||_||_|dvrt d |d t3j4| j6d |j|j j9d dd d }t3j4| j6d|j|j j9d dd d }|j;d|d|j;d|d|D||_||nd|_|j@rtBjEd|j>d|n|C||_d|z |_|j@rotBjEd|d|j<nId|_d|_|j@r/tBjEd|j>d|j<|j@rtBjEd|d|yy)N)rrr`set_distilled_trainingzModel z does not have 'set_distilled_training' method. TokenDistillationTask requires a model with a distillation head (e.g., DeiT distilled variants).T)r r rrrrzWteacher_model must be a model name string, nn.Module, or TokenDistillationTeacher, got )softhardzUnsupported distill_type 'z'. Must be 'soft' or 'hard'.rrrrrrGFrrH?z9TokenDistillationTask: Independent weights - task_weight=z, distill_weight=z8TokenDistillationTask: Complementary mode - task_weight=z;TokenDistillationTask: Default equal weights - task_weight=z$TokenDistillationTask: distill_type=z, temperature=)#r"r#rr0 ValueErrorr9r-rbr$r r%r)r*r rrrr+r,trainable_moduleteacherCrossEntropyLossrZr\r_r2r3rr4r5r]r^r`r&r')r6rXrYrZr[r\r]r^r_rrr`student_unwrappedrhr rrGrHr9s r:r#zTokenDistillationTask.__init__s eWE)7(*BC*44==>?ss  006 m%= >#G  s +z-/S+77K(11H.%2'! 7{{jj GM*3346  !. &/&;ATATAV(& / /9,Gcde e||  , ,V 4;;**  $q"a   ll  , ,U 3;;**  $q"a   ^\eL ]KEJ  *':D $8H8T$4Z]D !|| ##'#8#8"99JK^J_a )$4D !'*-='=D $|| ##3"44EdF^F^E_a (+D $$'D !|| ##'#8#8"99J4KcKcJdf << LL6|nNS^R_`  r; device_idsr>c ddlm}|jjD] }d|_ ||j fd|i||_|S)aPrepare task for distributed training. Wraps the student model in DistributedDataParallel (DDP) while leaving the frozen teacher model unwrapped. Args: device_ids: List of device IDs for DDP (e.g., [local_rank]) **ddp_kwargs: Additional arguments passed to DistributedDataParallel Returns: self (for method chaining) r)DistributedDataParallelFrk)torch.nn.parallelrmrh parameters requires_gradrg)r6rk ddp_kwargsDDPparams r:prepare_distributedz)TokenDistillationTask.prepare_distributedsQ" E\\,,. (E"'E  (!$D$9$9 _j _T^ _ r;r<r=c tj|jf||d||_|j|_|jjd||d||jS)z=Compile student eval/train forward and teacher logit forward.r@r!)r2rArg eval_modelrhrBs r:rAzTokenDistillationTask.compile5s`!& d.C.C rW[_ rcq r// JW4J>J$$$r;rDtargetc|j|}|\}}|j||}tj5|jj ||j |j}|j |j}ddd|jdk(rrtj||jz d} tj|jz d} tj| | dd|jdzz} n(jd} tj|| } |j |z|j"| zz} | ||| d S#1swYxYw) aForward pass with token distillation. Args: input: Input tensor [B, C, H, W] target: Target labels [B] Returns: Dictionary containing: - 'loss': Combined training loss (task + distillation) - 'output': Main head logits (for metrics) - 'task_loss': Classification loss component - 'distill_loss': Distillation loss component Nrcr)dim batchmeanT) reduction log_target)lossoutput task_loss distill_loss)rgrZr2no_gradrhrKrGrHdetachr\F log_softmaxr_kl_divargmax cross_entropyr^r])r6rDrwstudent_output main_logits dist_logitsrinput_kdteacher_logitsprob_sprob_tr teacher_hard total_losss r:rFzTokenDistillationTask.forwardBsR&..u5#1 [NN;7 ]]_ =||33E4;L;LdN^N^_H!\\(//*;D4D4D#D"MF88FFkVZ[_c_o_ost_tuL)00R08L??; EL**Y69Q9QT`9`` !"(    = =s AE%%E.) NNrcNNreNNT)NrM)r-rOrPrQr)r*rr%r rfloatr2rrboolr#listrtrArSrrFrTrUs@r:rWrWsh+b.259 &3704!$-1+/ g99g!bii1I!IJg  * g &.c] g  g"*%g'uoggU\\*gEKK(ggV*.  ! 6&"& % %3- %  %. <<. LL.  c5<< . r;rW)rQloggingtypingrrrr2torch.nnr)torch.nn.functional functionalr timm.modelsr timm.utilsrtaskr getLoggerr-r&r*r rWr!r;r:rs[P(( $# '  H %qQryyqQhk Lk r;