^j u dZddlZddlmZmZmZmZddlZddlm Z ddl m cm Z ddl mZmZddlmZddlmZej(eZGdd e j.Zd eee j.efd e j.d eed eej4deej6def dZGddeZGdde j.ZGddeZy)z5Knowledge distillation training tasks and components.N)DictOptionalTupleUnion) create_modelgroup_parameters) unwrap_model) TrainingTaskceZdZdZ ddeeejfdee de deedee jdee jf fd Z d e jd e jfd Z dd ed eededed df dZ dd e jded e jfdZ dd e jdee jdee jd e jfdZxZS)DistillationTeacheraeWrapper for a teacher model used in knowledge distillation. Creates and manages a pre-trained teacher model for knowledge distillation, handling model creation and normalization differences between teacher and student. Can be created from: - A model name string (creates the model internally with pretrained weights) - An existing nn.Module (wraps it with the necessary interface) Args: model_name_or_module: Either a model name string or an nn.Module num_classes: Number of output classes (required if model_name_or_module is a string) in_chans: Number of input channels (used if model_name_or_module is a string) pretrained_path: Optional path to pretrained weights (used if model_name_or_module is a string) device: Device to place the model on dtype: Model dtype (uses float32 if None) model_name_or_module num_classesin_chanspretrained_pathdevicedtypec t|t|trAtj d|dddi}|rt |||d<td|||||d|}n>t|tjr|}n!tdt|j|j||_t|} t!| d r9| j"j%d d } | j"j%d d } nd } d } t'j(| ||j+dddd} t'j(| ||j+dddd} |j-d| d|j-d| dd|_y)NzCreating KD teacher model: '' pretrainedT)filerpretrained_cfg_overlay) model_namerrrrz8model_name_or_module must be a string or nn.Module, got pretrained_cfgmean)g ףp= ?gv/?gCl?std)gZd;O?gy&1?g?rrr mean_kdF persistentstd_kd)super__init__ isinstancestr_loggerinfodictrnnModule TypeErrortype__name__evalmodelr hasattrrgettorchtensorviewregister_buffer_compiled_forward_features)selfrrrrrrpretrained_kwargsr1model_unwrappedrrrr" __class__s a/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/timm/task/distillation.pyr%zDistillationTeacher.__init__%s  *C 0 LL78L7MQO P!-t 4 >B( +?!":; !/'!  $ E,bii 8(EJ4PdKeKnKnJop    'u- ?$4 5"1155f>STD!0044UCCAr1aP YEB Xv%@*.'inputreturnc$t|jdrt|jds-td|jjjd|jj |}|jj |dS)Nforward_features forward_headzModel zi does not support feature extraction. Ensure the model has 'forward_features' and 'forward_head' methods.T pre_logits)r2r1 ValueErrorr<r/rBrC)r9r? feature_maps r=_forward_featuresz%DistillationTeacher._forward_features[stzz#56gdjjR`>a--6678VV jj11%8 zz&&{t&DDr>backendmode compile_modelcompile_featuresc |r(tj|jf||d||_|r(tj|jf||d||_|S)z;Compile teacher inference paths used by distillation tasks.rIrJ)r4compiler1rHr8)r9rIrJrKrLcompile_kwargss r=rOzDistillationTeacher.compiledsa tzz`7`Q_`DJ .3mm&&//! /D +  r>return_featuresc|r.|j|j|S|j|S|j|S)aGForward pass through teacher model. Args: input: Input tensor (should already be normalized for teacher) return_features: Whether to return pooled pre-logits features instead of logits Returns: Logits or pooled pre-logits features depending on return_features flag )r8rHr1)r9r?rQs r=forwardzDistillationTeacher.forwardxsD ..:66u==))%0 0::e$ $r> student_mean student_stdc|||Stj||jr"tj||jr|S||z|z|jz |jz S)aoNormalize input to match teacher's expected normalization. Args: input: Input tensor (already normalized for student) student_mean: Student normalization mean buffer [1, 3, 1, 1] student_std: Student normalization std buffer [1, 3, 1, 1] Returns: Input tensor normalized for the teacher model )r4equalrr")r9r?rTrUs r=normalize_inputz#DistillationTeacher.normalize_inputsa  ;#6L ;;|T\\ 2u{{;PTP[P[7\L #l2T\\AT[[PPr>)NNNN)inductorNTFF)NN)r/ __module__ __qualname____doc__rr'r+r,rintr4rrr%TensorrHboolrOrSrX __classcell__r<s@r=r r se**.-1-1+/4/"'RYY"74/"#4/ 4/ &c] 4/ U\\* 4/EKK(4/lEu||E E&"&"&%* 3-  #  .%*%<<%"%  %04826 Q<<Q#5<<0Q"%,,/ Q  Qr>r teacher student_modelrrrr@ct|tr|St|}|j}|j}t||||||S)aResolve teacher input to a DistillationTeacher instance. Args: teacher: Model name string, nn.Module, or DistillationTeacher student_model: Student model to infer num_classes/in_chans from pretrained_path: Optional path to teacher pretrained weights device: Device for teacher dtype: Dtype for teacher Returns: DistillationTeacher instance )rrrrrr)r&r r rr)rdrerrrstudent_unwrappedrrs r=_resolve_teacherrhsV&'./%]3#//K ))H $'  r>ceZdZdZ ddej deeej efde ej de edede e de e d e d e e jd e e jd effd Z dde eddfdZ ddede edej fdZde j&de j&deee j&ffdZxZS)LogitDistillationTaskaLogit-based knowledge distillation task. Performs distillation by matching student and teacher output logits using KL divergence with temperature scaling. Loss weighting supports two modes: 1. Independent weights: loss = task_loss_weight * task_loss + distill_loss_weight * distill_loss 2. Complementary mode: loss = task_loss_weight * task_loss + (1 - task_loss_weight) * distill_loss (used when only task_loss_weight is specified) Args: student_model: Student model to train teacher_model: Teacher model - can be a model name string, nn.Module, or DistillationTeacher criterion: Task loss function (default: CrossEntropyLoss) teacher_pretrained_path: Path to teacher pretrained weights (used when teacher_model is a string) loss_type: Type of distillation loss (currently only 'kl' supported) distill_loss_weight: Weight for distillation loss task_loss_weight: Weight for task loss temperature: Softmax temperature for distillation (typical values: 1-4) device: Device for task tensors/buffers dtype: Dtype for task tensors/buffers verbose: Enable info logging Example: >>> # With model name string (num_classes/in_chans inferred from student) >>> task = LogitDistillationTask( ... student_model=model, teacher_model='resnet50', ... criterion=nn.CrossEntropyLoss(), ... task_loss_weight=0.3, temperature=4.0, ... device=torch.device('cuda'), ... ) >>> # With raw model >>> task = LogitDistillationTask( ... student_model=model, teacher_model=my_teacher_model, ... criterion=nn.CrossEntropyLoss(), ... task_loss_weight=0.3, temperature=4.0, ... ) re teacher_model criterionteacher_pretrained_path loss_typedistill_loss_weighttask_loss_weight temperaturerrverbosec t|| | | t||||j|j} ||_| |_||ntj|_ ||_ ||_ |dk7rtd|dt|} tj| j d|j|jj#dddd}tj| j d |j|jj#dddd}|j%d |d |j%d |d |D||_||nd|_|j*rt,j/d|j(d|n|C||_d|z |_|j*rot,j/d|d|j&nId|_d|_|j*r/t,j/d|j(d|j&|j*rt,j/d|d|yy)NrrrrklzUnsupported loss_type 'z$'. Currently only 'kl' is supported.rrr rrrTFr rU?z9LogitDistillationTask: Independent weights - task_weight=, distill_weight=z8LogitDistillationTask: Complementary mode - task_weight=z;LogitDistillationTask: Default equal weights - task_weight=z!LogitDistillationTask: loss_type=z, temperature=)r$r%rhrrtrainable_modulerdr+CrossEntropyLossrlrnrqrFr r4r5rr6r7rorprrr(r))r9rerkrlrmrnrorprqrrrrrdrgrTrUr<s r=r%zLogitDistillationTask.__init__sb eWE#   # KK JJ  !. &/&;ATATAV"&  6ykAefg g)7||  , ,V 4;;**  $q"a   ll  , ,U 3;;**  $q"a   ^\eL ]KEJ  *':D $8H8T$4Z]D !|| ##'#8#8"99JK^J_a )$4D !'*-='=D $|| ##3"44EdF^F^E_a (+D $$'D !|| ##'#8#8"99J4KcKcJdf << LL3I;n[MZ  r> device_idsr@c ddlm}|jjD] }d|_ ||j fd|i||_|S)aPrepare task for distributed training. Wraps the student model in DistributedDataParallel (DDP) while leaving the frozen teacher model unwrapped. Args: device_ids: List of device IDs for DDP (e.g., [local_rank]) **ddp_kwargs: Additional arguments passed to DistributedDataParallel Returns: self (for method chaining) rDistributedDataParallelFrztorch.nn.parallelr}rd parameters requires_gradrxr9rz ddp_kwargsDDPparams r=prepare_distributedz)LogitDistillationTask.prepare_distributedDQ" E\\,,. (E"'E  (!$D$9$9 _j _T^ _ r>rIrJc tj|jf||d||_|j|_|jjd||ddd||jS)z=Compile student eval/train forward and teacher logit forward.rNTFrIrJrKrLr#)r4rOrx eval_modelrd)r9rIrJrPs r=rOzLogitDistillationTask.compile]ss!& d.C.C rW[_ rcq r//  "    $$$r>r?targetcl|j|}|j||}tj5|jj ||j |j}|j |jd}dddtj||jz d}tj|jz d}tj||dd|jd zz} |j|z|j| zz} | ||| d S#1swYxYw) aForward pass with logit distillation. Args: input: Input tensor [B, C, H, W] target: Target labels [B] Returns: Dictionary containing: - 'loss': Combined training loss (task + distillation) - 'output': Student logits (for metrics) - 'task_loss': Classification loss component - 'kd_loss': Logit distillation loss component FrQNr)dim batchmeanT) reduction log_targetlossoutput task_losskd_loss)rxrlr4no_gradrdrXrTrUdetachF log_softmaxrqkl_divrpro) r9r?rstudent_logitsrinput_kdteacher_logitsprob_sprob_tr total_losss r=rSzLogitDistillationTask.forwardps$..u5NN>6: ]]_ T||33E4;L;LdN^N^_H!\\(//*;U\SN T~0@0@@bI~0@0@@bI((66[TRVZVfVfjkVkl**Y69Q9QT[9[[ $"    T Ts AD**D3) NNruNNrvNNTNrZN)r/r\r]r^r+r,rr'r rfloatr4rrrar%listrrOr`rrSrbrcs@r=rjrjsh%V.259!3704!$-1+/ Q99Q!bii1D!DEQ  * Q &.c] Q  Q"*%Q'uoQQU\\*QEKK(QQj*.  ! 6&"&%%3-%  %&$ <<$ LL$  c5<< $ r>rjceZdZdZ d dej deej ffd ZdZd de fdZ de jd e e je jffd ZxZS) "FeatureDistillationTrainableModuleaTrainable module for feature distillation. Wraps student model and projection layer into a single module where all trainable forward operations happen inside forward(). This ensures proper DDP wrapping when the module is used with DistributedDataParallel. re projectionc>t|||_||_y)z Create trainable module wrapper for feature distillation. Args: student_model: Student model to train projection: Optional projection layer (Linear layer or None) N)r$r%studentr)r9rerr<s r=r%z+FeatureDistillationTrainableModule.__init__s $ $r>ct|j}t|ds tS|j Dchc]}d|z c}Scc}w)Nno_weight_decaystudent.)r rr2setr)r9rnames r=rz2FeatureDistillationTrainableModule.no_weight_decaysDt||,w 125L.5.E.E.GHd T!HHHs A coarsect|j}t|dsiSt||j |dt j dfd}|S)N group_matcher)rT)reverser)defaultcf|jdrj|tddSS)Nr) startswithr3len)rstudent_layer_map task_layers r=_matcherzBFeatureDistillationTrainableModule.group_matcher.._matchers4z*(,,T#j/2B-CZPP r>)r rr2rrmaxvalues)r9rrrrrs @@r=rz0FeatureDistillationTrainableModule.group_matchersat||,w0I,Wg6K6KSY6K6Zdhi*113Q?   r>r?r@c|jj|}|jj|}|jj|d}|j|j|}||fS)a Forward pass through student and projection. Args: input: Input tensor [B, C, H, W] Returns: Tuple of (student_logits, student_features) where features are optionally projected to match teacher dimension. TrD)rrBrCr)r9r?rGrstudent_featuress r=rSz*FeatureDistillationTrainableModule.forwardslll33E: 22;?<<44[T4R ?? &#/?@ ///r>rr[)r/r\r]r^r+r,rr%rrarr4r`rrSrbrcs@r=rrsl/3 %99 %!+ %I D 0U\\0eELL%,,4N.O0r>rceZdZdZ d#dej deeej efde ej de ede e de e d e e d e e d e e jd e e jd effd Zedej de fdZ d$de eddfdZ d%dede edej fdZd&de ej dede ej ffd Z d&de ej dedeeeee j0fffdZd'defdZ d(de eeeee j0ffdede ej deddf dZd e j0d!e j0deee j0ffd"ZxZS))FeatureDistillationTaskaFeature-based knowledge distillation task. Performs distillation by matching student and teacher intermediate features (pooled pre-logits) using MSE loss. Automatically creates a projection layer if student and teacher feature dimensions differ. Loss weighting supports two modes: 1. Independent weights: loss = task_loss_weight * task_loss + distill_loss_weight * distill_loss 2. Complementary mode: loss = task_loss_weight * task_loss + (1 - task_loss_weight) * distill_loss (used when only task_loss_weight is specified) Args: student_model: Student model to train teacher_model: Teacher model - can be a model name string, nn.Module, or DistillationTeacher criterion: Task loss function (default: CrossEntropyLoss) teacher_pretrained_path: Path to teacher pretrained weights (used when teacher_model is a string) distill_loss_weight: Weight for distillation loss task_loss_weight: Weight for task loss student_feature_dim: Student pre-logits dimension (auto-detected if None) teacher_feature_dim: Teacher pre-logits dimension (auto-detected if None) device: Device for task tensors/buffers dtype: Dtype for task tensors/buffers verbose: Enable info logging Example: >>> # With model name string (num_classes/in_chans inferred from student) >>> task = FeatureDistillationTask( ... student_model=model, teacher_model='resnet50', ... criterion=nn.CrossEntropyLoss(), ... distill_loss_weight=5.0, task_loss_weight=1.0, ... device=torch.device('cuda'), ... ) Nrerkrlrmrorpstudent_feature_dimteacher_feature_dimrrrrc t|| | | t||||j|j} | |_||nt j|_|D||_ ||nd|_ |jrtjd|jd|n|C||_ d|z |_ |jrotjd|d|jnId|_ d|_ |jr/tjd|jd|j||j|}||j| j}d} ||k7rU|jrtjd|d|t j |||j|j } n!|jrtjd t#|| |_t'|}t)j*|j,d |j|j j/d d d d }t)j*|j,d|j|j j/d d d d }|j1d|d|j1d|d|jrtjd|d|yy)Nrtrvz;FeatureDistillationTask: Independent weights - task_weight=rwz:FeatureDistillationTask: Complementary mode - task_weight=z=FeatureDistillationTask: Default equal weights - task_weight=zCreating projection layer: z -> rz.Feature dimensions match, no projection neededrr rrrTFr rUz%FeatureDistillationTask: student_dim=z, teacher_dim=)r$r%rhrrrdr+ryrlrorprrr(r)_detect_feature_dimr1Linearrrxr r4r5rr6r7)r9rerkrlrmrorprrrrrrrdrrgrTrUr<s r=r%z FeatureDistillationTask.__init__s eWE#   # KK JJ   &/&;ATATAV  *':D $8H8T$4Z]D !|| ##'#8#8"99JK^J_a )$4D !'*-='=D $|| ##3"44EdF^F^E_a (+D $$'D !|| ##'#8#8"99J4KcKcJdf  &"&":":="I   &"&":":7=="I  "5 5|| 12E1FdK^J_`#68KTXT_T_gkgqgqrJ|| MN B=R\ ])7||  , ,V 4;;**  $q"a   ll  , ,U 3;;**  $q"a   ^\eL ]KEJ << LL23>BUAVX  r>r1r@ct|}t|dr |jSt|dr |jSt d)z)Auto-detect feature dimension from model.head_hidden_size num_featureszCannot auto-detect feature dimension. Model must have 'head_hidden_size' or 'num_features' attribute, or you must specify student_feature_dim and teacher_feature_dim explicitly.)r r2rrrF)r1s r=rz+FeatureDistillationTask._detect_feature_dim\sNU# 5, -)) ) UN +%% %R r>rzc ddlm}|jjD] }d|_ ||j fd|i||_|S)aPrepare task for distributed training. Wraps the trainable module (student + projection) in DistributedDataParallel (DDP) while leaving the frozen teacher model unwrapped. Args: device_ids: List of device IDs for DDP (e.g., [local_rank]) **ddp_kwargs: Additional arguments passed to DistributedDataParallel Returns: self (for method chaining) rr|Frzr~rs r=rz+FeatureDistillationTask.prepare_distributedlrr>rIrJc tj|jjf||d|}||_tj|jf||d||_|j jd||ddd||S)z9Compile feature-distillation train and eval entry points.rNFTrr#)r4rOrxrrrd)r9rIrJrPrs r=rOzFeatureDistillationTask.compiles]]4#8#8#@#@o'X\o`no $ %  ! !! !  !    !    r>moduleemact|||}|}|:t|tr |jSt |ds |S|j }|:|S)N)rrr)r$get_eval_modelr&rrr2r)r9rrresolvedcurrentr<s r=rz&FeatureDistillationTask.get_eval_modelsc7)S)A!'#EF&7H-nnG ! r>c||n|j|}|iSt|}|jiSd|jjiS)Nrr)get_trainable_moduler r state_dict)r9rr trainables r=get_task_statez&FeatureDistillationTask.get_task_states^ "-43L3LQT3L3U >I (    'Ii22==?@@r> exclude_headct|j}t|jj }|r|dd}|j )|j |j j |S)N)r rxrrrrextend)r9rrrs r=get_clip_parametersz+FeatureDistillationTask.get_clip_parameterssh !6!67 )++6689 #CRJ    +   i22==? @r>statestrictc|sy||n|j|}||r tdtdt|}|jd}|y|j|r tdy|jj ||y)Nrz.Cannot load EMA task state before setup_ema().z2Cannot load task state without a trainable module.rz@Checkpoint has projection task state but task has no projection.)r)r RuntimeErrorr r3rload_state_dict)r9rrrrrprojection_states r=load_task_statez'FeatureDistillationTask.load_task_states !-43L3LQT3L3U >"#STTST T (  99\2  #     '"#eff ,,-=f,Mr>r?rc|j|\}}|j||}tj5|jj ||j |j}|j |jd}dddtj|}|j|z|j|zz} | |||dS#1swYExYw)aForward pass with feature distillation. Args: input: Input tensor [B, C, H, W] target: Target labels [B] Returns: Dictionary containing: - 'loss': Combined training loss (task + distillation) - 'output': Student logits (for metrics) - 'task_loss': Classification loss component - 'kd_loss': Feature distillation loss component TrNr) rxrlr4rrdrXrTrUrrmse_lossrpro) r9r?rrrrrteacher_featuresrrs r=rSzFeatureDistillationTask.forwards$,0+@+@+G((NN>6: ]]_ U||33E4;L;LdN^N^_H#||HOO,=t|T  U**-/?@**Y69Q9QT[9[[ $"    U Us ACC) NNNNNNNNTrr)NFr[)TNF)r/r\r]r^r+r,rr'r rrr_r4rrrar% staticmethodrrrrOrrr`rrrrSrbrcs@r=rrs L.25937041515-1+/ `99`!bii1D!DE`  * ` &.c] ` "*% `'uo`"*#`"*#`U\\*`EKK(``D 299   "*.  # 6&"&3-  0 Xbii%8 d W_`b`i`iWj +/ ARYY' A A c4U\\)** + A *. NDd3 +<&=!=>?NNRYY' N  N  N0! <<! LL!  c5<< ! r>r) r^loggingtypingrrrrr4torch.nnr+torch.nn.functional functionalr timm.modelsrr timm.utilsr taskr getLoggerr/r(r,r r'rrrhrjrrr#r>r=rs;// 6# '  H %OQ"))OQd"sBII'::;"yy""#"& "  $ "  "JK LK \=0=0@` l` r>