^jY:fddlmZddlZddlmZmZddlmZddlm Z m Z ddl m Z e rddl mZe jeZeZGdd eZeGd d Zdd ZddZ ddZddZ ddZ ddZ ddZGdded ZddZGddZ y) ) annotationsN)IterableSequence) dataclass) TYPE_CHECKINGAny)logging)PreTrainedConfigceZdZdZy)%AmbiguousGlobalPerLayerAttributeErrorzMRaised when a per-layer attribute is read from a heterogeneous global config.N)__name__ __module__ __qualname____doc__/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/integrations/heterogeneity/configuration_utils.pyr r #sWrr c,eZdZUded<ded<ded<y)_HeterogeneitySpecdict[int, dict[str, Any]]per_layer_overridesset[str]per_layer_attributesexplicit_per_layer_attributesN)r rr__annotations__rrrrr's22""#++rrc tj|}d|vrt|jd}t|tst|t s t dt|}td|Ds t d|rt||d<|S)Nskipz&`skip` must be an iterable of strings.c3<K|]}t|tywN) isinstancestr).0items r z-_normalize_layer_overrides..7s:T:dC(:sz!`skip` must contain only strings.) copydeepcopypopr r!r TypeErrorsetallsorted)layer_overrides normalizedrs r_normalize_layer_overridesr..s/J ~~f% dC 4(BDE E4y:T::?@ @ !'Jv  rr c|sy|j}|Dcgc]}|dks||k\s|}}|rtd|d|dycc}w)NrzG`per_layer_config` keys must be integer layer indices in the range [0, z); got .)num_hidden_layers ValueError)configrr1 layer_idxinvalid_layer_indicess r_validate_layer_indicesr6@su 00#6)a-9XiKi UVgUhi() ,  s??c8g}t|jD]o}|j|i}|jd|jdd}|jd|jdd}|\|_|j |q|rt d|dy)Nsliding_windowattention_chunk_sizezkThe following layers have the mutually exclusive `sliding_window` and `attention_chunk_size` both defined: z. To fix this, either remove a conflicting attribute from the global config,or set it to `None` in `per_layer_config` for the problematic layers.)ranger1get)_getattr_without_heterogeneous_validationappendr2)r3rproblematic_indicesr4r,r8r9s r1_validate_sliding_window_and_attention_chunk_sizer?Os6334 2 -11)R@(,, fNNO_aef  /22 "  < <=SUY Z   %*>*J  & &y 1 2y"#$T U  rct}|jD]}|j||jd|S)Nr)r)valuesupdatediscard)rrr,s r_get_per_layer_attributesrDisF%(U.5575##O45  ( rct|}t}|D]{}t||jk7r$|j |r-|j |?|j D]*}||vs|j |s|j |{}|rtdt|d|D]K}|j |s|j|}|j D]}||vs|||k(s||=Mt|jD] \}}|r ||= t|}t|||} | S)Nz&The following attributes are missing: zS Please define them globally, or provide them for every layer in `per_layer_config`)rrr) rDr)lenr1)_hasattr_without_heterogeneous_validationaddrAr2r+r<listitemsr) r3rr"missing_required_global_attributesattrr, global_valuer4rheterogeneity_specs r,_modify_config_and_create_heterogeneity_specrOrs%>>Q$R!*-&-  " #v'?'? ?CCDI266t<#6#=#=#? .!KKDQ:>>tD   *4V<^5_4`au v  .*??E GGM 299; *O&?4+@L+P#D) * *'++>+D+D+F&G/" ?#I./55HI+/1&C rc|jDcic]\}}t|t|}}}t||t ||t |||_ycc}}w)aRegister per-layer configuration overrides on a model config. In a heterogeneous model, individual layers can differ from the global config (e.g., different ``intermediate_size``, ``num_key_value_heads``, or entire sub-layers skipped via the ``skip`` attribute). This function validates the overrides and stores a ``_HeterogeneitySpec`` on ``config._heterogeneity_spec``. At model-init time, ``apply_heterogeneous_modeling`` reads this spec to patch each layer with its resolved config. Args: config: The global model config to modify in-place. per_layer_config: Mapping from layer index to a dictionary of attribute overrides. Only layers that differ from the global config need to be included. N)rJintr.r6r?rO_heterogeneity_spec)r3per_layer_configr4r,normalized_per_layer_overridess r_apply_heterogeneous_configrUsk.+;*@*@*B& &I I2?CC&"& F$BC5f>\]!MfVt!uF&sA ctj|}|jjdd|jdg|_|j D]\}}|dk(r t ||||S)NrRr)r%__dict__r'r;rrJsetattr)r3r, output_configrLvalues r_get_layer_configr[sxIIf%M4d;(,,VR8M&,,., e 6>  tU+, rc$eZdZddZddZddZy)_PerLayerConfigViewc||_yr)_config)selfr3s r__init__z_PerLayerConfigView.__init__s  rc.|jjSr)r_r1r`s r__len__z_PerLayerConfigView.__len__s||---rc|t|tr1t|jt |Dcgc]}|| c}S|dkr|t |z }|dks|t |k\r t d|j j}t|j |jj|iScc}w)Nrzlist index out of range) r slicer:indicesrF IndexErrorr_rRr[rr;)r`r4irNs r __getitem__z_PerLayerConfigView.__getitem__s i '%*I,=,=c$i,H%IJDGJ J q= T "I q=IT267 7!\\== LL  2 2 6 6y" E  Ks B9N)r3r returnNone)rkrQ)r4z int | slicerkz)PreTrainedConfig | list[PreTrainedConfig])r rrrardrjrrrr]r]s.  rr]c|j}i}t|jD]c}tj|j j |i}|jD]}||vs|j|||<|s_|||<e|Sr) rRr:r1r%r&rr;rr<)r3rNexplicit_per_layer_overridesr4r,rLs r!_get_explicit_per_layer_overridesros33#% 6334F --(:(N(N(R(RS\^`(ab&DD _D?*(.(X(XY](^% _ 6E ( 3F ('rceZdZdZdfd ZeddZeddZejddZeddZ eddZ e jddZ edd Z e jdd Z dd Z dd Z efdfd ZddZxZS)HeterogeneousConfigMixina_Mixin for heterogeneous per-layer config behavior. This mixin owns heterogeneity-specific state and rules. ``PreTrainedConfig`` assigns the ``per_layer_config`` property in the post-init phase and calls hook methods where heterogeneity needs to participate in the config lifecycle: attribute access, key iteration, and serialization. c t|djd}|K||jvr=t|dst d|d|d|dt j d|d t||S) NrWrR'allow_global_per_layer_attribute_access'z' is a per-layer attribute and may vary across layers. Access it via the individual layer configs instead (e.g. config.per_layer_config[i].z/). To read the global config value from config.z anyway, set `allow_global_per_layer_attribute_access` to `True` on the config. Warning: only do this if the caller can safely handle heterogeneous configs; code that assumes a homogeneous model may use the global value incorrectly.z5Reading global config value for per-layer attribute `z` on a heterogeneous config. Only do this if the caller can safely handle heterogeneous configs; code that assumes a homogeneous model may use the global value incorrectly.)super__getattribute__r;rr logger warning_once)r`keyrN __class__s rrvz)HeterogeneousConfigMixin.__getattribute__s#W5jAEEF[\  )(===w/0YZ?C5!LLO5Q""%'TU##KC5QBB w',,rct|dSNrR)hasattrrcs ris_heterogeneousz)HeterogeneousConfigMixin.is_heterogeneousst233rc2|jsyt|Sr)r~r]rcs rrSz)HeterogeneousConfigMixin.per_layer_configs$$"4((rcZ||jjddyt||yr|)rWr'rU)r`rSs rrSz)HeterogeneousConfigMixin.per_layer_config$s+  # MM  3T : #D*:;rcH|jsy|jjSr)r~rRrrcs rrz-HeterogeneousConfigMixin.per_layer_attributes,s $$''<<s?D ;rs # .!%&-   H % H XLX ,, , $    3L   4 - -3L--`v v5v v@ #" (#56 .("vvr