^j ddlZddlmZddlmZddlZddlmZddlmcm Z ddl m Z ddlmZddlmZmZddlmZdd lmZmZmZmZdd lmZdd lmZmZmZdd l m!Z!dd l"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5m6Z6m7Z7m8Z8e-deGdde#Z9e-deGdde+Z:edGddejvZ<Gdd ejvZ=d!ej|d"e?d#ej|fd$Z@ dbd%ejvd&ej|d'ej|d(ej|d)ej|dzd*eAd+eAd,ej|dzfd-ZBGd.d/ejvZCGd0d1ejvZDeGd2d3ejvZEGd4d5ejvZFGd6d7ejvZGGd8d9ejvZHd:ZIed; dcd!ej|dejdzd?eKdzf d@ZLedA dcd!ej|d=ejd>ejdzd?eKdzfdBZMeeLeMgGdCdDejvZNGdEdFe!ZOe-GdGdHe'ZPe-GdIdJePZQe-GdKdLePeZRGdMdNejvZSGdOdPePZTGdQdRejvZUdSe?d#eVe?fdTZW dddUe?dVe?dWe?dXe?d#ejf dYZYGdZd[ePZZe-d\Gd]d^ePZ[e-d\Gd_d`ePeZ\gdaZ]y)eN)Callable) dataclass)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_experts_implementationuse_kernel_forward_from_hubuse_kernel_func_from_hubuse_kernelized_func)force_accelerate_hooks)create_causal_maskcreate_recurrent_attention_mask!create_sliding_window_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupletorch_compilable_check)merge_with_config_defaults)capture_outputs)InklingAudioConfig InklingConfigInklingTextConfigInklingVisionConfigzL Base class for Inkling outputs, with hidden states and attentions.  custom_introc:eZdZUdZdZej dzed<y)InklingModelOutputWithPasta image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__r)torch FloatTensor__annotations__w/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/inkling/modeling_inkling.pyr(r(4s 59**T18r2r(zS Base class for Inkling causal language model (or autoregressive) outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) InklingCausalLMOutputWithPasta8 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr))r*r+r,r-r6r.r/r0r7r8rr9tupler:r)r1r2r3r5r5Ds &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r2r5RMSNormcheZdZddeddffd Zdej dej fdZdZxZ S) InklingRMSNormepsreturnNct|tjt j ||_||_y)z= InklingRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parameterr.onesweightvariance_epsilon)self hidden_sizer? __class__s r3rCzInklingRMSNorm.__init__ds1 ll5::k#:; #r2r9c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NT)keepdim) dtypetor.float32powmeanrsqrtrHrG)rIr9 input_dtypevariances r3forwardzInklingRMSNorm.forwardlsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r2c^t|jjd|jS)Nz, eps=)r;rGshaperH)rIs r3 extra_reprzInklingRMSNorm.extra_reprss* ))*+6$2G2G1HIIr2)ư>) r*r+r,floatrCr.TensorrXr[ __classcell__rKs@r3r>r>bs7$$$$;U\\;ell;Jr2r>ceZdZdZdedeffd Zdejdejdejdejfd ZxZ S) InklingRelativeLogitsa|hidden states conditioned relative position bias. `proj` is a trained bank of bias-vs-distance profiles; each token's `relative_states` mixes them into one bias value per backward distance (`sglang RelLogitsProj` + the FA4 `score_mod`, materialized densely). The bias is zero outside `0 <= distance < rel_extent`; causality and padding stay in the attention mask. d_rel rel_extentct|||_tjt j |||_yN)rBrCrdrDrEr.emptyproj)rIrcrdrKs r3rCzInklingRelativeLogits.__init__~s0 $LLUJ!?@ r2relative_statesquery_positions key_positionsr@cj||jzjdd}|dddf|dddfz ddddddf}|jd|jdz jg|j dddd}|j d|}|j|dk||jk\zdS)Nr rMrrN)rh transposeclamprdexpandrZgather masked_fill)rIrirjrk rel_logitsdistance gather_index position_biass r3rXzInklingRelativeLogits.forwards& 1<