^jddlZddlmZddlmZddlmZddlZddlm Z ddl m cm Z ddl mZddlmZddlmZdd lmZmZdd lmZdd lmZmZdd lmZdd lmZddl m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2m3Z3ddl4m5Z5m6Z6ddl7m8Z8ddl9m:Z:m;Z;e2jxe=Z>Gdde j~Z@Gdde j~ZAGddejj~ZBGdde j~ZCGdd e jZEGd!d"e j~ZFGd#d$e j~ZGd%ejd&ejd'ejd(eIejejffd)ZJd*ejd+eKd(ejfd,ZL d_d-e j~d.ejd/ejd0ejd1ejdzd2eMd3eMfd4ZN d_d-e j~d.ejd/ejd0ejd1ejdzd2eMd3eMfd5ZOGd6d7e j~ZPGd8d9eZQe0Gd:d;e+ZRe0Gd<d=eRZSGd>d?eReZTe0d@AeGdBdCe%ZUGdDdEejj~ZVGdFdGe j~ZWdHZXGdIdJe j~ZYdKejd.ejfdLZZd.ejd/ejdKejd(eIejejffdMZ[GdNdOe j~Z\GdPdQe j~Z]GdRdSeZ^GdTdUe j~Z_GdVdWe j~Z`GdXdYe j~ZaGdZd[eRZbGd\d]eReZcgd^Zdy)`N)Callable) dataclass)Optional)Llama4VisionConfig)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_maskcreate_chunked_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)maybe_autocastmerge_with_config_defaults)capture_outputs) Llama4ConfigLlama4TextConfigc\eZdZdeffd Zdej dej fdZxZS)Llama4TextExpertsconfigct||j|_|j|_|j |_|j|_tjtj|j|j d|j z|_ tjtj|j|j |j f|_ t|j|_yN)super__init__num_local_experts num_expertsintermediate_size hidden_size expert_dimnn Parametertorchzeros gate_up_projempty down_projr hidden_actact_fnselfr( __class__s u/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/llama4/modeling_llama4.pyr-zLlama4TextExperts.__init__8s !33!'!9!9!--00LLT5E5EtGWGWYZ]a]l]lYl)mnekk43C3CT__VZVfVf2g&hiV../ hidden_statesreturncv|j|jjdd|j}t j ||j}|j dd\}}t j ||j|z|j}|jd|j}|S)a2 This should really not be run on a single machine, as we are reaching compute bound: - the inputs are expected to be "sorted" per expert already. - the weights are viewed with another dim, to match num_expert, 1, shape * num_tokens, shape Args: hidden_states (torch.Tensor): (batch_size * token_num, hidden_size) selected_experts (torch.Tensor): (batch_size * token_num, top_k) routing_weights (torch.Tensor): (batch_size * token_num, top_k) Returns: torch.Tensor rr+dim) viewr7shaper1r5bmmchunkr;r9)r=rAgate_upgateup next_statess r?forwardzLlama4TextExperts.forwardBs&**4+<+<+B+B1+Er4K[K[\ ))M4+<+<====+biidkk$&7!7$..I !&&r4+;+;< r@) __name__ __module__ __qualname__r%r-r5TensorrO __classcell__r>s@r?r'r'7s+0/0U\\ellr@r'c&eZdZdfd ZdZxZS) Llama4TextMLPcft|| |j}||_t j |j |d|_t j |j |d|_t j ||j d|_ t|j|_ yNFbias) r,r-r0r(r3Linearr1 gate_projup_projr9r r: activation_fn)r=r(r0r>s r?r-zLlama4TextMLP.__init__Ys   $ & 8 8  6#5#57HuUyy!3!35FUS #4f6H6HuU#F$5$56r@c|j|j||j|z}|j|SN)r_r]r^r9)r=xr9s r?rOzLlama4TextMLP.forwardes7&&t~~a'89DLLOK ~~i((r@rarPrQrRr-rOrTrUs@r?rWrWXs  7)r@rWc8eZdZddeffd ZdZdZdZxZS)Llama4TextL2Normepsc0t|||_yra)r,r-rf)r=rfr>s r?r-zLlama4TextL2Norm.__init__ks r@c|tj|jdjdd|jzzSNr+rDT)keepdimr5rsqrtpowmeanrfr=rbs r?_normzLlama4TextL2Norm._normo45;;quuQx}}R}>IJJJr@c^|j|jj|Sra)rpfloattype_asros r?rOzLlama4TextL2Norm.forwardrs"zz!'')$,,Q//r@c d|jS)Nzeps=rfr=s r? extra_reprzLlama4TextL2Norm.extra_reprusdhhZ  r@)gư>) rPrQrRrsr-rprOrxrTrUs@r?rerejsEK0!r@rec2eZdZdfd ZdZdZdZxZS)Llama4TextRMSNormct|||_tjt j ||_y)z< Llama4RMSNorm is equivalent to T5LayerNorm N)r,r-rfr3r4r5onesweight)r=r1rfr>s r?r-zLlama4TextRMSNorm.__init__zs0 ll5::k#:; r@c|tj|jdjdd|jzzSrirkros r?rpzLlama4TextRMSNorm._normrqr@c||j|jj|}||jzSra)rprsrtr})r=rboutputs r?rOzLlama4TextRMSNorm.forwards0AGGI&..q1 ##r@c^t|jjd|jS)Nz, eps=)tupler}rHrfrws r?rxzLlama4TextRMSNorm.extra_reprs' ))*+6$((<)rPrQrRr-rprOrxrTrUs@r?rzrzys<K$=r@rzc(eZdZfdZfdZxZS) Llama4Routerct||j|jd|j|_|j |_yrY)r,r-r1r.r/num_experts_per_toktop_kr<s r?r-zLlama4Router.__init__s> ++V-E-EER!33// r@ctt||}tj||jd\}}tj |t djd||}tjjj|j j|j}||fS)Nr#rEz-inf) r,rOr5topkr full_likersscatter_r3 functionalsigmoidtodtype)r=rA router_logitsrouter_top_valuerouter_indices router_scoresr>s r?rOzLlama4Router.forwards 6 +0::mTZZUV+W(. uV}ENNqR`brs ++33M4G4G4IJMMmNaNab m++r@rcrUs@r?rrs0 ,,r@rc$eZdZfdZdZxZS) Llama4TextMoect||j|_|j|_|j |_t||_ t||_ t||_ yra)r,r-rrr1 hidden_dimr.r/r'expertsrrouterrW shared_expertr<s r?r-zLlama4TextMoe.__init__s[ //  ,,!33(0 "6* *62r@c|jd|j}|j|\}}|j|jdd}||j ddjddz}|j |}|j|}|j|j|jdd|jdjd||fS)NrDr#rrE) reshaperrrepeatrH transposerradd_sum)r=rArr routed_in routed_outouts r?rOzLlama4TextMoe.forwards%--b$//B '+{{='A$ }!(()<)s r?r-z"Llama4TextRotaryEmbedding.__init__s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L($( ZeD 0(..2BuUr@rz torch.deviceseq_lenrBz torch.Tensorc|jd}t|ddxs|j|jz}d}d|t j d|dtj j|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimN?rr+r)rr) rgetattrr1num_attention_headsr5arangeint64rrs)r(rrbaserFattention_factorrs r?rz9Llama4TextRotaryEmbedding.compute_default_rope_parameterss&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k )))r@cv|jddddfjj|jddd}|dddddfj}t |j j tr/|j j dk7r|j j nd}t|d5|j|j |zjdd}tjtj||}||jz}ddd|S#1swYSxYw) NrrDr#mpscpuF) device_typeenabledr+)rrsexpandrH isinstancertypestrr rrr5polar ones_liker)r=rb position_idsinv_freq_expandedposition_ids_expandedrfreqs freqs_ciss r?rOz!Llama4TextRotaryEmbedding.forwards!MM$4-8>>@GG HZHZ[\H]_acde ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk  U C ;&))!((36KKVVWXZ[\E EOOE$:EBI!D$:$::I ;   ; s =A'D..D8ra)NNN)rPrQrRr5rS__annotations__r%r- staticmethodrintrrsrno_gradrrOrTrUs@r?rrsllV/V *.+/"* 4'*(*t* ~u$ % **<U]]_  r@rxqxkrrBc &tj|jjg|jdddd}tj|jjg|jdddd}tj ||dddddddfzj d}tj ||dddddddfzj d}|j||j|fS)NrDr+r)r5view_as_complexrsrrH view_as_realflattenrt)rrrxq_xk_xq_outxk_outs r?apply_rotary_embrs    2 2 2 IBHHSbM I2 Iq I JC    2 2 2 IBHHSbM I2 Iq I JC   i1dA &> > ? G G JF   i1dA &> > ? G G JF >>" v~~b1 11r@rAn_repc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r#N)rHrr)rArbatchnum_key_value_headsslenrs r? repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr@modulequerykeyvalueattention_maskscalingdropoutc t||j}t||j} tj||j dd|z} || |z} t j j| d} t j j| ||j} tj| | } | j ddj} | | fS)Nr+rrDrEptrainingr#) rnum_key_value_groupsr5matmulrr3rsoftmaxrr contiguous rrrrrrrkwargs key_states value_states attn_weights attn_outputs r?eager_attention_forwardr s3 ; ;L==((6??([L,,|\:K''1-88:K  $$r@c t||j}t||j} tj||j dd|j dzz} || |z} t jj| d} t jj| ||j} tj| | } | j ddj} | | fS)Nr+rrDrErr#) rrr5rrrr3rrrrrrs r?vision_eager_attention_forwardr's3 ; ;L==((6??([L,,|\:K''1-88:K  $$r@ceZdZdZdeffd Z d dejdeejejfdejdzde dzd e e d eejejdzeejdzff d Z xZ S) Llama4TextAttentionz=Multi-headed attention from 'Attention Is All You Need' paperr(ct|||_||_t |d|j |j z|_|j |_|j |jz|_ |j|_|jdz|_ |j|_ |j|_ |j|_ |j|_d|_|j ||_t%j&|j |j |jz|j(|_t%j&|j |j|jz|j(|_t%j&|j |j|jz|j(|_t%j&|j |jz|j |j(|_|jj2r(|j"rt5|j6|_yyy)NrrTrZ)r,r-r( layer_idxrr1rrrrr attn_scale floor_scaleattn_temperature_tuningattention_dropout is_causalno_rope_layersuse_roper3r\attention_biasq_projk_projv_projo_proj use_qk_normre rms_norm_epsqk_normr=r(rr>s r?r-zLlama4TextAttention.__init__Cs  " F4F4F&JdJd4de #)#=#= $*$>$>&B\B\$\!#)#=#= }}d*  ++!--'-'E'E$!'!9!9--i8 ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii    : :T]] JQWQfQf  ii  & & 68J8JQWQfQf   ;; " "t}}+F,?,?@DL(5 "r@NrAposition_embeddingsrpast_key_valuesrrBc |jdd}g|d|j}|j|j|}|j |jg|d|j} |j |j|j dd} |jr)t|| |j|j\}} t|dr"|j|}|j| } |jr|js||j|jnd} t!j"|jd|j| z} t!j$t!j&| j)dz|j*z |j,zdz} | jd|dddfj/g|dd} || zj|j0}|j dd}| j dd} | |j3| | |j\} } t5j6|j8j:t<}|||| | |f|j>sdn |j@|jBd |\}}|jDg|djG}|jI|}||fS) NrDr#r+rrrr)rr)%rHrr rGrrrr rrrhasattrrrget_seq_lengthrr5rlog1pfloorrsrrrrupdater get_interfacer(_attn_implementationrrrrrrr)r=rArrrr input_shape hidden_shape query_statesrrpast_seen_tokens positions attn_scalesattention_interfacerrs r?rOzLlama4TextAttention.forwardas$))#2. 88b8$--8 {{=166|D 4T[[/44UkU2Ut}}U {{=166|DNNqRST =='7j*=*@*@ATAT*U( $L* 4 #<< 5Lj1J  ' ' Q`Ql==dnnMrs  ]%8%8%;MDXDXY\llI EKK):S)@DDTDT(TUVY]YhYhhknn &**A{21+EFMMNbP[Nb]^Nb`aNbcK(;6::<;M;MNL#--a3 ))!Q/  &'6'='=j,X\XfXf'g $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r@ra)rPrQrR__doc__r%r-r5rSrr rrrOrTrUs@r?rr@sGA/AF)- 8)||8)#5<<#=>8) t+ 8)  8) -. 8) u||U\\D0% 2E2LL M8)r@rc:eZdZfdZ d dej dej dzdej dzdedzdedzde ej ej fdzd e e d e eje ejejfdzffd Z xZS) Llama4TextDecoderLayerct||j|_||_t |||_||j v|_|jrt||_ nt||j|_ t|j|j|_t|j|j|_y)N)r0rv)r,r-r1rr self_attn moe_layers is_moe_layerr feed_forwardrWintermediate_size_mlprzrinput_layernormpost_attention_layernormrs r?r-zLlama4TextDecoderLayer.__init__s !--",VY?%):)::    -f 5D  -fHdHd eD 01C1CI\I\](9&:L:LRXReRe(f%r@NrArrr use_cacherrrBc |}|j|}|jd|||||d|\} } || z}|}|j|}|j|}|jr|\}} ||j |j z}|S)N)rArrrr3)r1r,r2r/r.rGrH) r=rArrrr3rrresidualattention_states_s r?rOzLlama4TextDecoderLayer.forwards!,,]; -dnn ' 3)+    !!#33 !55mD ))-8   , M1 =#5#5hnn#EE r@)NNNFN)rPrQrRr-r5rS LongTensorr boolrrr FloatTensorrOrTrUs@r?r*r*s g"/304(,!&HL ||  t+ &&-    $; #5<<#=>E -.  u  %(9(95;L;L(L"MPT"TT U r@r*cteZdZUeed<dZdZdgZdZdZ dZ dZ dZ e jfdZxZS)Llama4PreTrainedModelr()imagetextTrFct||t|jdr|jjn|jj j}t |trEtj|jd|tj|jd|yt |tr:tj|j|j|jyt |t rWtj|j"|j$tj|j&|j$yy)Ninitializer_ranger)rnstd)rB)r, _init_weightsrr(rA text_configrr'initnormal_r7r9Llama4VisionRotaryEmbeddingcopy_freqs_ci_compute_freqs_ciLlama4VisionModelclass_embeddingscalepositional_embedding_vlm)r=rrBr>s r?rCz#Llama4PreTrainedModel._init_weightss f%t{{$78 KK ) )((:: f/ 0 LL,,3C @ LL)) =  ; < JJv(@(@(O P  1 2 LL//V\\ B LL88fll K3r@)rPrQrRr$rinput_modalitiessupports_gradient_checkpointing_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendr5rrCrTrUs@r?r=r=sW (&*##4"5 N!"&U]]_LLr@r=c(eZdZUdgZdZdZeed<ee e dZ deffd Z e eee ddej$dzd ej&dzd ej$dzd edzd ej*dzd edzdeedeezfdZxZS)Llama4TextModelr*model)r?r() attentionsrArc  t|||j|_|j|_t j |j|j|j|_t jt|jDcgc]}t||c}|_ t|j|j|_t#||_d|_|j)ycc}w)Nrvr(F)r,r- pad_token_id padding_idx vocab_sizer3 Embeddingr1 embed_tokens ModuleListrangenum_hidden_layersr*layersrzrnormr rotary_embgradient_checkpointing post_initrs r?r-zLlama4TextModel.__init__s  !.. ++LL):):FQ>QR 36B&+#  isDN input_idsrrr inputs_embedsr3rrBc 4|du|duz r td|>|j|j|jjj}|r|t |j }|V||jnd}tj|jd|j|z}|jd}t|x} ts)|j ||||d} td i| td i| d} |} |j!| |} t#|j$d|j j&D].\} }|| f| |j j(| |||| d|} 0|j+| } t-| |r| Sd S) N:You must specify exactly one of input_ids or inputs_embedsr\rr#r)r(rkrrr)full_attentionchunked_attention)rrrr3r)last_hidden_staterr5) ValueErrorrarr}rr r(rr5rrH unsqueezerdictr rrg enumeratererd layer_typesrfr)r=rjrrrrkr3rr$causal_mask_mapping mask_kwargsrAfreq_cisi decoder_layers r?rOzLlama4TextModel.forward s  -t"; <YZ Z   --ill4;L;L;S;S;Z;Z.[\M 0*$++>O  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L?-F++!."0#2 , K#5"C{"C%?%N+%N#  & ??=,? )$++6U 8U8U*V W  A})24;;3J3J13MN) /#$, M  -0 &+/8O  >B  r@)NNNNNN)rPrQrR_no_split_modulesbase_model_prefixrOr%rrr*r_can_record_outputsr-rr!r"rr5r9rSr r;r:rrrrrOrTrUs@r?rXrXs12  )/& / .2.204(,26!%< ##d*<  t+< &&- <  < ((4/ < $;< +,<  ( (<  < r@rXcTeZdZUdgZdZddiZddiZeed<deffd Z e e dd e jd zd e jd zd e jd zd ed zde j d zde jd zded zdee jzdeedeezfdZxZS)Llama4ForCausalLMr*language_modelzlm_head.weightzmodel.embed_tokens.weightlm_headcolwise_gather_outputr(ct||t||_|j|_t j |j|jd|_|jyrY) r,r-rXrYr_r3r\r1rrir<s r?r-zLlama4ForCausalLM.__init__SsU  $V,  ++yy!3!3V5F5FUS  r@Nrjrrrrklabelsr3logits_to_keeprrBc j|jd||||||d| } | d} t|tr t| dn|} |j | dd| ddf} d}|*|j d| ||j jd| }t|| | j| j| jS)az labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Llama4ForCausalLM >>> model = Llama4ForCausalLM.from_pretrained("meta-llama4/Llama4-2-7b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("meta-llama4/Llama4-2-7b-hf") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rjrrrrkr3rN)logitsrr_)lossrrrArZr5) rYrrslicer loss_functionr(r_rrrArZ)r=rjrrrrkrr3rroutputsrA slice_indicesrrs r?rOzLlama4ForCausalLM.forward\sH$** )%+'     8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD%#33!//))   r@)NNNNNNNr)rPrQrRr{r|_tied_weights_keys_tp_planr%rr-rrr5r9rSr r;r:rrrrrrOrTrUs@r?rrLs&12(*,GH23H /.2.204(,26*.!%-.: ##d*:  t+: &&- :  : ((4/ :   4': $;: ell*: +,:  ' ': : r@rzQ Base class for Llava causal language model (or autoregressive) outputs.  custom_introceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) Llama4CausalLMOutputWithPasta3 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size (batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. NrrrrArZimage_hidden_states)rPrQrRr(rr5r;rrrr rArrZrr5r@r?rrs &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r@rc$eZdZfdZdZxZS)Llama4VisionMLP2c~t||j|_|j|_t j |j|j d|_t j |j|jd|_ t j|_ |j|_ yrY)r,r-r1r0r3r\projector_input_dimfc1projector_output_dimfc2GELUr_projector_dropoutrr<s r?r-zLlama4VisionMLP2.__init__s !--!'!9!999T33V5O5OV[\99V88&:U:U\abWWY// r@c|j|}|j|}tj||j|j}|j|j |S)Nr)rr_Frrrr=rAs r?rOzLlama4VisionMLP2.forwardsT/ **=9  -4<<$--X !!$((="9::r@rcrUs@r?rrs 0;r@rc$eZdZfdZdZxZS)Llama4MultiModalProjectorct|tj|jj |j jd|_yrY) r,r-r3r\ vision_configvision_output_dimrDr1linear_1r<s r?r-z"Llama4MultiModalProjector.__init__s?   2 2    * *  r@c(|j|}|Sra)r)r=image_featuresrAs r?rOz!Llama4MultiModalProjector.forwards n5 r@rcrUs@r?rrs  r@rc J|j\}}}ttj|}|j |||d}|j \}}}}|j ||t||zt||z }|j ddddj}|j |t||zt||zt||dzz }|j ddddj}|j |d|jd} | S)NrDrr+r#r)rHrmathsqrtrGsizepermuter) input_tensor shuffle_ratio batch_size num_patcheschannels patch_sizeheightwidthreshaped_tensor output_tensors r? pixel_shufflers%(4(:(:%J XTYY{+,J$$ZZLL*6*;*;*='Jx"'' FC @UUYZ r@NrArIrrrrBc |jdd}g|d|j}|j|j|}|j |j|} |j |j|} t || |\}} |jdd}| jdd} | jdd} tj|jjt} | ||| | df|jsdn |jddd|\} } | jg|dj!} |j#| } | | fS)NrD)rIr#r+rF)rrr )rHrr rGrrrrrrr(r rrrrrr)r=rArIrrrr!r"r#rrr'rrs r?rOzLlama4VisionAttention.forwardsj$))#2. 88b8$--8 {{=166|D [[/44\B {{=166|D #:<^f#g j#--a3 ))!Q/ #--a3 (?(M(M KK , ,.L) %8      % $}}C$2H2H %  % ! \*k));;;;FFH kk+. L((r@NN) rPrQrRrr-r5rSr rrrrOrTrUs@r?rrs [1 [&/3(, ')||'),,') t+ ')  ') -. ') u||U\\D0% 2E2LL M')r@rcVeZdZfdZdej dej fdZxZS)Llama4VisionMLPc&t|||_tj|_tj |j|jd|_ tj |j|jd|_ y)NTrZ) r,r-r(r3rr_r\r1r0rrr<s r?r-zLlama4VisionMLP.__init__Ise  WWY99V//1I1IPTU99V55v7I7IPTUr@rArBcl|j|}|j|}|j|}|Sra)rr_rrs r?rOzLlama4VisionMLP.forwardPs4/ **=9 / r@rrUs@r?rrHs$VU\\ellr@rc eZdZdeffd Z d dej dej dej dzdedzfdZxZ S) Llama4VisionEncoderLayerr(ct||j|_t||_t ||_tj|j|_ tj|j|_ yra) r,r-r1rr,rrr3 LayerNormr1r2r<s r?r-z!Llama4VisionEncoderLayer.__init__Xsb !--.v6"6*!||F,>,>?(* V5G5G(H%r@N hidden_staterIroutput_attentionsc|}|j|}|j|||\}}||z}|}|j|}|j|}||z}|f}|r||fz }|S)N)rIr)r1r,r2r)r=rrIrrr6rrs r?rOz Llama4VisionEncoderLayer.forwardbs ++L9 %)^^ )&4& " l  ,.  44\B xx - ,. /    &Gr@r) rPrQrRrr-r5rSr:rOrTrUs@r?rrWsZI1I/3)- ll,, t+   $; r@rceZdZdZdeffd Z d dejdejdejdzdedzd edzd edzd e e zfd Z xZ S)Llama4VisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`Llama4VisionEncoderLayer`]. Args: config: Llama4VisionConfig r(ct|||_tjt |j Dcgc] }t|c}|_d|_ ||_ycc}w)NF) r,r-r(r3rbrcrdrrerh)r=r(r8r>s r?r-zLlama4VisionEncoder.__init__sW  mmuU[UmUmOn$o!%=f%E$op &+# %psA*NrArIrroutput_hidden_states return_dictrBcz||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|jD]&} |r||fz}| ||||} |r || dfz}| d}(|r||fz}|st d|||fDSt |||S)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr5)rrrrIr#rc3&K|] }|| ywrar5.0vs r? z.Llama4VisionEncoder.forward..seqWXWderprArZ)r(rrrrerr) r=rArIrrrrencoder_statesall_attentions encoder_layer layer_outputss r?rOzLlama4VisionEncoder.forwards>2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e+>Vd  r@NNNN) rPrQrRr(rr-r5rSr:rrrOrTrUs@r?rrs1/3)-,0#'? ||? ,,?  t+ ?  $; ? #Tk ? D[?   ? r@rcVeZdZfdZdej dej fdZxZS)Llama4UnfoldConvolutionc<t||j}t|tr||f}t j j||j|_t j|j|dz|dz|jd|_ y)N) kernel_sizestriderr#FrZ) r,r-rrrr5r3Unfoldunfoldr\ num_channelsr1linear)r=r(rr>s r?r-z Llama4UnfoldConvolution.__init__s '' k3 '& 4Khhoo+fFWFWoX ii   +a. 0;q> A     r@rArBcp|j|}|jddd}|j|}|S)Nrr+r#)rrrrs r?rOzLlama4UnfoldConvolution.forwards8 M2 %--aA6  M2 r@rrUs@r?rrs#  U\\ellr@rc:eZdZdeffd ZedZdZxZS)rGr(cvt|||_|jd|j |dy)NrIFr)r,r-r(rrJr<s r?r-z$Llama4VisionRotaryEmbedding.__init__s6   Z)?)?)GTYZr@c|j|jz}tj|dztjj |dzd}tj ||ddgd}d|d<||z}||z}|j|jzdz}d|jd tjd|dd|dzj|z zz }|dzd |ddddfzjdd }|dzd |ddddfzjdd }tj ||gd jjd dddf} | j|j d dddkd} tjtjtj | tj"| gd } | S) Nr+rr#rrE)rDrDrr).NrD.) image_sizerr5rint32rcatr1rrrsrepeat_interleaver masked_fillrstackcossin) r(idximg_idx frequencies_x frequencies_yfreq_dim rope_freqfreqs_xfreqs_yrrxs r?rJz-Llama4VisionRotaryEmbedding._compute_freqs_cis6#4#44,,sAvU[[9AA#q&!L))Wgbqk2:# 3 %%)C)CCqH  " "< 0 Q!,->A ?EEG(R T "A%y1IdD!m4LL__`agi_j!A%y1IdD!m4LL__`agi_j 7G,"5;;=HHJ3PSRSPS8T!!'//"a";a"?C((eii6F RWHX5Y_a)bcr@cL|jj|jSra)rIrrrs r?rOz#Llama4VisionRotaryEmbedding.forwards}} 4 455r@) rPrQrRrr-rrJrOrTrUs@r?rGrGs+[1[ &6r@rGceZdZUdZdZdgZeed<deffd ZdZ dde jd e jdzd e dzd e dzd e dzd e ee jdfzf dZxZS)rK vision_model)r>rr(crt|||j|_|j|_|j|_|j |_|j|jzdzdz|_|jdz|_t||_ tj|jtj|jz|_tj|jtj|j |jz|_t!||_tj$|j|_tj$|j|_t+||_t/||_|j3y)Nr+r#r)r,r-rrr1rrrMrpatch_embeddingr3r4r5randnrLrNrGrotary_embeddingr layernorm_prelayernorm_postrrYrvision_adapterrir<s r?r-zLlama4VisionModel.__init__ sA   ++ ++!--"// OOt>1DqH''- 6v>!||DJJTEUEU9V,VW(* TZZ%++dN^N^`d`p`pBq5q(r% ;F C \\$*:*:; ll4+;+;<)0 9&A r@c|jS)zg This function is used to fetch the first embedding layer to activate grads on inputs. )rrws r?get_input_embeddingsz&Llama4VisionModel.get_input_embeddings&s###r@N pixel_valuesrrrrrB.c ||n|jj}||n|jj}||n|jj}|j\}}} } d} d} |j |} | j\}}}| j || z| z||} |jj| jdd| jd}tj| |gd} |dz }| j || z| ||} |jj| j| j}| |z} |j| } | j!|d|} |j#|}|j%| d|||}|j&} |j)| } | ddddddf} |j+| } |r |j,nd}|r|d}nd}|st/d | ||fDSt1| || S) a Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, MllamaVisionModel >>> checkpoint = "meta-llama/Llama-3.2-11B-Vision" >>> model = MllamaVisionModel.from_pretrained(checkpoint) >>> processor = AutoProcessor.from_pretrained(checkpoint) >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> output = model(**inputs) >>> print(output.last_hidden_state.shape) torch.Size([1, 1, 4, 1025, 7680]) ``` Nr#rrDrErr)rrrrIr+c3&K|] }|| ywrar5rs r?rz,Llama4VisionModel.forward..s_qQRQ^_rr)r(rrrrHrrrLrr5rrNrrrrrGrrYrprrrArr)r=rrrrrrbatch_size_times_num_tilesrrrnum_concurrent_media num_chunksrr8rrrLpositional_embeddingrIrrArZs r?rOzLlama4VisionModel.forward,sTD2C1N-TXT_T_TqTq$8$D $++JjJj &1% >b r@rKc#eZdZUddgZiZdZeed<deffd ZdZ dZ dZ d Z e ed ed dej"dedeedeezfdZdej0dej"dej"fdZe ed e d#dej0dzdej"dzdej4dzdej0dzdedzdej"dzdedzdej0dzdedzdedzdedzdedzd eej4zdeedeezfd!Z d$d"Z xZ!S)%Llama4ForConditionalGenerationr*rrYr(ct||t|j|_t ||_t|j|_ |jj|_ t|jdr|jj|_ n)|jjjxsd|_ |jy)Nr]rD)r,r-rKrrrmulti_modal_projectorrrDrr_rr(r]rir<s r?r-z'Llama4ForConditionalGeneration.__init__s  -f.B.BC%>v%F"/0B0BC ,,77 4;; / $ 8 8D  $ 7 7 D D JD  r@c6|jjSra)rget_output_embeddingsrws r?r'z4Llama4ForConditionalGeneration.get_output_embeddingss""88::r@c:|jj|yra)rset_output_embeddings)r=new_embeddingss r?r)z4Llama4ForConditionalGeneration.set_output_embeddingss 11.Ar@c:|jj|yra)r set_decoder)r=decoders r?r,z*Llama4ForConditionalGeneration.set_decoders ''0r@c6|jjSra)r get_decoderrws r?r/z*Llama4ForConditionalGeneration.get_decoders""..00r@F)tie_last_hidden_stateszOObtains image last hidden states from the vision tower and apply al projection.rrvision_feature_select_strategyrrBc |jDcic] \}}| || }}}|j|fi|Scc}}w)aj pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_select_strategy (`str`): The feature selection strategy used to select the vision feature from the vision backbone. Can be one of `"default"` or `"full"` )itemsr)r=rr1rkrs r?get_image_featuresz1Llama4ForConditionalGeneration.get_image_featuressG $*<<>C41aQ]!Q$CC t  888Ds ::rjrkrc |m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jdj|j }t||jdz|jk(d|d|jd|S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. rrDz6Image features and image tokens do not match, tokens: z , features: r)rr5tensorr(image_token_idlongrallrrrrrrHnumel)r=rjrkrspecial_image_maskn_image_tokenss r?get_placeholder_maskz3Llama4ForConditionalGeneration.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1/99"=@@AUAUV ]004 48L8L8N ND^DTT`aoauauvwax`y z "!r@Nrrrrr3rrrrc | | n|jj} | | n|jj} | | n|jj} |du|duz r t d| | t d||j |}||j ||dj}|jd|jd}|j|j|j|j}|j|||}|j||}|j d ||||| | | | | d |}|d }d}|<||dd|j"d d z dfj|j}|d ddddf|j|jd k7j%}|d d df|j|jd k7j%}n1|d ddddfj%}|d d dfj%}t'j(}||jd|jd|jdj|j}| s|f|d dz}||f|zS|St+|||j,|j.|j0| Sd S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, LlavaForConditionalGeneration >>> model = LlavaForConditionalGeneration.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf") >>> prompt = "USER: \nWhat's the content of the image? ASSISTANT:" >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs, max_new_tokens=15) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "USER: \nWhat's the content of the image? ASSISTANT: The image features a busy city street with a stop sign prominently displayed" ```NrmzdYou cannot specify both pixel_values and inputs_embeds at the same time, and must specify either oneT)rr1rrD)rkr) rrrrkr3rrrrrr#.)rrrrArZrr5)r(rrrrqrr5rprGrr%rrrr>masked_scatterrrHrr3CrossEntropyLossrrrArZ)r=rjrrrrrkr1rr3rrrrrr vision_flatprojected_vision_flatr<rrrshift_attention_mask shift_logits shift_labelsloss_fctrs r?rOz&Llama4ForConditionalGeneration.forwardsQd2C1N-TXT_T_TqTq$8$D $++JjJj &1%$>{$K$N$N$$m&9&9% !"&!:!:G\";" *889KMbcM%$%%  )%+'/!5#)      )(6a6<<?Q;N9O9Q6Q'R'U'UV\VcVc'd$%c3B3k23G3J3J6==3Y]^3^_jjl %c12g/C/F/Fv}}/UYZ/Z[ffh %c3B3k2==? %c12g99; **,H!!"l&7&7&;O>OPR>S>V>VWcWjWj>kDY,F'+'7D7V# CV C+#33!//))2>2J   QU   r@c ~|jj|f|||||d|} |s|jdds|| d<| S)N)rrkrris_first_iterationr3Tr)rprepare_inputs_for_generationget) r=rjrrkrrrrIr model_inputss r?rJzrSr r:rrrOrJrTrUs@r?r#r#sV13MNH  | ;B11 E2!rs9''9),9+, 9 + + 9t3 9 "))":?:K:K"]b]n]n". E2.215.204(,2659*.!%)-,0#'-.| ##d*| ''$.|  t+ | &&- |  | ((4/| ),d |   4'| $;|  $;| #Tk| D[| ell*| +,| - -!| 3 | B r@r#)r=rXrKrr#)r)ercollections.abcr dataclassesrtypingrr5torch.nnr3torch.nn.functionalrr/transformers.models.llama4.configuration_llama4rrrE activationsr cache_utilsr r generationr masking_utilsr rmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericr r!utils.output_capturingr"configuration_llama4r$r% get_loggerrPloggerModuler'rWrerzr\rrrrSrrrrrsrrrr*r=rXrrrrrrrrrrrrrrGrKr#__all__r5r@r?rfs $! N&!.)KB9G&jjG5@   H % B)BII)$ !uxx != =( ,299 ,"BII",? ?D 2  2  2|| 2 5<< %& 2 UU\\ U# U%,, U(% II% <<% % << % LL4' %  %%B% II% <<% % << % LL4' %  %%2Y)"))Y)x/7/dLOLL<[ +[ [ |L -L ^  9;9  90;uxx;"  ( )")) )!ELL!! : << :  :ll : 5<< %& :7)BII7)t bii )9)XO "))O dbii(6"))6