^j:<ddlmZddlmZddlZddlZddlmZddlmcm Z ddl m Z ddlmZddlmZmZmZddlmZdd lmZdd lmZdd lmZdd lmZmZdd lm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*ddl+m,Z,m-Z-ddl.m/Z/m0Z0ddl1m2Z2m3Z3m4Z4m5Z5m6Z6GddejnZ8GddejnZ9GddejnZ:GddeZ;dejxde=dejxfd Z> dXd!ejnd"ejxd#ejxd$ejxd%ejxdzd&e?d'e?d(e&e(fd)Z@d*ZAdYd+ZBGd,d-ejnZCGd.d/ejnZDe)Gd0d1e$ZEGd2d3eEZFGd4d5eEZGGd6d7eEZHd8ejxd9e=dzdejxfd:ZIGd;dZK d[d?ejxd@e=d=e=dAe=fdBZLdCejxdDejdEe=dFeNdGe=dejxf dHZOdIdejfdJejxdKe=dLe=dMeQdNe=dOejdeSejxejxffdPZTGdQdReEZUe)dSTGdUdVeEeZVgdWZWy)\)Callable)OptionalN)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)maybe_autocastmerge_with_config_defaults)OutputRecordercapture_outputs) BltConfigBltGlobalTransformerConfigBltLocalDecoderConfigBltLocalEncoderConfigBltPatcherConfigc$eZdZfdZdZxZS)BltMLPct|||_|j|_|j|_t j |j|jd|_t j |j|jd|_t j |j|jd|_ t|j|_ yNFbias) super__init__config hidden_sizeintermediate_sizennLinear gate_projup_proj down_projr hidden_actact_fnselfr+ __class__s o/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/models/blt/modeling_blt.pyr*zBltMLP.__init__5s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ c|j|j|j||j|z}|SN)r2r4r0r1)r6xr2s r8forwardzBltMLP.forward@s6NN4;;t~~a/@#ADLLQRO#ST r9)__name__ __module__ __qualname__r*r= __classcell__r7s@r8r$r$4s  0r9r$cheZdZddeddffd Zdej dej fdZdZxZ S) BltRMSNormepsreturnNct|tjt j ||_||_y)z9 BltRMSNorm is equivalent to T5LayerNorm N)r)r*r. Parametertorchonesweightvariance_epsilon)r6r,rEr7s r8r*zBltRMSNorm.__init__Fs1 ll5::k#:; #r9 hidden_statesc"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NT)keepdim) dtypetorIfloat32powmeanrsqrtrLrK)r6rM input_dtypevariances r8r=zBltRMSNorm.forwardNsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r9c^t|jjd|jS)Nz, eps=)tuplerKshaperLr6s r8 extra_reprzBltRMSNorm.extra_reprUs* ))*+6$2G2G1HIIr9)gư>) r>r?r@floatr*rITensorr=r^rArBs@r8rDrDEs7$$$$;U\\;ell;Jr9rDc eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)BltRotaryEmbeddinginv_freqNr+ct||j|_|j|_||_|j j d|_|j}|jdk7rt|j}||j |\}|_ |jd|d|jd|jdy)N rope_typedefaultrcF) persistentoriginal_inv_freq) r)r*max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr+rope_parametersrecompute_default_rope_parametersrattention_scalingregister_bufferclone)r6r+device rope_init_fnrcr7s r8r*zBltRotaryEmbedding.__init__\s "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L($( ZeD 0(..2BuUr9rqz torch.deviceseq_lenrFz torch.Tensorc|jd}t|ddxs|j|jz}d}d|t j d|dtj j|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimN?rrOrR)rqrR) rlgetattrr,num_attention_headsrIarangeint64rSr_)r+rqrsbasedimattention_factorrcs r8rmz2BltRotaryEmbedding.compute_default_rope_parametersls&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k )))r9c|jddddfjj|jddd}|dddddfj}t |j j tr/|j j dk7r|j j nd}t|d5|j|jzjdd}tj|dd }|j|jz}|j|jz} dddj|j  j|j fS#1swYAxYw) NrrPrmpscpuF) device_typeenabledrOr~rx)rcr_expandr\ isinstancerqtypestrr transposerIrepeat_interleavecosrnsinrSrR) r6r< position_idsinv_freq_expandedposition_ids_expandedrfreqsembrrs r8r=zBltRotaryEmbedding.forwards@!MM$4-8>>@GG HZHZ[\H]_acde ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk  U C 5&,,.1F1L1L1NNYYZ[]^_E))%;C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s =BFF r;NNN)r>r?r@rIr`__annotations__rr* staticmethodrintr[r_rmno_gradrr=rArBs@r8rbrbYsllVyV #'+/"*D *(*t* ~u$ % **:U]]_ < )&*<*<&BUBUV(263E3E6K^K^(_%"r9NrMcross_attention_statescross_attention_maskattention_maskfull_text_row_masked_out_maskrpast_key_values use_cacheposition_embeddingskwargsrFc |} |j|}|jd|||||| d| \}} | |z}|} |j|}|j|}| |z}|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs to be ignored, used for FSDP and other methods that injects code into the model )rMrrrrr)rrrr) r6rMrrrrrrrrrresidualself_attn_weightss r8r=zBltTransformerLayer.forwards>!,,]; ,:4>>, ')%+ 3 , , ( (!=0 !55mD /  =0 r9)NNNNNNFN)r>r?r@rr*rIr`r[ LongTensorrboolrr FloatTensorr=rArBs@r8rrs, ## #7;48.2RV04(,!&HL5||5!& t 35$llT1 5  t+ 5 (-U\\5<<-G'H4'O 5&&-55$;5#5<<#=>E5-.5 u  %(9(95;L;L(L"MPT"TT U5r9rrMn_reprFc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r\rreshape)rMrbatchnum_key_value_headsslenrvs r8 repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr9modulequerykeyvaluerscalingdropoutrc t||j}t||j} tj||j dd|z} || |z} t j j| dtjj|j} t j j| ||j} tj| | } | j ddj} | | fS)NrOrrP)r~rRptrainingr)rnum_key_value_groupsrImatmulrr. functionalsoftmaxrTrSrRrr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r8eager_attention_forwardrs3 ; ;C/0G3w;q>C/0G G r9c|eZdZdedeffd Z ddejdejdejfdZxZ S) rr+rcdt|||_|j|_|j |_|j |_|j|_|j |jz|_|j|jz|_ |jdz|_ ||_ d|_ tj|j |j|jzd|_tj|j |j|jzd|_tj|j |j|jzd|_tj|j|jz|j d|_y)NTFr')r)r*r+rz num_headsrr,rrvrrr is_causalr.r/q_projk_projv_projo_projrs r8r*zBltSelfAttention.__init__&s8  33~~ !--#)#=#= **dnn< $(NNd6N6N$N!}}d* "ii 0 0$..4==2PW\] ii 0 0$2J2JT]]2Zafg ii 0 0$2J2JT]]2Zafg ii >@P@PW\] r9rMrrc |j\}}}|j|} |j|} |j|} | j |||j |j jdd} | j |||j|j jdd} | j |||j|j jdd} |\} } t| | | | \} } | |j| | |j\} } tj|jjt }||| | | |f|j"sdn |j$|j&d|\}}|j)||dj+}|j-|}||fS)NrrOrrrP)sizerrrviewrrvrrrupdaterr get_interfacer+_attn_implementationrrrrrrr)r6rMrrrrbszq_len_ query_statesrrrrattention_interfacerrs r8r=zBltSelfAttention.forward9s&**, UA{{=1 [[/ {{=1 #((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm &S#7 jRUWZ#[ j  &'6'='=j,X\XfXf'g $J (?(M(M KK , ,.E) %8      % $}}C$,,LL %  % ! \"))#ub9DDF kk+. L((r9r;) r>r?r@rrr*rIr`r=rArBs@r8rr%sK^y^S^0 *)||*) *)#\\ *)r9rceZdZdZd dedededzffd Z ddejdejdzd ejdzd e e d e ejejdze ejdzff d Z xZ S)BltCrossAttentionz@P@PW\]  !1!1v7J7JK  !1!1v7J7JK r9rMrrrrFc `|j\}}}|j|}|j|}|j|||j|j j dd}|j|}|j|} |j|} | j|d|j|j j dd} | j|d|j|j j dd} tj|jjt} | ||| | |f|j sdn |j"|j$d|\} } | j'||dj)} |j+| } | |z} | | fS)z#Input shape: Batch x Time x ChannelrrOrPrr)rrrrrrvrrrrrrrr+rrrrrrrr)r6rMrrrrrrrrrrrrs r8r=zBltCrossAttention.forward}s&**, UA{{=1 {{<0 #((eT^^T]]S]]^_abc !%-C!D[[!78 {{#9: __S"d.F.F V``abdef #((b$2J2JDMMZddefhij (?(M(M KK , ,.E) %8      % $}}C$,,LL %  % ! \"))#ub9DDF kk+. !M1 L((r9r;NN)r>r?r@__doc__rrr*rIr`rrr[r=rArBs@r8rrfsFySsTz.7;.2 $)||$)!& t 3$) t+ $) +, $) u||U\\D0% 2E2LL M $)r9rceZdZUeed<dZdZdZdgZdZ dZ dZ dZ dZ eedeed d Zej&fd ZxZS) BltPreTrainedModelr+model)imagetextTrFr)indexr)rM attentionsc  t|||jj}t |t j rt|jdd}|7t|jdr!t|jjdd}| |j}|dz}tj|jd|d|zd|z|j,tj |j|jyt |t"t$fs|d vrqt|jdd}|t|dr |j&}|>d D]9}t||d}|t|d s |jj(d }n|y|dz}d D]v}t||d}|t|d s tj|jd|d|zd|zt|ddXtj |j*xt|dt|dd} | bt| d rVtj| jd|d|zd|zt| ddtj | j*yt |t,s|dk(rt|jdd}|7t|jdr!t|jj.dd}|7t|jdr!t|jjdd}d} ||dz} t|dt|dd} t|dd} t|dt|dd} | | fD]}|t|d s| xs|jj(ddz}tj|jd|d|zd|zt|ddktj |j*| t| d rt| jj(d}|dz}tj| jd|d|zd|zt| ddtj | j*yt |t j0rg|j2}|dz}tj|jd|d|zd|z|j*tj |j*yy)a Initialize BLT weights following the original ByteLatentTransformer: - Most weights are drawn from a truncated normal. - Scale is ~ 1 / sqrt(model_dim) (or 1 / sqrt(hidden_dim) for FFN outputs). - Norm layers are set to weight = 1, bias = 0. r,Nencoder_configrrr)rVstdab)MllamaTextSelfAttentionMllamaTextCrossAttention)rrrrdenserKrP)rrrr(rr  MllamaTextMLPdecoder_configr0fc1r1r2fc2r)r) _init_weightsr7r>rr. Embeddingryr+hasattrr embedding_diminit trunc_normal_rK padding_idxzeros_rrr,r\r(r$r r/ in_features)r6r class_namer,rr~nameproj proj_namerin_stdr0r1r2 hidden_dimout_stdfan_inr7s r8rz BltPreTrainedModel._init_weightss f%%%..  fbll +!$++}dCK"wt{{  f/1BC D W I $++}d;C{wv}=(({MD"646D'GD(,C"kk//3  {t)C< / vy$7#h(?&&  s(c' tVT2> DII. /VXwvw/MNF!gfh&?""MM3h#g 6640<KK ,  ff %)F!$++}dCK"wt{{ DII. /$H)E&--33A6 $d*""$$7l'k 9fd3?KK /  fbii (''F$,C    s(c'  {{& FKK(  )r9)r>r?r@rrbase_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_can_compile_fullgraph_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrrr_can_record_outputsrIrrrArBs@r8rrs} (&*#./"N "''(;1E$%5Q? U]]_JJr9rc\eZdZUeed<deeddiZdeffd Z dde jdzd e jdzd e jdzd e jdzd e jdzd e dzde jdzde dzde jdzdeefdZdZxZS)BltLocalEncoderr+encoder_attentionsr local_encoderr layer_namec t||d|_||_t j t |jDcgc]}t||c}|_ t||_ t j|j|j|jzd|_t j |j"|j|_t j |_|j(r |jnd}t |D]3}|j&j+t-|||j5|j/ycc}w)NFr+r out_featuresr(rr+rr,)r)r*gradient_checkpointingr+r. ModuleListrangenum_hidden_layersrlayersrb rotary_embr/r, cross_attn_kpatch_embedding_projectionr vocab_size embed_tokenscross_attn_layerscross_attn_all_layersappendr post_initr6r+r layers_to_addr7s r8r*zBltLocalEncoder.__init__Is!  &+# mmEJ6KcKcEd e  3 e  -F;*,))**++f.A.AA+ ' LL):):F>|L +33 2 21 5 8P8P PRVR]R]RiRi  $(;;#D#DC! ,MD,B,B9,M-".+8#9- -)& ,.D.G.G H[H[.\\ + ], ,222r9cF|jd}|jd}|jdjdd|jd}tj|||f|j |j }|j|d|dd}|ddd|ddf}|S) a Reduce variable length patches to single embedding per patch Note: this works with variable number of patches for different sequences in the batch It handles variable length patches by assuming that patch_lengths will be 0 for any extra patches on the *right*. Since there can be a variable number of patches this function also return the number of patches for each sequence in the batch. Any embeddings on the right that are not allocated to a patch (i.e. if the sum(patch_lengths[i]) < seq_len for any i) will be sent to a dummy patch, which is trimmed before returning. rrPrRrqramaxF)srcr~rreduce include_selfN)r\rrrIzerosrRrqscatter_reduce)r6rMmax_num_patchesrKrTrreduced_embeddingss r8rSzBltLocalEncoder.patch_reduces#((+ %++B/ ''+222r=;N;Nr;RS "[[ - 8 @S@S\i\p\p 0>> ? 03CO3CQ0FG!!r9 NNNNNNNNN)r>r?r@r!rrrr*r*rIrr`rrrrr=rSrArBs@r8r,r,Cs  !!n-=QSbc42.2-1,0.204(,6:"&)-23##d*23||d*23llT) 23  t+ 23 &&- 2323!& t 3234Z23<<$&23+,23h"r9r,ceZdZUeed<deffd Z d dejdzdejdzdejdzdejdzdejdzd e dzd ejdzd e e fd Z xZ S)BltLocalDecoderr+c t||d|_||_d|_t j t|jDcgc]}t||c}|_ t||_ t j|j|j|j zd|_t%|j|j&|_t j |_|j,r |jnd}t|D]3}|j*j/t1|||j5|j3ycc}w)NFTr2r3rrr5)r)r*r6r+cross_attn_decoderr.r7r8r9rr:rbr;r/hidden_size_globalr,r<r=rDrnormr@rArBrrCrDs r8r*zBltLocalDecoder.__init__s$  &+# "&mmEJ6KcKcEd e  3 e  -F;*,))11++f.A.AA+ ' v11v7J7JK !#4:4P4P00VW }- I  " " ) )!9RXRdRde   ! fs E%NrFrGrHrrrrIrc "|jd} |} |j|}|j| |jd|jjz|jj }||j s| |z} |Mtj|jd|jjdj| d}|j| |} tj| |jj|j} t!|j"D]N\} } | dk(s|jj$r!|j&| d| ||d|\}}| |z} | | f| ||d|} P|j)| }|S) NrrrMrPrrOrNr)r\r=rr+r<r,rgrIr{rqrrr;rPrrrQr:rAr@ri)r6rFrGrHrrrrIrrTrMrirVrWrlogitss r8r=zBltLocalDecoder.forwards#((+ % 66|D #++  **1- 0H0HH$++JaJa   #D,C,C)L8M   ]003Mr?r@r rr*rIrr`rrrr=rArBs@r8reres !!44.2-1,0.204(,6:.##d*.||d*.llT) .  t+ . &&- ..!& t 3.+,.r9rec eZdZUeed<deeddiZdeffd Z dde jd e jdzd e jdzd e dzd e ef d ZxZS)BltGlobalTransformerr+global_attentionsrglobal_transformerr/ct||||_tj|_t |jD]'}|j jt||)t||_ t|dd2tj|j|jd|_ntj"|_|j%y)Nr2encoder_cross_output_sizeFr')r)r*r+r.r7r:r8r9rBrrbr;ryr/rsr,token_embedding_projectionIdentityrCrs r8r*zBltGlobalTransformer.__init__s   mmo v778 GI KK  269E F G,F; 66 = I.0ii00&2D2D5/D +/1kkmD + r9NrGrrrrc |j\}}}|j|} tj| |jj|j } |Mt j|jd|jjdj|d}|j| |} t|jD]\} } | | f| ||d|} | S)NrrrMrrPrN)r\rtrPrr+rrIr{rqrrr;rQr:) r6rGrrrrrTrsrrMrrkrVs r8r=zBltGlobalTransformer.forwards"/!4!4 GQ77 F  -4;;3F3FQUQ^Q^_   ]003Mr?r@rrrrr*r*rIr`rrrrr=rArBs@r8roros &&^,z(process_patch_lengths..Ps6&#f+6srZrr)ritemdivmodextendrBmaxrIr_rRrqrQtensorrRanysumr\nonzero) rwrxrT processedseqr|length full_chunks remaindermax_lenpaddedrk last_nonzeros r8process_patch_lengthsr3s##A&JI!#'l )F[[]F%+F4D%E "K MM+,{: ; i(  )  !6I66G [[*g.m6I6IR_RfRf gFy)t 6 ',||F-BUBU^k^r^r'sF1mF m# $t ! Q##% Q7! ((Q(/779==?DDFJ =L=() Mr9ceZdZUeed<deffd Z ddejdzdejdzdejdzde dzdejdzd e dzd e dzd e dzd e dzd eefdZe ddZxZS) BltPatcherr+ct||t|j|_t j |_t|jjD]1}|jjt|j|3t j|jj|jj|_t!|jj|jj"|_t j&|jj|jjd|_|j+y)Nr2rFr')r)r*rbr+r;r.r7r:r8r9rBrrr>r,r?rDrrir/lm_headrCrs r8r*zBltPatcher.__init__bs  ,DKK@mmo t{{<<= LI KK  24;; J K LLL)?)?AXAXYt{{66DKKO  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L(;;')+%  & "oom\J[[ vE!-EXituM vdii 67$22>>f>MUUW&3&9&9"1&=# O  ! ==. /%# >M"JJ_-]5H5HQ^QeQeM.m=MN #]F::r9c|jd}tjddgtj|jj dj |d}|jd}|ddddf}||kD}|jd}tj||jj dj|d} tj| |} tj| | gd} tj||gd} | | j||} |jdj}| ddd|f} tj|| |zfd}tj|ddddf|dz }tj|ddddfdz |fd}||z dz}|S)z Computes patch lengths from token entropies. Depending on whether a threshold is provided, the function uses either: - Thresholding the entropy values (when `threshold` is set). rrrZNrMrPr)r\rIrlongrqrrepeatr{r full_likecatrrr)rrrrrT init_tokensoffset patch_maskrs token_indicessentinelpadded_indices padded_mask patch_startsmax_valid_patchespatch_start_ids last_token patch_endsrws r8rz'BltPatcher.patch_lengths_from_entropiess__Q'  LL!Quzz):J:J K U UVW X _ _`jlm n ""1%ae$ * ""1% WY5E5EFPPQRSZZ[egij ??=':M8#r?r@r"rr*rIrr`rrrrr_rrr=rrrArBs@r8rr_s /$.2.204(,26!%!%"&'+9;##d*9; t+9;&&- 9;  9; ((4/ 9;$;9;$J9;4<9;*9;+,9;v 33r9rprimectj|tj|j}tj|j d|j}||z}tj ||zdS)a A polynomial rolling hash algorithm that converts sequences of tokens into hash values. The hash is computed as: hash = (token_0 * prime^0 + token_1 * prime^1 + ... + token_n * prime^n) The rolling hash allows the model to efficiently identify and encode recurring byte-level patterns in the input text. Args: token_tensor (torch.Tensor): [batch_size, seq_len, group_size] containing token IDs to hash prime (int): Prime number used as the base for the polynomial hash. Returns: torch.Tensor: Hash values of shape [batch_size, seq_len] where each value represents the hash of the corresponding token group Example: >>> tokens = torch.tensor([[1, 2, 3], [4, 5, 6]]) >>> hashes = rolling_polynomial_hash(tokens, prime=31) >>> # hash[0] = 1*31^0 + 2*31^1 + 3*31^2 >>> # hash[1] = 4*31^0 + 5*31^1 + 6*31^2 rZrPrMr)rIrr|rqr{r\r) token_tensorr prime_tensorpowers prime_powerss r8rolling_polynomial_hashrsa.<<U[[ATATUL \\,,,R09L9L MF'L 99\L0b 99r9 token_ids group_sizemax_hashcZtj5|j\}}tj||dz tj|j }tj ||gd}|jd|d}t||} | |z} ddd| S#1swY SxYw)z1Hash token groups and map to range [0, max_hash].rrZrN) rIrr\r_r|rqrunfoldr) rrrrrTrspadding padded_tokenswindowshashes hash_valuess r8byte_group_hash_functionrs  ('oo G++j*q. T]TdTde 7I"6A>  &&q*a8(%8x'  (  ( s BB  B*local_encoder_tokensencoder_hash_tok_embedding$encoder_hash_byte_group_nb_functionsencoder_hash_byte_group_sizeencoder_hash_byte_group_vocabcgd}|j|}d}t|D]Y} || t|z} |D]A} t|| | |} | ||zz} ||| j |j z }|dz }C[|S)z=Compute token embeddings enhanced with hash-based embeddings.) ʚ;l21Ai oYlvtl.l}glAul 0lTlAK l|rr)r?r8rRrrSrq)rr.rrrrprimes embeddings embedding_idxfunc_nbrrhash_idsoffset_hash_idss r8compute_hash_embeddingsrs F++,@AJM=>wV,-6 J/0DjRWYvwH&9V)VVO 4_EHHIZIZ[ [J Q M   r9FrKrJrpatches_as_queriesr<rRcz|j\}}|j}|rp||z} |} tj||j dj dj |||} |j dj |||} no|} ||z} |j dj |||} tj||j dj dj |||} | | k(} |rdnd}| j ||} || | f}| j|k7rtd| jd|| j d} d| j|z }|j|jtjtj|j} | S) aR Prepare cross-attention mask for patch-based attention, following mllama's robust approach. This function creates masks that control which patches can attend to which other patches, with support for query/key role swapping and cross-attention multipliers. Args: patch_ids (torch.Tensor): Tensor of shape [batch_size, seq_len] containing patch ids. num_patches (int): Total number of patches. sequence_length (int): Length of the sequence. patches_as_queries (bool): If True, patches are used as queries, otherwise as keys. cross_attn_k (int): Cross-attention multiplier for repeating patches. dtype (torch.dtype): Data type for the output mask. Returns: Tuple[torch.Tensor, torch.Tensor]: - cross_attention_mask: 4D tensor [batch_size, 1, q_len, kv_len] rMrrPrrzCross attention mask shape z doesn't match expected rw) r\rqrIr{rrrrrS masked_fillrfinfomin)rKrJrrr<rRrTrsrqrkv_len q_patch_ids kv_patch_idsr repeat_dimexpected_shapeinverted_cross_attn_masks r8#_prepare_patch_cross_attention_maskr9s4$//J   Fl*  LLV 4 Yq\ Yr] VJ W 5  !**1-44ZgV |+))"-44Z+V LLV 4 > >q A K KA N U UV`bikv w  ',6)bJ/AA,T^A_!%0N!!^3)*>*D*D)EE]^l]m n  099!< #%9%<%/;;.    7t66 .-,    !88q!"u9M_A'%++A.+$11 &&  $##  /-&%ETE`OAAfj#6    '$+  r9c.|jjSr;r.r?r]s r8get_input_embeddingszBltModel.get_input_embeddings#s!!...r9c&||j_yr;r)r6rs r8set_input_embeddingszBltModel.set_input_embeddings&s*/'r9rsc|jd}tjtj|d|j|j |j dddddfgd}tj||j }|jd|jdjdkjddz S)NrrrZrPrrM) r\rIrr_rRrqcumsumr{rr)r6rwrsrTrtoken_positionss r8rz BltModel._patch_ids_from_lengths)s"((+ yy J1D1D]MaMab$$$,QV4    ,,w}7K7KL&&q)_-F-Fq-I-S-STV-WW\\ac\dghhhr9rm)r>r?r@rr*rrrIrr`rrrrrr[rr=rrrrrArBs@r8rrs y( .2-1.204(,26!%C ##d*C ||d*C  t+ C &&- C  C ((4/C $;C +,C  ( (C  C J/0 iU\\ iC iTYT`T` ir9rzB The Blt Text Model with a language modeling head on top. ) custom_introceZdZUeed<dZdZddiZdeffd Ze e dde jdzd e jdzd e jdzd e jdzd e jdzd ee je jfdzdedzde j dzde jdzdedzdee jzdeedeezfdZxZS)BltForCausalLMr+Frz'model.local_encoder.embed_tokens.weightzlm_head.weightc&t|||j|_|j|_t ||_tj|jj|jd|_ |jyr&) r)r*get_text_config text_configr>rrr.r/r r,rrCr5s r8r*zBltForCausalLM.__init__Ask  !113 ++f% yy!6!6!B!BFDUDU\ab  r9NrFrrrrrrrGlabelsrlogits_to_keeprrFc |jd|||||||| d| } | j}t| tr t | dn| }|j |dd|ddfj }d}| |j|| |jfi| }t||| j| j| jS)a cross_attention_states (`torch.FloatTensor`, *optional*): Output of the vision model, used for cross-attention. This tensor contains the processed image features that the language model will attend to. cross_attention_mask (`torch.Tensor` of shape `(batch_size, seq_length, max_num_images, max_num_tiles)`, *optional*): Cross-attention mask to control the interaction between text tokens and image tiles. This 4D tensor defines which image tiles each text token should attend to. For each text token (in seq_length): - 1 indicates the token **should attend** to the corresponding image tile - 0 indicates the token **should not attend** to the corresponding image tile full_text_row_masked_out_mask (`tuple[torch.Tensor, torch.Tensor]`, *optional*): A tuple containing two tensors that mask out rows in the cross-attention mechanism: - The first tensor has shape `(batch_size, 1, seq_length, 1)` and contains values of 0 or 1. A value of 0 indicates that the corresponding text token's entire row in the cross-attention matrix should be masked out (all image tokens ignored). - The second tensor has the same shape and is used internally to apply the masking during the forward pass of cross-attention layers. This mask is derived from the cross_attention_mask and is used to handle cases where a text token should not attend to any image token. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BltForCausalLM >>> model = BltForCausalLM.from_pretrained("itazap/blt-1b-hf") >>> tokenizer = AutoTokenizer.from_pretrained("itazap/blt-1b-hf") >>> prompt = "If I had to write a haiku, it would be:" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=40, do_sample=True, temperature=0.6) >>> result = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] >>> print(result) If I had to write a haiku, it would be: "Snowflakes gently fall" - simple, yet peaceful. I love the idea of snowflakes gently falling, each one ``` )rFrrrrrrGrN)lossrlrrMrr) rrrrslicerr_ loss_functionr>rrrMr)r6rFrrrrrrrGr rrroutputsrM slice_indicesrlrs r8r=zBltForCausalLM.forwardJs|$**  )%!5*G+'     11 8B>SV8W~ot4]k mA}a,?@AGGI  %4%%ffdooPPD%#33!//))   r9) NNNNNNNNNNr)r>r?r@rrr%r!_tied_weights_keysr*rrrIrr`r[rrrrrrrr=rArBs@r8r r 6s| "CEUVy.2.204:>8r-s.*% &!CC)/B9OKF&IIGERYY"JJ(><>)ryy>)B;) ;)|[[[|n"(n"bI(IX0-0f))t)X]XdXd)XB#BJ::<\a||),9<UX$#,,#!# #+. # #' # $' # \\#T % K ||K K K  K  K ;; K  5<< %&K \li!li^ g 'g  g T Mr9