^jddlZddlmZmZmZddlmZddlZddlm Z ddl m Z ddl m Z dd lmZmZmZmZdd lmZdd lmZdd lmZmZmZmZd e defdZd e defdZd e dee e ee e!ffdZ"GddZ#GddZ$y)N)floorgcdsqrt)Any)PreTrainedConfig)ContinuousBatchingConfig)is_flash_attention_requested) BlockManagerCacheAllocatorFullAttentionCacheAllocatorSlidingAttentionCacheAllocator)DistributedHelper)resolve_max_memory_percent) RequestState RequestStatusget_device_and_memory_breakdownloggerconfigreturncbt|dd}||St|dd}||Std|)z9Finds the number of key-value heads for the given config.num_key_value_headsNnum_attention_headszMnum_key_value_heads or num_attention_heads could not be found in the config: getattr ValueError)rkv_headss |/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/generation/continuous_batching/cache.pyfind_num_kv_headsr sMv4d;Hv4d;H eflemn ooct|dd}||St|dd}t|dd}||||zStd|)z.Finds the head dimension for the given config.head_dimN hidden_sizerzThead_dim or (hidden_size and num_attention_heads) could not be found in the config: r)rr#r$rs r find_head_dimr%*shvz40H&-6K!&*?F#6#B111 lmsltu vvr!c$t|dd}|3t|dddnd}t|jDcgc]}|}}i}t|D]\}}|j |g|gz||< t |j Dcgc] }t|c}}g} |jD]7\}}tdt||D]}| j||||z9| D cgc] } || d } } | | fScc}wcc}wcc} w)a Group layers depending on the attention mix, according to VLLM's hybrid allocator rules: - Layers in each group need to have the same type of attention - All groups have the same number of layers For a model with the following layer types: ["sliding", "full", "full", "sliding", "full", "full", "full", "full"] We would get four groups: [0, 3], [1, 2], [4,5] and [6,7]. layer_typesNsliding_windowsliding_attentionfull_attentionr) rrangenum_hidden_layers enumerategetrvalueslenitemsappend) rr' attn_type_ layer_countsi layer_typeindices group_size layer_groupslg group_typess rgroup_layers_by_attn_typer=8sD&-6K+26;KT+R+^'dt */0H0H*IJQyJ JL";/J :#/#3#3J#Cqc#I Z J<3F3F3HIs7|IJJL+113= Gq#g, 3 =A   A N ; < ==1==";r!u%=K=  $$#KJ>s DD.D ceZdZdZdZej fdededeje zde de e e fdejd d fd Zd ed ed efdZd ed ed efdZde d efdZdede d ed ed zfdZde d d fdZd efdZde dededeeed zdeeed d f dZde dededej4d d f dZdeded e e effdZdej4dej4d edeej4deej4d eej4ej4ff d!Zd"e d e fd#Zde d$eed efd%Z d&e!d'ed d fd(Z"d)eed*eed d fd+Z#d,e d efd-Z$d,e d.ee d eeeeeffd/Z%d1d0Z&y )2PagedAttentionCacheu Manages the cache for a paged attention mechanism, inspired by VLLM's hybrid allocator. The cache relies on making groups of layers to reduce the complexity of cache management and fragmentation. The cache uses a three-level hierarchy: - Pages: The smallest unit of cache, a page has a size of [num_heads, head_size], which is the space needed to store the key or value states for one token and one layer. For a model with only full-attention layers, to store the KV cache of one token, we need `2 * num_layers` pages: key and values each take `num_layers` pages. Pages are grouped into blocks: - Blocks: A block is a collection of `block_size` pages, serving as the allocation unit to reduce management complexity and fragmentation. Cache is allocated and freed block by block, not page by page. One block is allocated to one layer group, which only has one attention type, like full-attention or sliding-attention. If all layers in the model have the same attention type, then all layers will be in the same group. There is more than one group if and only if the model has a mixed attention types, like layers with full-attention and layers with sliding-attention. - Cache tensors: The physical supports for the cache. There are as many cache tensors as there are layer in a layer group, and the shape of the cache tensor is `[num_blocks * block_size, num_heads, head_size]`. Grouping layers into groups is useful because when we allocate one block to a group N, the block allocated is the same for all layers in group N, equivalently it is allocated across all cache tensors. This allows us to efficiently allocate and free blocks, and to efficiently read and write key and value states. For instance, imagine we have 8 blocks of cache and a model with two layer groups: a full-attention group with 3 layers and a sliding-attention group with 3 layers. At creation time, the physical cache tensors look like this: cache_tensor_0: □ □ □ □ □ □ □ □ cache_tensor_1: □ □ □ □ □ □ □ □ cache_tensor_2: □ □ □ □ □ □ □ □ where □ means the blocks is not allocated to any layer group yet. We have 3 cache tensors because there are 3 layers per group. We allocate 1 block to each group, after allocation, the cache tensors look like this: cache_tensor_0: ✖ ◉ □ □ □ □ □ □ cache_tensor_1: ✖ ◉ □ □ □ □ □ □ cache_tensor_2: ✖ ◉ □ □ □ □ □ □ where ✖ means the block is allocated to the full-attention group, and ◉ means the block is allocated to the sliding-attention group. Now, if we continue to generate, and the sliding window has been reached, we only need to allocate a new block for the full-attention group, and the cache tensors look like this: cache_tensor_0: ✖ ◉ ✖ □ □ □ □ □ cache_tensor_1: ✖ ◉ ✖ □ □ □ □ □ cache_tensor_2: ✖ ◉ ✖ □ □ □ □ □ And after further generation, when we need a new block allocated: cache_tensor_0: ✖ ◉ ✖ ✖ □ □ □ □ cache_tensor_1: ✖ ◉ ✖ ✖ □ □ □ □ cache_tensor_2: ✖ ◉ ✖ ✖ □ □ □ □ This would not have been possible if all layers were in the same group: we would have had to allocate a new block for the sliding-attention group, although it is not needed. rcontinuous_batching_configdevicedistributed_helpertp_plandtyperNc  ||_||_||_t||_t ||_|j|_|j|jkr%td|jd|jt|\}}t|d} t||_ i|_ i|_t|D]N\} } || dk(r |j nd} t| D]%\} }| | f|j|<| |j|<'Pd}dD]}||vrd|z|vrd }n|j"}|dkDrE|rC|j|zdk7rtd |jd |d |xj|zc_|j$ t'|dt)|||j|| j+\}}|dkDr{t-j.||g|jt,j0}|j3|t5|dj7t5|dj7}}||_||_|j:|jz|_t?j@d|j8d|j:d|j|jB}| |jD}||_!g|_#g|_$|dz|j|j|j f}|dz|jz|j|j f|_%||jz|_&||jzdz|_'|dz|jz|_(tS| D])}t-jT|jJ|j|j}t-jT|jJ|j|j}t,jVjY|t,jVjY||jFj[||jHj[||j]||j_d|j]||j_d,t?j@d|jJd|jFdj`d|jFdjc|jd|_2g|_3d|_4d|_5d|_6t|D]\} }|dk(r8to| |j|jd}|xjhdz c_4nq|dk(r^tq| |j|j |jN|jP}|xjjdz c_5|jr|_6ntd||jfj[||jdxr|dgk(|_:tw||j|dkD|_<d|_=d |_>y )a`Initialize a paged attention cache for efficient memory usage. Also turns in prefix sharing if the model has only full attention layers. Args: config: Model configuration continuous_batching_config: Continuous batching configuration containing cache parameters device: Device for the cache tensors distributed_helper: TP-aware helper. Used to dispatch attention heads and ensure coherent cache size tp_plan: Tensor parallelism plan dtype: Data type of the activation and the cache (for now, these are the same) zBlock size must be at least z , but got rr)r T)zlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzmodel.FzNumber of key value heads z+ must be divisible by tensor parallel size .N) cb_confighas_logit_processors)rrArEr<r9rBrEz1Paged cache initialized: self.max_batch_tokens = z, self.num_blocks = z, self.block_size = )rErBzself.cache_shape = z self.key_cache[0].shape = z self.key_cache[0].numel() = r*)allow_block_sharingzInvalid group type: )tp_on)?rrErBr rr%r# block_size_min_block_sizerr=r0 num_groupssliding_windowslayer_index_to_group_indicesr-r(tp_sizemax_memory_percentrPagedAttentionMemoryHandler%infer_max_batch_tokens_and_num_blockstorchtensorint64tp_all_reduce_minintitemmax_batch_tokens num_blocks num_pagesrinfomax_blocks_per_requestfallback_max_blocks_per_request key_cache value_cache cache_shaperead_trash_indexsentinel_indexwrite_trash_indexr+empty_dynamomark_static_addressr2viewfill_shapenumelrLgroup_cache_managersnum_full_attention_groupsnum_sliding_attention_groups%max_sliding_window_blocks_per_requestrr_max_blocks_per_requestuse_prefix_sharingr _block_manager_total_prefix_length_block_table_key)selfrrArBrCrDrEr:r<r9r6groupr(jlayerkv_is_tpkeyrSr]r^syncrablock_based_shaper4new_layer_key_cachenew_layer_value_cache group_typecms r__init__zPagedAttentionCache.__init__s(   ):&(A *62 5?? ??T11 1;Df$E! ka) l+!,.)!,/ =HAu6A!nH[6[V22abN%e, =5<=q611%8.<$$U+ = =M C7Nhn&?  %,, Q;8'''1Q6 01I1I0JJuv}u~~A  $ $ 0 $ & 8 8 @ &1Kbf g'B'A**#! ( 0 / 1 %* Q;<t{{Z_ZeZefD  0 0 6+.tAw||~+>DGLLN@Sj !1$4??: H0E0E/II^DOOK__tbfbqbqauvw"V>VX\XeXef'!^t>@X@XZ^ZgZgh *T__ <(4??:Q>",q.DOO!Cz" OA"'++d.>.>djjY]YdYd"e $)KK0@0@ [_[f[f$g ! MM - -.A B MM - -.C D NN ! !"5 6    # #$9 :  $ $%6 7 C I I! L ! & &'8 9* E K KA N O  *t''++GT^^A->-D-D,HHf$..YZJ[JaJaJcIghi$>#Q#Q :<!)*&,-)562&{3 1MAz--0DOOY]YqYqr..!3.223t(=(=t?R?RTXTjTj11Q61=?=W=W: #7 |!DEE  % % , ,R 0 1#'":":"`{O_N`?`*:tgXYkZ)*!!%r!num_requested_blocksallocated_blocksc||jz}|jr5t|j|z d}|t |||jzz }|S)aReturns the number of physical blocks needed to allocate (num_requested_blocks) blocks to a request that already has (allocated_blocks) blocks. The number of newly allocated blocks needed is predicted by the following rules: - for full attention groups: since there is no sliding window for full attention layers, one requested block is always equivalent to one newly allocated block for EACH full attention group - for sliding window groups: because of the sliding window, the number of blocks allocated to a request is capped. Using the number of already (allocated_blocks) we can compute the number of new blocks to actually allocate to the request, which can be lower than the number of requested blocks. That number is the same for all sliding window groups, as only one sliding window size is supported. r)rqrrmaxrsmin)ryrr needed_blocks blocks_lefts r blocks_neededz!PagedAttentionCache.blocks_needed)sY-t/M/MM  , ,dHHK[[]^_K S.BCdFgFgg gMr!cH|j|||jkS)zcReturns a boolean indicating if the allocation of (num_requested_blocks) blocks will be successful.)rget_num_free_blocks)ryrrs rwill_allocation_be_successfulz1PagedAttentionCache.will_allocation_be_successful<s%!!"68HITMeMeMgggr! request_idc@tfd|jDS)zfReturns the total number of physical blocks currently referenced by a request across all layer groups.c3hK|])}t|jjd+yw)N)r0 block_tabler.).0rrs r z4PagedAttentionCache.blocks_in_use..Bs'_r3r~~))*b9:_s/2)sumrp)ryrs `r blocks_in_usez!PagedAttentionCache.blocks_in_use@s_TE^E^___r!n_blocksc|j||syd}|jD]>}|j|||j}|t d|d|t ||}@|S)zAllocate cache blocks across all layer groups for a given request. Actual allocation is done by the cache managers, and this method only returns the maximum number of blocks actually allocated across all managers.NrzFailed to allocate z blocks for request )rrpallocate_blocksrvrr)ryrrr max_allocatedrnum_allocated_blockss rrz#PagedAttentionCache.allocate_blocksDs11(%> !L ^ B  !5!5j+|!\ ] ^  !$'(A(A:$N ` L##B$7$7 KQ]$^_ ` "r!rclt|jD]\}}|j|||||y)N)r-rpfill_block_table)ryrrrrr6rs rrz$PagedAttentionCache.fill_block_tableqs=t889 WEAr    K{ST~ V Wr!ci}|jdkDr||z|d<|jdkDr)|t||jjdz z|d<|S)zRetrieve the key sequence length for the given request_id across all layer types. Returns a dictionary of layer types to their corresponding key sequence lengths.rr*r r))rqrrrrr()ryrr seqlens_ks r get_seqlens_kz!PagedAttentionCache.get_seqlens_kwsb  ) )A -*5 *DI& '  , ,q 0-9C T[[MgMgjkMk   ! ! #q (   #4j A   #4l C|+ +--i8 Q    #4j A   #4l C$)$6$6wCS$T !&+&8&8!EU&V #"%&===%(;(;;FFrJTTUWXD$)$6$6wCS$T ! ! 1 1$ C&+&8&8!EU&V # # 3 3D, G   #4j A   #4l C%&===r!flash_attn_with_kvcache_fnc*|j|tj|jj }d|vrd|_|jSd|vrd|_|jSt dtj||jS)zA function to get the name of the block table key for the given flash_attn_with_kvcache_fn. The function's signature is only inspected once. This is necessary because different version of flash have different names for the block table key.r page_tablezOflash_attn_with_kvcache_fn does not have a block_table or page_table argument: )rxinspect signature parameterskeysr)ryr kwarg_namess rget_block_table_keyz'PagedAttentionCache.get_block_table_keys  (!++,FGRRWWYK +(5%$$$ ,(4% $$$!efmfwfwySgTfUV$$$r! prompt_idschd}g}tt||jzD]}|||jz|dz|jz}|jj ||d}|jj j |}|-|j||jj|n|rCtjd|dt|d|jd}||j|<t||jz} |xj| z c_ | S)aSearches for a prefix match in the cache for the given (prompts_ids). If one is found, we reference the matching blocks in the (request_id), increase the reference count of the blocks and return the number of blocks that match. If no prefix match is found, we return 0.Nr r)group_idzFound prefix match for request z with z blocks)r+r0rNrv compute_hash _hash_to_idr.r2increase_ref_countrdebugrprrw) ryrr current_hashrbtokensblock_idr prefix_lengths rsearch_prefix_matchz'PagedAttentionCache.search_prefix_matchs) s:$//9: ADOO 3q1u6OPF..;;L&[\;]L**66::<HH# ''1##66x@   LL::,fSQaMbLccjk l**1-B)9BNN: &,-?  !!]2!r!statenum_complete_blocksc|dk(s|jtjk(ry|jD][}|js|j j ||j|j|j|jz]y)aMarks the blocks allocated to a request (state) as complete if they are shareable and they have been computed in the forward pass. A complete block is a block where the KV cache has been fully computed: if the block has enough space to hold the cache for N tokens, the block is marked as complete when the cache data is present for the N tokens. If block sharing is off, this is a no-op.rN)rrr) statusrFINISHEDrpuses_block_sharingrv!mark_shareable_blocks_as_completerrinitial_tokensgenerated_tokens)ryrrrs rrz5PagedAttentionCache.mark_shareable_blocks_as_completes ! #u||}7M7M'M++ B$$##EE(;%'^^E4D4D%E % 4 4u7M7M MF r!list_source_blockslist_forked_blocksctj||jtj}tj||jtj}t |j |j D]y\}}|jd|j|j|j}|jd|j|j|j}||||<||||<{y)z;Copy the cache from the source blocks to the forked blocks.rJrN) rWrXrBint32rrcrdrlrNrr#)ryrr source_blocks forked_blocksrcrds r copy_cachezPagedAttentionCache.copy_caches %7 SXS^S^_  %7 SXS^S^_ &)$..$:J:J&K D "I{!r4??D>#6#6#8 9 :) )J   Z ( )r!)rN)'__name__ __module__ __qualname____doc__rOrWfloat16rr rBstrrdictrrErr[rboolrrrrrlistrTensorrrtuplerrrrrrrrrrr!rr?r?Ys6pO#]]S% S%%=S% s" S% . S% c3h S%{{S% S%j#QT&h#hY\haeh```   PS X[^bXb LL;>ll;> ;> & ;> %,,' ;> u||U\\) *;>z%c%c% ctCyS4|Z]bf"DT#YDDQTIDZ^DDsDsD$ 1c 1DQTI 1Z_`deh`ikopskt`tZu 1)r!r?c eZdZdZdZdZdededejde e de d d f d Z ed ee ee d fffd Zd e fdZd ee e ffdZde d zde d zded zd ee e ffdZdee d fde d zde d zded zd ee e ff dZde de d ee e ffdZdededed e fdZdee d fd ee d ffdZde de d e fdZy )rUuDetermines the optimal max batch tokens (M) and number of blocks (N) for the paged attention cache, given available GPU memory. The relation between N and number of blocks is: num_blocks = N // block_size. The memory footprint is a polynomial in M and N, where each term maps to a tensor allocated in ``ContinuousBatchingIOs._setup_static_tensors`` or ``PagedAttentionCache.__init__``: memory(M, N) = coeff_m · M + coeff_n · N + coeff_mn · M·N + coeff_mm · M² See ``_equation_coefficients`` for the breakdown. All three solving modes (auto, fixed-N, fixed-M) reduce to solving this equation, which is at most quadratic in one variable. i rrArEr<r9rNc||_||_||_||_|j|_t |t |z|_t||_ ||_ t|jrd|_ n d|vrdnd|_ |j|_ |j|j|_ |jrdnd|_|j"rdnd|_|j'|_y)avInitialize the memory handler. Args: - config: the model configuration - continuous_batching_config: the continuous batching configuration - dtype: the data type of the activation and the cache - group_types: the list of all attention group types, formatted as strings - group_size: the size (in layers) of an attention group rr)rKr N)rrH cache_dtypeactivation_dtyperNr%r page_sizer0rPr9r num_attention_masksrarbreturn_logprobsnum_output_rowsuse_async_batching io_multiplierget_available_memoryavailable_memory)ryrrArEr<r9s rrz$PagedAttentionMemoryHandler.__init__>s 3  %4??&v.1B61JJk*$ ( 4'(D $,?;,NqTUD $&@&W&W#  & & .*D*d*dD '$>$N$NqTU"<"O"OQUV $ 9 9 ;r!.c|jjt|jz}|j}i}|jj|j j z}||jjtjj zz }|dddf|d<|j j |jj|zd|zzz}d|z|j j z}||ddf|d<|S)Nrlm_headrK attention) rrr%rr$ritemsize vocab_sizerWfloat32)rymem_per_q_tokenmem_per_k_or_v_tokenpeaksdelta_mdelta_ns ractivation_peakz+PagedAttentionMemoryHandler.activation_peakes++99M$++NRY>Y=ZZ]^_r!c|jj}|jj}|||j||S||)|j ||d\}}|j||S|j ddd\}}t |j |}t||j}|j ||d\}}|j||S)aXInfers max_batch_tokens and num_blocks based on the available memory and the size of the activation peaks. If neither value is provided, we use a default value of 8192 for max_batch_tokens, apply bounds depending on the available VRAM, and solve for num_blocks. If one value is provided, the other is found using a linear solve.N)cache_fill_per_batchg?)r]r^r+) rHr]r^_check_footprint_solve_for_peaksr_default_max_batch_tokensr_min_max_batch_tokens)ryr]r^upper_bound_vramr4s rrVzAPagedAttentionMemoryHandler.infer_max_batch_tokens_and_num_blockss  >>::^^..   'J,B(()9:F F  ':+A+/+@+@ *4,A, ( j(()9:F F#33!!$4 ! t==?OP/1K1KL'+'<'<=Mzpt'<'u$*$$%5zBBr!r]r^r+c$g}|jjD],}|j||||\}}|j||f.t |Dcgc]}|d c}} t |Dcgc]}|d c}} | | fScc}wcc}w)aReturns max_batch_tokens and num_blocks so that their memory footprint is within the available memory for all activation peaks. If neither value is given, a value must be provided for cache_fill_per_batch: this means we solve for both varibles by saying each batch fill a certain percentage of the cache (eg, if cache_fill_per_batch is 0.01, each batch will fill 1% of the cache).rr )r%r/_solve_for_peakr2r) ryr]r^r+ solutions peak_deltasmnsolutionfinal_mfinal_ns rr-z,PagedAttentionMemoryHandler._solve_for_peakss //668 %K'' 5EzSghDAq   aV $ %9=xx{=>9=xx{=>>=s  B1 B peakc|j|\}}}}|k|i| td|} |j|| z|| dzzz||| zz|j } t | | z}t | |j z}||fS|B|} t |j|| zz || dzzz ||| zzz } | |j z}||fS|A||j z} t |j|||| zz|| z|jz }||fS)z~Returns a couple of `(max_batch_tokens, num_blocks)` that satisfy the memory constraint for the given activation peak.z>m must be provided if max_batch_tokens and num_blocks are NonerK)_equation_coefficientsr_solve_quadraticrr[rNr) ryr:r]r^r+rcncmncmmr5r_MNs rr2z+PagedAttentionMemoryHandler._solve_for_peaksX 66t<BS  # (:#+ !abb$A--cAgad .BBaKRVRgRgQghI"9q=1 Y4??:J ++  At44rAv=ad JrTWZ[T[|\]I"doo5J ++  %T__,A"4#8#8b37lBQRFUYUjUjLj#kl ++r!c|j||}||jkDrtd|d|j|dks|dkrtd|d|||fS)zDChecks if the footprint of the cache is within the available memory.zMemory footprint z is more than available memory rz#Invalid values: max_batch_tokens = z, num_blocks = )compute_memory_footprintr MemoryErrorr)ryr]r^memory_footprints rr,z,PagedAttentionMemoryHandler._check_footprints889I:V d33 3#$4#55TUYUjUjTkl  q J!OBCSBTTcdncopq q++r!arcc|dk(rt| |z S|dzd|z|zz }|dkrtd|d| t|zd|zz }|dkrtd|dtt|S)u_Largest positive root of a·x² + b·x + c = 0. Falls back to linear when a == 0. Rounded down.rrKr@z!No real solution (discriminant = )zNo positive solution (root = )r[rrr)ryrGrrH discriminantroots rr=z,PagedAttentionMemoryHandler._solve_quadratics 6rAv; !ta!eai' ! @aPQ QT,''AE2 !8r?r@rFs rrDz4PagedAttentionMemoryHandler.compute_memory_footprints   ( ((//1 OD#::4@ BC!AvQq1rds!! 3F9tt*6ZZ p. p3 p w* ws w%&6%5d3iRVWZR[A[;\%BR)R)jy$y$r!