^jx dZddlZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z ddl mZddlmZddlmZer2ddlZddlZddlZdd lmZmZmZmZmZdd lmZdd lmZdd l m!Z!d dl"m#Z#ejHe%Z&dZ'GddejPZ)GddZ*Gdde+Z,Gdde-Z.Gdde/Z0ddiddddddd id!d"ddd#d$d%d&d'id%d(d)d*d+d,d"d-Z1d.d/d0e2dzfd1Z3d2e2d0e2fd3Z4d4e2d0e5e2dzfd5Z6d6gd7d%d&d'id&d'id8d9d:d;Z7dd?iZ8d`d.d/d0e2dzfd@Z9dAe-dBe2d0e:e-e-dzffdCZ;dDe5e<d0e=fdEZ>dFez z<|im_start|>assistant tool_callsTjson)opencloserepeatscontent)defaults start_anchorfields)stcetcschemaz9[^>\n]+)>(?P.*?)arrayobjecttypestringz<[^>\n]+)>\s*(?P.*?)\s*)r@zx-regex-key-valuename arguments)r@ properties)zx-regex-iteratorr@items)) qwen2 qwen2_moeqwen2_vl qwen2_5_vlqwen3 qwen3_moe qwen3_nextqwen3_vl qwen3_vl_moe)qwen3_5 qwen3_5_moemodelr returnc t|d|}t|dd}t|dd}t|dd}t|dd}d}|r;|r9|r7d|jdivr#id|ddid }d D]}||vs||||<n`n^|r |r |r |d d}nO|jj t fd t j Dd} | y| d | d| d}}}|j|} |j|} || | dS)afReturn tool call config for the model, or ``None`` if tool calls are not supported. Returns a dict with: - ``schema`` (`dict`): Schema to pass to ``tokenizer.parse_response(block, schema)``. - ``stc_id`` (`int`): Token ID of the start-of-tool-call delimiter. - ``etc_id`` (`int`): Token ID of the end-of-tool-call delimiter. tokenizer stc_tokenN etc_tokenresponse_templateresponse_schemar2r:)r8r:)r9start_anchor_patternrEc32K|]\}}|vs |ywr%r).0typesv model_types r z'get_tool_call_config..s_xua:Y^K^_s r;r<r=)r=stc_idetc_id)getattrgetconfigr_next_TOOL_CALL_FALLBACKSrFconvert_tokens_to_ids) processorrRrUr;r<rXrYr= anchor_keyfallbackrarbr_s @r get_tool_call_configrlsR ; :I )[$ /C )[$ /C +>Ei):DAOF s(\=N=R=RS[]_=`-`#%6x%@%NO  C J..%6z%Bz"   .|<\\,, _+?+E+E+G_aef  #E?HUOXh=O&S  , ,S 1F  , ,S 1F& AAr tool_callc|jd|}|jdi}|dt|tstj|dS|dS)aNormalize a parsed tool call to ``{"name": str, "arguments": str}``. Different models return different structures from ``parse_response``: - Gemma: ``{"function": {"name": ..., "arguments": {...}}}`` (nested, arguments as dict) - Qwen: ``{"name": ..., "arguments": {...}}`` (flat, arguments as dict) The OpenAI API expects ``arguments`` as a JSON **string**, so we ``json.dumps`` it. functionrDrCrB)rd isinstancer)r3dumps)rmrorDs r _normalize_tool_callrrsX}}Z3H ["-I 2Parse tool calls from generated token IDs using ``tokenizer.parse_response``. Args: processor: The processor or tokenizer. generated_ids: Token IDs from generation. Passed directly to ``parse_response`` which decodes them internally, preserving special tokens that ``skip_special_tokens=True`` would strip (e.g. Gemma's ``<|tool_call>``). schema: The tool call schema (from ``response_schema`` or ``_TOOL_CALL_FALLBACKS``). Returns a list of ``{"name": str, "arguments": str}`` dicts, or ``None`` if none found. )prefixr2N)parse_responserpdictlistrr)ri generated_idsr=parsedrmr2s r parse_tool_callsr{sy % %mVB % GF&$LF$: %  fd #CIJi&y1JJJ#:--KsA"zz)thinkingr7zK(?:)?(?P.*?)(?P.*?)(?:<\|[^|<>\s]+\|>)?\Z)r@rEzx-regex)startendr=gemma4)z <|channel>thought z )r}r~c t|d| |jjj t fdt j Dt}|dDcgc]} j|}} j|d}t fd|Ds|d jfvryt dd}|rd|d vs td }|||d }|t|||d <|Scc}w) aReturn reasoning config for the model, or ``None`` if not supported. The config drives both streaming detection (token IDs) and post-hoc parsing (response schema). Returns a dict with: - ``start_ids`` (`list[int]`): Token ID sequence that opens a thinking block. - ``end_id`` (`int`): Token ID that closes the block. - ``schema`` (`dict`): Response schema with ``thinking`` / ``content`` properties for :func:`parse_reasoning`. - ``start_in_thinking`` (`bool`, only when ``input_ids`` is given): Whether the rendered prompt already opened an unclosed thinking block (prefilled by the template), so the model's output begins inside the block. rUc34K|]\}}|k(s |ywr%r)r\kr^r_s r r`z'get_reasoning_config..sCtq!1 ?Cs r}r~c3>K|]}|djfvywr%) unk_token_id)r\tidrUs r r`z'get_reasoning_config..s! FS34//0 0 FsNrYr|rEr=) start_idsend_idr=start_in_thinking) rcrer_lowerrf_THINKING_TOKENSrF_DEFAULT_THINKING_TOKENSrhanyr_starts_in_thinking) rirR input_idsthinking_tokenstrrr=rer_rUs @@r get_reasoning_configrs ; :I((..0JC'--/C O>MW=UV003VIV  , ,_U-C DF FI FF&UY[d[q[qTrJrY 14 8F zVL%99)(3!*fOF&9)Y&O"# MWs&C1r7reasoning_configc|j||d}|r(|jdd}|r|jdd|fS|jdrd|fS|dfS)uSplit generated output into ``(content, reasoning_content)`` via ``parse_response``. If the schema's regex matches (closing marker present), use it. For prompts that prefill the opener (QwQ-32B, DeepSeek-R1) the entire output is reasoning until ```` arrives — when that's truncated, fall back to treating all decoded text as reasoning. Returns ``(content, None)`` otherwise. r=r|rtr7rN)rvrd)riryr7rrz reasonings r parse_reasoningr%sm % %m5Eh5O PF JJz2. ::i,i7 7/07{ D=rrc t|dr|j}|r't|dtrt |dk7ry|d}t |}dD]/}t |||zk\st ||z }|||z ||k(s/yy)uyTrue if the rendered prompt ends with an unclosed thinking block. Some reasoning-model chat templates prefill the thinking opener as the final prompt tokens (e.g. DeepSeek-R1, QwQ-32B emit ``\n`` at the end when ``add_generation_prompt=True``). In those cases the model resumes *inside* the block, so its output contains only ``...reasoninganswer`` with no opening tag — the streamer must start with ``_inside_thinking=True``. The prefill always lands at the tail of the prompt (optionally followed by a single whitespace token like ``\n``), so we only inspect the last few tokens. tolistrrF)rrT)hasattrrrprxlen)rrntrailingr~s r rr9sy(#$$& Z ! d3 y>Q aL  IA y>Q\ )i.8+Cq3'94  rtoken_idcl|jy|jr||jk(rd|_yy|jt|j}||k7rg|_y|jj |t|jt|jk(rd|_g|_y)uUMutate ``streamer``'s thinking state; return ``True`` if ``token_id`` is a start or end token. Shared between :class:`DirectStreamer` and :class:`CBStreamer` — both track the same four attributes (``_thinking_start_ids``, ``_thinking_end_id``, ``_inside_thinking``, ``_thinking_prefix``) and need identical edge handling. FT)_thinking_start_ids_inside_thinking_thinking_end_idr_thinking_prefixappend)streamerrexpecteds r _advance_thinking_staterUs##+  x00 0(-H %++C0I0I,JKH8$&! $$X. 8 $ $%X-I-I)JJ$(!$&! rcpeZdZdZdedefdZdededzddfd Zded ededzddfd Z deddfd Z dd Z y)DownloadAggregatora Aggregates byte-progress across multiple concurrent download tqdm bars. huggingface_hub opens one tqdm bar per file shard. This class tracks them all and emits a single aggregate ``{"stage": "download", "progress": {...}}`` event whenever any updates. enqueuemodel_idc<||_||_i|_d|_yr%)rrRbarslast_emitted_current)r&rrs r r'zDownloadAggregator.__init__us  79 04!rbar_idtotalNrScFd|f|j|<|jy)z6Register a new download bar with its total byte count.rNr_emit)r&rrs r registerzDownloadAggregator.register{sJ & rcurrentcF||f|j|<|jy)z>Update a bar's current byte count and emit aggregate progress.Nr)r&rrrs r updatezDownloadAggregator.updates$e, & rcyr%r)r&rs r r5zDownloadAggregator.close rcTtd|jjD}||jk(ry||_|jjDcgc] \}}| | }}}|r t|nd}|j d|j d||ddycc}}w)Nc3&K|] \}}| ywr%r)r\c_s r r`z+DownloadAggregator._emit..s;1!;loadingdownloadrrstatusrRstageprogress)sumrvaluesrrrR)r& agg_currentrrtotals agg_totals r rzDownloadAggregator._emits; (8(8(:;; $33 3 $/! $ 0 0 2D1am!DD#)CKt  ##(3iH   Es  B$*B$rSN) rrrr(rr)r'intrrr5rrrr rrnsu 55C5 s3:$ S3sTzd  C D  rrcallbackrcNddlm}tGfdd|}|S)u Create a tqdm subclass that routes progress to a callback. Bars with ``unit="B"`` are download bars — aggregated via ``DownloadAggregator``. Other bars (e.g. "Loading weights") emit ``weights`` stage events. Args: callback (`callable`): Called with a dict payload ``{"status": "loading", "model": ..., "stage": ..., "progress": ...}``. model_id (`str`): The model ID (included in progress payloads). Returns: A tqdm subclass that forwards progress to *callback*. r)tqdmcLeZdZfdZdfd ZfdZfdZxZS).make_progress_tqdm_class..ProgressTqdmc |jdxsd|_d|d<t| |i|d|_d|_|jdk(r7t ||_j|j|jyy)NunititTdisablerB) rdsse_unitsuperr'r last_emittedid_bar_idrr)r&argskwargs __class__download_aggregators r r'z7make_progress_tqdm_class..ProgressTqdm.__init__sw"JJv.6$DM $F9  G d -f -DF "D }}#!$x #,,T\\4::F$rcX|d}|xj|z c_|jdk(r2j|j|j|jy|j|j k7r6|j|_dd|j|jddyyNrrrweightsrr)rrrrrr)r&rrrrs r rz5make_progress_tqdm_class..ProgressTqdm.updatesy FFaKF}}##**4<<L4,,,$(FF!"+!)!*04$L -rc 3K|jD]}|xjdz c_|jdk(r2j|j|j|j nN|j|j k7r5|j|_dd|j|j dd|ywr)iterablerrrrrr)r&itemrrrs r __iter__z7make_progress_tqdm_class..ProgressTqdm.__iter__s  ! ==C''..t||TVVTZZPVVt000(,D%&/%-%.48FFTZZ(P   sB;B>cv|jdk(rj|jt|y)Nr)rr5rr)r&rrs r r5z4make_progress_tqdm_class..ProgressTqdm.closes*}}##))$,,7 GMOr)r)rrrr'rrr5 __classcell__)rrrrs@r ProgressTqdmrs G " "  rr) tqdm.autorr)rr base_tqdmrrs`` @r make_progress_tqdm_classrs/,,Xx@00y0d rceZdZdZ ddddej dej dededzd edzf d Z dd Z dd Z dd Z y)DirectStreameraStreamer for ``model.generate()`` (used by :class:`GenerateManager`). Implements the ``put``/``end`` protocol that ``model.generate()`` expects: generate calls ``put(token_tensor)`` after each decode step, and ``end()`` when generation is complete. Tokens are decoded incrementally via the Rust ``DecodeStream`` (O(1) per token) and pushed as text to an asyncio.Queue. NrUtokenizers.Tokenizerloopqueueskip_special_tokens tool_configrcddlm}||_||_||_|g||_|r|dnd|_|r|dnd|_d|_|r|dnd|_ |r|dnd|_ t|xr|jd |_ g|_d |_t!j"|_d|_g|_y) a Args: tokenizer: The Rust tokenizer (``tokenizer._tokenizer``). loop (`asyncio.AbstractEventLoop`): The event loop to push decoded text to. queue (`asyncio.Queue`): The queue that receives decoded text chunks. skip_special_tokens (`bool`, *optional*, defaults to `True`): Whether to strip special tokens during decoding. tool_config (`dict`, *optional*): Tool call config from ``get_tool_call_config``. When set, tokens between stc/etc delimiters (inclusive) are suppressed from the queue so tool call markup is never streamed to the client. reasoning_config (`dict`, *optional*): Thinking config from ``get_reasoning_config``. When set, tokens between start/end delimiters are wrapped as :class:`ReasoningText` so handlers route them to ``reasoning_content``. r DecodeStreamraNrbFrrrT)tokenizers.decodersr _tokenizer_loop_queue_decode_stream_stc_id_etc_id_inside_tool_callrrboolrdrr_first threadingEvent _cancelled total_tokensgenerated_token_ids)r&rUrrrrrrs r r'zDirectStreamer.__init__s. 5#  *2/BC0;{8, 0;{8, !&DT#3K#@Z^ >N 0 :TX $%5%c:J:N:NOb:c d+- #//+.0 rc|jjr t|jrd|_y|j D]}|xj dz c_|j j|||jk(rd|_ n||jk(rd|_ t||}|jj|j|}||js||jk(s|r|jr t!|}|j"j%|j&j(|y)zHCalled by ``model.generate()`` after each decode step with new token(s).FNrT)ris_setr+rrrrrrrrrrsteprrr.rcall_soon_threadsafer put_nowait)r&valueris_start_or_end_tokentexts r putzDirectStreamer.puts ?? ! ! #&( ( ;;DK   JH    "   $ $ + +H 54<<')-&T\\)).&$;D($K !&&++DOOXFD|t55T\\9QUj$$$T* JJ + +DKK,B,BD I! Jrcd|jj|jjdy)z;Called by ``model.generate()`` when generation is complete.N)rr rr r&s r r~zDirectStreamer.end,s  '' (>(>Erc8|jjy)zWSignal cancellation. The next ``put()`` call will raise and abort ``model.generate()``.N)rsetrs r cancelzDirectStreamer.cancel0s r)TNN)r  torch.TensorrSNr) rrrr(asyncioAbstractEventLooprrrwr'rr~rrrr rrsu%)#'(,'1)'1'''1}} '1 " '1 D[ '1+'1RJ4FrrceZdZdZ ddddedddej d ejd edzd edzfd Z dd Z ddZ ddZ y) CBStreameraStreamer for continuous batching (used by :class:`CBGenerateManager`). Same ``put``/``end`` protocol as :class:`DirectStreamer`, but called manually by :class:`CBGenerateManager` instead of by ``model.generate()``: ``put(output)`` receives a CB ``GenerationOutput``, decodes new tokens, and pushes text to the asyncio.Queue. ``end()`` signals the stream is complete. N cb_managerr request_idrUrrrrrcnddlm}||_||_||_||_||_|gd|_|r|dnd|_|r|dnd|_ d|_ |r|dnd|_ |r|d nd|_ t|xr|jd |_g|_d|_d|_g|_y) aY Args: cb_manager (`ContinuousBatchingManager`): The CB manager instance. request_id (`str`): The request ID to track in the CB scheduler. tokenizer: The Rust tokenizer (``tokenizer._tokenizer``). loop (`asyncio.AbstractEventLoop`): The event loop to push decoded text to. queue (`asyncio.Queue`): The queue that receives decoded text chunks. tool_config (`dict`, *optional*): Tool call config (see ``DirectStreamer``). reasoning_config (`dict`, *optional*): Thinking config (see ``DirectStreamer``). rrTraNrbFrrr)rr_cb _request_idrrrrrrrrrrrdrr _prev_lenrr) r&rrrUrrrrrs r r'zCBStreamer.__init__>s( 5%  #*2t40;{8, 0;{8, !&DT#3K#@Z^ >N 0 :TX $%5%c:J:N:NOb:c d+-.0 rc:|j|jd}t|j|_|D]}|xjdz c_|jj |||j k(rd|_n||jk(rd|_t||}|jj|j|}||js||jk(s|r|jr t|}|jj!|y)zLDecode new tokens from a CB ``GenerationOutput`` and push text to the queue.NrTF)generated_tokensrrrrrrrrrrr rrr.rr )r&output new_tokensrr rs r rzCBStreamer.putes,,T^^-=> V445" )H    "   $ $ + +H 54<<')-&T\\)).&$;D($K !&&++DOOXFD|t55T\\9QUj$$$T* KK " "4 (! )rc:|jjdy)zSignal end of stream.N)rr rs r r~zCBStreamer.end{s t$rcN|jj|jy)zCancel the CB request.N)rcancel_requestrrs r rzCBStreamer.cancels  0 01rNN)r"rrSNr) rrrr(r)rrrrwr'rr~rrrr rr5sz$((,%1/%1%1* %1 '' %1 }} %1D[%1+%1N),%2rrseedc0ddl}|j|y)z8Set the PyTorch random seed for reproducible generation.rN)torch manual_seed)r(r*s r set_torch_seedr,sEdrcvddl}|jjr|jjyy)z+Empty the CUDA cache if a GPU is available.rN)r*cuda is_available empty_cache)r*s r reset_torch_cacher1s* zz   !rcJeZdZdZdZddZdefdZdej fdZ y) InferenceThreadzPersistent thread for ``model.generate()`` calls. ``torch.compile`` with CUDA graphs stores state in thread-local storage. All inference must run on the same thread to avoid corrupted graph state. ct|_tj|jd|_|j j y)NT)targetdaemon)rrrThread_run_threadr}rs r r'zInferenceThread.__init__s3"W  ''tyyF  rrSNcD |jj\}}}}} ||i|}||j|j|n|j|Z#t$r:}||j|j |n|j |Yd}~?d}~wwxYwr%)rrdr  set_result Exception set_exception)r&fnrrfuturerresultes r r8zInferenceThread._runs-1[[__-> *Bffd ,T,V,#--f.?.?H%%f- ,#--f.B.BAF((+  ,s8A B%0BBcZt}|jj||||df|S)ESubmit a callable to the inference thread. Returns a blocking Future.N)rrr)r&r>rrr?s r submitzInferenceThread.submits) T66489 rctj}|j}|jj |||||f|SzOSubmit a callable to the inference thread. Returns an awaitable asyncio.Future.)rget_running_loop create_futurerr)r&r>rrrr?s r async_submitzInferenceThread.async_submits>'')##% T66489 rr) rrrr(r'r8rrDrrIrrr r3r3s-  ,V 7>>rr3ceZdZdZddZe dddd d d eddd ed edzdedzdee jdffdZ eddd d d eddd edeee e e ff dZeddZy)BaseGenerateManageruBase class for generation managers. Subclasses: - :class:`GenerateManager` — sequential ``model.generate()`` on a persistent thread. - :class:`CBGenerateManager` — continuous batching with paged attention. rRr gen_configr rSNcy)z:Initialize continuous batching. No-op for non-CB managers.Nrr&rRrLs r init_cbzBaseGenerateManager.init_cbrri(ProcessorMixin | PreTrainedTokenizerFastinputsrrrzDirectStreamer | CBStreamercy)aStart streaming generation. Args: model (`PreTrainedModel`): The loaded model. processor: The processor or tokenizer for decoding. inputs (`dict`): Tokenized inputs (tensors for sequential, lists for CB). gen_config (`GenerationConfig`): Generation parameters. request_id (`str`): Unique request identifier. tool_config (`dict`, *optional*): Tool call config from ``get_tool_call_config``. When set, tool call tokens (between stc/etc) are suppressed from output. reasoning_config (`dict`, *optional*): Thinking config from ``get_reasoning_config``. When set, thinking tokens are wrapped as :class:`ReasoningText`. Returns: `tuple[asyncio.Queue, DirectStreamer | CBStreamer]`: A ``(queue, streamer)`` pair where *queue* yields ``str | _StreamError | None`` and *streamer* exposes ``.total_tokens`` and ``.cancel()``. Nr)r&rRrirRrLrrrs r generate_streamingz&BaseGenerateManager.generate_streamingrPrc Kyw)aRun generation to completion. Args: model (`PreTrainedModel`): The loaded model. processor: The processor or tokenizer for decoding. inputs (`dict`): Tokenized inputs (tensors for sequential, lists for CB). gen_config (`GenerationConfig`): Generation parameters. request_id (`str`): Unique request identifier. Returns: `tuple[str, int, list[int]]`: ``(text, input_len, generated_ids)``. Nr)r&rRrirRrLrs r generate_non_streamingz*BaseGenerateManager.generate_non_streamingsscy)z/Stop the generation manager and free resources.Nrrs r stopzBaseGenerateManager.stoprPrrRr rLr rSNr'r)rrrr(rOrrwr)tuplerrrTrrxrVrXrrr rKrKsI$((,   >   '    D[ +  w}};; <  :   >   '     sCc" #  *>>rrKceZdZdZdZ dddddded d d ed edzd edzdeeje ffdZ ddddded d d edeee dff dZ dedefdZdedejfdZddZy)GenerateManagerzFSequential generation via ``model.generate()`` on a persistent thread.c"t|_yr%)r3r9rs r r'zGenerateManager.__init__s &( rNrRr rirQrRrLr rrrrSc tj tj t|d|j}t | ||} i|| ||d t drd d<d  fd } |j|  | fS) zLStart streaming generation via ``model.generate()`` on the inference thread.rUrr)rgeneration_configrU has_talkerrgeneration_modec  jdiy#t$rjjdYyt$r8}jjt t |Yd}~yd}~wwxYw)Nr)generater+r r r<r"r))rA gen_kwargsrrRrs r r8z0GenerateManager.generate_streaming.._runsm R,,' B))%*:*:DA R))%*:*:LQcB|jj|g|i|S)rC)r9rDr&r>rrs r rDzGenerateManager.submit;s#"t||""27777rcB|jj|g|i|SrF)r9rIros r rIzGenerateManager.async_submit?s#(t||((=d=f==rcyr%rrs r rXzGenerateManager.stopCrrr'r)rrrr(r'rwr)rZrrrrTrrVrrrDrIrXrrr r\r\sP)$((, >  '   D[+ w}}n, -B. .>. . ' .  . sC' (.(88v8>x>W^^> rr\ceZdZdZdddZdd Zdefd Zd eddfd Z dddd dde ddd ede dzde dzde e jeffdZddd dde ddd ede eeeeff dZeddZddZy)CBGenerateManagerauContinuous batching generation via paged attention. Translates between the handler's text-level asyncio.Queue and CB's token-level interface. Per-request: ``max_new_tokens``, ``eos_token_id``. The CB manager is initialized lazily on the first request via :meth:`ensure_initialized`, using that request's ``gen_config`` for shared sampling params (temperature, top_p, do_sample). .. todo:: Remove :meth:`init_cb` when CB supports per-request generation config. At that point, ``gen_config`` can be passed directly to ``add_request`` and the CB manager no longer needs a shared config. Nc d|_||_yr%)r _cb_config)r& cb_configs r r'zCBGenerateManager.__init__Vs59#rrRr rLr rSc|jy|j||j|_|jjy)atInitialize the CB manager on first call with the request's generation config. .. todo:: Remove when CB supports per-request generation config. Args: model (`PreTrainedModel`): The loaded model (must support ``init_continuous_batching``). gen_config (`GenerationConfig`): Generation config used for shared sampling params. N)r`continuous_batching_config)rinit_continuous_batchingrur}rNs r rOzCBGenerateManager.init_cbZsA 88  11(T__2  rcR|jduxs|jjduS)zJWhether the CB worker is healthy. ``True`` before ``init_cb()`` is called.N)r fatal_errorrs r is_alivezCBGenerateManager.is_aliveks%xx4?488#7#74#??rrc|j<|jj%td|d|jjyy)uRaise :class:`CBWorkerDeadError` if the CB worker has died. Called at request entry to fail fast — submitting to a dead worker would otherwise enqueue the request into a void where it never gets processed. Nz,CB worker is dead and cannot accept request : )rr{r1)r&rs r _check_alivezCBGenerateManager._check_aliveosN 88 DHH$8$8$D#>zl"TXXMaMaLbc %E rrirQrRrrc  |j}| td|j|tj} tj |d} |j | |d|j|j}t|d|j} t|j|| | || fd} |j||  fS)zFStart streaming CB generation. Registers a per-request output handler.3CB manager not initialized. Call `init_cb()` first.rT)r streamingmax_new_tokens eos_token_idrUr_c` j||j5jt|jj y|j rj yy#t $r-}jtt|Yd}~yd}~wwxYwr%)rerrorr r"r~ is_finishedr<r))r"rAr text_queues r _on_outputz8CBGenerateManager.generate_streaming.._on_outputs < V$<<+)),v||*DELLN'')LLN* <%%l3q6&:;; ._on_results;;=!!&)!rF)rrrrzCB worker died during request r~zCB generation failed for Trh)rrrrrrGrHrrrrrr{r1r!rj)r&rRrirRrLrrrrmrrr@ryrr?s @r rVz(CBGenerateManager.generate_non_streamings5XX :TU U *%;'  N '')##% * "":z:  !%44#00   << #~~)'*H TVW]WcWcVd(eff!::,bWX X//  4HY --sB%D!(D)A7D!c|j|jj td|jjjS)z*The CB scheduler (for testing/monitoring).z.Continuous batching processor not initialized.)rbatch_processorr schedulerrs r rzCBGenerateManager.schedulers? 88 txx77?OP Pxx''111rcX|j|jjddyy)NT)blocktimeout)rrXrs r rXzCBGenerateManager.stops% 88  HHMMaM 0 rr%)rvContinuousBatchingConfig | NonerYr')rSrr)rrrr(r'rOrr|r)rrwrZrrrrTrrxrVpropertyrrXrrr rsrsGs  $"@$@ s t $$((,8$ 8$>8$ 8$ ' 8$  8$D[8$+8$ w}}j( )8$t/. /.>/. /. ' /.  /. sCc" #/.b22 1rrscjeZdZdZ ddededdfdZdd d ed efd Zdd eded e fdZ ddZ d efdZ y)GenerationStatea'Shared generation state across all handlers. Manages per-model :class:`GenerateManager` instances (each with its own :class:`InferenceThread` so different models can run concurrently while ``torch.compile`` / CUDA graphs require same-model-same-thread) and a single :class:`CBGenerateManager` for continuous batching. Args: continuous_batching (`bool`, *optional*, defaults to `False`): Whether to use continuous batching with paged attention instead of sequential ``model.generate()`` calls. Ncontinuous_batchingcompilervrcX||_||_||_i|_d|_d|_yr%)_continuous_batching_compileru_generate_managers _cb_manager _cb_model_id)r&rrrvs r r'zGenerationState.__init__s2 %8! #>@59(,rrRr modalityrSc|jsyt|dxr|tjk(}|s,tj |j jd|S)aWCheck if continuous batching can be used for this model and modality. Args: model (`PreTrainedModel`): The loaded model. modality (`Modality`): The detected model modality (LLM, VLM, etc.). Returns: `bool`: ``True`` if CB is enabled and the model supports it, ``False`` otherwise. FryzM does not support continuous batching. Falling back to sequential generation.)rrrrlogger warning_oncerr)r&rRrcans r use_continuous_batchingz'GenerationState.use_continuous_batching s]((e78UX=U   ??++,-99  rruse_cbcZ|rv|j|k7r-|j!|jjd|_|j"t|j|_||_|jS||j vrt |j |<|j |S)afReturn a per-model generation manager, lazily created on first request. Args: model_id (`str`): The model ID in ``'model_id@revision'`` format. use_cb (`bool`): Whether to return a CB manager or a sequential one. Returns: `BaseGenerateManager`: Either a `GenerateManager` or `CBGenerateManager`. N)rv)rrrXrsrurr\)r&rrs r get_managerzGenerationState.get_manager s   H,##/$$))+'+D$'#4t#O $,!## # 422 20?0AD # #H -&&x00rc`|j"|jjd|_yy)z$Stop any active generation managers.N)rrXrs r shutdownzGenerationState.shutdown7s-    '    ! ! ##D  (rcV|jduxs|jjS)zTWhether the CB worker is healthy. ``True`` if CB is disabled or not yet initialized.N)rr|rs r is_cb_alivezGenerationState.is_cb_alive=s'4'F4+;+;+D+D+FFr)FFNFr) rrrr(rr'rrr)rKrrrrrr rrs} %*7; -! - -5 -->(W[(1C11BU1.$ GTGrrc eZdZUdZdZedzed<eZee ed< dddde de dzfd Z d e d dfd Z ed dd e fdZd e d ee ddffdZ dd e ddded dfdZedee ded ee fdZy) BaseHandleraShared logic for chat completion and responses handlers. Provides model resolution, generation config building, and SSE formatting. Generation is delegated to the shared :class:`GenerationState`. Args: model_manager (`ModelManager`): Handles model loading, caching, and lifecycle. generation_state (`GenerationState`): Shared state managing per-model generation managers. N_valid_params_class_unused_fields model_managerrgeneration_statechat_template_kwargsc6||_||_|xsi|_yr%)rrr)r&rrrs r r'zBaseHandler.__init__Rs! + 0$8$>B!rbodyrScddlm}t|j}|j1|t |jdtz }|r |dd|||j z}|rtjd|yy) zMValidate request fields against the handler's params class and unused fields.r HTTPExceptionN__mutable_keys__iz"Unexpected fields in the request:  status_codedetailz,Ignoring unsupported fields in the request: ) fastapirrkeysrrcrrr)r&rr input_keys unexpectedunuseds r _validate_requestzBaseHandler._validate_request\s)%  # # /#gd.F.FHZ\_\a&bbJ#>`ak`l||jjk7r%|dd|jjd|d|jj|d<|jj |d}|jj |\}}|||fS) zfApply force_model, load model + processor. Returns ``(model_id, model, processor)``. rrrRizServer is pinned to 'z'; requested 'z'.r)rrr force_modelrdprocess_model_nameload_model_and_processor)r&rr requestedrrRris r _resolve_modelzBaseHandler._resolve_modelps *    ) ) 5)I$d6H6H6T6T)T# #3D4F4F4R4R3SSabkallno!..::DM%%88gG--FFxPy ))rmodel_generation_configr rcBddlm}|jd|d itj|d}n7t j |}|j|jdkrd|_|jd+t|d|_ t|ddk(rd|_ |jdt|d|_ |jd t|d |jjr|jd |_|rd|_|S) a'Build a GenerationConfig from shared params (temperature, top_p, seed, generation_config JSON). Subclasses should call ``super()._build_generation_config(...)`` then apply endpoint-specific params (``max_tokens``, ``max_output_tokens``, etc.). Args: body (`dict`): The raw request body. model_generation_config (`GenerationConfig`): The model's default generation config (will be deep-copied). use_cb (`bool`, *optional*, defaults to `False`): Whether continuous batching is active. If ``True``, disables the model's internal KV cache (CB manages its own paged cache). Returns: `GenerationConfig`: A new config with request-specific overrides applied. r)r r`i temperaturegFtop_pr(staticr) transformersr rdr3loadscopydeepcopyrfloatr do_samplerr,rrcache_implementation use_cache)r&rrrr r`s r _build_generation_configz$BaseHandler._build_generation_configs( 2 88' ( 4 0 Y4::dCV>W3X Y  $ .E F  //7;L;[;[^b;b37!0 88M " .,1$}2E,F  )T-()S0.3!+ 88G  (&+DM&:  # 88F  ' 4< (  ) ).?.T.T.\5=  2 */  '! rmessagesrcg}|D]v}|dgd}d|vrzg}|dD]k}tj|}|jdxs|}t|dtrt j |d|d<|j|m||d<d|vr|d|d<d|vrgn|jdxsg}t|trd|d g}|D]h} | d } | d vr|djd| dd (| d vrT|tjtjfvr2| d } t| tr| d} |djd| d| dk(re|tjk(rR| d} t| tr#| d} | jd}|rd|d| n| } n| } |djd| d| dk(rA|tjtjfvr|djd| ddd0| dk(s7|tjk(sL|djd| dddk|tjk(rdjd|dD|d<|j|y|S)a=Convert OpenAI-format messages to the format expected by HF processors. All modalities extract text. VLM additionally handles ``image_url`` and ``video_url``. MULTIMODAL handles all of the above plus ``input_audio`` and ``audio_url``. For LLMs, the content parts are collapsed into a plain text string. Args: messages (`list[dict]`): OpenAI-format chat messages. modality (`Modality`): The model modality (LLM, VLM, or MULTIMODAL). Returns: `list[dict]`: Processor-compatible messages. role)rr7r2rorD tool_call_idr7r)r@rr@)r input_text output_text) image_url input_imagerurlimage)r@r input_audiodataformatz data:audio/z;base64,audio video_urlvideo audio_url c3&K|] }|d yw)rNr)r\rs r r`zABaseHandler.get_processor_inputs_from_messages..s,R1QvY,Rr)rrrdrpr)r3rrrrrrwrjoin)rrprocessor_inputsmessagerzr2tcr> raw_contentr7 content_typerr audio_b64fmts r "get_processor_inputs_from_messagesz.BaseHandler.get_processor_inputs_from_messagess7 ,G%fo"=Fw& !,/*Br*B +1rB!"[/37*.**R _*E;%%b) * (2|$()0)@~&!- 7"gkk)>T>ZXZK+s+(. DE & d&v #HH9%,,fgfo-VW!%AAhS[S_S_aiatatRuFu!+.C!#t,!%j9%,,gc-JK"]2x8CVCV7V")-"8K!+t4$/$7 )ooh7HK C5 DQZ)9%,,gc-JK![0X(,,PXPcPcAd5d9%,,ggkFZ[`Fa-bc![0XATAT5T9%,,ggkFZ[`Fa-bc7 d<8<<'$'HH,Ry@Q,R$Ry!  # #F +o7 ,p rr%r)rrrr(rr@__annotations__rrr)rrwr'r staticmethodrrZrrrrxrrrrr rrBs (,+"uNCH$ -1 ?%?*?#Tk ? Yd Yt YG6G3GG *4*E#7HJt2t,u*,W\0!0!3E0!OS0! 0!dH T$ZH 8H X\]aXbH H rrr%r)Mr(rrenumr3rabcrrcollections.abcrconcurrent.futuresrrrtypingrtransformers.utilsr pydantic tokenizersr*rr r r r r:transformers.generation.continuous_batching.continuous_apir4transformers.generation.continuous_batching.requestsr5transformers.generation.continuous_batching.schedulerrrr get_loggerrr X_REQUEST_IDEnumrr"r<r+r)r.rr1rgrwrlrrrxr{rrrrZrrrrrrr@rrrr,r1r3rKr\rsrrrrr rs #$% &eUO+   H % tyyP9PC 05)+#%   ( \ #X. (-l"   !?1h(B+<(B(BVDT".t.T T@Q.6[ *) b , 7 M +<QUX\Q\DsdW\]`behlbl]lWm(d3iD82( ( VDxD3D4DNRRjL2L2^!&&RA>#A>HD )D Ng1+g1TNGNGb~ ~ r