^jQ#dZddlmZddlZddlmZddlZddZddZ ddZ ddZ d \Z Z dd Zdd Zdd Zdd Zee e e eeedZehdZddZej,dZddZddZd dZgdZy)!zThis file contains the parsers used by chat response parsing. Each parser takes a chunk of captured text and parses it into a single key in the output message dictionary. Functions are generally boilerplate.) annotationsN)AnycJ|jddr|jS|S)NstripT)getrtextargss z/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/utils/chat_parsing/content_parsers.py_textr s 88GT24::<<<c,tt||SN)intr rs r _intrs uT4 !!r c,tt||Sr)floatr rs r _floatr!s tT" ##r c:t||jdvS)N)true1)r lowerrs r _boolr%s t  " " $ 55r )c  |jdg}|jdd}|rt|vst|vr td|}g |D]`\}}t j |dzt j |z}d fd }t j |||tj}b|rt j dd |}t D]8\} } |jt| ttj| }: tj|S#tj$rS} |jd rt||cYd } ~ S||k(rtd |d | | td|d|d | | d } ~ wwxYw)aJSON parser with optional dialect knobs for LLM-emitted quirks. `args`: - `unquoted_keys` (bool): quote bare-identifier keys before parsing. - `string_delims` ([[open, close], ...]): strings delimited by these custom markers are pre-extracted, then restored as standard JSON strings. - `allow_non_json` (bool): return stripped text if parsing fails. string_delims unquoted_keysFzSjson: input contains reserved sentinel characters (\x01/\x02); cannot parse safely.z(.*?)c|j|jdttdz tS)N)appendgroup _LAX_OPENlen _LAX_CLOSE)mcaptureds r _capturez_json.._captureBs4 OOAGGAJ '[X!2 3J<@ @r flagsz(?<=[{,])(\w+):z"\1":allow_non_jsonNz5json parser could not parse region as JSON. Content: z Error: z:json: could not parse after dialect transforms. Original: z Transformed: )r&rreturnstr)rr#r% ValueErrorreescapesubDOTALL enumeratereplacejsondumpsloadsJSONDecodeErrorr ) r r rrworkingopen_dclose_dpatternr(iser's @r _jsonr@.sHH_b1MHH_e4M)t+zT/ApqqGH(F))F#h.71CC A&&(G299EF&&+Xw?(#P1//YKs:,"?AOP zz'""    88$ %t$ $ d?UVZU]]fghfijkqr r{)A3 H   s$ D!!F4FF,FFcd||St||jdd|jdiS)Nnamer r ) parse_contentr)raw value_parsers r _sub_parserF[s8 l..vv> @P@PQWY[@\ ]]r c|jd}| td|jd}|jdd}i}tj||tjD]}|j }|jd}|td|t |jd d |} ||vr3|r1t||ts ||g||<||j| | ||<|S) zParse shallow XML-ish tags into a dict. `tag_pattern` regex must have named groups `key` and `value`. Optional `value_parser` recurses; `merge_duplicates` collects duplicate keys into a list. tag_patternz1xml-inline: 'tag_pattern' content_arg is requiredrEmerge_duplicatesFr)keyz@xml-inline: tag_pattern must have a named group 'key'. Pattern: value) rr.r/finditerr2 groupdictrF isinstancelistr!) r r rHrEmergeoutr&groupsrJrKs r _xml_inlinerTas((=)KLMM88N+L HH' /EC [[d")) < jj ;_`k_lmn n6::gr2LA #:%c#h-H:C HOOE "CH  Jr c>|jdd}|jdd}|jd}i}|j|D]Q}t||}|r||vr|j|d\}}t||t||}}t||||<S|S)z7Parse line-delimited `keyvalue` pairs into a dict.line_sep kv_sep:rEr )rsplitr rF) r r rVrXrErRlinekvs r _kv_linesr^{sxx D)H XXh $F88N+LC 8$-T4 vT) zz&!$1Q~uQ~1A|,A - Jr )r rrboolr5z xml-inlinezkv-lines>rr_r rc t|||Sr)CONTENT_PARSERS)r rBr s r rCrCs 4 t ,,r z \{(\w+)\}c t|tr.|jDcic]\}}|t||c}}St|tr|Dcgc]}t||c}St|t s|St j|}|s|S|jd}||vrtd|dt|||Scc}}wcc}w)ztRecursively walk a transform template, which is used to restructure parsed output into the actual shape we want.r ztransform placeholder '{z}' is not defined. Available: ) rOdictitems_apply_transformrPr- _PLACEHOLDER fullmatchr"KeyErrorsorted) transformscoper\r]wholerJs r reres)T":C//:KL$!Q#Au--LL)T"4=>q E*>> i %  " "9 -E  ++a.C %23%7VW]^cWdVefgg :M>s CCcJt|tr"|jD]}t||yt|tr|D]}t||yt|t syt j|r't j|st|d|dyy)a$Walk a transform template and reject any string that mixes a `{name}` placeholder with literal text. Only whole-string placeholders (e.g. `"{content}"`) and plain literals are supported. Called from the template loader so authors get a clear error at load time, not at parse time.Nz: transform string z mixes a {placeholder} with literal text. Use either a whole-string placeholder (e.g. "{content}") or a plain literal; string interpolation is not supported.) rOrcvaluesvalidate_transform_stringsrPr-rfsearchrgr.)rkrjr]s r roros )T"!!# 1A &ua 0 1)T" 1A &ua 0 1 i %9%l.D.DY.Og( 65 5  /P%r c t||j|j}|j|S|jrt |t s/td|jdt|jdg}|D]j}t |ts/td|jdt|jd|jt|ji||l|St|ji|d|iS)a~Run `body` through the field's content parser, then optionally apply the transform template. When `transform_each` is set, the parsed content must be a list and the template is applied to each element (with the element's keys unpacked into the template scope, alongside any regex captures). `field` is a `spec.Field`; typed via duck-typing to avoid a cyclic import.zField 'z@': transform_each requires the parsed content to be a list, got .z?': transform_each requires each list element to be a dict, got content)rCrs content_argsrjtransform_eachrOrPr.rBtype__name__rcr!re)bodyfieldcapturesrKrRitems r process_fieldr|s $ u/A/A BE   %&%**&E{++,A/  PDdD) ejj\*:../q2 JJ'9MH9M9MN O  P EOO-K-K)U-K LLr )raSTREAMABLE_PARSERSrCr|ro)r r-r rcr,r-)r r-r rcr,r)r r-r rcr,r)r r-r rcr,r_)r r-r rcr,r)rDr-rEz dict | Noner,r)r r-r rcr,rc)r r-rBr-r rcr,r)rjrrkrcr,r)rkr-rjrr,None)rxr-rzrcr,r)__doc__ __future__rr5typingrregexr/r rrrr#r%r@rFrTr^ra frozensetr}rCcompilerfreror|__all__r r rsj# ="$6 ' :*Z^ 4$     ?@-rzz,' $ .M8 r