^j: :ddlZddlmZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZddlm Z ddl!m"Z"dd l#m$Z$dd l%m&Z&dd l'm(Z(dd l)m*Z*dd l+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6ddl7m8Z8ddl9m:Z:ddl;mZ>ddl?m@Z@ddlAmBZBddlCmDZDddlEmFZFddlGmHZHddlImJZJddlKmLZLddlMmNZNddlOmPZPid e&d!e*d"e,d#e<d$e"d%eFd&eHd'e6d(e8d)e0d*e>d+e@d,e.d-e2d.eNd/e(d0ePeLe4e4e$eDeBeJe:d1ZQid ed!e d"e d)e d#ed$ed%ed&ed'ed(ed+ed,e d-e d*ed.ed/e d0eeeeeeeeed1ZReeee e eeefZSejeUZVGd2d3ZWGd4d5ZXd6eYfd7ZZd8eYfd9Z[d:Z\y);N) AutoConfig)logging) AqlmConfigAutoRoundConfig AwqConfigBitNetQuantConfigBitsAndBytesConfigCompressedTensorsConfig EetqConfigFbgemmFp8ConfigFineGrainedFP8ConfigFourOverSixConfig FPQuantConfigGemmaQuantizationConfig GPTQConfig HiggsConfig HqqConfig MetalConfig Mxfp4ConfigQuantizationConfigMixinQuantizationMethod QuantoConfig QuarkConfig SinqConfig SpQRConfig TorchAoConfig VptqConfig) HfQuantizer)AqlmHfQuantizer)AutoRoundQuantizer) AwqQuantizer)BitNetHfQuantizer)Bnb4BitHfQuantizer)Bnb8BitHfQuantizer)CompressedTensorsHfQuantizer)EetqHfQuantizer)FbgemmFp8HfQuantizer)FineGrainedFP8HfQuantizer)FourOverSixHfQuantizer)FPQuantHfQuantizer)GemmaQuantizer)GptqHfQuantizer)HiggsHfQuantizer)HqqHfQuantizer)MetalHfQuantizer)Mxfp4HfQuantizer)QuantoHfQuantizer)QuarkHfQuantizer)SinqHfQuantizer)SpQRHfQuantizer)TorchAoHfQuantizer)VptqHfQuantizerawqbitsandbytes_4bitbitsandbytes_8bitgptqaqlmquantoquark fouroversixfp_quanteetqhiggshqqzcompressed-tensors fbgemm_fp8torchaobitnetvptq)spqrfp8mxfp8z auto-roundmxfp4metalsinqgemmac6eZdZdZedefdZedZy)AutoQuantizationConfigz The Auto-HF quantization config class that takes care of automatically dispatching to the correct quantization config given a quantization config stored in a dictionary. quantization_config_dictc v|jd}|jdds|jddr*|jddrdnd}tj|z}n | td|tvr,td|d t t jt|}|j|S) N quant_method load_in_8bitF load_in_4bit_4bit_8bitThe model's quantization config from the arguments has no `quant_method` attribute. Make sure that the model has been correctly quantizedUnknown quantization type, got  - supported types are: ) getrBITS_AND_BYTES ValueError AUTO_QUANTIZATION_CONFIG_MAPPINGlistAUTO_QUANTIZER_MAPPINGkeys from_dict)clsrRrTsuffix target_clss g/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/transformers/quantizers/auto.pyrcz AutoQuantizationConfig.from_dicts/33NC # ' ' >BZB^B^_motBu 8 < <^U SWY`F-<>>( # #rhrQcreZdZdZedeezfdZedZedeezdedzfdZ e dZ y) AutoHfQuantizerz The Auto-HF quantizer class that takes care of automatically instantiating to the correct `HfQuantizer` given the `QuantizationConfig`. rjc zt|trtj|}|j}|t j k(r2t|ts td|jr|dz }n|dz }|tvr,td|dttjt|}||fi|S)NzZFound `quant_method=bitsandbytes` but `quantization_config` is not a `BitsAndBytesConfig`.rXrWrZr[) isinstancerwrQrcrTrr]r TypeErrorrUrar^r`rb)rdrjrprTrfs rg from_configzAutoHfQuantizer.from_configs )4 0"8"B"BCV"W *77  -<< <13EFp#//' ' 5 51,@/44678:  ,L9 -888rhc Ptj|fi|}|j|S)N)rQrlr})rdrorprjs rgrlzAutoHfQuantizer.from_pretraineds*4DDEbmflm233rhquantization_config_from_argsNc|d}nd}t|tr;t|trtj|}ntj|}|g|j j |j j k7r:td|j j d|j j dt|trgt|trW|j}|jD]\}}t||||r |dt|jdz }|dk7r2t|ttt fst#j$||St&j)||S)z handles situations where both quantization_config from args and quantization_config from model config are present. zYou passed `quantization_config` or equivalent parameters to `from_pretrained` but the model you're loading already has a `quantization_config` attribute. The `quantization_config` from the model will be used.zThe model is quantized with z but you are passing a z| config. Please make sure to pass the same quantization config class to `from_pretrained` with different loading attributes.z"However, loading attributes (e.g. z]) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.)r{rwrrcrQ __class__rrr^LOADING_ATTRIBUTES_CONFIG_TYPESget_loading_attributesitemssetattrr`rbrrrwarningswarnloggerinfo)rdrjr warning_msgloading_attr_dictattrvals rgmerge_quantization_configsz*AutoHfQuantizer.merge_quantization_configss ) 4y  K )4 07I&5&?&?@S&T#&<&F&FGZ&[# * 5#--66:W:a:a:j:jj./B/L/L/U/U.VVmoLoVoVo_o_n``FF  )+J KPZ )+JQ !> T T V .446 8 c+T37 8!!CDIZI_I_IaDbCcdA BB " Z0CkS^`tEu%v MM+ &#" KK $""rhc ^|jdd}|jdds|jddr*|jddrdnd}tj|z}n | td|tvr8t j d|d ttjd yy ) NrTrUFrVrWrXrYrZr[z~. Hence, we will skip the quantization. To remove the warning, you can delete the quantization_config attribute in config.jsonT) r\rr]r^r_rwarningr`rarb)rRrTres rgsupports_quant_methodz%AutoHfQuantizer.supports_quant_methods/33NDI # ' ' >BZB^B^_motBu 8 < <^U SWY`F-<.register_config_fn-sH 5 5xx/CDE E#67HI I36(0 rhrk)rrs` rgregister_quantization_configr*s rhnamecfd}|S)zRegister a custom quantizer.c~tvrtddt|ts t d|t<|S)Nz Quantizer 'rz!Quantizer must extend HfQuantizer)rar^rr r|)rdrs rgregister_quantizer_fnz1register_quantizer..register_quantizer_fn=sG ) ){4&0DEF F#{+?@ @'*t$ rhrk)rrs` rgregister_quantizerr:s ! rhcBt|ddxst|jddd}|du}|rtj|sd}|s|G|rtj |||_n||_tj |j |}nd}||j|||j|}|j|}|j|}t|j dds&|j j}t|d||d <|||fS) NrjT)decoderF) pre_quantized) device_map weights_only dequantizevaluequant) rmget_text_configryrrrjr}validate_environmentupdate_device_mapupdate_tp_planupdate_ep_planrT) configrjrr user_agentquantization_params_from_configr hf_quantizerrTs rgget_hf_quantizerrJsE&-f6KT&R'V]t,.CTW#44?M_BBCbc +7 )8)S)S/1D*F &*=F &&22  & &'3  ))!% * "33J? ,,V4,,V4|77uM';;HHL"),"NJw   ++rh)]rmodels.auto.configuration_autorutilsrutils.quantization_configrrrr r r r r rrrrrrrrrrrrrrrrrbaser quantizer_aqlmr!quantizer_auto_roundr" quantizer_awqr#quantizer_bitnetr$quantizer_bnb_4bitr%quantizer_bnb_8bitr&quantizer_compressed_tensorsr'quantizer_eetqr(quantizer_fbgemm_fp8r)quantizer_finegrained_fp8r*quantizer_fouroversixr+quantizer_fp_quantr,quantizer_gemmar-quantizer_gptqr.quantizer_higgsr/ quantizer_hqqr0quantizer_metalr1quantizer_mxfp4r2quantizer_quantor3quantizer_quarkr4quantizer_sinqr5quantizer_spqrr6quantizer_torchaor7quantizer_vptqr8rar_r get_loggerrrrrQrystrrrrrkrhrgrs76+4'/22F+6@92++-)--/-++1+ <++ O   O     )" O  >6&!  !" O#$  $'$    9>$ 9$+$+$ J $  J $  J $ l$ [$$$ $ 9$1$/$ [$}$  !$" J#$$   !!    $3$ :  #   H %&#&#Rll^   !S ! $,rh