^j^ddlZddlZddlZddlZddlmZddlmZddlZddl Z ddl m cm Z ddl mZddlmZddlmZmZmZddlmZddlmZmZmZmZmZmZmZdd lm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&ejNe(Z)d e jTjVde,ede-defdZ.Gddej^Z0GddeZ1y)N)Callable)Any)counters)ir)SubgraphCPUBenchmarkRequestSubgraphGPUBenchmarkRequest TensorMeta)KernelTemplate)Buffer FixedLayoutget_free_symbolsget_symbolic_inputsgm_original_output_stridesir_node_to_tensorLayout) benchmarker)do_bench_using_profilingV) OrderedSetgminputsnamereturncddlm}tjj} |tj_||||tj_S#|tj_wxYw)a.Inline a subgraph by converting its FX operations to individual IR nodes. This converts a subgraph to multiple ComputedBuffer nodes (fusable), enabling epilogue fusion with subsequent operations. Returns: TensorBox containing the final operation result as individual IR nodes r)process_subgraph_nodes)torch._inductor.loweringrrgraphmodule)rrrroriginal_modules k/var/www/ramen.bs-engineer-server.com/venv/lib/python3.12/site-packages/torch/_inductor/codegen/subgraph.pyinline_subgraph_to_ir_nodesr"&sH@ggnnO)%b&1(s AA,ceZdZdZ ddedeedededede fd e e ee ge jffdzd dffd Zd ee fd Zd ede fde ee fd dfdZd efdZd e fdZd eezfdZed eezfdZddZdee de jd efdZdee de jd dfdZd efdZd ej<fdZd e ee ffdZ d efdZ!xZ"S)SubgraphChoiceCallerz Represents a Subgraph Autotuning choice, and the subgraph can be any arbitrary GraphModule. Compiles the Subgraph down to a module for benchmarking. Nr input_nodeslayout description make_fx_graph. input_gen_fnsrc t |||||g}g|_tj5t |j D]\}} tt| jddk(sJtt| jddk(sJ| jj|jt| |)||vr%|jj||| |jjt| d ddd|||_t!|j||_t%|j |_|j)|_d|_i|_i|_d|_d|_t6j8rDtj5|j;|_|j=|_dddyy#1swYxYw#1swYyxYw)NT) unbacked_onlyr)replace_symbols_with_hints)super__init__benchmark_inputsr fake_mode enumerater%lenr get_size get_stridedata freeze_layoutappendrrrexample_inputsr sym_inputs_compute_sym_input_valuessym_input_values decompositiondecomposition_kwargsconfig_patches_compiled_module_bmreqconfigpipeline_max_autotune_gemm_compile_for_benchmarking_create_benchmark_request) selfrr%r&r'r(r) trace_inputsiinp __class__s r!r.zSubgraphChoiceCaller.__init__Bs {FK@ " [[ #D$4$45 3+CLLN$OPTUUUU+CNN,.>? $ > > @9=46!.0%)   , , ?(,(F(F(H%"<<>  ? ? -U  V ? ?sC5G8+H8HH c ht|jDcgc]}t|ds|jc}}i}t |j |j D]\}}t|tjs!t |j|jD]]\}}t|tjrt|||j<9t||vsGt||t|<_g}|jD]} t| tjr-| j|vr|j!|| jJt"j$j&j(j+| d} |j!t| |Scc}w)a)Extract concrete dimension values for sym_inputs from benchmark_inputs. The compiled module expects symbolic dimension values as runtime arguments. This maps each symbolic variable to its concrete value from the benchmark tensors. Used for range based autotuning. r)fallback)rr9hasattrrzipr%r/ isinstancetorchTensorr3shapesympySymbolintstrr7rrsizevars shape_envoptimization_hint) rEssym_input_namessym_name_to_valueinp_node benchmark_inpsym_dim actual_dimresultsym_varhints r!r:z.SubgraphChoiceCaller._compute_sym_input_valuessj%!__ C60BQVV C  -/'*4+;+;T=R=R'S J #Hm-6+.%%')<)<,J'GZ"'5<<8:=j/)',,7W8:=j/)#g,7 J J )G'5<<0W\\EV5V / =>ww''11CCGVWCX c$i(  )  - Ds F/F/r<kwargsc ||_||_y)zHCache decomposition function and kwargs for range-based dispatch lookup.N)r<r=)rEr<rds r!cache_decompositionz(SubgraphChoiceCaller.cache_decompositions+$*!c"d|jdS)NzSubgraphCaller()rrEs r!__str__zSubgraphChoiceCaller.__str__s  1--rgc @ddlm}|jjddjdd}|jr |j n |j }tjd|j|j|jJ||j|tjjtjjtjjtjjtjj tjj"d| }|jD]@}||j$|j<|j&j)|jBtj*|5d d d d d d |j,}t/j0|5|j2||j5cdddcdddS#1swYnxYw dddy#1swYyxYw) zCCompile the subgraph for benchmarking, returns the compiled module.r) GraphLoweringz::_.z#Benchmark compile %s: sym_inputs=%sN benchmark_) rr8rX cpp_wrapperaot_modeextern_node_serializer is_inference is_backwardrFATEN) max_autotunemax_autotune_gemmmax_autotune_gemm_backendsbenchmark_fusionrB)torch._inductor.graphrnrreplacer9r8r/logdebugrrr _shape_envrrrsrtrurv graph_inputsgraph_input_namesr7set_graph_handlerr>rApatchruncompile_to_module)rErn safe_namecompile_inputsbm_graph_loweringsym_inpbenchmark_configs r!rCz.SubgraphChoiceCaller._compile_for_benchmarkings7II%%dC088cB $(??D  8M8M  7DOOTww""")ww)gg((++WW%%#$77#A#A--++i[)   EG;B  * *7<< 8  / / 6 6w|| D E !2 3 =!&%*.4$).3 0 %% 0 ./ =%!%%~6(::< = = = = = = = = = =s$"*H G>+ H>H HHc |jJdtj|j}tj|j}|jj j dk(rt}nt}||j||t|jj|jj|jS)z0Create a benchmark request for async autotuning.z9Module must be compiled before creating benchmark requestcpu) kernel_nameinput_tensor_metaoutput_tensor_meta extra_args module_pathmodule_cache_keyr;)r?r from_irnodesr%r&devicetyperrrtuple__file__keyr;)rErr bmreq_clss r!rDz.SubgraphChoiceCaller._create_benchmark_requests$$0 G 0'33D4D4DE'44T[[A ;;   " "e +3I3I /1w--66!2266!22  rgc@|jJd|jS)z`Benchmark request for async autotuning. Pre-compiled when pipeline_max_autotune_gemm is enabled.zMbmreq accessed but pipeline_max_autotune_gemm was not enabled during __init__)r@rks r!bmreqzSubgraphChoiceCaller.bmreqs* {{& [ &{{rgcH|j|j|_yy)zKEnsure the module is compiled. Used for lazy compilation in non-async path.N)r?rCrks r!_ensure_compiledz%SubgraphChoiceCaller._ensure_compileds$  ($($B$B$DD ! )rgargsoutcX|j|jj|jdtffd }|j rt j|Stjr t|St j|t jgS)z>Regular benchmarking: compile if needed, then use benchmarker.rcgSN)rbm_funcr9sr!fnz*SubgraphChoiceCaller.benchmark..fn s/Z/$/0 0rg)r) rr?callr;r_benchmark_with_cudagraphsrbenchmark_gpu_with_cuda_graphrA/profile_bandwidth_with_do_bench_using_profilingr benchmark infer_device)rErrrrr9s ` @@r!rzSubgraphChoiceCaller.benchmarks '',,**  1C 1  * *<) rr TensorBoxcreateSubgraphBufferr&r%r8rr>rks r! output_nodez SubgraphChoiceCaller.output_node'sww"""||""   {{ ,,77#22"ii6:6I6It22    PT    rgc d|jdS)zRInformation returned here is logged to the autotune log file when that is enabled.subgraph)backendrrjrks r! info_dictzSubgraphChoiceCaller.info_dict4s"99  rgc d|jS)N subgraph_rjrks r!autoheuristic_idz%SubgraphChoiceCaller.autoheuristic_id;s499+&&rgr)rN)#__name__ __module__ __qualname____doc__rVlistr rrrdictrUrPrQr.r:rfrlrCrrrDpropertyrrfloatrrrrrrrr __classcell__rIs@r!r$r$<sJN@?@?&\@? @?  @?  S) @?C3%*=!>>?$F@? @?D49@+%c3h/+9=c3h+ +..,=3,=\ $'B B 2 $'B BE  tCy u||  &D$s)D%,,D4D  #    R\\   4S> '#'rgr$cZeZdZdZej Zdeffd Z ddede e de de de fd ed eee e gej"ffdzd e d efd Z ddede e de fde e de eee fde de fdzd eee e gej"ffdzde eee fdzd e efdZde de fd eee fd efdZd eee fd dfdZdede e de fde e d dfdZ dde e de de fd eee fde de fdzd eee e gej"ffdzd e f dZxZS)SubgraphTemplatea  A template for subgraph evaluation to be used in autotuning. This class allows creating customized subgraphs that can be appended as choices during the autotuning process, enabling the selection of optimal implementations for complex operations. rc&t||y)z Initialize a subgraph template. Args: name: The name of this template graph: The FX graph rjN)r-r.)rErrIs r!r.zSubgraphTemplate.__init__Js d#rgNr%r&r(.r'r)rdrc \t|dttj|||||S)a Generate a SubgraphChoiceCaller instance for autotuning. Args: name: The name for this subgraph choice input_nodes: List of input nodes to the subgraph layout: Memory layout information for the output make_fx_graph: Callable that creates the FX graph for this subgraph description: Optional description of this choice input_gen_fns: Optional dict mapping input indices to tensor generators **kwargs: Additional keyword arguments Returns: SubgraphChoiceCaller: A callable object that can be used for autotuning ro)rr%r&r'r(r))r$nextr index_counter)rErr%r&r(r'r)rds r!generatezSubgraphTemplate.generateWs;4$64 0 > >?@A##''   rgdecompositionsnon_tensor_args default_implconfig_patches_listc 8|sgSt|t|k(s Jdt|dt|d||Dcgc]}i}}t||D cgc]\} } |j|| | ||} } } |j||| | d} g} t|||D]\} }}ddlm}| |ddt dtd t fd ttt fd t fd }|j| |} |j|d ||| |d| j|}|j!| |||_| j%|| Scc}wcc} } w#|$r7tjd| jtddxxdz cc<YwxYw)a Generate multiple SubgraphChoiceCaller instances for custom op autotuning. This method extends SubgraphTemplate to support custom op decompositions, allowing multiple implementations to compete in autotuning. Args: name: Base name for the choices decompositions: List of decomposition functions to compete in autotuning input_nodes: List of tensor inputs. All tensor arguments must be passed here. non_tensor_args: List of non-tensor kwargs only, one dict per corresponding decomposition. default_impl: Default implementation for layout inference input_gen_fns: Optional dict mapping input indices to tensor generators config_patches_list: Optional list of config patches per decomposition Returns: List of SubgraphChoiceCaller instances for autotuning z>decompositions and non_tensor_args must have same length, got z decompositions and z kwargsr)_ShapeEnvGuardError)decomp decomp_kwargsrr.rrc ddlm}ddlm}|}tj j }||jntj}|5|tj|fi||d|cdddS#1swYyxYw)Nr)make_fx)select_decomp_tablesymbolic)decomposition_table tracing_mode) "torch.fx.experimental.proxy_tensorrr<rrr0rXerror_on_new_guards contextlib nullcontext functoolspartial)rrrrrrrX guard_ctxs r!r(zBSubgraphTemplate.generate_custom_op_choices..make_fx_graphsG?&9&;#KK11 !,113#//1 7!))&BMB,?%/ s "BB roz CustomOp )rr%r&r(r'r)z5Skipping decomposition %s: adds guards during tracinginductorcustom_op_decomp_guard_skipsrK)r2rN_infer_custom_op_layout_validate_layout_equivalence%torch.fx.experimental.symbolic_shapesrrrrrV_generate_variant_namerrr~inforrfr>r7)rErrr%rrr)rrorrdlayoutsr&choicesrr>rr( variant_namechoices r!generate_custom_op_choicesz+SubgraphTemplate.generate_custom_op_choiceszs8I>"c/&:: ~&'';C!2"> ">#&no"F    ( (VV\=    ))$H.058 O-@6 ; # 1FM> R.40=   c*  $CH~   8 66v}ML  6<.1 +!"/"+FOO+< ="/ '"  & &v} =$2F ! NN6 "w; #zY#? v' KOO$%CDID  s EE;)E9FFrcddl|j}|s|Sdtdtffd dj fdt |j D}|d|S)zLGenerate a descriptive name for a decomposition variant with its parameters.rNvrcxt|}jdd|}|r|djrd|z}|S)z7Convert a value to a valid Python identifier component.z [^a-zA-Z0-9_]ror)rVsubisdigit)rrZres r!sanitize_valuez?SubgraphTemplate._generate_variant_name..sanitize_values<AA'a0AQqT\\^!GHrgroc3>K|]\}}|d|yw)roNr).0krrs r! z:SubgraphTemplate._generate_variant_name..s+ +/1aqc>!$% & s)rrrrVrsorteditems)rErrd base_name param_suffixrrs @@r!rz'SubgraphTemplate._generate_variant_namesh OO   c c xx 39&,,.3I  Al^,,rgc|jD]<\}}t|tjtfs'Jd|dt |dy)z8Validate that kwargs contains only non-tensor arguments.zkwargs['z'] contains tensor zo. Tensor arguments should be in input_nodes, not kwargs. Only scalar/non-tensor parameters should be in kwargs.N)rrOrPrQr r)rErdrvalues r!_validate_non_tensor_kwargsz,SubgraphTemplate._validate_non_tensor_kwargssZ ,,. JC!%%,,)?@ 3%24;-@IJ @ rgop_namerc,|sy|d}t|dddD]\}}|j|j|j|jf|j|j|j|jfk7sdt d|d||j d|jd|jd|jd|jd |dj d|jd|jd|jd|jd y) zXEnsure all layouts have consistent stride, device, dtype, and sizes for fair autotuning.NrrK)startzLayout mismatch in custom op 'z': decomposition 'z ' produces (z, z) but 'ri)r1rdtypesizestrideAssertionErrorr)rErrr referencerGr&s r!rz-SubgraphTemplate._validate_layout_equivalences( AJ "712;a8 IAv v||V[[&--H    M %4WI>&&4Q&7&@&@%AB bb R W*1-6678!(()IOO+NbQZQaQaPbbc e rgfunction_decompositionc ,ddl}ddlm}|j||j5g}t |D]\} } |r| |vr || | } n| j } |jjj| } | j}|jjj|}tj| || j| j} |j| |j |fi|}||}t#|tj$sJdt'|dt)|j*|j,|j.|j1cdddS#1swYyxYw)zInfer output layout for custom ops using the default implementation when available. Note that the Subgraph assumes custom ops return exactly one tensor output. TODO: Add support for multiple output custom ops. rNr)r rz#Expected single tensor output, got z:. Multi-output custom ops not yet supported in autotuning.)rr r r )rtorch._inductor.virtualizedrrr0r1r3rrWoptimization_hintsr4rP empty_strided get_dtype get_devicer7rrOrQrr rr rRr )rEr%rrdrr)rrr8rGrH fake_tensor raw_shapeconcrete_shape raw_strideconcrete_strideroutputs r!rz(SubgraphTemplate._infer_custom_op_layout*ss 1 ((0 [[ N#K0 33 Q-%7"2-"23"7K # I%&WW%5%5%H%H%SN!$!1J&'gg&6&6&I&I*&UO"'"5"5&'!mmo"~~/ #K %%k2 3 #""#9DVDB(Ffell3 5d6l^DKL 3 }}ll\\}} 7   s EF  F)N)NNN)NN)rrrr itertoolscountrrVr.rr rrrrrUrPrQr$rrrrrrrrs@r!rr?s$IOO%M $ $&IM! ! &\!  !  S) !  ! C3%*=!>>?$F! !  ! R37IM;?rrXc3h/0r&\ r d38n- r sCx(4/ rC3%*=!>>?$Fr"$sCx.1D8r " #rh-sCx(-26sCx.- -2$sCx.TXc3h/0f   >37IM 3&\3!)c 23S#X 3 sCx(4/ 3 C3%*=!>>?$F 3 3rgr)2rrrloggingcollections.abcrtypingrrSrPtorch._inductor.config _inductorrAtorch._dynamo.utilsrtorch._inductorr torch._inductor.autotune_processrrr torch._inductor.codegen.commonr torch._inductor.irr r r rrrr$torch._inductor.runtime.benchmarkingrtorch._inductor.utilsrrrtorch.utils._ordered_setr getLoggerrr~fx GraphModulerrVr" ChoiceCallerr$rrrgr!r/s$ ''( :=:)/g!) )&*3i)7:)),@'2??@'F^~^rg