跳过的函数#
创建日期：2025 年 7 月 28 日 | 最后更新日期：2025 年 7 月 29 日
摘要
有时，
torch.compile会完全放弃编译某个函数，而是改为立即执行它，这可能导致优化机会的丢失。有一些方法可以规避跳过的函数，以便在有问题的代码周围重新启用跟踪。
有时，torch.compile 配合 fullgraph=False 在遇到图中断或其他编译器错误时，无法恢复跟踪。在许多此类情况下，torch.compile 将完全跳过编译该函数并立即执行它。
请注意，跳过仅应用于当前函数，而不应用于任何嵌套函数调用。torch.compile 仍会尝试编译嵌套调用。
def inner1(x):
return x + 1
def inner2(x):
return x + 2
@torch.compile
def fn(x):
x = inner1(x)
torch._dynamo.skip_frame()
x = inner2(x)
fn(torch.randn(3))
ChromiumEventLogger initialized with id f1893c87-5fd6-4b16-b03d-bbdfdad40be9
torchdynamo start compiling fn /tmp/ipykernel_1130/2126697152.py:5, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/2126697152.py", line 10, in <module>
fn(torch.randn(3))
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/2126697152.py:5
create_env
TRACE starts_line /tmp/ipykernel_1130/2126697152.py:7 in fn
x = inner1(x)
TRACE LOAD_GLOBAL inner1 []
TRACE LOAD_FAST x [LazyVariableTracker()]
TRACE CALL_FUNCTION 1 [LazyVariableTracker(), LazyVariableTracker()]
INLINING <code object inner1 at 0x7f06d8074030, file "/tmp/ipykernel_1130/2126697152.py", line 1>, inlined according trace_rules.lookup inlined by default
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE starts_line /tmp/ipykernel_1130/2126697152.py:2 in inner1 (inline depth: 1)
return x + 1
TRACE LOAD_FAST x []
TRACE LOAD_CONST 1 [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]
TRACE RETURN_VALUE None [TensorVariable()]
DONE INLINING <code object inner1 at 0x7f06d8074030, file "/tmp/ipykernel_1130/2126697152.py", line 1>
TRACE STORE_FAST x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/2126697152.py:8 in fn
torch._dynamo.skip_frame()
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]
TRACE LOAD_ATTR skip_frame [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
Skipping frame Skip frame due to `torch._dynamo.skip_frame()`. Message: None fn /tmp/ipykernel_1130/2126697152.py 5
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
torchdynamo start compiling inner1 /tmp/ipykernel_1130/2126697152.py:1, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/2126697152.py", line 10, in <module>
fn(torch.randn(3))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
return fn(*args, **kwargs)
Step 1: torchdynamo start tracing inner1 /tmp/ipykernel_1130/2126697152.py:1
create_env
TRACE starts_line /tmp/ipykernel_1130/2126697152.py:2 in inner1
return x + 1
TRACE LOAD_FAST x []
TRACE LOAD_CONST 1 [LazyVariableTracker()]
TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE RETURN_VALUE None [TensorVariable()]
Step 1: torchdynamo done tracing inner1 (RETURN_VALUE)
RETURN_VALUE triggered compile
COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_1130/2126697152.py, line 2 in inner1>], graph_break=False)
TRACED GRAPH
===== __compiled_fn_2_315fb0aa_568d_461e_8ece_e93221874b7a =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# File: /tmp/ipykernel_1130/2126697152.py:2 in inner1, code: return x + 1
add: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None
return (add,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # return x + 1 # mp/ipykernel_1130/2126697152.py:2 in inner1
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # return x + 1 # mp/ipykernel_1130/2126697152.py:2 in inner1
Guard eval latency = 766.36 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
skipping: inner (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_compile.py)
skipping: disable (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/decorators.py)
skipping: innermost_fn (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)
skipping: __init__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)
skipping: __init__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)
skipping: nothing (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)
skipping: __call__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)
skipping: _fn (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py)
skipping: skip_frame (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/decorators.py)
torchdynamo start compiling inner2 /tmp/ipykernel_1130/2126697152.py:3, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/2126697152.py", line 10, in <module>
fn(torch.randn(3))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
return fn(*args, **kwargs)
Step 1: torchdynamo start tracing inner2 /tmp/ipykernel_1130/2126697152.py:3
create_env
TRACE starts_line /tmp/ipykernel_1130/2126697152.py:4 in inner2
return x + 2
TRACE LOAD_FAST x []
TRACE LOAD_CONST 2 [LazyVariableTracker()]
TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 2)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE RETURN_VALUE None [TensorVariable()]
Step 1: torchdynamo done tracing inner2 (RETURN_VALUE)
RETURN_VALUE triggered compile
COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_1130/2126697152.py, line 4 in inner2>], graph_break=False)
TRACED GRAPH
===== __compiled_fn_4_d65a3269_1392_4a3f_b227_2a5402485c26 =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# File: /tmp/ipykernel_1130/2126697152.py:4 in inner2, code: return x + 2
add: "f32[3][1]cpu" = l_x_ + 2; l_x_ = None
return (add,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # return x + 2 # mp/ipykernel_1130/2126697152.py:4 in inner2
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # return x + 2 # mp/ipykernel_1130/2126697152.py:4 in inner2
Guard eval latency = 94.20 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
skipping: remove (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/weak.py)
skipping: __hash__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/weak.py)
在上面的示例中，torch.compile 将跟踪 fn（包括 inner1），直到 skip_frame。然后 fn 被跳过并立即执行 — inner1 和 inner2 在被调用时会被编译。
跳过函数可能会导致优化机会的丢失，因此检查您想要编译的代码是否被跳过非常重要，如果被跳过，则应设法规避跳过。
循环中的图中断#
torch.compile 在循环中发生图中断时无法恢复跟踪
@torch.compile
def fn(x):
for i in range(5):
x = x + 1
if i == 3:
torch._dynamo.graph_break()
return x
fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_1130/2044822433.py:1, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/2044822433.py", line 8, in <module>
fn(torch.randn(3))
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/2044822433.py:1
create_env
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:3 in fn
for i in range(5):
TRACE LOAD_GLOBAL range []
TRACE LOAD_CONST 5 [LazyVariableTracker()]
TRACE CALL_FUNCTION 1 [LazyVariableTracker(), ConstantVariable(int: 5)]
TRACE GET_ITER None [RangeVariable()]
TRACE FOR_ITER 40 [RangeIteratorVariable()]
TRACE STORE_FAST i [RangeIteratorVariable(), ConstantVariable(int: 0)]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:4 in fn
x = x + 1
TRACE LOAD_FAST x [RangeIteratorVariable()]
TRACE LOAD_CONST 1 [RangeIteratorVariable(), LazyVariableTracker()]
TRACE BINARY_ADD None [RangeIteratorVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_FAST x [RangeIteratorVariable(), TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:5 in fn
if i == 3:
TRACE LOAD_FAST i [RangeIteratorVariable()]
TRACE LOAD_CONST 3 [RangeIteratorVariable(), ConstantVariable(int: 0)]
TRACE COMPARE_OP == [RangeIteratorVariable(), ConstantVariable(int: 0), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 38 [RangeIteratorVariable(), ConstantVariable(bool: False)]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:6 in fn
torch._dynamo.graph_break()
TRACE JUMP_ABSOLUTE 8 [RangeIteratorVariable()]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:3 in fn
for i in range(5):
TRACE FOR_ITER 40 [RangeIteratorVariable()]
TRACE STORE_FAST i [RangeIteratorVariable(), ConstantVariable(int: 1)]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:4 in fn
x = x + 1
TRACE LOAD_FAST x [RangeIteratorVariable()]
TRACE LOAD_CONST 1 [RangeIteratorVariable(), TensorVariable()]
TRACE BINARY_ADD None [RangeIteratorVariable(), TensorVariable(), ConstantVariable(int: 1)]
TRACE STORE_FAST x [RangeIteratorVariable(), TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:5 in fn
if i == 3:
TRACE LOAD_FAST i [RangeIteratorVariable()]
TRACE LOAD_CONST 3 [RangeIteratorVariable(), ConstantVariable(int: 1)]
TRACE COMPARE_OP == [RangeIteratorVariable(), ConstantVariable(int: 1), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 38 [RangeIteratorVariable(), ConstantVariable(bool: False)]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:6 in fn
torch._dynamo.graph_break()
TRACE JUMP_ABSOLUTE 8 [RangeIteratorVariable()]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:3 in fn
for i in range(5):
TRACE FOR_ITER 40 [RangeIteratorVariable()]
TRACE STORE_FAST i [RangeIteratorVariable(), ConstantVariable(int: 2)]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:4 in fn
x = x + 1
TRACE LOAD_FAST x [RangeIteratorVariable()]
TRACE LOAD_CONST 1 [RangeIteratorVariable(), TensorVariable()]
TRACE BINARY_ADD None [RangeIteratorVariable(), TensorVariable(), ConstantVariable(int: 1)]
TRACE STORE_FAST x [RangeIteratorVariable(), TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:5 in fn
if i == 3:
TRACE LOAD_FAST i [RangeIteratorVariable()]
TRACE LOAD_CONST 3 [RangeIteratorVariable(), ConstantVariable(int: 2)]
TRACE COMPARE_OP == [RangeIteratorVariable(), ConstantVariable(int: 2), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 38 [RangeIteratorVariable(), ConstantVariable(bool: False)]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:6 in fn
torch._dynamo.graph_break()
TRACE JUMP_ABSOLUTE 8 [RangeIteratorVariable()]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:3 in fn
for i in range(5):
TRACE FOR_ITER 40 [RangeIteratorVariable()]
TRACE STORE_FAST i [RangeIteratorVariable(), ConstantVariable(int: 3)]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:4 in fn
x = x + 1
TRACE LOAD_FAST x [RangeIteratorVariable()]
TRACE LOAD_CONST 1 [RangeIteratorVariable(), TensorVariable()]
TRACE BINARY_ADD None [RangeIteratorVariable(), TensorVariable(), ConstantVariable(int: 1)]
TRACE STORE_FAST x [RangeIteratorVariable(), TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:5 in fn
if i == 3:
TRACE LOAD_FAST i [RangeIteratorVariable()]
TRACE LOAD_CONST 3 [RangeIteratorVariable(), ConstantVariable(int: 3)]
TRACE COMPARE_OP == [RangeIteratorVariable(), ConstantVariable(int: 3), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 38 [RangeIteratorVariable(), ConstantVariable(bool: True)]
TRACE starts_line /tmp/ipykernel_1130/2044822433.py:6 in fn
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch [RangeIteratorVariable()]
TRACE LOAD_ATTR _dynamo [RangeIteratorVariable(), LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [RangeIteratorVariable(), LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [RangeIteratorVariable(), LazyVariableTracker()]
Graph break in user code at /tmp/ipykernel_1130/2044822433.py:6
Graph Break Reason: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
User code traceback:
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/2044822433.py", line 8, in <module>
fn(torch.randn(3))
File "/tmp/ipykernel_1130/2044822433.py", line 6, in fn
torch._dynamo.graph_break()
Skipping frame because there is a graph break in a for/while loop
<FrameSummary file /tmp/ipykernel_1130/2044822433.py, line 6 in fn>
Skipping frame Skipping frame because there is a graph break in a for/while loop
<FrameSummary file /tmp/ipykernel_1130/2044822433.py, line 6 in fn> fn /tmp/ipykernel_1130/2044822433.py 1
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
skipping: graph_break (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/decorators.py)
tensor([4.9693, 5.4235, 4.7725])
在此示例中，我们可以通过展开循环来避免跳过
@torch.compile
def fn(x):
def inner(i):
nonlocal x
x = x + 1
if i == 3:
torch._dynamo.graph_break()
inner(0)
inner(1)
inner(2)
inner(3)
inner(4)
return x
fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_1130/617960493.py:1, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/617960493.py", line 14, in <module>
fn(torch.randn(3))
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/617960493.py:1
create_env
TRACE starts_line /tmp/ipykernel_1130/617960493.py:3 in fn
def inner(i):
TRACE LOAD_CLOSURE x []
TRACE BUILD_TUPLE 1 [CellVariable()]
TRACE LOAD_CONST <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3> [TupleVariable(length=1)]
TRACE LOAD_CONST fn.<locals>.inner [TupleVariable(length=1), ConstantVariable(code: <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>)]
TRACE MAKE_FUNCTION 8 [TupleVariable(length=1), ConstantVariable(code: <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>), ConstantVariable(str: 'fn.<locals>.inner')]
TRACE STORE_FAST inner [NestedUserFunctionVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:8 in fn
inner(0)
TRACE LOAD_FAST inner []
TRACE LOAD_CONST 0 [NestedUserFunctionVariable()]
TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 0)]
INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/617960493.py:5 in inner (inline depth: 1)
x = x + 1
TRACE LOAD_DEREF x []
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=True), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE LOAD_CONST 1 [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]
TRACE STORE_DEREF x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:6 in inner (inline depth: 1)
if i == 3:
TRACE LOAD_FAST i []
TRACE LOAD_CONST 3 [ConstantVariable(int: 0)]
TRACE COMPARE_OP == [ConstantVariable(int: 0), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>
TRACE POP_TOP None [ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:9 in fn
inner(1)
TRACE LOAD_FAST inner []
TRACE LOAD_CONST 1 [NestedUserFunctionVariable()]
TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 1)]
INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/617960493.py:5 in inner (inline depth: 1)
x = x + 1
TRACE LOAD_DEREF x []
TRACE LOAD_CONST 1 [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]
TRACE STORE_DEREF x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:6 in inner (inline depth: 1)
if i == 3:
TRACE LOAD_FAST i []
TRACE LOAD_CONST 3 [ConstantVariable(int: 1)]
TRACE COMPARE_OP == [ConstantVariable(int: 1), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>
TRACE POP_TOP None [ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:10 in fn
inner(2)
TRACE LOAD_FAST inner []
TRACE LOAD_CONST 2 [NestedUserFunctionVariable()]
TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 2)]
INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/617960493.py:5 in inner (inline depth: 1)
x = x + 1
TRACE LOAD_DEREF x []
TRACE LOAD_CONST 1 [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]
TRACE STORE_DEREF x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:6 in inner (inline depth: 1)
if i == 3:
TRACE LOAD_FAST i []
TRACE LOAD_CONST 3 [ConstantVariable(int: 2)]
TRACE COMPARE_OP == [ConstantVariable(int: 2), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>
TRACE POP_TOP None [ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:11 in fn
inner(3)
TRACE LOAD_FAST inner []
TRACE LOAD_CONST 3 [NestedUserFunctionVariable()]
TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 3)]
INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/617960493.py:5 in inner (inline depth: 1)
x = x + 1
TRACE LOAD_DEREF x []
TRACE LOAD_CONST 1 [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]
TRACE STORE_DEREF x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:6 in inner (inline depth: 1)
if i == 3:
TRACE LOAD_FAST i []
TRACE LOAD_CONST 3 [ConstantVariable(int: 3)]
TRACE COMPARE_OP == [ConstantVariable(int: 3), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: True)]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:7 in inner (inline depth: 1)
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
empty checkpoint
FAILED INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>
Graph break in user code at /tmp/ipykernel_1130/617960493.py:7
Graph Break Reason: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
User code traceback:
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/617960493.py", line 14, in <module>
fn(torch.randn(3))
File "/tmp/ipykernel_1130/617960493.py", line 11, in fn
inner(3)
File "/tmp/ipykernel_1130/617960493.py", line 7, in inner
torch._dynamo.graph_break()
Restarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysis
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/617960493.py:1
create_env
TRACE starts_line /tmp/ipykernel_1130/617960493.py:3 in fn
def inner(i):
TRACE LOAD_CLOSURE x []
TRACE BUILD_TUPLE 1 [CellVariable()]
TRACE LOAD_CONST <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3> [TupleVariable(length=1)]
TRACE LOAD_CONST fn.<locals>.inner [TupleVariable(length=1), ConstantVariable(code: <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>)]
TRACE MAKE_FUNCTION 8 [TupleVariable(length=1), ConstantVariable(code: <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>), ConstantVariable(str: 'fn.<locals>.inner')]
TRACE STORE_FAST inner [NestedUserFunctionVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:8 in fn
inner(0)
TRACE LOAD_FAST inner []
TRACE LOAD_CONST 0 [NestedUserFunctionVariable()]
TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 0)]
INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/617960493.py:5 in inner (inline depth: 1)
x = x + 1
TRACE LOAD_DEREF x []
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=True), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE LOAD_CONST 1 [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]
TRACE STORE_DEREF x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:6 in inner (inline depth: 1)
if i == 3:
TRACE LOAD_FAST i []
TRACE LOAD_CONST 3 [ConstantVariable(int: 0)]
TRACE COMPARE_OP == [ConstantVariable(int: 0), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>
TRACE POP_TOP None [ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:9 in fn
inner(1)
TRACE LOAD_FAST inner []
TRACE LOAD_CONST 1 [NestedUserFunctionVariable()]
TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 1)]
INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/617960493.py:5 in inner (inline depth: 1)
x = x + 1
TRACE LOAD_DEREF x []
TRACE LOAD_CONST 1 [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]
TRACE STORE_DEREF x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:6 in inner (inline depth: 1)
if i == 3:
TRACE LOAD_FAST i []
TRACE LOAD_CONST 3 [ConstantVariable(int: 1)]
TRACE COMPARE_OP == [ConstantVariable(int: 1), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>
TRACE POP_TOP None [ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:10 in fn
inner(2)
TRACE LOAD_FAST inner []
TRACE LOAD_CONST 2 [NestedUserFunctionVariable()]
TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 2)]
INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/617960493.py:5 in inner (inline depth: 1)
x = x + 1
TRACE LOAD_DEREF x []
TRACE LOAD_CONST 1 [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]
TRACE STORE_DEREF x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:6 in inner (inline depth: 1)
if i == 3:
TRACE LOAD_FAST i []
TRACE LOAD_CONST 3 [ConstantVariable(int: 2)]
TRACE COMPARE_OP == [ConstantVariable(int: 2), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>
TRACE POP_TOP None [ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:11 in fn
inner(3)
TRACE LOAD_FAST inner []
TRACE LOAD_CONST 3 [NestedUserFunctionVariable()]
TRACE CALL_FUNCTION 1 [NestedUserFunctionVariable(), ConstantVariable(int: 3)]
COMPILING GRAPH due to GraphCompileReason(reason='Call to `torch._dynamo.graph_break()`\n Explanation: User-inserted graph break. Message: None\n Hint: Remove the `torch._dynamo.graph_break()` call.\n\n Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`\n\n For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html', user_stack=[<FrameSummary file /tmp/ipykernel_1130/617960493.py, line 11 in fn>, <FrameSummary file /tmp/ipykernel_1130/617960493.py, line 7 in inner>], graph_break=True)
TRACED GRAPH
===== __compiled_fn_8_bd29ec73_fdf6_4524_93cf_b1317d70e0d1 =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# File: /tmp/ipykernel_1130/617960493.py:5 in inner, code: x = x + 1
x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None
# File: /tmp/ipykernel_1130/617960493.py:5 in inner, code: x = x + 1
x_1: "f32[3][1]cpu" = x + 1; x = None
# File: /tmp/ipykernel_1130/617960493.py:5 in inner, code: x = x + 1
x_2: "f32[3][1]cpu" = x_1 + 1; x_1 = None
return (x_2,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=2), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_1130/617960493.py:5 in inner
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_1130/617960493.py:5 in inner
Guard eval latency = 105.35 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
skipping: _create_nested_fn (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py)
torchdynamo start compiling inner /tmp/ipykernel_1130/617960493.py:3, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/617960493.py", line 14, in <module>
fn(torch.randn(3))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
return fn(*args, **kwargs)
Step 1: torchdynamo start tracing inner /tmp/ipykernel_1130/617960493.py:3
create_env
TRACE starts_line /tmp/ipykernel_1130/617960493.py:5 in inner
x = x + 1
TRACE LOAD_DEREF x []
TRACE LOAD_CONST 1 [LazyVariableTracker()]
TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=False, dynamism=None, is_derefed_cell_contents=True), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_DEREF x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:6 in inner
if i == 3:
TRACE LOAD_FAST i []
TRACE LOAD_CONST 3 [LazyVariableTracker()]
TRACE COMPARE_OP == [LazyVariableTracker(), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: True)]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:7 in inner
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
Graph break (user stack suppressed due to duplicate graph break) in user code at /tmp/ipykernel_1130/617960493.py:7
Graph Break Reason: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
Restarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysis
Step 1: torchdynamo start tracing inner /tmp/ipykernel_1130/617960493.py:3
create_env
TRACE starts_line /tmp/ipykernel_1130/617960493.py:5 in inner
x = x + 1
TRACE LOAD_DEREF x []
TRACE LOAD_CONST 1 [LazyVariableTracker()]
TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=False, dynamism=None, is_derefed_cell_contents=True), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_DEREF x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:6 in inner
if i == 3:
TRACE LOAD_FAST i []
TRACE LOAD_CONST 3 [LazyVariableTracker()]
TRACE COMPARE_OP == [LazyVariableTracker(), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: True)]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:7 in inner
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
COMPILING GRAPH due to GraphCompileReason(reason='Call to `torch._dynamo.graph_break()`\n Explanation: User-inserted graph break. Message: None\n Hint: Remove the `torch._dynamo.graph_break()` call.\n\n Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`\n\n For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html', user_stack=[<FrameSummary file /tmp/ipykernel_1130/617960493.py, line 7 in inner>], graph_break=True)
TRACED GRAPH
===== __compiled_fn_12_0e162e3e_3da5_4276_8a0a_f801e0b4125e =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# File: /tmp/ipykernel_1130/617960493.py:5 in inner, code: x = x + 1
x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None
return (x,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['i'], accessed_by=FrameLocalsGuardAccessor(key='i', framelocals_idx=0), type=<class 'int'>, tag_safe=(True, False)
| | +- EQUALS_MATCH: L['i'] == 3 # if i == 3: # mp/ipykernel_1130/617960493.py:6 in inner
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=1), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_1130/617960493.py:5 in inner
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_1130/617960493.py:5 in inner
| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(False, False)
| | +- GuardManager: source=G['torch'], accessed_by=DictGetItemGuardAccessor('torch'), type=<class 'module'>, tag_safe=(False, False)
| | | +- ID_MATCH: ___check_obj_id(G['torch'], 139667418978544) # torch._dynamo.graph_break() # mp/ipykernel_1130/617960493.py:7 in inner
| | | +- GuardManager: source=G['torch']._dynamo, accessed_by=GetAttrGuardAccessor(_dynamo), type=<class 'module'>, tag_safe=(False, False)
| | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo, 139666007729360) # torch._dynamo.graph_break() # mp/ipykernel_1130/617960493.py:7 in inner
| | | | +- GuardManager: source=G['torch']._dynamo.graph_break, accessed_by=GetAttrGuardAccessor(graph_break), type=<class 'function'>, tag_safe=(True, False)
| | | | | +- GuardManager: source=G['torch']._dynamo.graph_break.__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)
| | | | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo.graph_break.__code__, 139665819779296) # torch._dynamo.graph_break() # mp/ipykernel_1130/617960493.py:7 in inner
Guard eval latency = 94.90 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
torchdynamo start compiling torch_dynamo_resume_in_inner_at_7 /tmp/ipykernel_1130/617960493.py:7, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/617960493.py", line 14, in <module>
fn(torch.randn(3))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
return fn(*args, **kwargs)
File "/tmp/ipykernel_1130/617960493.py", line 11, in fn
inner(3)
Step 1: torchdynamo start tracing torch_dynamo_resume_in_inner_at_7 /tmp/ipykernel_1130/617960493.py:7
create_env
TRACE starts_line /tmp/ipykernel_1130/617960493.py:7 in torch_dynamo_resume_in_inner_at_7
torch._dynamo.graph_break()
TRACE LOAD_CONST True []
TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]
TRACE LOAD_FAST ___stack0 []
TRACE LOAD_CONST False [LazyVariableTracker()]
TRACE STORE_FAST __is_tracing_resume_prologue [LazyVariableTracker(), ConstantVariable(bool: False)]
TRACE JUMP_ABSOLUTE 36 [LazyVariableTracker()]
TRACE POP_TOP None [LazyVariableTracker()]
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
Skipping frame because no content in function call torch_dynamo_resume_in_inner_at_7 /tmp/ipykernel_1130/617960493.py 7
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
torchdynamo start compiling torch_dynamo_resume_in_fn_at_11 /tmp/ipykernel_1130/617960493.py:11, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/617960493.py", line 14, in <module>
fn(torch.randn(3))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
return fn(*args, **kwargs)
Step 1: torchdynamo start tracing torch_dynamo_resume_in_fn_at_11 /tmp/ipykernel_1130/617960493.py:11
create_env
TRACE starts_line /tmp/ipykernel_1130/617960493.py:11 in torch_dynamo_resume_in_fn_at_11
inner(3)
TRACE LOAD_CONST True []
TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]
TRACE LOAD_FAST ___stack0 []
TRACE LOAD_CONST False [LazyVariableTracker()]
TRACE STORE_FAST __is_tracing_resume_prologue [LazyVariableTracker(), ConstantVariable(bool: False)]
TRACE JUMP_ABSOLUTE 54 [LazyVariableTracker()]
TRACE POP_TOP None [LazyVariableTracker()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:12 in torch_dynamo_resume_in_fn_at_11
inner(4)
TRACE LOAD_FAST inner []
TRACE LOAD_CONST 4 [LazyVariableTracker()]
TRACE CALL_FUNCTION 1 [LazyVariableTracker(), ConstantVariable(int: 4)]
INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/617960493.py:5 in inner (inline depth: 1)
x = x + 1
TRACE LOAD_DEREF x []
TRACE LOAD_CONST 1 [LazyVariableTracker()]
TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=False, dynamism=None, is_derefed_cell_contents=True), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_DEREF x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:6 in inner (inline depth: 1)
if i == 3:
TRACE LOAD_FAST i []
TRACE LOAD_CONST 3 [ConstantVariable(int: 4)]
TRACE COMPARE_OP == [ConstantVariable(int: 4), ConstantVariable(int: 3)]
TRACE POP_JUMP_IF_FALSE 30 [ConstantVariable(bool: False)]
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object inner at 0x7f06795e8500, file "/tmp/ipykernel_1130/617960493.py", line 3>
TRACE POP_TOP None [ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/617960493.py:13 in torch_dynamo_resume_in_fn_at_11
return x
TRACE LOAD_DEREF x []
TRACE RETURN_VALUE None [TensorVariable()]
Step 1: torchdynamo done tracing torch_dynamo_resume_in_fn_at_11 (RETURN_VALUE)
RETURN_VALUE triggered compile
COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_1130/617960493.py, line 13 in torch_dynamo_resume_in_fn_at_11>], graph_break=False)
TRACED GRAPH
===== __compiled_fn_16_80d96e3e_b87e_4bb6_a339_82a2f4f49856 =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# File: /tmp/ipykernel_1130/617960493.py:5 in inner, code: x = x + 1
x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None
return (x,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=6), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_1130/617960493.py:5 in inner
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_1130/617960493.py:5 in inner
| +- GuardManager: source=L['inner'], accessed_by=FrameLocalsGuardAccessor(key='inner', framelocals_idx=3), type=<class 'function'>, tag_safe=(True, False)
| | +- GuardManager: source=L['inner'].__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)
| | | +- ID_MATCH: ___check_obj_id(L['inner'].__code__, 139665782768896) # inner(4) # mp/ipykernel_1130/617960493.py:12 in torch_dynamo_resume_in_fn_at_11
Guard eval latency = 100.60 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
skipping because no torch.* remove /opt/conda/envs/py_3.10/lib/python3.10/weakref.py 370
tensor([5.9183, 5.3813, 4.1516])
总的来说，解决导致跳过的图中断也将解决跳过本身。
上下文管理器中的图中断#
另一个常见的无法恢复的图中断示例是大多数上下文管理器中的图中断
class CustomCtxManager:
def __enter__(self):
pass
def __exit__(self, exc_type, exc_value, traceback):
pass
@torch.compile
def fn(x):
with CustomCtxManager():
x = x + 1
torch._dynamo.graph_break()
return x + 1
fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_1130/4148913404.py:6, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/4148913404.py", line 12, in <module>
fn(torch.randn(3))
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/4148913404.py:6
create_env
TRACE starts_line /tmp/ipykernel_1130/4148913404.py:8 in fn
with CustomCtxManager():
TRACE LOAD_GLOBAL CustomCtxManager []
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
TRACE SETUP_WITH 48 [GenericContextWrappingVariable(CustomCtxManager)]
INLINING <code object __enter__ at 0x7f06dd1163f0, file "/tmp/ipykernel_1130/4148913404.py", line 2>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/4148913404.py:3 in __enter__ (inline depth: 1)
pass
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object __enter__ at 0x7f06dd1163f0, file "/tmp/ipykernel_1130/4148913404.py", line 2>
TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/4148913404.py:9 in fn
x = x + 1
TRACE LOAD_FAST x [WithExitFunctionVariable()]
TRACE LOAD_CONST 1 [WithExitFunctionVariable(), LazyVariableTracker()]
TRACE BINARY_ADD None [WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_FAST x [WithExitFunctionVariable(), TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/4148913404.py:10 in fn
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch [WithExitFunctionVariable()]
TRACE LOAD_ATTR _dynamo [WithExitFunctionVariable(), LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [WithExitFunctionVariable(), LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), LazyVariableTracker()]
empty checkpoint
run_gc_after_compile: running gc
Graph break: skip: from user code at:
File "/tmp/ipykernel_1130/4148913404.py", line 10, in fn
torch._dynamo.graph_break()
Traceback (most recent call last):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper
return inner_fn(self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2320, in CALL_FUNCTION
self.call_function(fn, args, {})
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function
self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type]
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward
return getattr(self.realize(), name)(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function
unimplemented_v2(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1624, in __call__
result = self._inner_convert(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__
result = _compile(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1433, in _compile
guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function
return function(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner
return _compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner
dynamo_output = compile_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame
bytecode, tracer_output = transform_code_object(code, transform)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object
tracer_output = transformations(instructions, code_options)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform
tracer_output = trace_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame
run_tracer()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer
tracer.run()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1487, in run
while self.step():
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step
self.dispatch_table[inst.opcode](self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 910, in wrapper
unimplemented_v2(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 595, in unimplemented_v2
raise Unsupported(msg) from from_exc
torch._dynamo.exc.Unsupported: Graph break under GenericContextWrappingVariable
Explanation: Attempted to graph break in an active context manager(s) that doesn't support graph breaking.
Hint: Move the offending context manager(s) to outside the compiled region.
Hint: This graph break may have been caused by an earlier graph break. Resolving the earlier graph break may resolve this one.
Developer debug context: Active generic context managers: [GenericContextWrappingVariable(CustomCtxManager)]
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0066.html
from user code:
File "/tmp/ipykernel_1130/4148913404.py", line 10, in fn
torch._dynamo.graph_break()
Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
WON'T CONVERT fn /tmp/ipykernel_1130/4148913404.py line 6
due to:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper
return inner_fn(self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2320, in CALL_FUNCTION
self.call_function(fn, args, {})
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function
self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type]
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward
return getattr(self.realize(), name)(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function
unimplemented_v2(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1624, in __call__
result = self._inner_convert(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__
result = _compile(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1433, in _compile
guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function
return function(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner
return _compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner
dynamo_output = compile_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame
bytecode, tracer_output = transform_code_object(code, transform)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object
tracer_output = transformations(instructions, code_options)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform
tracer_output = trace_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame
run_tracer()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer
tracer.run()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1487, in run
while self.step():
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step
self.dispatch_table[inst.opcode](self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 910, in wrapper
unimplemented_v2(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 595, in unimplemented_v2
raise Unsupported(msg) from from_exc
torch._dynamo.exc.Unsupported: Graph break under GenericContextWrappingVariable
Explanation: Attempted to graph break in an active context manager(s) that doesn't support graph breaking.
Hint: Move the offending context manager(s) to outside the compiled region.
Hint: This graph break may have been caused by an earlier graph break. Resolving the earlier graph break may resolve this one.
Developer debug context: Active generic context managers: [GenericContextWrappingVariable(CustomCtxManager)]
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0066.html
from user code:
File "/tmp/ipykernel_1130/4148913404.py", line 10, in fn
torch._dynamo.graph_break()
Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
Traceback (most recent call last):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper
return inner_fn(self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2320, in CALL_FUNCTION
self.call_function(fn, args, {})
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function
self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type]
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward
return getattr(self.realize(), name)(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function
unimplemented_v2(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1624, in __call__
result = self._inner_convert(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__
result = _compile(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1433, in _compile
guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function
return function(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner
return _compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner
dynamo_output = compile_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame
bytecode, tracer_output = transform_code_object(code, transform)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object
tracer_output = transformations(instructions, code_options)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform
tracer_output = trace_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame
run_tracer()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer
tracer.run()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1487, in run
while self.step():
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step
self.dispatch_table[inst.opcode](self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 910, in wrapper
unimplemented_v2(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 595, in unimplemented_v2
raise Unsupported(msg) from from_exc
torch._dynamo.exc.Unsupported: Graph break under GenericContextWrappingVariable
Explanation: Attempted to graph break in an active context manager(s) that doesn't support graph breaking.
Hint: Move the offending context manager(s) to outside the compiled region.
Hint: This graph break may have been caused by an earlier graph break. Resolving the earlier graph break may resolve this one.
Developer debug context: Active generic context managers: [GenericContextWrappingVariable(CustomCtxManager)]
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0066.html
from user code:
File "/tmp/ipykernel_1130/4148913404.py", line 10, in fn
torch._dynamo.graph_break()
Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
skipping because no torch.* __enter__ /tmp/ipykernel_1130/4148913404.py 2
skipping because no torch.* __exit__ /tmp/ipykernel_1130/4148913404.py 4
tensor([2.5926, 1.2684, 2.1012])
我们可以通过将图中断移到上下文管理器外部来避免跳过
@torch.compile
def fn(x):
with CustomCtxManager():
x = x + 1
torch._dynamo.graph_break()
with CustomCtxManager():
return x + 1
fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_1130/2124425154.py:1, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/2124425154.py", line 8, in <module>
fn(torch.randn(3))
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/2124425154.py:1
create_env
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:3 in fn
with CustomCtxManager():
TRACE LOAD_GLOBAL CustomCtxManager []
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
TRACE SETUP_WITH 30 [GenericContextWrappingVariable(CustomCtxManager)]
INLINING <code object __enter__ at 0x7f06dd1163f0, file "/tmp/ipykernel_1130/4148913404.py", line 2>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/4148913404.py:3 in __enter__ (inline depth: 1)
pass
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object __enter__ at 0x7f06dd1163f0, file "/tmp/ipykernel_1130/4148913404.py", line 2>
TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:4 in fn
x = x + 1
TRACE LOAD_FAST x [WithExitFunctionVariable()]
TRACE LOAD_CONST 1 [WithExitFunctionVariable(), LazyVariableTracker()]
TRACE BINARY_ADD None [WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_FAST x [WithExitFunctionVariable(), TensorVariable()]
TRACE POP_BLOCK None [WithExitFunctionVariable()]
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:3 in fn
with CustomCtxManager():
TRACE LOAD_CONST None [WithExitFunctionVariable()]
TRACE DUP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE DUP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]
TRACE CALL_FUNCTION 3 [WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]
INLINING <code object __exit__ at 0x7f06795e9a50, file "/tmp/ipykernel_1130/4148913404.py", line 4>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/4148913404.py:5 in __exit__ (inline depth: 1)
pass
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object __exit__ at 0x7f06795e9a50, file "/tmp/ipykernel_1130/4148913404.py", line 4>
TRACE POP_TOP None [ConstantVariable(NoneType: None)]
TRACE JUMP_FORWARD 46 []
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:5 in fn
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
Graph break in user code at /tmp/ipykernel_1130/2124425154.py:5
Graph Break Reason: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
User code traceback:
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/2124425154.py", line 8, in <module>
fn(torch.randn(3))
File "/tmp/ipykernel_1130/2124425154.py", line 5, in fn
torch._dynamo.graph_break()
Restarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysis
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/2124425154.py:1
create_env
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:3 in fn
with CustomCtxManager():
TRACE LOAD_GLOBAL CustomCtxManager []
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
TRACE SETUP_WITH 30 [GenericContextWrappingVariable(CustomCtxManager)]
INLINING <code object __enter__ at 0x7f06dd1163f0, file "/tmp/ipykernel_1130/4148913404.py", line 2>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/4148913404.py:3 in __enter__ (inline depth: 1)
pass
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object __enter__ at 0x7f06dd1163f0, file "/tmp/ipykernel_1130/4148913404.py", line 2>
TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:4 in fn
x = x + 1
TRACE LOAD_FAST x [WithExitFunctionVariable()]
TRACE LOAD_CONST 1 [WithExitFunctionVariable(), LazyVariableTracker()]
TRACE BINARY_ADD None [WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_FAST x [WithExitFunctionVariable(), TensorVariable()]
TRACE POP_BLOCK None [WithExitFunctionVariable()]
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:3 in fn
with CustomCtxManager():
TRACE LOAD_CONST None [WithExitFunctionVariable()]
TRACE DUP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE DUP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]
TRACE CALL_FUNCTION 3 [WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]
INLINING <code object __exit__ at 0x7f06795e9a50, file "/tmp/ipykernel_1130/4148913404.py", line 4>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/4148913404.py:5 in __exit__ (inline depth: 1)
pass
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object __exit__ at 0x7f06795e9a50, file "/tmp/ipykernel_1130/4148913404.py", line 4>
TRACE POP_TOP None [ConstantVariable(NoneType: None)]
TRACE JUMP_FORWARD 46 []
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:5 in fn
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
COMPILING GRAPH due to GraphCompileReason(reason='Call to `torch._dynamo.graph_break()`\n Explanation: User-inserted graph break. Message: None\n Hint: Remove the `torch._dynamo.graph_break()` call.\n\n Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`\n\n For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html', user_stack=[<FrameSummary file /tmp/ipykernel_1130/2124425154.py, line 5 in fn>], graph_break=True)
TRACED GRAPH
===== __compiled_fn_20_310a0629_3dcb_48ab_9c3e_00c0e2147a39 =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# File: /tmp/ipykernel_1130/2124425154.py:4 in fn, code: x = x + 1
x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None
return (x,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_1130/2124425154.py:4 in fn
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_1130/2124425154.py:4 in fn
| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(False, False)
| | +- GuardManager: source=G['torch'], accessed_by=DictGetItemGuardAccessor('torch'), type=<class 'module'>, tag_safe=(False, False)
| | | +- ID_MATCH: ___check_obj_id(G['torch'], 139667418978544) # torch._dynamo.graph_break() # mp/ipykernel_1130/2124425154.py:5 in fn
| | | +- GuardManager: source=G['torch']._dynamo, accessed_by=GetAttrGuardAccessor(_dynamo), type=<class 'module'>, tag_safe=(False, False)
| | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo, 139666007729360) # torch._dynamo.graph_break() # mp/ipykernel_1130/2124425154.py:5 in fn
| | | | +- GuardManager: source=G['torch']._dynamo.graph_break, accessed_by=GetAttrGuardAccessor(graph_break), type=<class 'function'>, tag_safe=(True, False)
| | | | | +- GuardManager: source=G['torch']._dynamo.graph_break.__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)
| | | | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo.graph_break.__code__, 139665819779296) # torch._dynamo.graph_break() # mp/ipykernel_1130/2124425154.py:5 in fn
| | +- GuardManager: source=G['CustomCtxManager'], accessed_by=DictGetItemGuardAccessor('CustomCtxManager'), type=<class 'type'>, tag_safe=(True, False)
| | | +- ID_MATCH: ___check_obj_id(G['CustomCtxManager'], 94270933536464) # with CustomCtxManager(): # mp/ipykernel_1130/2124425154.py:3 in fn
Guard eval latency = 102.29 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
torchdynamo start compiling torch_dynamo_resume_in_fn_at_5 /tmp/ipykernel_1130/2124425154.py:5, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/2124425154.py", line 8, in <module>
fn(torch.randn(3))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
return fn(*args, **kwargs)
Step 1: torchdynamo start tracing torch_dynamo_resume_in_fn_at_5 /tmp/ipykernel_1130/2124425154.py:5
create_env
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:5 in torch_dynamo_resume_in_fn_at_5
torch._dynamo.graph_break()
TRACE LOAD_CONST True []
TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]
TRACE LOAD_FAST ___stack0 []
TRACE LOAD_CONST False [LazyVariableTracker()]
TRACE STORE_FAST __is_tracing_resume_prologue [LazyVariableTracker(), ConstantVariable(bool: False)]
TRACE JUMP_ABSOLUTE 66 [LazyVariableTracker()]
TRACE POP_TOP None [LazyVariableTracker()]
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:6 in torch_dynamo_resume_in_fn_at_5
with CustomCtxManager():
TRACE LOAD_GLOBAL CustomCtxManager []
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
TRACE SETUP_WITH 98 [GenericContextWrappingVariable(CustomCtxManager)]
INLINING <code object __enter__ at 0x7f06dd1163f0, file "/tmp/ipykernel_1130/4148913404.py", line 2>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/4148913404.py:3 in __enter__ (inline depth: 1)
pass
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object __enter__ at 0x7f06dd1163f0, file "/tmp/ipykernel_1130/4148913404.py", line 2>
TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:7 in torch_dynamo_resume_in_fn_at_5
return x + 1
TRACE LOAD_FAST x [WithExitFunctionVariable()]
TRACE LOAD_CONST 1 [WithExitFunctionVariable(), LazyVariableTracker()]
TRACE BINARY_ADD None [WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE starts_line /tmp/ipykernel_1130/2124425154.py:6 in torch_dynamo_resume_in_fn_at_5
with CustomCtxManager():
TRACE POP_BLOCK None [WithExitFunctionVariable(), TensorVariable()]
TRACE ROT_TWO None [WithExitFunctionVariable(), TensorVariable()]
TRACE LOAD_CONST None [TensorVariable(), WithExitFunctionVariable()]
TRACE DUP_TOP None [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE DUP_TOP None [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]
TRACE CALL_FUNCTION 3 [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]
INLINING <code object __exit__ at 0x7f06795e9a50, file "/tmp/ipykernel_1130/4148913404.py", line 4>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/4148913404.py:5 in __exit__ (inline depth: 1)
pass
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
DONE INLINING <code object __exit__ at 0x7f06795e9a50, file "/tmp/ipykernel_1130/4148913404.py", line 4>
TRACE POP_TOP None [TensorVariable(), ConstantVariable(NoneType: None)]
TRACE RETURN_VALUE None [TensorVariable()]
Step 1: torchdynamo done tracing torch_dynamo_resume_in_fn_at_5 (RETURN_VALUE)
RETURN_VALUE triggered compile
COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_1130/2124425154.py, line 6 in torch_dynamo_resume_in_fn_at_5>], graph_break=False)
TRACED GRAPH
===== __compiled_fn_23_8a9660f0_2df4_4aac_841f_98c9836b0a0e =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# File: /tmp/ipykernel_1130/2124425154.py:7 in torch_dynamo_resume_in_fn_at_5, code: return x + 1
add: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None
return (add,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=3), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # return x + 1 # mp/ipykernel_1130/2124425154.py:7 in torch_dynamo_resume_in_fn_at_5
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # return x + 1 # mp/ipykernel_1130/2124425154.py:7 in torch_dynamo_resume_in_fn_at_5
| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(True, False)
| | +- GuardManager: source=G['CustomCtxManager'], accessed_by=DictGetItemGuardAccessor('CustomCtxManager'), type=<class 'type'>, tag_safe=(True, False)
| | | +- ID_MATCH: ___check_obj_id(G['CustomCtxManager'], 94270933536464) # with CustomCtxManager(): # mp/ipykernel_1130/2124425154.py:6 in torch_dynamo_resume_in_fn_at_5
Guard eval latency = 931.58 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
tensor([3.2876, 2.6963, 0.7453])
有些上下文管理器在图中断后 Dynamo 可以恢复。其中一些可以在 torch/_dynamo/variables/torch.py 中的 supported_ctx_manager_classes 中找到。总的来说，在 torch/_dynamo/variables/ctx_manager.py 中由 ContextWrappingVariable 子类表示的任何上下文管理器都支持在图中断后恢复。例如
import contextlib
@torch.compile
def fn(x):
with contextlib.nullcontext():
with torch.no_grad():
x = x + 1
torch._dynamo.graph_break()
return x + 1
fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_1130/3152636365.py:2, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/3152636365.py", line 9, in <module>
fn(torch.randn(3))
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/3152636365.py:2
create_env
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:4 in fn
with contextlib.nullcontext():
TRACE LOAD_GLOBAL contextlib []
TRACE LOAD_ATTR nullcontext [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
TRACE SETUP_WITH 106 [NullContextVariable()]
TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:5 in fn
with torch.no_grad():
TRACE LOAD_GLOBAL torch [WithExitFunctionVariable()]
TRACE LOAD_ATTR no_grad [WithExitFunctionVariable(), LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), LazyVariableTracker()]
TRACE SETUP_WITH 74 [WithExitFunctionVariable(), GradModeVariable()]
TRACE POP_TOP None [WithExitFunctionVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:6 in fn
x = x + 1
TRACE LOAD_FAST x [WithExitFunctionVariable(), WithExitFunctionVariable()]
TRACE LOAD_CONST 1 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
TRACE BINARY_ADD None [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_FAST x [WithExitFunctionVariable(), WithExitFunctionVariable(), TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:7 in fn
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch [WithExitFunctionVariable(), WithExitFunctionVariable()]
TRACE LOAD_ATTR _dynamo [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
Graph break in user code at /tmp/ipykernel_1130/3152636365.py:7
Graph Break Reason: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
User code traceback:
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/3152636365.py", line 9, in <module>
fn(torch.randn(3))
File "/tmp/ipykernel_1130/3152636365.py", line 7, in fn
torch._dynamo.graph_break()
Restarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysis
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/3152636365.py:2
create_env
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:4 in fn
with contextlib.nullcontext():
TRACE LOAD_GLOBAL contextlib []
TRACE LOAD_ATTR nullcontext [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
TRACE SETUP_WITH 106 [NullContextVariable()]
TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:5 in fn
with torch.no_grad():
TRACE LOAD_GLOBAL torch [WithExitFunctionVariable()]
TRACE LOAD_ATTR no_grad [WithExitFunctionVariable(), LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), LazyVariableTracker()]
TRACE SETUP_WITH 74 [WithExitFunctionVariable(), GradModeVariable()]
TRACE POP_TOP None [WithExitFunctionVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:6 in fn
x = x + 1
TRACE LOAD_FAST x [WithExitFunctionVariable(), WithExitFunctionVariable()]
TRACE LOAD_CONST 1 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
TRACE BINARY_ADD None [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_FAST x [WithExitFunctionVariable(), WithExitFunctionVariable(), TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:7 in fn
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch [WithExitFunctionVariable(), WithExitFunctionVariable()]
TRACE LOAD_ATTR _dynamo [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
COMPILING GRAPH due to GraphCompileReason(reason='Call to `torch._dynamo.graph_break()`\n Explanation: User-inserted graph break. Message: None\n Hint: Remove the `torch._dynamo.graph_break()` call.\n\n Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`\n\n For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html', user_stack=[<FrameSummary file /tmp/ipykernel_1130/3152636365.py, line 7 in fn>], graph_break=True)
TRACED GRAPH
===== __compiled_fn_26_e309f364_c8e6_4cac_b8d7_644e34b36641 =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# No stacktrace found for following nodes
_set_grad_enabled = torch._C._set_grad_enabled(False); _set_grad_enabled = None
# File: /tmp/ipykernel_1130/3152636365.py:6 in fn, code: x = x + 1
x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None
# No stacktrace found for following nodes
_set_grad_enabled_1 = torch._C._set_grad_enabled(True); _set_grad_enabled_1 = None
return (x,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_1130/3152636365.py:6 in fn
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_1130/3152636365.py:6 in fn
| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(False, False)
| | +- GuardManager: source=G['torch'], accessed_by=DictGetItemGuardAccessor('torch'), type=<class 'module'>, tag_safe=(False, False)
| | | +- ID_MATCH: ___check_obj_id(G['torch'], 139667418978544) # with torch.no_grad(): # mp/ipykernel_1130/3152636365.py:5 in fn
| | | +- GuardManager: source=G['torch']._dynamo, accessed_by=GetAttrGuardAccessor(_dynamo), type=<class 'module'>, tag_safe=(False, False)
| | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo, 139666007729360) # torch._dynamo.graph_break() # mp/ipykernel_1130/3152636365.py:7 in fn
| | | | +- GuardManager: source=G['torch']._dynamo.graph_break, accessed_by=GetAttrGuardAccessor(graph_break), type=<class 'function'>, tag_safe=(True, False)
| | | | | +- GuardManager: source=G['torch']._dynamo.graph_break.__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)
| | | | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo.graph_break.__code__, 139665819779296) # torch._dynamo.graph_break() # mp/ipykernel_1130/3152636365.py:7 in fn
| | | +- GuardManager: source=G['torch'].no_grad, accessed_by=GetAttrGuardAccessor(no_grad), type=<class 'type'>, tag_safe=(True, False)
| | | | +- ID_MATCH: ___check_obj_id(G['torch'].no_grad, 94270896281648) # with torch.no_grad(): # mp/ipykernel_1130/3152636365.py:5 in fn
| | +- GuardManager: source=G['contextlib'], accessed_by=DictGetItemGuardAccessor('contextlib'), type=<class 'module'>, tag_safe=(False, False)
| | | +- ID_MATCH: ___check_obj_id(G['contextlib'], 139667464279104) # with contextlib.nullcontext(): # mp/ipykernel_1130/3152636365.py:4 in fn
| | | +- GuardManager: source=G['contextlib'].nullcontext, accessed_by=GetAttrGuardAccessor(nullcontext), type=<class 'abc.ABCMeta'>, tag_safe=(True, False)
| | | | +- ID_MATCH: ___check_obj_id(G['contextlib'].nullcontext, 94270821252912) # with contextlib.nullcontext(): # mp/ipykernel_1130/3152636365.py:4 in fn
Guard eval latency = 80.12 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
skipping because no torch.* __init__ /opt/conda/envs/py_3.10/lib/python3.10/contextlib.py 732
skipping because no torch.* __enter__ /opt/conda/envs/py_3.10/lib/python3.10/contextlib.py 735
skipping: __init__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/grad_mode.py)
skipping: __enter__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/grad_mode.py)
skipping: __exit__ (reason: in skipfiles, file: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/grad_mode.py)
skipping because no torch.* __exit__ /opt/conda/envs/py_3.10/lib/python3.10/contextlib.py 738
torchdynamo start compiling torch_dynamo_resume_in_fn_at_7 /tmp/ipykernel_1130/3152636365.py:7, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/3152636365.py", line 9, in <module>
fn(torch.randn(3))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
return fn(*args, **kwargs)
Step 1: torchdynamo start tracing torch_dynamo_resume_in_fn_at_7 /tmp/ipykernel_1130/3152636365.py:7
create_env
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:7 in torch_dynamo_resume_in_fn_at_7
torch._dynamo.graph_break()
TRACE LOAD_CONST True []
TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]
TRACE LOAD_FAST ___stack0 []
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
TRACE NOP None [NullContextVariable()]
TRACE SETUP_WITH 96 [NullContextVariable()]
TRACE POP_TOP None [WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE NOP None [WithExitFunctionVariable()]
TRACE LOAD_FAST ___stack1 [WithExitFunctionVariable()]
TRACE LOAD_CONST False [WithExitFunctionVariable(), LazyVariableTracker()]
TRACE CALL_FUNCTION 1 [WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(bool: False)]
TRACE NOP None [WithExitFunctionVariable(), GradModeVariable()]
TRACE SETUP_WITH 56 [WithExitFunctionVariable(), GradModeVariable()]
TRACE POP_TOP None [WithExitFunctionVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE NOP None [WithExitFunctionVariable(), WithExitFunctionVariable()]
TRACE LOAD_FAST ___stack2 [WithExitFunctionVariable(), WithExitFunctionVariable()]
TRACE LOAD_CONST False [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
TRACE STORE_FAST __is_tracing_resume_prologue [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(bool: False)]
TRACE JUMP_ABSOLUTE 158 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
TRACE POP_TOP None [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:8 in torch_dynamo_resume_in_fn_at_7
return x + 1
TRACE LOAD_FAST x [WithExitFunctionVariable(), WithExitFunctionVariable()]
TRACE LOAD_CONST 1 [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker()]
TRACE BINARY_ADD None [WithExitFunctionVariable(), WithExitFunctionVariable(), LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:5 in torch_dynamo_resume_in_fn_at_7
with torch.no_grad():
TRACE POP_BLOCK None [WithExitFunctionVariable(), WithExitFunctionVariable(), TensorVariable()]
TRACE ROT_TWO None [WithExitFunctionVariable(), WithExitFunctionVariable(), TensorVariable()]
TRACE LOAD_CONST None [WithExitFunctionVariable(), TensorVariable(), WithExitFunctionVariable()]
TRACE DUP_TOP None [WithExitFunctionVariable(), TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE DUP_TOP None [WithExitFunctionVariable(), TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]
TRACE CALL_FUNCTION 3 [WithExitFunctionVariable(), TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]
TRACE POP_TOP None [WithExitFunctionVariable(), TensorVariable(), ConstantVariable(NoneType: None)]
TRACE starts_line /tmp/ipykernel_1130/3152636365.py:4 in torch_dynamo_resume_in_fn_at_7
with contextlib.nullcontext():
TRACE POP_BLOCK None [WithExitFunctionVariable(), TensorVariable()]
TRACE ROT_TWO None [WithExitFunctionVariable(), TensorVariable()]
TRACE LOAD_CONST None [TensorVariable(), WithExitFunctionVariable()]
TRACE DUP_TOP None [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None)]
TRACE DUP_TOP None [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]
TRACE CALL_FUNCTION 3 [TensorVariable(), WithExitFunctionVariable(), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None), ConstantVariable(NoneType: None)]
TRACE POP_TOP None [TensorVariable(), ConstantVariable(NoneType: None)]
TRACE RETURN_VALUE None [TensorVariable()]
Step 1: torchdynamo done tracing torch_dynamo_resume_in_fn_at_7 (RETURN_VALUE)
RETURN_VALUE triggered compile
COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_1130/3152636365.py, line 4 in torch_dynamo_resume_in_fn_at_7>], graph_break=False)
TRACED GRAPH
===== __compiled_fn_31_a495bc45_720f_4a95_a0de_36220f81d126 =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# No stacktrace found for following nodes
_set_grad_enabled = torch._C._set_grad_enabled(False); _set_grad_enabled = None
# File: /tmp/ipykernel_1130/3152636365.py:8 in torch_dynamo_resume_in_fn_at_7, code: return x + 1
add: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None
# No stacktrace found for following nodes
_set_grad_enabled_1 = torch._C._set_grad_enabled(True); _set_grad_enabled_1 = None
return (add,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=5), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # return x + 1 # mp/ipykernel_1130/3152636365.py:8 in torch_dynamo_resume_in_fn_at_7
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # return x + 1 # mp/ipykernel_1130/3152636365.py:8 in torch_dynamo_resume_in_fn_at_7
| +- GuardManager: source=L['___stack0'], accessed_by=FrameLocalsGuardAccessor(key='___stack0', framelocals_idx=2), type=<class 'abc.ABCMeta'>, tag_safe=(True, False)
| | +- ID_MATCH: ___check_obj_id(L['___stack0'], 94270821252912) # torch._dynamo.graph_break() # mp/ipykernel_1130/3152636365.py:7 in torch_dynamo_resume_in_fn_at_7
| +- GuardManager: source=L['___stack1'], accessed_by=FrameLocalsGuardAccessor(key='___stack1', framelocals_idx=3), type=<class 'type'>, tag_safe=(True, False)
| | +- ID_MATCH: ___check_obj_id(L['___stack1'], 94270896125728) # torch._dynamo.graph_break() # mp/ipykernel_1130/3152636365.py:7 in torch_dynamo_resume_in_fn_at_7
Guard eval latency = 187.05 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
tensor([4.2707, 1.3815, 3.5170])
Try 块中的图中断#
Try 块中的图中断无法恢复
@torch.compile
def fn(x):
try:
x = x + 1
torch._dynamo.graph_break()
return x + 1
except Exception as e:
pass
fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_1130/2546874632.py:1, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/2546874632.py", line 9, in <module>
fn(torch.randn(3))
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/2546874632.py:1
create_env
TRACE starts_line /tmp/ipykernel_1130/2546874632.py:3 in fn
try:
TRACE SETUP_FINALLY 30 []
TRACE starts_line /tmp/ipykernel_1130/2546874632.py:4 in fn
x = x + 1
TRACE LOAD_FAST x []
TRACE LOAD_CONST 1 [LazyVariableTracker()]
TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_FAST x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/2546874632.py:5 in fn
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
empty checkpoint
run_gc_after_compile: running gc
Graph break: skip: from user code at:
File "/tmp/ipykernel_1130/2546874632.py", line 5, in fn
torch._dynamo.graph_break()
Traceback (most recent call last):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1624, in __call__
result = self._inner_convert(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__
result = _compile(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1433, in _compile
guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function
return function(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner
return _compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner
dynamo_output = compile_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame
bytecode, tracer_output = transform_code_object(code, transform)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object
tracer_output = transformations(instructions, code_options)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform
tracer_output = trace_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame
run_tracer()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer
tracer.run()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1487, in run
while self.step():
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step
self.dispatch_table[inst.opcode](self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper
return inner_fn(self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2320, in CALL_FUNCTION
self.call_function(fn, args, {})
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function
self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type]
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward
return getattr(self.realize(), name)(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function
unimplemented_v2(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
from user code:
File "/tmp/ipykernel_1130/2546874632.py", line 5, in fn
torch._dynamo.graph_break()
Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
WON'T CONVERT fn /tmp/ipykernel_1130/2546874632.py line 1
due to:
Traceback (most recent call last):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1624, in __call__
result = self._inner_convert(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__
result = _compile(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1433, in _compile
guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function
return function(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner
return _compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner
dynamo_output = compile_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame
bytecode, tracer_output = transform_code_object(code, transform)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object
tracer_output = transformations(instructions, code_options)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform
tracer_output = trace_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame
run_tracer()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer
tracer.run()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1487, in run
while self.step():
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step
self.dispatch_table[inst.opcode](self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper
return inner_fn(self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2320, in CALL_FUNCTION
self.call_function(fn, args, {})
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function
self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type]
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward
return getattr(self.realize(), name)(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function
unimplemented_v2(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
from user code:
File "/tmp/ipykernel_1130/2546874632.py", line 5, in fn
torch._dynamo.graph_break()
Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
Traceback (most recent call last):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1624, in __call__
result = self._inner_convert(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 688, in __call__
result = _compile(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1433, in _compile
guarded_code, tracer_output = compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 92, in wrapper_function
return function(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1117, in compile_inner
return _compile_inner(code, one_graph, hooks)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1151, in _compile_inner
dynamo_output = compile_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1032, in compile_frame
bytecode, tracer_output = transform_code_object(code, transform)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 1592, in transform_code_object
tracer_output = transformations(instructions, code_options)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1004, in transform
tracer_output = trace_frame(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 312, in _fn
return fn(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 815, in trace_frame
run_tracer()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 797, in run_tracer
tracer.run()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1487, in run
while self.step():
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1348, in step
self.dispatch_table[inst.opcode](self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 904, in wrapper
return inner_fn(self, inst)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2320, in CALL_FUNCTION
self.call_function(fn, args, {})
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1266, in call_function
self.push(fn.call_function(self, args, kwargs)) # type: ignore[arg-type]
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/lazy.py", line 212, in realize_and_forward
return getattr(self.realize(), name)(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py", line 1515, in call_function
unimplemented_v2(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/exc.py", line 596, in unimplemented_v2
raise Unsupported(msg)
torch._dynamo.exc.Unsupported: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
from user code:
File "/tmp/ipykernel_1130/2546874632.py", line 5, in fn
torch._dynamo.graph_break()
Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo"
tensor([2.6188, 3.9058, 1.7391])
我们可以通过将图中断移到 Try 块外部来避免跳过
@torch.compile
def fn(x):
try:
x = x + 1
except Exception as e:
pass
torch._dynamo.graph_break()
try:
return x + 1
except Exception as e:
pass
fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_1130/3015389759.py:1, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/3015389759.py", line 12, in <module>
fn(torch.randn(3))
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/3015389759.py:1
create_env
TRACE starts_line /tmp/ipykernel_1130/3015389759.py:3 in fn
try:
TRACE SETUP_FINALLY 14 []
TRACE starts_line /tmp/ipykernel_1130/3015389759.py:4 in fn
x = x + 1
TRACE LOAD_FAST x []
TRACE LOAD_CONST 1 [LazyVariableTracker()]
TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_FAST x [TensorVariable()]
TRACE POP_BLOCK None []
TRACE JUMP_FORWARD 50 []
TRACE starts_line /tmp/ipykernel_1130/3015389759.py:7 in fn
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
Graph break in user code at /tmp/ipykernel_1130/3015389759.py:7
Graph Break Reason: Call to `torch._dynamo.graph_break()`
Explanation: User-inserted graph break. Message: None
Hint: Remove the `torch._dynamo.graph_break()` call.
Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`
For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html
User code traceback:
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/3015389759.py", line 12, in <module>
fn(torch.randn(3))
File "/tmp/ipykernel_1130/3015389759.py", line 7, in fn
torch._dynamo.graph_break()
Restarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysis
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/3015389759.py:1
create_env
TRACE starts_line /tmp/ipykernel_1130/3015389759.py:3 in fn
try:
TRACE SETUP_FINALLY 14 []
TRACE starts_line /tmp/ipykernel_1130/3015389759.py:4 in fn
x = x + 1
TRACE LOAD_FAST x []
TRACE LOAD_CONST 1 [LazyVariableTracker()]
TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE STORE_FAST x [TensorVariable()]
TRACE POP_BLOCK None []
TRACE JUMP_FORWARD 50 []
TRACE starts_line /tmp/ipykernel_1130/3015389759.py:7 in fn
torch._dynamo.graph_break()
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]
TRACE LOAD_ATTR graph_break [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
COMPILING GRAPH due to GraphCompileReason(reason='Call to `torch._dynamo.graph_break()`\n Explanation: User-inserted graph break. Message: None\n Hint: Remove the `torch._dynamo.graph_break()` call.\n\n Developer debug context: Called `torch._dynamo.graph_break()` with args `[]`, kwargs `{}`\n\n For more details about this graph break, please visit: https://meta-pytorch.github.io/compile-graph-break-site/gb/gb0025.html', user_stack=[<FrameSummary file /tmp/ipykernel_1130/3015389759.py, line 7 in fn>], graph_break=True)
TRACED GRAPH
===== __compiled_fn_35_b21a68f3_1e9b_48d8_96cb_7258a569244f =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# File: /tmp/ipykernel_1130/3015389759.py:4 in fn, code: x = x + 1
x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None
return (x,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = x + 1 # mp/ipykernel_1130/3015389759.py:4 in fn
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = x + 1 # mp/ipykernel_1130/3015389759.py:4 in fn
| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(False, False)
| | +- GuardManager: source=G['torch'], accessed_by=DictGetItemGuardAccessor('torch'), type=<class 'module'>, tag_safe=(False, False)
| | | +- ID_MATCH: ___check_obj_id(G['torch'], 139667418978544) # torch._dynamo.graph_break() # mp/ipykernel_1130/3015389759.py:7 in fn
| | | +- GuardManager: source=G['torch']._dynamo, accessed_by=GetAttrGuardAccessor(_dynamo), type=<class 'module'>, tag_safe=(False, False)
| | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo, 139666007729360) # torch._dynamo.graph_break() # mp/ipykernel_1130/3015389759.py:7 in fn
| | | | +- GuardManager: source=G['torch']._dynamo.graph_break, accessed_by=GetAttrGuardAccessor(graph_break), type=<class 'function'>, tag_safe=(True, False)
| | | | | +- GuardManager: source=G['torch']._dynamo.graph_break.__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)
| | | | | | +- ID_MATCH: ___check_obj_id(G['torch']._dynamo.graph_break.__code__, 139665819779296) # torch._dynamo.graph_break() # mp/ipykernel_1130/3015389759.py:7 in fn
Guard eval latency = 1165.35 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
torchdynamo start compiling torch_dynamo_resume_in_fn_at_7 /tmp/ipykernel_1130/3015389759.py:7, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/3015389759.py", line 12, in <module>
fn(torch.randn(3))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
return fn(*args, **kwargs)
Step 1: torchdynamo start tracing torch_dynamo_resume_in_fn_at_7 /tmp/ipykernel_1130/3015389759.py:7
create_env
TRACE starts_line /tmp/ipykernel_1130/3015389759.py:7 in torch_dynamo_resume_in_fn_at_7
torch._dynamo.graph_break()
TRACE LOAD_CONST True []
TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]
TRACE LOAD_FAST ___stack0 []
TRACE LOAD_CONST False [LazyVariableTracker()]
TRACE STORE_FAST __is_tracing_resume_prologue [LazyVariableTracker(), ConstantVariable(bool: False)]
TRACE JUMP_ABSOLUTE 70 [LazyVariableTracker()]
TRACE POP_TOP None [LazyVariableTracker()]
TRACE starts_line /tmp/ipykernel_1130/3015389759.py:8 in torch_dynamo_resume_in_fn_at_7
try:
TRACE SETUP_FINALLY 84 []
TRACE starts_line /tmp/ipykernel_1130/3015389759.py:9 in torch_dynamo_resume_in_fn_at_7
return x + 1
TRACE LOAD_FAST x []
TRACE LOAD_CONST 1 [LazyVariableTracker()]
TRACE BINARY_ADD None [LazyVariableTracker(), ConstantVariable(int: 1)]
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE POP_BLOCK None [TensorVariable()]
TRACE RETURN_VALUE None [TensorVariable()]
Step 1: torchdynamo done tracing torch_dynamo_resume_in_fn_at_7 (RETURN_VALUE)
RETURN_VALUE triggered compile
COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_1130/3015389759.py, line 9 in torch_dynamo_resume_in_fn_at_7>], graph_break=False)
TRACED GRAPH
===== __compiled_fn_38_8e0bfecd_ee9a_456b_b671_069df5fbbef7 =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# File: /tmp/ipykernel_1130/3015389759.py:9 in torch_dynamo_resume_in_fn_at_7, code: return x + 1
add: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None
return (add,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=3), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # return x + 1 # mp/ipykernel_1130/3015389759.py:9 in torch_dynamo_resume_in_fn_at_7
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # return x + 1 # mp/ipykernel_1130/3015389759.py:9 in torch_dynamo_resume_in_fn_at_7
Guard eval latency = 1074.67 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
tensor([1.9553, 2.5961, 3.2747])
编译器错误#
某些编译器错误将导致函数被跳过。其他编译器错误将导致硬错误而不是跳过的函数。
处理跳过的函数#
总的来说，您可以通过修复导致函数被跳过的根本图中断或错误来解决跳过的函数。
如果导致跳过函数的图中断/错误难以修复，请考虑将其隔离到自己的函数中，以便只跳过最小部分的代码。
def inner1(x):
return x + 1
def inner2(x):
return x + 2
@torch.compile
def fn(x):
x = inner1(x)
def problematic_code():
torch._dynamo.skip_frame()
problematic_code()
x = inner2(x)
fn(torch.randn(3))
torchdynamo start compiling fn /tmp/ipykernel_1130/273153676.py:5, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/273153676.py", line 12, in <module>
fn(torch.randn(3))
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/273153676.py:5
create_env
TRACE starts_line /tmp/ipykernel_1130/273153676.py:7 in fn
x = inner1(x)
TRACE LOAD_GLOBAL inner1 []
TRACE LOAD_FAST x [LazyVariableTracker()]
TRACE CALL_FUNCTION 1 [LazyVariableTracker(), LazyVariableTracker()]
INLINING <code object inner1 at 0x7f06dbee1630, file "/tmp/ipykernel_1130/273153676.py", line 1>, inlined according trace_rules.lookup inlined by default
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE starts_line /tmp/ipykernel_1130/273153676.py:2 in inner1 (inline depth: 1)
return x + 1
TRACE LOAD_FAST x []
TRACE LOAD_CONST 1 [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]
TRACE RETURN_VALUE None [TensorVariable()]
DONE INLINING <code object inner1 at 0x7f06dbee1630, file "/tmp/ipykernel_1130/273153676.py", line 1>
TRACE STORE_FAST x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/273153676.py:8 in fn
def problematic_code():
TRACE LOAD_CONST <code object problematic_code at 0x7f06792a80e0, file "/tmp/ipykernel_1130/273153676.py", line 8> []
TRACE LOAD_CONST fn.<locals>.problematic_code [ConstantVariable(code: <code object problematic_code at 0x7f06792a80e0, file "/tmp/ipykernel_1130/273153676.py", line 8>)]
TRACE MAKE_FUNCTION 0 [ConstantVariable(code: <code object problematic_code at 0x7f06792a80e0, file "/tmp/ipykernel_1130/273153676.py", line 8>), ConstantVariable(str: 'fn.<locals>.problematic_code')]
TRACE STORE_FAST problematic_code [NestedUserFunctionVariable()]
TRACE starts_line /tmp/ipykernel_1130/273153676.py:10 in fn
problematic_code()
TRACE LOAD_FAST problematic_code []
TRACE CALL_FUNCTION 0 [NestedUserFunctionVariable()]
INLINING <code object problematic_code at 0x7f06792a80e0, file "/tmp/ipykernel_1130/273153676.py", line 8>, inlined according trace_rules.lookup inlined by default
TRACE starts_line /tmp/ipykernel_1130/273153676.py:9 in problematic_code (inline depth: 1)
torch._dynamo.skip_frame()
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]
TRACE LOAD_ATTR skip_frame [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
SKIPPED INLINING <code object problematic_code at 0x7f06792a80e0, file "/tmp/ipykernel_1130/273153676.py", line 8>: Skip frame due to `torch._dynamo.skip_frame()`. Message: None
Graph break in user code at /tmp/ipykernel_1130/273153676.py:10
Graph Break Reason: SKIPPED INLINING <code object problematic_code at 0x7f06792a80e0, file "/tmp/ipykernel_1130/273153676.py", line 8>: Skip frame due to `torch._dynamo.skip_frame()`. Message: None
User code traceback:
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/273153676.py", line 12, in <module>
fn(torch.randn(3))
File "/tmp/ipykernel_1130/273153676.py", line 10, in fn
problematic_code()
Restarting analysis due to _dynamo/symbolic_convert.py:249 in fail_and_restart_analysis
Step 1: torchdynamo start tracing fn /tmp/ipykernel_1130/273153676.py:5
create_env
TRACE starts_line /tmp/ipykernel_1130/273153676.py:7 in fn
x = inner1(x)
TRACE LOAD_GLOBAL inner1 []
TRACE LOAD_FAST x [LazyVariableTracker()]
TRACE CALL_FUNCTION 1 [LazyVariableTracker(), LazyVariableTracker()]
INLINING <code object inner1 at 0x7f06dbee1630, file "/tmp/ipykernel_1130/273153676.py", line 1>, inlined according trace_rules.lookup inlined by default
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE starts_line /tmp/ipykernel_1130/273153676.py:2 in inner1 (inline depth: 1)
return x + 1
TRACE LOAD_FAST x []
TRACE LOAD_CONST 1 [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 1)]
TRACE RETURN_VALUE None [TensorVariable()]
DONE INLINING <code object inner1 at 0x7f06dbee1630, file "/tmp/ipykernel_1130/273153676.py", line 1>
TRACE STORE_FAST x [TensorVariable()]
TRACE starts_line /tmp/ipykernel_1130/273153676.py:8 in fn
def problematic_code():
TRACE LOAD_CONST <code object problematic_code at 0x7f06792a80e0, file "/tmp/ipykernel_1130/273153676.py", line 8> []
TRACE LOAD_CONST fn.<locals>.problematic_code [ConstantVariable(code: <code object problematic_code at 0x7f06792a80e0, file "/tmp/ipykernel_1130/273153676.py", line 8>)]
TRACE MAKE_FUNCTION 0 [ConstantVariable(code: <code object problematic_code at 0x7f06792a80e0, file "/tmp/ipykernel_1130/273153676.py", line 8>), ConstantVariable(str: 'fn.<locals>.problematic_code')]
TRACE STORE_FAST problematic_code [NestedUserFunctionVariable()]
TRACE starts_line /tmp/ipykernel_1130/273153676.py:10 in fn
problematic_code()
TRACE LOAD_FAST problematic_code []
TRACE CALL_FUNCTION 0 [NestedUserFunctionVariable()]
COMPILING GRAPH due to GraphCompileReason(reason='SKIPPED INLINING <code object problematic_code at 0x7f06792a80e0, file "/tmp/ipykernel_1130/273153676.py", line 8>: Skip frame due to `torch._dynamo.skip_frame()`. Message: None', user_stack=[<FrameSummary file /tmp/ipykernel_1130/273153676.py, line 10 in fn>], graph_break=True)
TRACED GRAPH
===== __compiled_fn_41_87503dc9_8f1b_409f_85cb_de53e8f746d5 =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# File: /tmp/ipykernel_1130/273153676.py:2 in inner1, code: return x + 1
x: "f32[3][1]cpu" = l_x_ + 1; l_x_ = None
return (x,)
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=0), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = inner1(x) # mp/ipykernel_1130/273153676.py:7 in fn
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = inner1(x) # mp/ipykernel_1130/273153676.py:7 in fn
| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(True, False)
| | +- GuardManager: source=G['inner1'], accessed_by=DictGetItemGuardAccessor('inner1'), type=<class 'function'>, tag_safe=(True, False)
| | | +- GuardManager: source=G['inner1'].__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)
| | | | +- ID_MATCH: ___check_obj_id(G['inner1'].__code__, 139667436344880) # x = inner1(x) # mp/ipykernel_1130/273153676.py:7 in fn
Guard eval latency = 880.19 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
torchdynamo start compiling problematic_code /tmp/ipykernel_1130/273153676.py:8, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/273153676.py", line 12, in <module>
fn(torch.randn(3))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
return fn(*args, **kwargs)
Step 1: torchdynamo start tracing problematic_code /tmp/ipykernel_1130/273153676.py:8
create_env
TRACE starts_line /tmp/ipykernel_1130/273153676.py:9 in problematic_code
torch._dynamo.skip_frame()
TRACE LOAD_GLOBAL torch []
TRACE LOAD_ATTR _dynamo [LazyVariableTracker()]
TRACE LOAD_ATTR skip_frame [LazyVariableTracker()]
TRACE CALL_FUNCTION 0 [LazyVariableTracker()]
Skipping frame Skip frame due to `torch._dynamo.skip_frame()`. Message: None problematic_code /tmp/ipykernel_1130/273153676.py 8
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
torchdynamo start compiling torch_dynamo_resume_in_fn_at_10 /tmp/ipykernel_1130/273153676.py:10, stack (elided 5 frames):
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py_3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
app.launch_new_instance()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tornado/platform/asyncio.py", line 211, in start
self.asyncio_loop.run_forever()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
self._run_once()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
handle._run()
File "/opt/conda/envs/py_3.10/lib/python3.10/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 519, in dispatch_queue
await self.process_one()
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 508, in process_one
await dispatch(*args)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 400, in dispatch_shell
await result
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 368, in execute_request
await super().execute_request(stream, ident, parent)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 767, in execute_request
reply_content = await reply_content
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 455, in do_execute
res = shell.run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ipykernel/zmqshell.py", line 577, in run_cell
return super().run_cell(*args, **kwargs)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3006, in run_cell
result = self._run_cell(
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3061, in _run_cell
result = runner(coro)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/async_helpers.py", line 129, in _pseudo_sync_runner
coro.send(None)
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3266, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3445, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3505, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/tmp/ipykernel_1130/273153676.py", line 12, in <module>
fn(torch.randn(3))
File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 832, in compile_wrapper
return fn(*args, **kwargs)
Step 1: torchdynamo start tracing torch_dynamo_resume_in_fn_at_10 /tmp/ipykernel_1130/273153676.py:10
create_env
TRACE starts_line /tmp/ipykernel_1130/273153676.py:10 in torch_dynamo_resume_in_fn_at_10
problematic_code()
TRACE LOAD_CONST True []
TRACE STORE_FAST __is_tracing_resume_prologue [ConstantVariable(bool: True)]
TRACE LOAD_FAST ___stack0 []
TRACE LOAD_CONST False [LazyVariableTracker()]
TRACE STORE_FAST __is_tracing_resume_prologue [LazyVariableTracker(), ConstantVariable(bool: False)]
TRACE JUMP_ABSOLUTE 32 [LazyVariableTracker()]
TRACE POP_TOP None [LazyVariableTracker()]
TRACE starts_line /tmp/ipykernel_1130/273153676.py:11 in torch_dynamo_resume_in_fn_at_10
x = inner2(x)
TRACE LOAD_GLOBAL inner2 []
TRACE LOAD_FAST x [LazyVariableTracker()]
TRACE CALL_FUNCTION 1 [LazyVariableTracker(), LazyVariableTracker()]
INLINING <code object inner2 at 0x7f06796d61e0, file "/tmp/ipykernel_1130/273153676.py", line 3>, inlined according trace_rules.lookup inlined by default
wrap_to_fake L['x'] (3,) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>], dynamic_strides=[<DimDynamic.INFER_STRIDE: 4>], constraint_sizes=[None], constraint_strides=[None], specialize_on=[[]], view_base_context=None, tensor_source=LocalSource(local_name='x', is_input=True, dynamism=None, is_derefed_cell_contents=False), shape_env_to_source_to_symbol_cache={}) <class 'torch.Tensor'>
create_graph_input L_x_ L['x'] FakeTensor(..., size=(3,)) at debug_level 0 before=False
TRACE starts_line /tmp/ipykernel_1130/273153676.py:4 in inner2 (inline depth: 1)
return x + 2
TRACE LOAD_FAST x []
TRACE LOAD_CONST 2 [TensorVariable()]
TRACE BINARY_ADD None [TensorVariable(), ConstantVariable(int: 2)]
TRACE RETURN_VALUE None [TensorVariable()]
DONE INLINING <code object inner2 at 0x7f06796d61e0, file "/tmp/ipykernel_1130/273153676.py", line 3>
TRACE STORE_FAST x [TensorVariable()]
TRACE LOAD_CONST None []
TRACE RETURN_VALUE None [ConstantVariable(NoneType: None)]
Step 1: torchdynamo done tracing torch_dynamo_resume_in_fn_at_10 (RETURN_VALUE)
RETURN_VALUE triggered compile
COMPILING GRAPH due to GraphCompileReason(reason='return_value', user_stack=[<FrameSummary file /tmp/ipykernel_1130/273153676.py, line 11 in torch_dynamo_resume_in_fn_at_10>], graph_break=False)
TRACED GRAPH
===== __compiled_fn_45_6b698e84_de83_47c8_b624_23174d97626e =====
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/_lazy_graph_module.py class GraphModule(torch.nn.Module):
def forward(self, L_x_: "f32[3][1]cpu"):
l_x_ = L_x_
# File: /tmp/ipykernel_1130/273153676.py:4 in inner2, code: return x + 2
x: "f32[3][1]cpu" = l_x_ + 2; l_x_ = x = None
return ()
Step 2: calling compiler function eager
Step 2: done compiler function eager
produce_guards
track_symint L['x'].size()[0] 3 None
track_symint L['x'].stride()[0] 1 None
track_symint L['x'].storage_offset() 0 None
Skipping guard L['x'].size()[0] == 3
Skipping guard L['x'].stride()[0] == 1
Skipping guard L['x'].storage_offset() == 0
GUARDS:
TREE_GUARD_MANAGER:
+- RootGuardManager
| +- LAMBDA_GUARD: torch._functorch.aot_autograd.utils.top_saved_tensors_hooks ids == None # _dynamo/output_graph.py:688 in init_ambient_guards
| +- DEFAULT_DEVICE: utils_device.CURRENT_DEVICE == None # _dynamo/output_graph.py:676 in init_ambient_guards
| +- GLOBAL_STATE: ___check_global_state()
| +- TORCH_FUNCTION_MODE_STACK: ___check_torch_function_mode_stack()
| +- GuardManager: source=L['x'], accessed_by=FrameLocalsGuardAccessor(key='x', framelocals_idx=3), type=<class 'torch.Tensor'>, tag_safe=(True, False)
| | +- TENSOR_MATCH: check_tensor(L['x'], Tensor, DispatchKeySet(CPU, BackendSelect, ADInplaceOrView, AutogradCPU), torch.float32, device=None, requires_grad=False, size=[3], stride=[1]) # x = inner2(x) # mp/ipykernel_1130/273153676.py:11 in torch_dynamo_resume_in_fn_at_10
| | +- NO_HASATTR: hasattr(L['x'], '_dynamo_dynamic_indices') == False # x = inner2(x) # mp/ipykernel_1130/273153676.py:11 in torch_dynamo_resume_in_fn_at_10
| +- GuardManager: source=G, accessed_by=GlobalsGuardAccessor, type=<class 'dict'>, tag_safe=(True, False)
| | +- GuardManager: source=G['inner2'], accessed_by=DictGetItemGuardAccessor('inner2'), type=<class 'function'>, tag_safe=(True, False)
| | | +- GuardManager: source=G['inner2'].__code__, accessed_by=CodeGuardAccessor, type=<class 'code'>, tag_safe=(True, False)
| | | | +- ID_MATCH: ___check_obj_id(G['inner2'].__code__, 139665783742944) # x = inner2(x) # mp/ipykernel_1130/273153676.py:11 in torch_dynamo_resume_in_fn_at_10
Guard eval latency = 98.28 us
put_code_state: no cache key, skipping
run_gc_after_compile: running gc
