Crash on AST with misordered linenos

Compile 3.11 branch
./python.exe -m ensurepip
./python.exe -m pip install pytest-cov
./python.exe -m pytest

Crashes with Assertion failed: (i->i_end_lineno >= i->i_lineno), function write_location_info_long_form, file compile.c, line 7532.

lldb stack trace:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = hit program assert
    frame #0: 0x00007fff203cd92e libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff203fc5bd libsystem_pthread.dylib`pthread_kill + 263
    frame #2: 0x00007fff20351406 libsystem_c.dylib`abort + 125
    frame #3: 0x00007fff203507d8 libsystem_c.dylib`__assert_rtn + 314
  * frame #4: 0x000000010027c183 python.exe`write_location_info_long_form.cold.2 at compile.c:7532:5 [opt]
    frame #5: 0x000000010018f469 python.exe`write_location_info_long_form(a=<unavailable>, i=<unavailable>, length=<unavailable>) at compile.c:7532:5 [opt]
    frame #6: 0x000000010018f27d python.exe`write_location_info_entry(a=0x00007ffeefbfe5d0, i=0x000000010388ac98, isize=1) at compile.c:7590:5 [opt]
    frame #7: 0x000000010018d2cd python.exe`assemble_emit_location(a=0x00007ffeefbfe5d0, i=0x000000010388ac98) at compile.c:7605:12 [opt]
    frame #8: 0x000000010017e9fa python.exe`assemble(c=<unavailable>, addNone=<unavailable>) at compile.c:8411:18 [opt]
    frame #9: 0x000000010017d4a3 python.exe`compiler_mod(c=0x00007ffeefbfe6b8, mod=<unavailable>) at compile.c:2203:10 [opt]
    frame #10: 0x000000010017d2b3 python.exe`_PyAST_Compile(mod=0x00000001038852b0, filename=0x000000010654d630, flags=<unavailable>, optimize=<unavailable>, arena=0x00000001064b0100) at compile.c:581:10 [opt]
    frame #11: 0x00000001001634e0 python.exe`builtin_compile_impl(module=<unavailable>, source=0x00000001065a3fc0, filename=0x000000010654d630, mode=<unavailable>, flags=<unavailable>, dont_inherit=1, optimize=-1, feature_version=-1) at bltinmodule.c:818:33 [opt]
    frame #12: 0x0000000100161fb6 python.exe`builtin_compile(module=<unavailable>, args=0x00007ffeefbfe7f0, nargs=<unavailable>, kwnames=<unavailable>) at bltinmodule.c.h:328:20 [opt]
    frame #13: 0x00000001000cbb08 python.exe`cfunction_vectorcall_FASTCALL_KEYWORDS(func=0x0000000100b9f4d0, args=0x0000000100c6fe00, nargsf=<unavailable>, kwnames=0x0000000102c72260) at methodobject.c:443:24 [opt]
    frame #14: 0x0000000100087018 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x0000000100b9f4d0, args=0x0000000100c6fe00, nargsf=9223372036854775811, kwnames=0x0000000102c72260) at pycore_call.h:92:11 [opt]
    frame #15: 0x0000000100086fa0 python.exe`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:299:12 [opt]
    frame #16: 0x0000000100170d7c python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6fd70, throwflag=<unavailable>) at ceval.c:4826:23 [opt]
    frame #17: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #18: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000104737d80, locals=0x0000000000000000, args=0x00007ffeefbfeb80, argcount=3, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #19: 0x0000000100087330 python.exe`_PyFunction_Vectorcall(func=0x0000000104737d80, stack=0x00007ffeefbfeb80, nargsf=<unavailable>, kwnames=0x0000000000000000) at call.c:0:45 [opt]
    frame #20: 0x0000000100089fd8 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x0000000104737d80, args=0x00007ffeefbfeb80, nargsf=3, kwnames=0x0000000000000000) at pycore_call.h:92:11 [opt]
    frame #21: 0x0000000100089644 python.exe`method_vectorcall(method=<unavailable>, args=0x00000001064c9788, nargsf=<unavailable>, kwnames=0x0000000000000000) at classobject.c:89:18 [opt]
    frame #22: 0x0000000100086f3d python.exe`_PyVectorcall_Call(tstate=0x0000000100486930, func=(python.exe`method_vectorcall at classobject.c:44), callable=0x000000010640d310, tuple=0x00000001064c9770, kwargs=0x0000000000000000) at call.c:245:16 [opt]
    frame #23: 0x0000000100087148 python.exe`_PyObject_Call(tstate=0x0000000100486930, callable=0x000000010640d310, args=0x00000001064c9770, kwargs=0x0000000000000000) at call.c:328:16 [opt]
    frame #24: 0x0000000100087209 python.exe`PyObject_Call(callable=<unavailable>, args=<unavailable>, kwargs=<unavailable>) at call.c:355:12 [opt]
    frame #25: 0x0000000100177419 python.exe`do_call_core(tstate=0x0000000100486930, func=0x000000010640d310, callargs=0x00000001064c9770, kwdict=0x0000000000000000, use_tracing=<unavailable>) at ceval.c:0:13 [opt]
    frame #26: 0x00000001001733b0 python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6f580, throwflag=<unavailable>) at ceval.c:5431:22 [opt]
    frame #27: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #28: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000104053490, locals=0x0000000000000000, args=0x00000001064c0a48, argcount=1, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #29: 0x0000000100087330 python.exe`_PyFunction_Vectorcall(func=0x0000000104053490, stack=0x00000001064c0a48, nargsf=<unavailable>, kwnames=0x00000001064c8f50) at call.c:0:45 [opt]
    frame #30: 0x00000001000866b2 python.exe`_PyObject_FastCallDictTstate(tstate=0x0000000100486930, callable=0x0000000104053490, args=0x00007ffeefbfef70, nargsf=1, kwargs=0x00000001064c6d50) at call.c:152:15 [opt]
    frame #31: 0x00000001000875e4 python.exe`_PyObject_Call_Prepend(tstate=<unavailable>, callable=<unavailable>, obj=0x000000010622f3a0, args=<unavailable>, kwargs=<unavailable>) at call.c:482:24 [opt]
    frame #32: 0x00000001000ee1f9 python.exe`slot_tp_call(self=0x000000010622f3a0, args=0x000000010046c5a0, kwds=0x00000001064c6d50) at typeobject.c:7604:15 [opt]
    frame #33: 0x000000010008689c python.exe`_PyObject_MakeTpCall(tstate=0x0000000100486930, callable=0x000000010622f3a0, args=0x0000000100c6f408, nargs=<unavailable>, keywords=0x0000000104694960) at call.c:214:18 [opt]
    frame #34: 0x0000000100087049 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x000000010622f3a0, args=0x0000000100c6f408, nargsf=9223372036854775808, kwnames=0x0000000104694960) at pycore_call.h:90:16 [opt]
    frame #35: 0x0000000100086fa0 python.exe`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:299:12 [opt]
    frame #36: 0x0000000100170d7c python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6f380, throwflag=<unavailable>) at ceval.c:4826:23 [opt]
    frame #37: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #38: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000102881b20, locals=0x0000000102830350, args=0x0000000000000000, argcount=0, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #39: 0x0000000100166615 python.exe`PyEval_EvalCode(co=<unavailable>, globals=0x0000000102830350, locals=0x0000000102830350) at ceval.c:1207:21 [opt]
    frame #40: 0x0000000100163b09 python.exe`builtin_exec_impl(module=<unavailable>, source=0x000000010617f230, globals=0x0000000102830350, locals=0x0000000102830350, closure=<unavailable>) at bltinmodule.c:1080:17 [opt]
    frame #41: 0x000000010016224d python.exe`builtin_exec(module=<unavailable>, args=0x0000000100c6f180, nargs=2, kwnames=0x0000000000000000) at bltinmodule.c.h:465:20 [opt]
    frame #42: 0x00000001000cbb08 python.exe`cfunction_vectorcall_FASTCALL_KEYWORDS(func=0x0000000100b9f6b0, args=0x0000000100c6f180, nargsf=<unavailable>, kwnames=0x0000000000000000) at methodobject.c:443:24 [opt]
    frame #43: 0x0000000100087018 python.exe`_PyObject_VectorcallTstate(tstate=0x0000000100486930, callable=0x0000000100b9f6b0, args=0x0000000100c6f180, nargsf=9223372036854775810, kwnames=0x0000000000000000) at pycore_call.h:92:11 [opt]
    frame #44: 0x0000000100086fa0 python.exe`PyObject_Vectorcall(callable=<unavailable>, args=<unavailable>, nargsf=<unavailable>, kwnames=<unavailable>) at call.c:299:12 [opt]
    frame #45: 0x0000000100170d7c python.exe`_PyEval_EvalFrameDefault(tstate=0x0000000100486930, frame=0x0000000100c6f0d8, throwflag=<unavailable>) at ceval.c:4826:23 [opt]
    frame #46: 0x000000010016680d python.exe`_PyEval_EvalFrame(tstate=<unavailable>, frame=<unavailable>, throwflag=<unavailable>) at pycore_ceval.h:66:16 [opt]
    frame #47: 0x0000000100166726 python.exe`_PyEval_Vector(tstate=0x0000000100486930, func=0x0000000102c5eaf0, locals=0x0000000000000000, args=0x0000000102894ba8, argcount=2, kwnames=<unavailable>) at ceval.c:6468:24 [opt]
    frame #48: 0x0000000100087330 python.exe`_PyFunction_Vectorcall(func=0x0000000102c5eaf0, stack=0x0000000102894ba8, nargsf=<unavailable>, kwnames=0x0000000000000000) at call.c:0:45 [opt]
    frame #49: 0x0000000100086f3d python.exe`_PyVectorcall_Call(tstate=0x0000000100486930, func=(python.exe`_PyFunction_Vectorcall at call.c:385), callable=0x0000000102c5eaf0, tuple=0x0000000102894b90, kwargs=0x0000000000000000) at call.c:245:16 [opt]
    frame #50: 0x0000000100087148 python.exe`_PyObject_Call(tstate=0x0000000100486930, callable=0x0000000102c5eaf0, args=0x0000000102894b90, kwargs=0x0000000000000000) at call.c:328:16 [opt]
    frame #51: 0x0000000100087209 python.exe`PyObject_Call(callable=<unavailable>, args=<unavailable>, kwargs=<unavailable>) at call.c:355:12 [opt]
    frame #52: 0x00000001001d9f3d python.exe`pymain_run_module(modname=<unavailable>, set_argv0=<unavailable>) at main.c:300:14 [opt]
    frame #53: 0x00000001001d982d python.exe`pymain_run_python(exitcode=0x00007ffeefbff7dc) at main.c:595:21 [opt]
    frame #54: 0x00000001001d9718 python.exe`Py_RunMain at main.c:680:5 [opt]
    frame #55: 0x00000001001d99af python.exe`pymain_main(args=<unavailable>) at main.c:710:12 [opt]
    frame #56: 0x00000001001d99fb python.exe`Py_BytesMain(argc=<unavailable>, argv=<unavailable>) at main.c:734:12 [opt]
    frame #57: 0x0000000100001999 python.exe`main(argc=<unavailable>, argv=<unavailable>) at python.c:15:12 [opt]
    frame #58: 0x00007fff20417f3d libdyld.dylib`start + 1

Your environment

CPython versions tested on: 3.11 current head
Operating system and architecture: macOS

I put print statements in that show the file it crashes on is '/usr/local/lib/python3.11/site-packages/pytest_cov/plugin.py', but just running Python on that file doesn't reproduce the crash. I'll try to debug some more.

Simple repro:

from ast import *
tree = Module(body=[
    Import(names=[alias(name='builtins', lineno=1, col_offset=0)], lineno=1, col_offset=0), 
    Import(names=[alias(name='traceback', lineno=0, col_offset=0)], lineno=0, col_offset=1)
], type_ignores=[])
compile(tree, "x", "exec", dont_inherit=True)

% ./python.exe wronglinenos.py 
Assertion failed: (i->i_end_lineno >= i->i_lineno), function write_location_info_long_form, file compile.c, line 7532.
zsh: abort      ./python.exe wronglinenos.py

944fffe is the first bad commit

commit 944fffee8916cb94321fa33cd3a43f4108717746
Author: Mark Shannon <mark@hotpy.org>
Date:   Thu Apr 21 16:10:37 2022 +0100

    GH-88116: Use a compact format to represent end line and column offsets. (GH-91666)
    
    * Stores all location info in linetable to conform to PEP 626.
    
    * Remove column table from code objects.
    
    * Remove end-line table from code objects.
    
    * Document new location table format

@markshannon

@markshannon

Looks like the ast defaults the end_lineno to None, which then becomes 0 in the compiler. Maybe the ast should default it be equal to lineno? Or should None become -1 in c?

Both, perhaps.
If no end_lineno is specified, then it should probably default to lineno
If the end_lineno is None then we should convert that to -1, as we do for lineno.

@pablogsal

cc @pablogsal

Both, perhaps.
If no end_lineno is specified, then it should probably default to lineno
If the end_lineno is None then we should convert that to -1, as we do for lineno.

That's what we do in some parts that deal with these information. I can prepare a PR

The question is if we should manage this in the AST creation routines or in the compiler. I assume that the first is more resilient but what do others think?

@bluetech

FWIW, this seems to affect pytest's assertion rewriting as well. As soon as I'm running pytest against e.g.:

def test_foo():
    pass

with a Python 3.11.0b1 configured with --with-pydebug, I get:

collecting ... python: Python/compile.c:7531: write_location_info_long_form: Assertion `i->i_end_lineno >= i->i_lineno' failed.
Fatal Python error: Aborted

Current thread 0x00007f2555185740 (most recent call first):
  File ".../site-packages/_pytest/assertion/rewrite.py", line 361 in _rewrite_test
  File ".../site-packages/_pytest/assertion/rewrite.py", line 159 in exec_module
  ...

which is here:

https://github.com/pytest-dev/pytest/blob/c988e49af63a50f0704a46a93e73463666451af0/src/_pytest/assertion/rewrite.py#L361

and indeed i_end_lineno is 0:

(gdb) pp i
i = 
   autoderefcount="1",[
      i_opcode = <int> = {"100"}
      i_oparg = <int> = {"0"}
      i_target = <struct basicblock_*> = {"0x0"}
      i_except = <struct basicblock_*> = {"0x0"}
      i_lineno = <int> = {"1"}
      i_end_lineno = <int> = {"0"}
      i_col_offset = <int> = {"0"}
      i_end_col_offset = <int> = {"0"}
   ],<struct instr> = {"{...}"}

Turning pytest's assertion rewriting off with --assert=plain helps, and I can confirm it works fine before 944fffe.

Passing the end_lineno where we generate fake imports for the pytest assertion rewriting helpers seems to help:

diff --git i/src/_pytest/assertion/rewrite.py w/src/_pytest/assertion/rewrite.py
index 81096764e..5d7392dd4 100644
--- i/src/_pytest/assertion/rewrite.py
+++ w/src/_pytest/assertion/rewrite.py
@@ -727,7 +727,7 @@ def run(self, mod: ast.Module) -> None:
                 ast.alias("_pytest.assertion.rewrite", "@pytest_ar"),
             ]
         imports = [
-            ast.Import([alias], lineno=lineno, col_offset=0) for alias in aliases
+            ast.Import([alias], lineno=lineno, end_lineno=item.end_lineno, col_offset=0) for alias in aliases
         ]
         mod.body[pos:pos] = imports

cc @bluetech - though I suppose we don't necessarily need to fix this in pytest, hence I didn' t open an issue over there.

Another project affected in the wild seems to be flask:

$ python3.11 -X faulthandler -c "import flask; flask.Flask('test')"
python: Python/compile.c:7534: write_location_info_long_form: Assertion `i->i_end_lineno >= i->i_lineno' failed.
Fatal Python error: Aborted

Current thread 0x00007fd44da08740 (most recent call first):
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 1073 in _compile_builder
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 885 in compile
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 805 in bind
  File ".../lib/python3.11/site-packages/werkzeug/routing.py", line 1546 in add
  File ".../lib/python3.11/site-packages/flask/app.py", line 1086 in add_url_rule
  File ".../lib/python3.11/site-packages/flask/scaffold.py", line 56 in wrapper_func
  File ".../lib/python3.11/site-packages/flask/app.py", line 511 in __init__
  File "<string>", line 1 in <module>

I'm working on a patch, will have a draft PR ready soon

@pablogsal

@pablogsal Do you have time to work on this, or should I fix it?

@pablogsal

@pablogsal Do you have time to work on this, or should I fix it?

I have opened a PR for this.

gh-92597: Ensure that AST nodes without explicit end positions can be…

pythongh-92597: Ensure that AST nodes without explicit end positions …

00381: pythongh-92597: Ensure that AST nodes without explicit end pos…

pythongh-92597: Improve error message for AST nodes with invalid ranges

gh-92597: Ensure that AST nodes without explicit end positions can be…

pythongh-92597: Improve error message for AST nodes with invalid rang…

gh-92597: Improve error message for AST nodes with invalid ranges (GH…

Navigation Menu

Repository navigation

Crash on AST with misordered linenos #92597

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Crash on AST with misordered linenos #92597

Description

Activity

JelleZijlstra commented on May 10, 2022

kumaraditya303 commented on May 10, 2022

iritkatriel commented on May 10, 2022

markshannon commented on May 10, 2022

iritkatriel commented on May 11, 2022

pablogsal commented on May 11, 2022

pablogsal commented on May 11, 2022

The-Compiler commented on May 25, 2022

The-Compiler commented on May 25, 2022

pablogsal commented on May 25, 2022

markshannon commented on May 27, 2022

pablogsal commented on May 30, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions