镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

libvmaf/feature: avoid static close() shadowing close(2) (fixes Darwin LTO miscompile) - #1696

Open
StormBytePP wants to merge 1 commit into
Netflix:masterfrom
StormBytePP:lto_darwin_miscompile
Open

StormBytePP wants to merge 1 commit into
Netflix:masterfrom
StormBytePP:lto_darwin_miscompile

Conversation

@StormBytePP

@StormBytePP StormBytePP commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

libvmaf/feature: rename static close() callbacks to close_fex()

Summary

Thirteen feature extractors define their close callback as

static int close(VmafFeatureExtractor *fex)

This name is the same as POSIX close(2). When libvmaf is statically linked with LTO on Apple platforms, the linker has bound other code's close(fd) calls to one of these static functions. The result is a silent wrong-call-target miscompile that crashes the process. This PR renames the callbacks so libvmaf no longer defines any symbol called close.

This is not an API or ABI change. The renamed functions are static. Nothing outside their own file refers to them by name, and they are reached only through VmafFeatureExtractor.close. No public header, exported symbol, struct layout or runtime behaviour changes.

What happens

We statically link libvmaf, FFmpeg and our own library into one image with Apple Clang and -flto (macOS, arm64). With LTO, ld64 resolved the close(fd) call in FFmpeg's libavformat/file.c (file_close()) to one of libvmaf's internal close(VmafFeatureExtractor *) functions instead of libSystem's close.

The file descriptor is then treated as a VmafFeatureExtractor *. The first thing each of these callbacks does is load fex->priv, which is at offset 0x38 on 64-bit:

field offset
name 0x00
init 0x08
extract 0x10
flush 0x18
close 0x20
merge 0x28
options 0x30
priv 0x38

With fd 3 this faults at 3 + 0x38 = 0x3b, which is exactly what we see:

<paste the Darwin crash report / backtrace here>
EXC_BAD_ACCESS (SIGSEGV) at 0x000000000000003b
  ... file_close (libavformat/file.c)
  ... avformat_close_input

The crash happens on the normal path: open a file, decode successfully, then call avformat_close_input().

Why it only shows up on Darwin + LTO

  • Without LTO, each static close stays private to its object file and the build works.
  • On Linux (GCC and Clang, with and without LTO) and on Windows (clang-cl), the same sources and link configuration do not show the problem.
  • On macOS with Apple Clang + LTO, libvmaf's bitcode and the close(fd) caller go through the same LTO link, and the call resolves to the wrong function.

So this cannot be reproduced on Linux or Windows. It is still a serious bug, for three reasons:

  • It is silent. Nothing fails at build time; the program just calls the wrong function with an int that is used as a pointer.
  • It affects code that is not libvmaf. Any close(fd) in the same LTO image is at risk, not only libvmaf's own code.
  • What happens depends on which close the linker chooses and on the fd value. Here it crashed, but with another layout it could read or free unrelated memory instead.

Precedent: FFmpeg made the same change

FFmpeg had the same pattern in its parsers and renamed them for the same reason:

  • FFV1 parser: close → renamed in FFmpeg commit 584c41a4ac.
  • APV parser: static av_cold void close(AVCodecParserContext *) → apv_close, merged upstream (#25211)

Once our APV fix was in, libvmaf's 13 static close functions were the only definitions of that name left in our link. Renaming them is what fixed the crash.

The change

  • close() → close_fex() in ciede.c, float_adm.c, float_ansnr.c, float_moment.c, float_motion.c, float_ms_ssim.c, float_psnr.c, float_ssim.c, float_vif.c, integer_adm.c, integer_ssim.c and integer_vif.c.
    integer_motion.c already uses close_fex, so this follows an existing name in the tree.
  • close() → close_temporal() in speed.c, next to the existing close_chroma() in the same file. cambi.c already uses close_cambi.

26 lines changed: each function definition and its .close = ... initializer.

Why fix it in libvmaf

We can't control every downstream toolchain, and libvmaf is commonly linked statically into larger media stacks such as FFmpeg builds. The rename costs nothing, matches names already used in the tree, and removes a whole class of symbol collision for every consumer.

Testing

  • libvmaf builds cleanly. nm on the built library shows no close symbol, only close_fex / close_temporal locals.
  • Our macOS (Apple Clang, arm64) LTO CI used to crash in file_close() → avformat_close_input(). With this patch it passes, without any workaround in FFmpeg.
  • Linux and Windows builds and tests are unchanged.

Thirteen feature extractors define their close callback as

    static int close(VmafFeatureExtractor *fex)

That name shadows close(2) from <unistd.h>. In C, a static function is
private to its translation unit. LTO toolchains do not always keep it
that way. When libvmaf is linked into an application with LTO together
with other code that calls close(fd), Apple's ld64 has bound those
close(fd) calls to one of these static functions. The file descriptor
is then used as a VmafFeatureExtractor pointer. The first thing every
one of these callbacks does is load fex->priv, which is at offset 0x38,
so the process dies with EXC_BAD_ACCESS at fd + 0x38 (0x3b for fd 3).

We hit this in a macOS arm64 build with Apple Clang and -flto that
statically links libvmaf with FFmpeg. The crash was in
libavformat's file_close() during avformat_close_input(). It goes away
without LTO, and the same build works on Linux and Windows. No LTO
mode is needed to reproduce it on its own; the only requirement is
that libvmaf's bitcode and a close(fd) caller end up in the same LTO
link. FFmpeg removed the same pattern from its parsers for the same
reason (584c41a4ac, FFV1 parser, and the APV parser after it).

This change renames the callbacks:

  - close() -> close_fex() in ciede.c, float_adm.c, float_ansnr.c,
    float_moment.c, float_motion.c, float_ms_ssim.c, float_psnr.c,
    float_ssim.c, float_vif.c, integer_adm.c, integer_ssim.c and
    integer_vif.c. integer_motion.c already uses this name.
  - close() -> close_temporal() in speed.c. It sits next to
    close_chroma() in the same file.

This is not an API or ABI change. The functions are static, nothing
outside their own file refers to them by name, and they are still
reached only through VmafFeatureExtractor.close. No public header,
exported symbol or struct layout changes, and runtime behaviour is
the same. The patch only removes the libc name from libvmaf's
internal symbols. After it, the library defines no symbol named
close.

Signed-off-by: David C. Manuelda <StormByte@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant