[lusoris]

integer_adm_scale0 of adm_cuda differs from the CPU by up to 0.41 on frames of 17 to 64 pixels. The rounding of the contrast masking sum explains about 90% of the scale 0 error summed over the sweep below. This is bug 2 of #1564 for scale 0; the same change covers the scale 1 to 3 reduction, where no frame of the sweep shows an effect.

Cause

The CPU (adm_cm(), i4_adm_cm()) sums a whole row of cubed contrast masking values in accum_inner and applies (accum_inner + add_shift_inner_accum) >> shift_inner_accum once per row. The CUDA kernels apply that shift to every partial sum before the atomic add:

• adm_cm_line_kernel (scale 0, fused): to the warp-reduced sum of a tile of 32 columns by 8 rows;
• adm_cm_reduce_line_kernel (scales 1 to 3): to the warp-reduced sum of 128 columns.

Rounding is not additive, so the accumulators drift from the CPU's by O(1) per extra warp. In the large accumulators of busy frames that is below the precision of the final float score. In the near-zero accumulators of smooth content it is a large relative error: a row sum the CPU rounds down to 0 becomes 1 in a tile that holds eight rows.

A second defect sits in the scale 0 setup. adm_cm_device() computed the rounding constant of the horizontal and vertical cubes on the host as 1 << (shift - 1). The shift is ceil(log2(w) - 4), which is 0 for a scale 0 band of 9 to 16 samples (frame widths 17 to 32), so the constant is 2^31 on x86 and the CPU's is 0.

Reproducer

Master 8e7a1ac4e, -Denable_cuda=true, RTX 4090. The CLI prints six digits, so the numbers below are from a C API program that prints %.17g; test_cuda_adm_cm_row_rounding makes the same comparison. A structured 38x38 8-bit frame (a ramp with a 3x3 checkerboard, the lower right quadrant mirrored; the generator is the test's), integer_adm_scale0:

	integer_adm_scale0
scalar CPU	0.638810
adm_cuda, master	0.953082
adm_cuda, this branch	0.638810

212 configurations (17x17 to 64x64 and fifteen other sizes up to 320x180, 8 and 10 bits, random and structured content, two frames), integer_adm_scale0 of adm_cuda against the scalar CPU, largest absolute difference over the configurations of the size group:

frame size	master	this branch
17..28 square	3.2e-1	7.8e-2
29..34 square	2.2e-1	0
35..64 square	3.3e-1	4.4e-4
17x64 to 130x40	4.1e-1	6.7e-2
96x96 to 320x180	6.7e-2	1.5e-5

Rows (size, bit depth) with a scale 0 difference: 88 of 106 on master, 30 on this branch. Sum over the rows of the largest difference: 16.85 on master, 1.68 on this branch. What remains on the branch is not this cause: the 17 to 28 pixel rows and 17x64 / 64x17 are the border clamp of the scale 0 contrast masking (next change in the series), and 55x55 and 320x180 at 8 bits are the angle test of the decouple stage (a separate change).

Fix

adm_cm_line_kernel stores its masked values per sample in tmp_accum, as i4_adm_cm_line_kernel already does. adm_cm_reduce_line_kernel is launched for both paths with one block per band and row (it already carried the constants of the scale 0 bands): the block cubes the values, reduces them in shared memory and thread 0 applies the shift once. The rounding constants of the reduction are shift ? 1u << (shift - 1) : 0. The WarpShift host structure and its host-side constants are removed.

Tests

test_cuda_adm_cm_row_rounding (new, libvmaf/test/, built with enable_cuda; it reports a skip when no CUDA device can be opened) compares integer_adm_scale0 of adm_cuda with the scalar CPU adm on 38x38, 48x48, 64x64, 130x40 and 200x120 at 8 and 10 bits, random and structured content, with a tolerance of 1e-9. Unpatched master fails it (38x38, 8 bit, random: 9.7e-6, structured: 0.31); this branch passes it. All sizes have both dimensions of 33 or more, where the scalar CPU on master equals the CPU with the fixes of #1599 and #1600.

Validation

Master 8e7a1ac4e, 2026-10-01. x86-64 Linux, GCC 16.2.1, nvcc 13.4, RTX 4090.

Rebased on master 9e48141b (2026-10-02). On that base, with -Denable_cuda=true -Denable_float=true -Denable_checkasm=true, meson test gives 28 of 29 (the failure is test_cuda_pic_preallocation, which also fails on unpatched 9e48141b, 27 of 28 there), and the CPU -Db_sanitize=address,undefined -Db_lto=false build gives 22 of 25 on master and here (test_predict and test_pic_preallocation abort under LeakSanitizer and checkasm aborts on a heap-buffer-overflow in adm_dwt2_16 (integer_adm.c:2603); all three also abort on unpatched 9e48141b). The CLI output of the three Netflix pairs (--gpumask 0, vmaf_v0.6.1) against unpatched 9e48141b: identical on all three pairs (src01 76.668905, checkerboard 1 px 35.068667, 10 px 7.985899). The sweeps and the other measurements below were taken on 8e7a1ac4e and not repeated; the files and x86 code paths they depend on are unchanged since (the one upstream change in between is an arm64-only ADM kernel).

• Real pairs, this branch against master and against the scalar CPU, adm_cuda at %.17g (integer_adm2, scales 0 to 3, numerators and denominators): src01 576x324 8 bit (48 frames) and 10 bit (3 frames), checkerboard 1 px and 10 px 1920x1080 (3 frames each), KristenAndSara 1280x720, akiyo 352x288, sparks 480x270 10 bit (5 frames): identical to both. The 160x90 clip of the Netflix test resources (48 frames) is identical to master; its remaining difference to the CPU is the border of scales 1 to 3 (previous change). On the 18x22 akiyo clip the largest scale 0 difference to the CPU goes from 7.2e-2 to 2.4e-2. No golden assertion changes.
• 1920x1080 structured (8 and 10 bit), 1280x720 and 960x540 structured, 3840x2160 bbb (2 frames): master and the merge of the five branches both equal the CPU on every output. The scale 1 to 3 reduction therefore has no measured effect on any frame I ran; it is changed because it is the same kernel and the CPU's contract is one rounding per row.
• Cost: not measured. The scale 0 path adds one launch, and the reduction has one block per row where it had one per 512 columns.
• Merging with the other CUDA ADM branches of the series: libvmaf/test/meson.build conflicts textually (every branch adds its test at the same place); adm_cm.cu and integer_adm_cuda.c merge cleanly. cuda: keep kernel parameter structs and accumulators out of local memory #1595 and cuda: build the kernels with clang without the CUDA toolkit headers #1596 change the same WarpShift lines of adm_cm.cu, which this change removes: whichever lands second needs a rebase.
• Full meson test on the merge of the five branches: 28 of 29 pass. test_cuda_pic_preallocation aborts with SIGSEGV in test_cuda_picture_preallocation_method_host_pinned; it does the same on unpatched master.

The workflow run on this PR needs a maintainer's approval.