JIT: stale ZF after rotate causes wrong-direction branch in if (BitOperations.RotateRight(x, n) == 0) pattern #129288

Summary

The x64 JIT emits ror reg, 1 followed directly by jne to test whether
BitOperations.RotateRight(value, 1) is non-zero, but ROR does not
modify ZF on x64. The jne therefore reads stale ZF from a prior
unrelated cmp, producing wrong-direction branches and corrupting control
flow. Discovered by an experimental in-development fuzzer.

Root cause

GenTree::SupportsSettingZeroFlag() in
gentree.cpp:21685
treats rotates the same as shifts:

#if defined(TARGET_XARCH)
    if (OperIs(GT_LSH, GT_RSH, GT_RSZ, GT_ROL, GT_ROR))
    {
        // Shift/Rotate instructions do not update the flags in case of count being zero.
        return gtGetOp2()->IsNeverZero();
    }

Lowering uses this signal to drop a TEST/CMP before a zero-comparison
branch. That is correct for shifts (which set ZF when count != 0) but
wrong for rotates: per the Intel SDM, ROL/ROR/RCL/RCR only
affect CF (and OF for 1-bit form). SF, ZF, AF, PF are not
affected, regardless of count. So even with a non-zero count, a rotate
does not produce a meaningful ZF.

GT_ROL and GT_ROR were added to the supports-zero-flag set in
#113037 (Apr 2025) and
the partial repair in
#117960 (Jul 2025)
moved the IsNeverZero check but kept rotates in the same block.

The proposed fix is to split them:

if (OperIs(GT_LSH, GT_RSH, GT_RSZ))
{
    // Shift instructions do not update the flags in case of count being zero.
    return gtGetOp2()->IsNeverZero();
}
// GT_ROL / GT_ROR omitted: rotates never modify ZF on xarch.

Repro

C# source (verified to throw on FullOpts, succeed on MinOpts and tiered):

using System;
using System.Numerics;
using System.Runtime.CompilerServices;

public static class Program
{
    private static ulong Input_p0 = 0UL;
    private static ulong Input_p1 = 0UL;
    private static int[] __trace = new int[64];
    private static int __traceLen = 0;

    [MethodImpl(MethodImplOptions.NoInlining)]
    public static ulong Fn(ulong p0, ulong p1)
    {
        unchecked
        {
            ulong v4 = 0UL, v11 = 0UL, v30 = 0UL, v33 = 0UL, v34 = 0UL;
            int v12 = 0;
        b0:
            __trace[__traceLen++] = 0;
            v4 = BitOperations.RotateLeft(p0, (int)(ulong)0x7FFFFFFFFFFFFFFFUL);
        b1:
            __trace[__traceLen++] = 1;
        b2:
            __trace[__traceLen++] = 2;
            v11 = BitOperations.RotateRight(v4, 1);
            v12 = (v11 < 1UL) ? 1 : 0;
            if (v12 != 0) goto b3;
            goto b4;
        b3:
            __trace[__traceLen++] = 3;
            goto b6;
        b4:
            __trace[__traceLen++] = 4;
            goto b7;
        b6:
            __trace[__traceLen++] = 6;
            goto b0;
        b7:
            __trace[__traceLen++] = 7;
            v34 = v33 % v30;   // dead in source semantics
            goto b0;
        }
    }

    public static int Main()
    {
        ulong result = Fn(Input_p0, Input_p1);
        Console.WriteLine($"{result:X16}");
        return 0;
    }
}

With DOTNET_TieredCompilation=0 (FullOpts), this throws
System.DivideByZeroException on the very first iteration of the loop.

Under the source semantics, control should go
b0 → b1 → b2 → b3 → b6 → b0 → ... repeatedly: v4 is
RotateLeft(p0=0, ...) = 0, so v11 = RotateRight(0, 1) = 0, so
v12 = (v11 < 1) ? 1 : 0 = 1, so the branch v12 != 0 is taken to
b3. Block b7 (with the v33 % v30 where both are 0) is
unreachable. With __trace sized at 64, the correct semantics eventually
overflow the trace array on iteration ~13 with
IndexOutOfRangeException; with default tiered or JitMinOpts=1 that
is what is observed.

Generated code (offending fragment, FullOpts x64)

G_M44088_IG04:        ; BB07 (main loop body)
    ...
    cmp   ebx, r11d            ; bounds check for __trace store (sets ZF)
    jae   SHORT G_M44088_IG06
    mov   dword ptr [rdx+4*rbx+0x10], 2
    ror   rax, 1               ; ROR does NOT modify ZF
    jne   G_M44088_IG03        ; reads STALE ZF from the cmp above

G_M44088_IG03:        ; BB04 -> b4/b7
    ...
    xor   eax, eax
    xor   r11d, r11d
    xor   edx, edx
    div   rdx:rax, r11         ; <-- DivideByZeroException

The cmp ebx, r11d immediately preceding the rotate-then-branch
compares the trace cursor with __trace.Length = 64. Since
ebx != 64, ZF = 0, and the jne is taken — into the unreachable
b4/b7 path that does % 0.

Environment

• HEAD of main as of 2026-06-10 (Checked CoreCLR, x64 Windows).
• Repros on corerun with DOTNET_TieredCompilation=0.
• Discovered using ReifyCs (in-development C# fuzzer); seed 2000017594.

Suggested test

A regression test could synthesize any
if (BitOperations.RotateRight(x, n) == 0) pattern preceded by a
flag-setting cmp whose ZF differs from "is rotate result zero",
asserting the branch direction matches the interpreter.