[shadye-6]

Summary

Fix matrix stack underflow in swPopMatrix() for all three matrix modes in src/external/rlsw.h.

Closes #6361

Problem

during initialization, projection, modelview, and texture matrix stack counters are initialized to 1, representing their initial matrices. but, swPopMatrix() previously checked whether the respective counter was <= 0 before decrementing it.
When popping a stack containing only its initial matrix (counter == 1), the check was bypassed. The counter was decremented to zero, and the subsequent - 1 index calculation underflowed the uint32_t counter to 4294967295.
resulted in RLSW.currentMatrix pointing outside the corresponding matrix stack. subsequent matrix operations then caused out-of-bounds writes and AddressSanitizer segmentation faults.
affected SW_PROJECTION, SW_MODELVIEW, and SW_TEXTURE.

Solution

Changed the underflow checks for all three matrix stacks from <= 0 to <= 1.
This prevents popping the initial matrix, sets SW_STACK_UNDERFLOW, and preserves the existing matrix stack state.

Testing

I verified using a standalone rlsw test harness compiled with Clang 18.1.3, AddressSanitizer, and UndefinedBehaviorSanitizer:

• Before the fix: All three matrix modes produced UBSan out-of-bounds diagnostics followed by ASan segmentation faults when the invalid matrix pointer was used.
• After the fix: All three modes correctly returned SW_STACK_UNDERFLOW (1284) without sanitizer errors.
• Normal swPushMatrix(), matrix transformations, and swPopMatrix() operations were tested across all three modes and continued to work correctly.

standalone reproducer was tested on Ubuntu 24.04.3 LTS under WSL2.

[shadye-6]

hey @raysan5, just following up on this PR, it's a pretty small fix, could you pls review? thanks for your time.

[raysan5]

@Bigfoot71 it looks good to me, despite the apparently AI generated issue, but please, could you take a second look to confirm merge?