XXHashJNI uses a stale local class reference for OutOfMemoryError and does not check createState results #100

File: src/jni/net_jpountz_xxhash_XXHashJNI.c

1. Stale local reference (line 30)

init stores the result of FindClass(env, "java/lang/OutOfMemoryError") in the static OutOfMemoryError field. That is a local reference, and it is freed once init returns. throw_OOM (line 34) then calls ThrowNew with the dangling reference.

You can reach this from the public API without memory pressure. A zero-length FileChannel.map(mode, 0, 0) buffer reports isDirect() == true but has a NULL native address, so GetDirectBufferAddress returns NULL and throw_OOM runs:

ByteBuffer bb = FileChannel.open(tmp, READ, WRITE).map(READ_WRITE, 0, 0);
XXHashFactory.nativeInstance().hash32().hash(bb, 0, 0, 0);

Observed with 1.11.2 on JDK 25:

• the first call threw NoSuchMethodError: ReadOnlyBufferException.<init>(String), because the freed slot now held a different class;
• the second call crashed the JVM with SIGSEGV;
• with -Xcheck:jni it failed with FATAL ERROR ... Bad global or local ref.

Fix: promote the reference to a global one, the way LZ4JNI.c already does (init_class_ref):

jclass c = (*env)->FindClass(env, "java/lang/OutOfMemoryError");
if (c) { OutOfMemoryError = (*env)->NewGlobalRef(env, c); (*env)->DeleteLocalRef(env, c); }

Optionally, XXH32BB/XXH64BB could return the hash of the empty input when the address is NULL and len == 0, instead of throwing.

2. Missing NULL checks on XXH*_createState (lines 92, 202)

The result of XXH32_createState() / XXH64_createState() is passed straight to XXH*_reset. If malloc fails, that dereferences NULL and crashes the JVM. Fix: if (state == NULL) { throw_OOM(env); return 0; }. This depends on fix 1.

Related: #83 (same class of problem in LZ4JNI.c).