java.lang.IllegalArgumentException: Bad position on Android 4.x

[dmitrychistyakov]

simple packString causes exception on Android 4.x, Android 5.x and later works fine (MsgPack 0.8.11)

String sampleText = "Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.";
            Map<String, String> sampleMap =  new HashMap<>();
            for(int i=0; i<100; i++) {
                sampleMap.put(String.valueOf(i), sampleText);
            }
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            MessagePacker packer = MessagePack.newDefaultPacker(out);
            packer.packMapHeader(sampleMap.size());
            for (Map.Entry<String, String> entry : sampleMap.entrySet()) {
                packer.packString(entry.getKey());
                packer.packString(entry.getValue());
            }

Exception stackTrace:

java.lang.IllegalArgumentException: Bad position (limit 7621): -568
	at java.nio.Buffer.positionImpl(Buffer.java:364)
	at java.nio.Buffer.position(Buffer.java:358)
	at java.nio.charset.CharsetEncoderICU.setPosition(CharsetEncoderICU.java:249)
	at java.nio.charset.CharsetEncoderICU.implFlush(CharsetEncoderICU.java:159)
	at java.nio.charset.CharsetEncoder.flush(CharsetEncoder.java:526)
	at org.msgpack.core.MessagePacker.encodeStringToBufferAt(MessagePacker.java:490)
	at org.msgpack.core.MessagePacker.packString(MessagePacker.java:549)
	...

[komamitsu]

@dmitrychistyakov Thanks for reporting the issue. I reproduced it and will look into it.

[komamitsu]

@dmitrychistyakov
The following code that doesn't use MessagePack API sets ByteBuffer's position to wrong position right after calling CharsetEncoder#encode() and finally throws an exception when calling CharsetEncoder#flush with Android 4.4 (Kitkat).

        String sampleText = "Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industry's standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum.";

        byte[] buf = new byte[8192];
        ByteBuffer origBB = ByteBuffer.wrap(buf);
        int padding = 573;
        origBB.position(padding);
        ByteBuffer bb = origBB.slice();
        CharBuffer cb = CharBuffer.wrap(sampleText);
        CharsetEncoder encoder = Charset.forName("UTF-8").newEncoder()
                .onMalformedInput(CodingErrorAction.REPLACE)
                .onUnmappableCharacter(CodingErrorAction.REPLACE);
        CoderResult cr = encoder.encode(cb, bb, true);
        if (cr.isError()) {
            try {
                cr.throwException();
            } catch (CharacterCodingException e) {
                throw new RuntimeException("Failed to encode", e);
            }
        }
        Log.i(MainActivity.class.getSimpleName(), "bb.position=" + bb.position());
        if (!cr.isUnderflow() || cr.isOverflow()) {
            throw new RuntimeException("Unexpected code result: " + cr);
        }
        encoder.flush(bb);

It seems this issue is a Kitkat's bug that's related to https://android.googlesource.com/platform/libcore/+/91bb19d2ef9822cfb50fd1191cdb3ee06e2939b9%5E%21/#F1 and https://android.googlesource.com/platform/libcore/+/4e4b93ffe988b0a597835d852489573b57e0f3fe%5E%21/#F0. Current msgpack-java passes a ByteBuffer that includes a byte array with offset as an internal buffer to CharsetEncoder#encode and runs into the Kitkat's issue.

I think the following change would be a workaround for you. With this configuration, MessagePacker won't use CharsetEncorder when packing a string and I don't think the change significantly affects your application's performance.

-   MessagePacker packer = MessagePack.newDefaultPacker(out);
+   MessagePacker packer = new MessagePack.PackerConfig().withSmallStringOptimizationThreshold(Integer.MAX_VALUE).newPacker(out);

[komamitsu]

#409 takes care of this issue.