With the default Encoding.UTF8, output starting with a BOM followed by a real U+FEFF loses both, so the same bytes decode differently per Encoding instance #145

What's wrong

AsyncProcessStreamReader builds its readers with detectEncodingFromByteOrderMarks: false (RunCommand/AsyncProcessStreamReader.cs:29-33) and then removes one leading U+FEFF itself in StripLeadingByteOrderMark (AsyncProcessStreamReader.cs:216-235). The remarks there assume that turning detection off "also turned off the stripping that came with it".

That's only true for encodings with no preamble. StreamReader skips the encoding's own preamble whenever encoding.Preamble.Length > 0, whatever detectEncodingFromByteOrderMarks is set to. OutputHandler defaults to Encoding.UTF8 (OutputHandler.cs:47), which has a preamble. So for the default handler:

1. StreamReader drops the BOM.
2. StripLeadingByteOrderMark then drops the next U+FEFF, which is real content.

The same double strip happens with Encoding.Unicode, Encoding.BigEndianUnicode and Encoding.UTF32.

Reproduction

A command writes the bytes EF BB BF EF BB BF 41 (a BOM, then U+FEFF, then A):

await RunCommand.ExecuteAsync("printf", [@"\357\273\277\357\273\277A"],
    new OutputHandler(s => sb.Append(s), null, encoding));

The underlying StreamReader behaviour is easy to confirm on its own. With detectEncodingFromByteOrderMarks: false, new StreamReader(new MemoryStream(bytes), Encoding.UTF8, false).ReadToEnd() returns FEFF 41, while the same call with new UTF8Encoding(false) returns FEFF FEFF 41.

Why it matters

The impact is low, but the data loss is silent and it hits the default configuration. Take a BOM-emitting tool that echoes a file which itself starts with a BOM: its captured output loses a character. Output should also not depend on which of two equivalent UTF-8 Encoding instances the caller passed.

Suggested fix

Strip in exactly one place. Either:

• only call the manual strip when outputHandler.Encoding.GetPreamble().Length == 0, or
• build the readers from a preamble-free copy of the encoding (for example new UTF8Encoding(false, throwOnInvalid) with the caller's settings preserved, or a wrapper whose Preamble is empty) and keep the manual strip.

Acceptance criteria

• A regression test feeds a doubled BOM through Encoding.UTF8 and through new UTF8Encoding(false), and both deliver "A".
• The same test covers Encoding.Unicode, so the UTF-16 path is pinned too.
• The existing single-BOM tests from OutputHandler.Encoding is silently overridden by a byte order mark, and decode failures are sometimes discarded #76 keep passing.

Related, but not a duplicate: #76 (closed) is the fix that introduced this stripping.