Repository navigation
IPC ArrowFileWriter can produce entirely zero-filled encapsulated messages #409
Description
Describe the bug, including details regarding any error messages, version, and platform.
Environment
Apache.Arrow/Apache.Arrow.Compression23.0.0,net472, x64, Windows 11ArrowFileWriterover a plainFileStream, writing record batches continuously for the duration of a recording
Background
We are using Apache.Arrow to save continuous streaming data from ONIX hardware, where our project (OpenEphys.Onix1) is hosted as a package that can run in Bonsai-Rx.
Problem
Rarely, on one specific computer, a file comes out with an entire encapsulated message (continuation marker, metadata, and body) replaced by zeros. The footer is intact and its Block offsets are correct for every batch, including all batches after the zeroed one, so the file reads until it hits the gap and then raises ArrowInvalid: Unexpected empty message in IPC file format. Happens with and without Zstd compression, always early in the recording. Individual valid batches can be read and concatenated together, but this leaves a gap in the recorded samples as confirmed by our clock parameter.
ArrowFileWriter takes the block offset from BaseStream.Position and the following batch is recorded at the correct offset; we believe this points to Arrow having issued the writes and FileStream having accepted them, but the bytes were lost before being saved to disk. We'd like to confirm if this is the case before concluding it's environmental.
We are tracking this in our own issue in case it is how we are utilizing the API. For more details (e.g., byte offsets, and affected files), see open-ephys/bonsai-onix1#683. We have been unable to replicate this on two other systems.
Our usage
ArrowWriter.cs— construction,WriteRecordBatch, andWriteEndon disposeArrowBatchWriter.cs— batches are accumulated with an RxBuffer(timeout, count), soWriteRecordBatchcalls are serialized but arrive on different threads (producer thread on a count flush, thread-pool timer thread on a timeout flush)
Note also that our ArrowBuffers wrap unmanaged memory via a custom MemoryManager<byte> to handle OpenCV.Net.Mat objects.
I'd be happy to answer any questions you might have about our usage, and if there is any other information I can provide to narrow this down please let me know.
Questions
- Is there any path in
ArrowFileWriterthat advancesBaseStream.Positionwithout writing bytes (a seek, or padding emitted by seeking)? - Is there anything we should be doing differently?
FileOptions.WriteThrough, periodicFlush(true)on the base stream, or otherwise to keep a lower layer from silently dropping a write like this?
Activity
CurtHagenlocher commentedon Sep 27, 2026
Thanks for the detailed report. I don't think it's Arrow that's writing the zeros. The pattern better fits something outside your process shrinking the file back to an earlier size while it's still being written.
Can ArrowFileWriter advance Position without writing bytes?
No. ArrowFileWriter and ArrowStreamWriter never call Seek or SetLength on the base stream. They only read
BaseStream.Position, to record the block offsets for the footer. All padding is written as real zero bytes, not by
seeking past it. So Position only moves when a Write call succeeds.
The continuation marker (0xFFFFFFFF) and the message header come from Arrow's own memory, filled just before each
write. That memory isn't released or reused until the write returns. This also holds on net462/net472, where writes go
through a temporary pooled copy. Your custom MemoryManager could at worst affect the data values in the body.
It can't zero out the marker or header. Your Rx usage also looks fine: Buffer(timeSpan, count) hands off each batch
under a lock, so WriteRecordBatch calls don't overlap even when they arrive on different threads.
A few details from open-ephys/bonsai-onix1#683 stand out:
aren't aligned to disk sectors or pages. A disk or cache fault loses whole sectors or pages, so it wouldn't line up
with Arrow's messages in file after file.
One explanation covers both failures: another process shrinks the file back to an earlier size while your writer still
has it open. Suppose the file is cut back to the end of batch 2 after batch 3 has been written. Your FileStream still
points past batch 3, so batch 4 lands at the correct offset and Windows fills the gap with zeros. That's exactly what
you're seeing. The same thing happening after the footer is written leaves a file of whole batches with no footer.
The writer is idle between batches, so the size the file gets cut back to is almost always a batch boundary. A sync
client that notices a new, growing file and acts on it soon after it appears would explain why this only happens
early. Your FileStream is opened with FileShare.Read, so an ordinary program couldn't open the file for writing. That
points at something with a kernel-mode driver, such as a sync client's on-demand sync driver or antivirus software.
I haven't reproduced this, so please treat it as a hypothesis. It would explain why only one machine is affected.
How to confirm it
On the affected machine, run Process Monitor with a filter on the output folder or file name. Look for
SetEndOfFileInformationFile,SetAllocationInformationFileorWriteFileon the.arrowfile from any process other thanBonsai, especially the Synology Drive processes. Synology Drive's client logs from around the start of the recording
may also help.
Would FileOptions.WriteThrough or Flush(true) help?
Probably not. Those settings protect against losing data in a crash or power loss. They can't stop another process
from truncating or rewriting the file. The practical fix is to record to a folder that isn't synced and move each file
into the synced folder once it's closed. Alternatively, write under a temporary name or extension the sync client is
set to ignore and rename the file when it's complete.
If Process Monitor shows the file being changed by something else, please post what you find here. If it shows nothing
unusual, we can try to dig a little further on the Arrow side.