ShreddedVariantArrayBuilder writes shredded field groups nullable, and nulls the group for a missing field #499

Describe the bug, including details regarding any error messages, version, and platform.

ShreddedVariantArrayBuilder declares every shredded field group nullable: each field of an object's typed_value struct, and an array's element group. When an object lacks one of the shredded fields, it marks that by setting the group itself to null (BuildElementGroupArray: if (r.IsMissing) validity.Append(false)).

The Parquet VariantShredding spec has no null group. Each shredded object field is a required group holding optional value and optional typed_value, and so is an array's element. A field missing from a present object is spelled as a non-null group with both value and typed_value null.

A Parquet writer that maps Arrow nullability to repetition therefore writes an OPTIONAL group with nulls at the group's definition level. Spark 4.1.2 (with delta-spark 4.3.1) refuses such a file as soon as one object row lacks a shredded field:

[MALFORMED_VARIANT] Variant binary is malformed. Please check the data source is valid.

Files where every object row has every shredded field read fine, which makes this easy to miss in tests.

Repro

Apache.Arrow.Operations 23.0.0. Current main has the same code (BuildObjectTyped, BuildArrayTyped, BuildElementGroupArray):

var schema = ShredSchema.ForObject(new Dictionary<string, ShredSchema>
{
    ["a"] = ShredSchema.Primitive(ShredType.Int8),
});
var values = new List<VariantValue>
{
    VariantValue.FromObject(new Dictionary<string, VariantValue> { ["a"] = VariantValue.FromInt8(1) }),
    VariantValue.FromObject(new Dictionary<string, VariantValue> { ["b"] = VariantValue.FromInt8(1) }), // lacks "a"
};
var (metadata, rows) = VariantShredder.Shred(values, schema);
VariantArray array = ShreddedVariantArrayBuilder.Build(schema, metadata, rows);

var typed = (StructArray)array.TypedValueArray;
var aField = ((StructType)typed.Data.DataType).Fields[0];
Console.WriteLine(aField.IsNullable);         // True  -- expected False
Console.WriteLine(typed.Fields[0].IsNull(1)); // True  -- expected False, with value and typed_value both null

Expected

• Each object field's Field and the list's element Field are nullable: false.
• A missing field is a valid group slot with value and typed_value both null. The children already hold nulls there, so the only change is to stop nulling the group and stop declaring it nullable.
• Any reader-side code that treats a null group as "missing" also has to accept both-children-null. A spec-conforming file from Spark or DuckDB is already shaped that way.

Workaround

clast-project/engineered-wood rewrites the builder's output: it pushes each group's nulls into its two children and drops the group's validity and nullability, recursing through nested objects and list elements. With that rewrite, Spark 4.1.2, DuckDB 2.0 and pyarrow all read the files correctly (clast-project/engineered-wood#679, clast-project/engineered-wood#683).

Component(s)

C#