镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

ShreddedVariantArrayBuilder throws InvalidCastException for Decimal16 typed values #498

Open

Description

@CurtHagenlocher

Describe the bug, including details regarding any error messages, version, and platform.

ShreddedVariantArrayBuilder.Build throws InvalidCastException whenever a shred schema has a Decimal16 typed value and a row shreds into it. Shredding a variant into a 16-byte decimal column therefore fails for every input; the size of the value doesn't matter.

Repro, against the Apache.Arrow.Operations 23.0.0 package on .NET 10 (Windows):

using System.Data.SqlTypes;
using Apache.Arrow.Operations.Shredding;
using Apache.Arrow.Scalars.Variant;

var schema = ShredSchema.Primitive(ShredType.Decimal16);
// Fits System.Decimal; the failure does not depend on the magnitude.
var values = new List<VariantValue> { VariantValue.FromDecimal16(SqlDecimal.Parse("1.50")) };
var (metadata, rows) = VariantShredder.Shred(values, schema);
var array = ShreddedVariantArrayBuilder.Build(schema, metadata, rows); // throws
System.InvalidCastException: Unable to cast object of type 'System.Data.SqlTypes.SqlDecimal' to type 'System.Decimal'.
   at Apache.Arrow.Operations.Shredding.ShreddedVariantArrayBuilder.BuildDecimalArray(ShredType shredType, IList`1 typedValues, MemoryAllocator allocator)
   at Apache.Arrow.Operations.Shredding.ShreddedVariantArrayBuilder.BuildPrimitiveTyped(...)
   at Apache.Arrow.Operations.Shredding.ShreddedVariantArrayBuilder.BuildTypedValueArray(...)
   at Apache.Arrow.Operations.Shredding.ShreddedVariantArrayBuilder.Build(...)

Cause (as of main at bb2f7ed). For a Decimal16 slot, VariantShredder extracts the typed value as a SqlDecimal (VariantShredder.cs line 320: case ShredType.Decimal16: return value.AsSqlDecimal();). BuildDecimalArray in ShreddedVariantArrayBuilder.cs assumes every value is a System.Decimal in two places:

  • it unboxes each value with b.Append((decimal)v), which throws for a SqlDecimal;
  • it takes the column's scale from v is decimal d values only, so a Decimal16 column's scale would come out as 0 even without the cast.

Expected. The rows build a Decimal128 typed column holding the values at their scale, the same way Decimal4 and Decimal8 already work. Values outside System.Decimal's range (the reason Decimal16 goes through SqlDecimal) should keep their exact digits. Decimal128Array.Builder has an Append(SqlDecimal) overload, which would avoid narrowing through System.Decimal.

As far as I can see there's no test that builds a shredded array with a Decimal16 typed value. The existing Decimal16 tests in ShreddedVariantReaderTests read arrays that were constructed directly.

Found while adding variant statistics to EngineeredWood's Delta writer (clast-project/engineered-wood#676).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions