[Feature Request]: Avoid reflective lookup and interning in BigQuery Storage Write API unknown-column collectors #40492

What would you like to happen?

Avoid repeated reflective lookup and string interning in the unknown-column collectors used by TableRowToStorageApiProto.messageFromMap.

These collectors are TableRow instances. Unknown-column capture, nested collector attachment, and empty-STRUCT pruning currently use their GenericData map operations. Those operations consult ClassInfo.getFieldInfo and intern field names, including ordinary column names that are not annotated Java model fields. With withAutoSchemaUpdate(true), this bookkeeping also runs for known nested fields, so large repeated STRUCTs pay the cost even when no unknown columns appear.

Proposal

Use the existing backing map returned by TableRow.getUnknownKeys() for ordinary collector keys. Cache the annotated model-field names once and keep the original TableRow setter and map operations for those names.

For collector emptiness, use backing-map emptiness plus getF() == null only when f is the model's sole annotated field. Every non-null f counts as an entry, including an empty cell list. If a future model adds fields, fall back to the original isEmpty() behavior.

Keep the collector representation, eager allocation, repeated positions, empty-container pruning, encoded unknown JSON, and model-field exceptions unchanged. This does not change payload conversion, descriptor lookup, required-field validation, timestamps, the cell-format path, or public converter signatures. It adds no dependency or configuration.

Measurements

A downstream Beam 2.68.0 backport was tested in one matched, backlogged Dataflow staging pair. Both variants used 40 fixed n2-standard-4 workers in the same zone, Streaming Engine, and no autoscaling. Measurement followed a 600-second warmup and covered five settled intervals, about 1,664 seconds of offset capture per variant.

That is 27.1% more committed offsets/s and 20.3% lower estimated CPU per committed offset. The CPU estimate uses observed GCE CPU utilization and the fixed vCPU count, not profiler percentages. All five intervals favored the candidate, but its backlog still grew; this does not establish keep-up.

In a matching settled CPU-profile window, control String.intern accounted for 15.4% of global self CPU. It fell below the candidate's visible top-100 cutoff of 0.163%, not to a proven zero.

An overlap check matched 1,040,000 Kafka coordinates without duplicates or side-only coordinates. Coordinate equality is not full payload parity. These are downstream staging results, not a benchmark of the current master port or a production guarantee. Full pipeline payload parity, long-run validation, and attributable allocation/GC savings remain open.

Related

#40188 / #40203 address descriptor lookup, and #40189 / #40204 address required-field validation. Collector access is an independent cost and does not need either change to land first.

Draft implementation: #40491. It includes collector regression tests and documents the local validation setup and full-master build limitation.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Prism Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner