镜像站点 · 本页由第三方 GitHub 只读镜像提供，非 GitHub 官方站点，不接受任何登录或凭据输入。前往 github.com
Skip to content

feat(bigquery): support arrowSerializationOptions in createQueryArrowStream - #9603

Merged
ohmayr merged 2 commits into
mainfrom
feat/bigquery-arrow-serialization-options
Oct 21, 2026
Merged

feat(bigquery): support arrowSerializationOptions in createQueryArrowStream#9603
ohmayr merged 2 commits into
mainfrom
feat/bigquery-arrow-serialization-options

Conversation

@ohmayr

@ohmayr ohmayr commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Callers need to choose the timestamp precision of Arrow results (for example picoseconds for TIMESTAMP(12) columns), and the choice has to be the same on every page. This adds arrowSerializationOptions to ArrowQuery and exports the ArrowSerializationOptions type, both marked beta.

  • Defaults picosTimestampPrecision to TIMESTAMP_PRECISION_MICROS and sends the same value on jobs.query, on ReadRows for streams/_default (including resumed streams), and in readOptions on CreateReadSession. It's kept out of jobs.insert.
  • Rejects LZ4_FRAME and ZSTD bufferCompression, since apache-arrow doesn't bundle those codecs.
  • Requires @google-cloud/bigquery-storage >= 6.3.0 (already enforced on main by chore(bigquery): require bigquery-storage >= 6.3.0 for Arrow #9675), whose ReadRowsRequest carries arrowSerializationOptions, so no extra version check is needed.

@product-auto-label product-auto-label Bot added the api: bigquery Issues related to the BigQuery API. label Oct 13, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Oct 13, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for arrowSerializationOptions (specifically picosTimestampPrecision) in createQueryArrowStream. It adds validation and normalization for these options, refactors the internal streaming helper functions to use a shared execution context (ArrowStreamContext), and ensures options are correctly propagated to the BigQuery Storage Read API. Comprehensive unit and system tests are also added. The reviewer suggests spreading the original options object in resolveArrowSerializationOptions to ensure future-proofing and compatibility with other serialization options.

Comment thread handwritten/bigquery/src/arrow.ts Resolved
@ohmayr
ohmayr added this pull request to stack #9597 October 13, 2026 07:08
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch from 7aa4ae9 to c763965 Compare October 13, 2026 08:41
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch 2 times, most recently from 3cb0725 to 9d90ad6 Compare October 13, 2026 10:27
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch 3 times, most recently from 0db4d25 to f4e7b32 Compare October 13, 2026 12:00
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch from f4e7b32 to 1250acc Compare October 13, 2026 19:26
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch from 1250acc to 1c4630a Compare October 15, 2026 09:55
quirogas
quirogas reviewed Oct 15, 2026
View reviewed changes
Comment thread handwritten/bigquery/src/arrow.ts Resolved
Comment thread handwritten/bigquery/system-test/arrow.ts Resolved
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch 2 times, most recently from 4b102fe to b686ec6 Compare October 15, 2026 18:27
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch from b686ec6 to d486ca5 Compare October 15, 2026 19:30
@ohmayr ohmayr mentioned this pull request Oct 15, 2026
Merged
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch 2 times, most recently from abdda6d to 646b497 Compare October 16, 2026 02:19
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch 2 times, most recently from 3ad895b to 646b497 Compare October 16, 2026 21:50
@ohmayr
ohmayr marked this pull request as ready for review October 16, 2026 21:53
@ohmayr
ohmayr requested review from a team as code owners October 16, 2026 21:53
quirogas
quirogas reviewed Oct 20, 2026
View reviewed changes
Comment thread handwritten/bigquery/src/arrow.ts Resolved
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch from 646b497 to 888e454 Compare October 20, 2026 22:02
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch from 48dd111 to d7863ee Compare October 21, 2026 21:28
quirogas
quirogas approved these changes Oct 21, 2026
View reviewed changes

@quirogas quirogas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Base automatically changed from feat/bigquery-arrow-stream-resume to main October 21, 2026 23:04
ohmayr added 2 commits October 21, 2026 16:04
@ohmayr
feat(bigquery): add arrowSerializationOptions to createQueryArrowStream
43e9768 
Accept arrowSerializationOptions (beta, like createQueryArrowStream) on
ArrowQuery in createQueryArrowStream and default picosTimestampPrecision
to TIMESTAMP_PRECISION_MICROS when unset or
PICOS_TIMESTAMP_PRECISION_UNSPECIFIED. Forward the normalized
options across jobs.query, ReadRows on streams/_default, and
CreateReadSession for destination tables, and keep them out of
jobs.insert requests. Reject LZ4_FRAME and ZSTD bufferCompression values
upfront since apache-arrow does not bundle decompression codecs.
@ohmayr
test(bigquery): cover Arrow precision on destination table reads
ad859eb 
Add a system test that sends each picosTimestampPrecision with an
explicit destination table, so the option goes on CreateReadSession.
With BIGQUERY_PICOSECOND_SUPPORT=true it also checks that a TIMESTAMP(12)
column is cut to microseconds by default and keeps all 12 digits with
TIMESTAMP_PRECISION_PICOS.
@ohmayr
ohmayr force-pushed the feat/bigquery-arrow-serialization-options branch from d7863ee to ad859eb Compare October 21, 2026 23:05
@ohmayr
ohmayr merged commit 2449fe4 into main Oct 21, 2026
46 checks passed
@ohmayr
ohmayr deleted the feat/bigquery-arrow-serialization-options branch October 21, 2026 23:23
@release-please release-please Bot mentioned this pull request Oct 21, 2026
Merged
shivanee-p pushed a commit that referenced this pull request Oct 23, 2026
@release-please
chore: release main (#9684)
b3a9687 
🤖 I have created a release *beep* *boop*
---


<details><summary>common: 8.2.2</summary>

##
[8.2.2](common-v8.2.1...common-v8.2.2)
(2026-10-23)


### Bug Fixes

* **common:** Preserve {{projectId}} token during URI path encoding
([#9668](#9668))
([14b0cc6](14b0cc6))
</details>

<details><summary>gapic-generator: 5.7.2</summary>

##
[5.7.2](gapic-generator-v5.7.1...gapic-generator-v5.7.2)
(2026-10-23)


### Bug Fixes

* **generator:** Compile protos before running tests in generator-tests
([#9701](#9701))
([cde8ae3](cde8ae3))
* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>gaxios: 8.3.2</summary>

##
[8.3.2](gaxios-v8.3.1...gaxios-v8.3.2)
(2026-10-23)


### Bug Fixes

* **gaxios:** Narrow fetch types to call signature for @types/bun
([#9699](#9699))
([2a34528](2a34528))
</details>

<details><summary>bigquery: 9.5.0</summary>

##
[9.5.0](bigquery-v9.4.0...bigquery-v9.5.0)
(2026-10-23)


### Features

* **bigquery:** Allow WRITE_APPEND in createQueryArrowStream
([#9648](#9648))
([2062403](2062403))
* **bigquery:** Resume interrupted Arrow ReadRows streams
([#9602](#9602))
([8372c9e](8372c9e))
* **bigquery:** Stream multi-page Arrow results from default job stream
([#9598](#9598))
([1aa6eb4](1aa6eb4))
* **bigquery:** Support arrowSerializationOptions in
createQueryArrowStream
([#9603](#9603))
([2449fe4](2449fe4))
* **bigquery:** Support explicit destination tables in
createQueryArrowStream
([#9601](#9601))
([f22e2dd](f22e2dd))
* **bigquery:** Support jobs.insert in createQueryArrowStream
([#9600](#9600))
([4c414c6](4c414c6))
* **bigquery:** Support maxResults in createQueryArrowStream
([#9605](#9605))
([a7fdd05](a7fdd05))
* **bigquery:** Wait for unfinished jobs before streaming Arrow results
([#9599](#9599))
([5e465b7](5e465b7))


### Bug Fixes

* **bigquery:** Preserve timestamp nanoseconds when microseconds are
zero
([#9650](#9650))
([f6ca61d](f6ca61d))
* **bigquery:** Raise when an Arrow batch arrives before its schema
([#9681](#9681))
([576a42e](576a42e))
</details>

<details><summary>datastore: 11.3.1</summary>

##
[11.3.1](datastore-v11.3.0...datastore-v11.3.1)
(2026-10-23)


### Bug Fixes

* **datastore:** Align Transaction JSDoc
[@extends](https://github.com/extends) tag with DatastoreRequest
([#9671](#9671))
([733e9f1](733e9f1))
</details>

<details><summary>datamanager: 1.4.0</summary>

##
[1.4.0](datamanager-v1.3.0...datamanager-v1.4.0)
(2026-10-23)


### Features

* Update API sources and regenerate
([#9472](#9472))
([514e2f6](514e2f6))


### Bug Fixes

* Update package.json and .gitignore across packages
([#9378](#9378))
([9c54cec](9c54cec))
</details>

<details><summary>agentidentity: 0.3.1</summary>

##
[0.3.1](agentidentity-v0.3.0...agentidentity-v0.3.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>agentregistry: 0.6.1</summary>

##
[0.6.1](agentregistry-v0.6.0...agentregistry-v0.6.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>apptopology: 0.4.1</summary>

##
[0.4.1](apptopology-v0.4.0...apptopology-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>bigquery-storage-api: 0.6.1</summary>

##
[0.6.1](bigquery-storage-api-v0.6.0...bigquery-storage-api-v0.6.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>bigtable-api: 0.11.1</summary>

##
[0.11.1](bigtable-api-v0.11.0...bigtable-api-v0.11.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>commerceproducer: 0.7.1</summary>

##
[0.7.1](commerceproducer-v0.7.0...commerceproducer-v0.7.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>datastore-api: 0.5.1</summary>

##
[0.5.1](datastore-api-v0.5.0...datastore-api-v0.5.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>firestore-api: 0.7.1</summary>

##
[0.7.1](firestore-api-v0.7.0...firestore-api-v0.7.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>ftp: 0.4.1</summary>

##
[0.4.1](ftp-v0.4.0...ftp-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>logging-api: 0.5.1</summary>

##
[0.5.1](logging-api-v0.5.0...logging-api-v0.5.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>policy-troubleshooter-iam: 0.11.1</summary>

##
[0.11.1](policy-troubleshooter-iam-v0.11.0...policy-troubleshooter-iam-v0.11.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>productregistry: 2.2.1</summary>

##
[2.2.1](productregistry-v2.2.0...productregistry-v2.2.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>spanner-api: 0.6.1</summary>

##
[0.6.1](spanner-api-v0.6.0...spanner-api-v0.6.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>speech: 8.2.1</summary>

##
[8.2.1](speech-v8.2.0...speech-v8.2.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>tasks: 7.3.1</summary>

##
[7.3.1](tasks-v7.3.0...tasks-v7.3.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>workloadidentity: 0.4.1</summary>

##
[0.4.1](workloadidentity-v0.4.0...workloadidentity-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>isochrones: 0.3.1</summary>

##
[0.3.1](isochrones-v0.3.0...isochrones-v0.3.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>inventories: 0.15.1</summary>

##
[0.15.1](inventories-v0.15.0...inventories-v0.15.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>loyaltycustomers: 0.4.1</summary>

##
[0.4.1](loyaltycustomers-v0.4.0...loyaltycustomers-v0.4.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>reports: 0.18.1</summary>

##
[0.18.1](reports-v0.18.0...reports-v0.18.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

<details><summary>reviews: 0.9.0</summary>

##
[0.9.0](reviews-v0.8.0...reviews-v0.9.0)
(2026-10-23)


### Features

* **google/shopping/merchant/reviews/v1:** Onboard a new library
([#9691](#9691))
([f369d0b](f369d0b))
</details>

<details><summary>storage-control: 0.13.1</summary>

##
[0.13.1](storage-control-v0.13.0...storage-control-v0.13.1)
(2026-10-23)


### Bug Fixes

* Remove compile-protos from prepare
([#9690](#9690))
([1a4c864](1a4c864))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@quirogas quirogas quirogas approved these changes
@shivanee-p shivanee-p shivanee-p approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

api: bigquery Issues related to the BigQuery API.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ohmayr @quirogas @shivanee-p