This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 6, 2026. It is now read-only.
Optional switch to non-nullable dtypes in to_dataframe #1345
Description
Release 3.0.0 introduced the use of nullable Int64 and boolean dtypes (pandas extension dtypes) (#786)
However the pandas extension dtypes are not widely supported across the pandas api. This might lead to issues using numba in pandas operations.
Please see my bug report in the pandas project for example:
pandas-dev/pandas#46867
I would propose to introduce an optional switch to pre-3.0.0 behaviour.
Reacted by Tim Sweña (Swast)
Activity
derHeinzer commentedon Sep 12, 2022
To make things worse, not all widely-used machine learning libraries can handle pandas extension dtypes. I just figured out, the training of a lightgbm Regressor ist not possible with such input.
I did not check every library I know, but I am pretty sure there must be others.
tswast commentedon Sep 14, 2022
Thank you so much for the feedback! I hope that the nullable dtypes can stay as the default, as they more accurately reflect the BigQuery data types, but you make some good points that there are still use cases where the old behavior would be desirable.
tswast commentedon Mar 9, 2023
Related: #954 Support for string[pyarrow] dtype (edit: fixed issue number)
I'm thinking we add
int_dtype,bool_dtype,string_dtype,float_dtype,time_dtype,timestamp_dtype,date_dtype,datetime_dtypeas arguments toto_dataframe.Alternative 1: expose typemapper directly, but doubtful that a lambda function that inspects an arrow schema would be all that understandable to the average pandas user.
Alternative 2: expose
use_nullable_dtypesargument to match https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html and similar methods. It defaults to True to avoid lossy conversions as we have attempted to do in google-cloud-bigquery v3 -- technically another breaking change though since we'll want to change to use Float64Dtype and StringDtype too, which we aren't currently.tswast commentedon Mar 23, 2023
With #1529, you'll be able to explicitly set int_dtype to
Noneto use legacy dtypes.chelsea-lin commentedon Mar 24, 2023
The changes have been merged.