[Python][Docs] Revise Python Documentation

Describe the enhancement requested

This is the umbrella ticket for ongoing efforts to improve and revise the PyArrow Python User Guide and API reference documentation.

Many sections of the guide need refreshing, especially based on what users frequently search for and the kinds of issues commonly reported. We’ve already started prioritizing topics through research using Matomo web analytics and GitHub issues. In the future, insights from Kapa AI will also be incorporated. Comments on the priorities are welcome!

We’ll open sub-issues for specific tasks as we go. Everyone is welcome to contribute—whether you're experienced or just getting started, your help in making the PyArrow docs better is appreciated! ❤️

Suggested Focus Areas (in order of priority):

Parquet module
Dataset module
Table, RecordBatch, Schema and data types
Getting Started
Pandas integration
IPC

Note: The User Guide is the main focus of this revision effort, but improving the API reference documentation is also important. Analytics show it receives a significant amount of traffic, so we'll consider enhancements there as well.

Existing Documentation Issues

Connected

Extras

make sure pandas examples are used only in places, where pyarrow integration with pandas is explained.

Component(s)

Documentation, Python

@thisisnic

cc @thisisnic @raulcd @rok @ianmcook @pitrou for initial thoughts =)

Many sections of the guide need refreshing, especially based on what users frequently search for and the kinds of issues commonly reported. We’ve already started prioritizing topics through research using Matomo web analytics and GitHub issues.

That seems like a useful approach. I'm waiting for more details on the proposed changes :)

Here's the link to the analysis of questions asked to the kapa.ai bot over 3 months - https://thisisnic.github.io/arrow-kapa-ai-analysis/

I think there's a bug with how entries for the API Reference are currently being collected.

From the most recent few versions, I've noticed the following are missing:

pyarrow
- arange
pyarrow.compute
- bottom_k_unstable
- inverse_permutation
- random
- rank_normal
- rank_quantile
- scatter
- top_k_unstable
- winsorize

The most recent addition that is present seems to be

https://arrow.apache.org/docs/20.0/python/generated/pyarrow.compute.pivot_wider.html

I'm assuming this wasn't intentional, since they all do have docstrings:

import pyarrow.compute as pc

print(pc.bottom_k_unstable.__doc__)

Select the indices of the bottom-k ordered elements from
array- or table-like data.

This is a specialization for :func:`select_k_unstable`. Output is not
guaranteed to be stable.

Parameters
----------
values : Array, ChunkedArray, RecordBatch, ...

print(pc.rank_normal.__doc__)

Compute normal (gaussian) ranks of an array.

This function computes a normal (gaussian) rank of the input array.
By default, null values are considered greater than any other ...

@dangotbanned

This is not intentional, no. Unfortunately the kernels have to be listed specifically in this list:

https://github.com/apache/arrow/blob/main/docs/source/python/api/compute.rst

to appear in the API docs (other sections also). Similar issue has been opened here:

#45322

but for other kernels. We should add all that are listed here also. Would you be interested in making a contribution for this @dangotbanned? It would need a separate issue though.

@AlenkaF

Thanks @AlenkaF

Would you be interested in making a contribution for this?

I am interested, but realistically I don't think I'd get back to this for a while, sadly.
If anyone else wants to take it, feel free 🙌

It would need a separate issue though.

Now that I can commit to 😄

GH-47686: [Docs][Python] Split the Python Parquet docs into separate …

Navigation Menu

[Python][Docs] Revise Python Documentation #46601

Describe the enhancement requested

Suggested Focus Areas (in order of priority):

Existing Documentation Issues

Connected

Extras

Component(s)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Python][Docs] Revise Python Documentation #46601

Description

Describe the enhancement requested

Suggested Focus Areas (in order of priority):

Existing Documentation Issues

Connected

Extras

Component(s)

Activity

AlenkaF commented on May 27, 2025

pitrou commented on May 28, 2025

thisisnic commented on Oct 2, 2025

dangotbanned commented on Nov 9, 2025

AlenkaF commented on Nov 10, 2025

dangotbanned commented on Nov 11, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions