镜像站点 · 本页由第三方 GitHub 只读镜像提供，非 GitHub 官方站点，不接受任何登录或凭据输入。前往 github.com
Skip to content

C API: Consider adding public PyLong_AsByteArray() and PyLong_FromByteArray() functions #111140

Closed
Closed
C API: Consider adding public PyLong_AsByteArray() and PyLong_FromByteArray() functions#111140
Labels
3.13only security fixestopic-C-APItype-featureA feature request or enhancement

Description

@vstinner
vstinner
opened on Oct 20, 2023

Feature or enhancement

The private _PyLong_AsByteArray() and _PyLong_FromByteArray() functions were removed in Python 3.13: see PR #108429.

@scoder asked what is the intended replacement for _PyLong_FromByteArray().

The replacement for _PyLong_FromByteArray() is PyObject_CallMethod((PyObject*)&PyList_Type, "from_bytes", "s#s", str, len, "big") but I'm not sure what is the easy way to set the signed parameter to True (default: signed=False).

The replacement for _PyLong_AsByteArray() is PyObject_CallMethod(my_int, "to_bytes", "ns", length, "big"). Same, I'm not sure how to easy set the signed parameter to True (default: signed=False).

I propose to add public PyLong_AsByteArray() and PyLong_FromByteArray() functions to the C API.

Python 3.12 modified PyLongObject: it's no longer a simple array of digits, but it's now a more less straightforward _PyLongValue structure which requires using unstable functions to access small "compact" values:

  • PyUnstable_Long_IsCompact()
  • PyUnstable_Long_CompactValue()

So having a reliable and simple way to import/export a Python int object as bytes became even more important.

A code search for _PyLong_AsByteArray in PyPI top 5,000 projects found 12 projects using it:

  • Cython (0.29.36)
  • blspy (2.0.2)
  • catboost (1.2)
  • fastobo (0.12.2)
  • gevent (22.10.2)
  • guppy3 (3.1.3)
  • line_profiler (4.0.3)
  • msgspec (0.16.0)
  • orjson (3.9.1)
  • pickle5 (0.0.12)
  • pyodbc (4.0.39)
  • rlp (3.0.0)

Linked PRs

Activity

  1. vstinner
    added
    type-featureA feature request or enhancement
    topic-C-API
    on Oct 20, 2023
  2. vstinner
    changed the title [-] C API: Consider adding a public PyLong_AsByteArray() and PyLong_FromByteArray() functions[/-] [+] C API: Consider adding public PyLong_AsByteArray() and PyLong_FromByteArray() functions[/+] on Oct 20, 2023

  3. scoder commented on Oct 21, 2023

    @scoder
    scoder
    on Oct 21, 2023
    Contributor

    Thanks for creating the issue. I agree that the functions should be added. The current replacements seem awful for this kind of basic functionality. Going through an expensive Python call like this for converting between PyLong and large C integer types (int128_t) seems excessive.

    Note that at least a couple of projects that you list use Cython implemented parts and thus probably just mention the function in there. I'm sure something like line_profiler would never end up calling it.

  4. serhiy-storchaka commented on Oct 21, 2023

    @serhiy-storchaka
    serhiy-storchaka
    on Oct 21, 2023
    Member

    It was already discussed several times. This API lacks some features which would be needed for general use. You need to know the size of the resulting bytes array. Calculating it is not trivial, especially for negative integers. Also, it would be core convenient to support "native" ending, not just "big"/"littel".

    I have been thinking about implementing a similar API for mpz_import/mpz_export in GMP (https://gmplib.org/manual/Integer-Import-and-Export) or mp_unpack/mp_unpack in libtommath (https://github.com/libtom/libtommath). It is powerful and a kind of standard. It is general enough to support internal PyLong represenation and the marshal format used for long integers (15 bits packed in 16-bit words). But it is only for unsigned integers, you are supposed to store the sign separately. It should be extended to support negative integers in several formats, for convenience and for performance.

  5. vstinner commented on Oct 21, 2023

    @vstinner
    vstinner
    on Oct 21, 2023
    MemberAuthor

    I'm not sure that passing the endian as a string is efficient if this function is part of hot code.

  6. serhiy-storchaka commented on Oct 21, 2023

    @serhiy-storchaka
    serhiy-storchaka
    on Oct 21, 2023
    Member

    Not as a string. Just 3-variant value native/little/big instead of boolean little/big.

  7. vstinner commented on Oct 21, 2023

    @vstinner
    vstinner
    on Oct 21, 2023
    MemberAuthor

    Not as a string. Just 3-variant value native/little/big instead of boolean little/big.

    Sorry, I was confused between C API (int for the endian) and the Python API (string for the endian).

  8. scoder commented on Oct 21, 2023

    @scoder
    scoder
    on Oct 21, 2023
    Contributor

    It was already discussed several times. This API lacks some features which would be needed for general use.

    Ok, then please put the existing function back in until there is a public replacement.

  9. scoder commented on Oct 21, 2023

    @scoder
    scoder
    on Oct 21, 2023
    Contributor

    I created PR #111162

  10. vstinner commented on Oct 22, 2023

    @vstinner
    vstinner
    on Oct 22, 2023
    MemberAuthor

    This API lacks some features which would be needed for general use.

    Which features are missing?

    Do you have links to previous discussions if it was discussed previously?

  11. vstinner commented on Oct 22, 2023

    @vstinner
    vstinner
    on Oct 22, 2023
    MemberAuthor

    You need to know the size of the resulting bytes array.

    wcstombs() can be called with NULL buffer to compute the buffer size. It avoids to have to provide a second API "calculate the buffet size".

    I suppose that a common use case is also to convert a Python int object to C int type for which there is no C API. Like int128_t

  12. scoder commented on Oct 23, 2023

    @scoder
    scoder
    on Oct 23, 2023 via email
    Contributor

  13. serhiy-storchaka commented on Oct 23, 2023

    @serhiy-storchaka
    serhiy-storchaka
    on Oct 23, 2023
    Member

    https://mail.python.org/archives/list/python-ideas@python.org/thread/V2EKXMKSQV25BMRPMDH47IM2OYCLY2TF/

  14. encukou commented on Oct 23, 2023

    @encukou
    encukou
    on Oct 23, 2023
    Member

    I'm partial to API like mpz_export that accepts a buffer and length, and can:

    • Successfully fill the buffer, and output the length
    • Not fill the buffer, but output the necessary length

    This allows handling the common cases with a minimum of function calls:

    • If the size is already known, you only need one function call
    • If the caller can pre-allocate a buffer, and the value happens to fit, this is again one function call
    • If the size is unknown, and a pre-allocated buffer can't be used or turns out to be too small, it's two function calls

    But, IMO we also need general API for exporting/importing Python objects to/from buffers (see e.g. #15 in capi-workgroup/problems), and it would be good to make this consistent.

    I'd prefer adding the original functions back until we design a proper replacement.

  15. 75 remaining items

  16. diegorusso
    added a commit that references this issue on Apr 17, 2024

    pythongh-111140: PyLong_From/AsNativeBytes: Take *flags* rather than …

    29db213
  17. scoder
    added 3 commits that reference this issue on May 3, 2024

    Use the large PyLong conversion functions in Py3.13.

    eb68bce

    Use the new large PyLong conversion functions in Py3.13. (GH-5997)

    dc63743

    Use the new large PyLong conversion functions in Py3.13. (GH-5997)

    cff39ef

  18. scoder commented on May 5, 2024

    @scoder
    scoder
    on May 5, 2024
    Contributor

    The interface seems complete and usable now. Is this done now or is there anything left for this ticket to stay open?

  19. gpshead
    added a commit that references this issue on May 5, 2024

    pythongh-111140: minor docs typos cleanup in the C example API calls.

    d8cfb91

  20. gpshead commented on May 5, 2024

    @gpshead
    gpshead
    on May 5, 2024
    Member

    Looking things over I like the C API that what was settled upon. It seems to address all of the needs from our earlier discussions.

  21. gpshead
    closed this as completedon May 5, 2024
  22. gpshead
    added
    3.13only security fixes
    on May 5, 2024
  23. gpshead
    added a commit that references this issue on May 5, 2024

    gh-111140: minor docs typos cleanup in the C example API calls. (#118612

    b744fa5
  24. SonicField
    added a commit that references this issue on May 8, 2024

    pythongh-111140: minor docs typos cleanup in the C example API calls. (…

    1eb0a54
  25. jessecomeau87
    added a commit that references this issue on May 20, 2024

    gh-106320: Re-add _PyLong_FromByteArray(), _PyLong_AsByteArray() and …

    8d46cb3
  26. Xmader
    added a commit that references this issue on Aug 22, 2024

    fix: Python 3.13 added the `PyLong_AsNativeBytes` API, but changed th…

    428353c
  27. Glyphack
    added a commit that references this issue on Sep 2, 2024

    pythongh-106320: Re-add some PyLong/PyDict C-API functions (GH-python…

    4aa7b76
  28. LukasWoodtli
    added a commit that references this issue on Jan 22, 2025

    pythongh-111140: Improve PyLong_AsNativeBytes API doc example & impro…

    b1b27e6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.13only security fixestopic-C-APItype-featureA feature request or enhancement

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions