Repository navigation
Deprecate and schedule removal of collections.abc.ByteString and typing.ByteString #91896
Description
The current docstring of collections.abc.ByteString is:
"""This unifies bytes and bytearray.
XXX Should add all their methods.
"""
Let's do that last thing. This will be useful for typing code that accepts both bytes and bytearray, especially with my proposal in PEP-688 to make bytes no longer acceptable as a shortcut for bytearray in the type system.
cc @rhettinger for collections.abc
Activity
pythongh-91896: Add abstract methods to collections.abc.ByteString
rhettinger commentedon Apr 25, 2022
The general rule is that we can never add methods to ABCs once they are published. The purpose of an ABC is promise that a minimal set of methods are available. If
isinstance(x, SomeABC)returns true, the methods in the ABC are expected to be present. Adding methods presents a problem for existing code that has registered a class as being compliant with SomeABC. If it lacks the new methods, then the registered promise is invalid.Perhaps because this is an empty ABC that is almost entirely unused, this might be okay. On the other hand, because it is an empty ABC that is almost entirely unused, there is almost nothing to be gained by adding it. AFAICT no one has ever asked for or needed this since the ABC was added 15 years ago. That suggests that there is no problem to be solved here.
JelleZijlstra commentedon Apr 25, 2022
Right, the ABC is hardly useful with no methods. I searched on
grep.appand found no uses of ByteString.register except for one ofmemoryviewin what appears to be an old copy oftyping.py.Well, I'm asking for it now. The use case is type annotating code that accepts both
bytesandbytearray.rhettinger commentedon Apr 25, 2022
If you were to do it correctly, the procedure would be to subclass ByteString and add the new methods in a subclass. Loosely, this is similar to why we had to add IterableUserDict in Python 2 rather than modifying the existing UserDict.
To annotate code that accepts both bytes or bytearray wouldn't the correct way be to write:
b: bytes | bytearrayjust like you would withtuple | list?One other thought: the name ByteString wasn't very good to begin as is suggests
bytes | str. If you really must do this, it would be better to create a new, well-named ABC with the requisite methods and even leave the old ABC around (it isn't hurting anything) or deprecate it.rhettinger commentedon Apr 25, 2022
I forgot to mention that the docs promise something narrower that the bytes/bytearray API. It says, "ABCs for read-only and mutable sequences". There is no promise of the extra methods found in bytes or bytearray. Existing code reasonably by subclassing from ByteString and not providing or expecting any of the stringlike methods.
All around, I think this proposal is a contract violation and that a new ABC should be created. A scan on grep.app is insufficient to show this won't be a breaking change (most of the world's Python code isn't publicly visible).
JelleZijlstra commentedon Apr 25, 2022
That documentation is for Sequence, MutableSequence, and ByteString together (https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence). "Read-only and mutable sequences" is a good description for the first two, but the documentation really doesn't tell me what ByteString is good for.
ByteString is also documented at https://docs.python.org/3/library/typing.html#typing.ByteString, but that documentation has a couple of problems:
memoryviewis not in fact registered as a ByteStringByteStringis not in fact generic (and it doesn't make sense for it to be generic)That's a reasonable point. If we can't use the existing ByteString ABC for
bytes | bytearray, I don't think it's worth creating a new ABC—as you said above, the union annotation is good enough.But if we keep ByteString as is, with no methods, I have no idea what it's useful for. We occasionally get people trying to use the ABC in type annotations, so the current state causes confusion.
Perhaps we could deprecate ByteString, or explicitly document its limited use.
rhettinger commentedon Apr 25, 2022
I vote for deprecation because the name is bad (implying
bytes | str) and it would just be a continuing point of confusion.[-]Give collections.abc.ByteString some methods[/-][+]Deprecate collections.abc.ByteString[/+]serhiy-storchaka commentedon Apr 26, 2022
The name is good to me. It implies the bytes-like object with
strmethods (find(),lower(),isspace(), etc).The terms "buffer", "bytes-like", "bytestring" and "bytes string" are used loosely in the documentation, but there are several meanings of bytes-likeness:
len()which returns the size in bytes.str(exceptencode()of course).Unfortunately there are no strongly defined terms and corresponding abstract classes, protocols or types in the code to express the requirements precisely.
ankith26 commentedon Aug 17, 2022
Hi! I'm one of the contributors to the pygame project.
I'm not sure whether this is the best best place to be asking, but it is relevant to the usage of
ByteString.So we have a function implemented with the python C API, and it uses
y#, which according to the docs is a format string for a generic "bytes-like" sized object. The term "bytes-like" is defined here in the glossary which gives me the impression that any function usingy#must accept a wide range of "byte-like" objects. Simply usingbytes | bytearraywould be narrow, and probably miss some kinds of objects. The same also mentions that "bytes-like object" is an object supporting the C level buffer protocol (which, is also not fully exposed on the python end in my understanding)I was looking for a suitable ABC to typehint this, and the closest thing I could find that already exists is
ByteString.The next closest thing I found is
typing.SupportsByteswhich by the naming, gives me the impression that this is what I'm looking for, but weirdly enough, the concretebytesobject itself does not confirm to this protocol (due to missing the__bytes__method)I suppose a set of ABCs for the buffer protocol (if this were to be added) would be the closest replacement to
ByteString, and would also work for my usecase (typing the C levely#arg format)TeamSpen210 commentedon Aug 17, 2022
The problem with buffers is that it doesn't have a visible Python API, so the ABC would be really weird, being entirely empty. See previous discussion at python/typing#593 and then PEP 688.
18 remaining items
Remove use of collections.abc.ByteString
Remove use of typing.ByteString
gh-91896: Improve visibility of `ByteString` deprecation warnings (#1…
gh-91896: Fixup some docs issues following ByteString deprecation (#1…
Merge branch 'main' into supermon
gh-91896: Revert some very noisy DeprecationWarnings for `ByteString` (…
Merge branch 'main' into supermon
Merge branch 'main' into cached_property_docs
[-]Deprecate collections.abc.ByteString[/-][+]Deprecate collections.abc.ByteString and typing.ByteString[/+][-]Deprecate collections.abc.ByteString and typing.ByteString[/-][+]Deprecate and schedule removal of collections.abc.ByteString and typing.ByteString[/+]mypy: comply with strict mode and resolve all 'type: ignore' cases
mypy: comply with strict mode and resolve all 'type: ignore' cases
mypy: comply with strict mode and resolve all 'type: ignore' cases
mypy: comply with strict mode and resolve all 'type: ignore' cases
mypy: comply with strict mode and resolve all 'type: ignore' cases