Repository navigation
Improve the Efficiency of Python3.11.1 __getattr__ #102213
Description
Feature or enhancement
__getattr__ in Python3.11.1 is much slower than @Property and visiting an object's attribute. It's even slower than Python3.10.4.
_PyObject_GenericGetAttrWithDict is the key reason. If Python3 fails finding an attribute in normal ways, it will return NULL and raise an exception. But raising an exception has performance cost. Python3.11.1 add set_attribute_error_context to support Fine Grained Error Locations in Tracebacks. It makes things worser.
Pitch
We can use this test code:
import time
import sys
class A:
def foo(self):
print("Call A.foo!")
def __getattr__(self, name):
return 2
@property
def ppp(self):
return 3
class B(A):
def foo(self):
print("Call B.foo!")
class C(B):
def __init__(self) -> None:
self.pps = 1
def foo(self):
print("Call C.foo!")
def main():
start = time.time()
for i in range(1, 1000000):
pass
end = time.time()
peer = end - start
c = C()
print(f"Python version of {sys.version}")
start = time.time()
for i in range(1, 1000000):
s = c.pps
end = time.time()
print(f"Normal getattr spend time: {end - start - peer}")
start = time.time()
for i in range(1, 1000000):
s = c.ppa
end = time.time()
print(f"Call __getattr__ spend time: {end - start - peer}")
start = time.time()
for i in range(1, 1000000):
s = c.ppp
end = time.time()
print(f"Call property spend time: {end - start - peer}")
if __name__ == "__main__":
main()The result shows how slow __getattr__ is:
Python version of 3.11.1 (main, Dec 26 2022, 16:32:50) [GCC 8.3.0]
Normal getattr spend time: 0.03204226493835449
Call __getattr__ spend time: 0.4767305850982666
Call property spend time: 0.06345891952514648
When we define __getattr__, failed to find an attribute is what we expected. If we can get this result and then call __getattr__ without exception handling, it will be faster.
I tried to modify Python3.11.1 like this:
- add a new function in
object.c:
PyObject *
PyObject_GenericTryGetAttr(PyObject *obj, PyObject *name)
{
return _PyObject_GenericGetAttrWithDict(obj, name, NULL, 1);
}- change
typeobject.c:
if (getattribute == NULL ||
(Py_IS_TYPE(getattribute, &PyWrapperDescr_Type) &&
((PyWrapperDescrObject *)getattribute)->d_wrapped ==
(void *)PyObject_GenericGetAttr))
// res = PyObject_GenericGetAttr(self, name);
res = PyObject_GenericTryGetAttr(self, name);
else {
Py_INCREF(getattribute);
res = call_attribute(self, getattribute, name);
Py_DECREF(getattribute);
}
if (res == NULL) {
if (PyErr_ExceptionMatches(PyExc_AttributeError))
PyErr_Clear();
res = call_attribute(self, getattr, name);
}
Py_DECREF(getattr);
return res;Rebuild python, it really become faster: spend time: 0.13772845268249512.
Previous discussion
getattr is much slower in Python3.11
Activity
Fidget-Spinner commentedon Feb 25, 2023
Could you submit a pull request please? I'll try to review it.
wangxiang-hz commentedon Mar 8, 2023
I have already submitted a pull request, but the default assigned reviewer is too busy to review it. Could you please review it and provide some suggestions when you are available? Thank you very much.
Fidget-Spinner commentedon Mar 8, 2023
Sorry. I didn't see the pull request (pull requests linked to an issue don't ping me even though I'm subscribe to the issue). I'll review it. Thanks!
wangxiang-hz commentedon Mar 10, 2023
Thank you. I know that contributors in open source communities are very busy, so I didn't want to bother you with a “polite ping” :) I really appreciate your review.
gh-102213: Optimize the performance of `__getattr__` (GH-102248)
pythongh-102213: Optimize the performance of `__getattr__` (pythonGH-…
Revert "pythongh-102213: Optimize the performance of `__getattr__` (p…
gh-102213: Revert "gh-102213: Optimize the performance of `__getattr_…
pythongh-102213: Revert "pythongh-102213: Optimize the performance of…
gh-102213: Optimize the performance of `__getattr__` (GH-103761)
Merge branch 'main' into supermon
Merge branch 'main' into inlinecomp2