ElementTree.iterparse "leaks" file descriptor when not exhausted

The PR #31696 attempts to fix the "leak" of file descriptors when the iterator is not exhausted. That PR fixes the warning, but not the underlying issue that the files aren't closed until the next tracing garbage collection cycle.

Note that there isn't truly a leak of file descriptors. The file descriptors are eventually closed when the file object is finalized (at cyclic garbage collection). The point of the ResourceWarning (in my understanding) is that waiting until the next garbage collection cycle means that you may temporarily have a lot of unwanted open file descriptors, which could exhaust the global limit or prevent successful writes to those files on Windows.

# run with ulimit -Sn 1000
import xml.etree.ElementTree as ET
import tempfile

import gc
gc.disable()

def run():
    with tempfile.NamedTemporaryFile("w") as f:
        f.write("<document />junk")

        for i in range(10000):
            it = ET.iterparse(f.name)
            del it

run()

On my system, after lowering the file descriptor limit to 1000 (via ulimit -Sn 1000) I get:

OSError: [Errno 24] Too many open files: '/tmp/tmpwwmd9gp6'

Linked PRs

I think this would require something like PEP 533 “Deterministic cleanup for iterators“ to solve.

In this case, I think you can fix it just by avoiding reference cycles that involve the generator. Below is an example refactoring of iterparse.

def iterparse(source, events=None, parser=None):
    # Use the internal, undocumented _parser argument for now; When the
    # parser argument of iterparse is removed, this can be killed.
    pullparser = XMLPullParser(events=events, _parser=parser)
    _root = None

    def iterator(source):
        nonlocal _root
        close_source = False
        try:
            if not hasattr(source, "read"):
                source = open(source, "rb")
                close_source = True
            yield None
            while True:
                yield from pullparser.read_events()
                # load event buffer
                data = source.read(16 * 1024)
                if not data:
                    break
                pullparser.feed(data)
            root = pullparser._close_and_return_root()
            yield from pullparser.read_events()
            _root = root
        finally:
            if close_source:
                source.close()

    class IterParseIterator(collections.abc.Iterator):
        def __init__(self, source):
            self.it = iterator(source)
        def __next__(self):
            return next(self.it)
        @property
        def root(self):
            nonlocal _root
            return _root
    it = IterParseIterator(source)
    del iterator, IterParseIterator

    next(it)
    return it

This avoids two reference cycles that captured the generator created by iterator(source):

Classes naturally form reference cycles, so move iterator(source) to the instance instead of a class attribute
Move _root up so that iterator(source) doesn't refer to the instance it.

I'm not sure about a few things including:

How do you test this?
Is it.root important? It's not documented. Is it only for testing?

EDIT: the problem is not capturing of the file descriptor, but the generator iterator(source).

@colesbury

@colesbury Doe the issue persist if you add gc.collect() after the del it?

@rhettinger

@rhettinger everything is cleaned up properly once the garbage collector runs. So adding a gc.collect() after the del it would collect the generator and close the file descriptor.

pythongh-101438: possible fix for ElementTree.iterparse leak

pythongh-101438: Avoid reference cycle in ElementTree.iterparse.

gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH-114269)

pythongh-101438: Avoid reference cycle in ElementTree.iterparse. (pyt…

[3.12] gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH…

[3.11] gh-101438: Avoid reference cycle in ElementTree.iterparse. (GH…

pythongh-101438: Avoid reference cycle in ElementTree.iterparse. (pyt…

Navigation Menu

Repository navigation

ElementTree.iterparse "leaks" file descriptor when not exhausted #101438

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ElementTree.iterparse "leaks" file descriptor when not exhausted #101438

Description

Linked PRs

Activity

ericvsmith commented on Jan 30, 2023

colesbury commented on Jan 30, 2023

rhettinger commented on Jan 30, 2023

colesbury commented on Jan 30, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions