镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

Check for IOBase instead of BufferedIOBase - #8

Open
wilson1yan wants to merge 1 commit into
danijar:mainfrom
wilson1yan:patch-2
Open

wilson1yan wants to merge 1 commit into
danijar:mainfrom
wilson1yan:patch-2

Conversation

@wilson1yan

Copy link
Copy Markdown
Contributor

No description provided.

@wilson1yan wilson1yan changed the title Check fo IOBase instead of BufferedIOBase Check for IOBase instead of BufferedIOBase Dec 24, 2024
@danijar

danijar commented Dec 28, 2024

Copy link
Copy Markdown
Owner

Why does it need to be IOBase? For general (unbuffered) IO objects, read() and write() may do partial work and return early, but the code on the outside currently doesn't support that.

@wilson1yan

Copy link
Copy Markdown
Contributor Author

It's just that GFile internally is not a BufferedIOBase and is an IOBase. So the assert cases the code the code to always crash

@psarka

psarka commented Dec 11, 2025

Copy link
Copy Markdown

Same for UPath, https://pypi.org/project/universal-pathlib/ :(

Any hints on what is needed to make it work?

@danijar

danijar commented Dec 13, 2025 •

Copy link
Copy Markdown
Owner

Could you try making it explicit that it's a buffered IO object?

class MarkBuffered(BufferedIOBase):
  def __init__(self, obj):
    self.obj = obj
  def __getattr__(self, name):
    return getattr(self.obj, name)

obj = MarkBuffered(obj)

We should find an easier solution for this since it seems to be a common catch. But just blindly accepting IOBase can lead to difficult to debug errors, if the file object is actually unbuffered. Adding an optional kwarg is_buffered=None could work, where the user can pass IOBase objects when setting is_buffered=True.

But let's first see if the above workaround even works.

@psarka

psarka commented Dec 15, 2025 •

Copy link
Copy Markdown

Edit: I got it to work with a more elaborate shim (but not with __getattr__ trick):

class MarkBuffered(BufferedIOBase):
    def __init__(self, obj):
        self.obj = obj

    def writable(self):
        return self.obj.writable()

    def close(self):
        return self.obj.close()

    def write(self, b):
        return self.obj.write(b)


class WPath(UPath):

    def open(self, mode='r', **kwargs) -> BufferedIOBase:
        return MarkBuffered(super().open(mode, **kwargs))


base_dir = WPath('s3://some-folder')
with DatasetWriter(base_dir, SPEC, encoders) as writer:
    writer.append(...)

In my case, UPath when opened resolves into s3fs.S3File which inherits from fsspec.spec.AbstractBufferedFile but this one does not inherit from io.BufferedIOBase, only from io.IOBase. Apparently on purpose, but (to my understanding) not because of reasons that affect granular: fsspec/filesystem_spec#1583

@danijar
danijar force-pushed the main branch 5 times, most recently from 2767092 to 9b547e5 Compare January 2, 2026 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants