Repository navigation
I can open an S3 file in root, but not in a subdirectory #62
Description
Originally posted here:
https://stackoverflow.com/questions/57227152/opening-a-file-in-pyfilesystems-s3-filesystem-fails
I'm able to open a file in the root of an S3 bucket without a problem, however I cannot open a file with a standard path prefix:
>>> s3fs.listdir('dfparks/test')
['test.txt']
>>> s3fs.open('dfparks/test/test.txt')
I get the error:
fs.errors.ResourceNotFound: resource 'dfparks/test/test.txt' not found
I've tried every variant of the path / s3 url I can think of but I always get the not found error.
Activity
mrk-its commentedon Nov 17, 2019
Could you try to reproduce this using code from #60? (you can simply install
fs-s3fs-ngpackage - it is the same as fs-s3fs==1.1.1 but with #60 and #61 applied)willmcgugan commentedon Nov 17, 2019
@davidparks21 Did you create the test directory with S3FS?
S3FS has some limitations, which are documented here.
davidparks21 commentedon Nov 17, 2019
I understand now, that does appear to be the issue, I hadn't caught that detail in the docs. This might be worth a big red flag in the docs.
@mrk-its your PR is brilliant, it solved my next problem too. I was doing a test to see if I could open a large file (3+ GB) and only read the first few bytes of its header. With
fs_s3fs_ngit worked no problem, but withfs_s3fswithstrict=FalseI was able to open the file it appears, but it started reading the entire file in.I'll watch your PRs, you've got my vote for sure! :)