[dlmarion]

Add an external tablet file type and persist its marker in tablet metadata while preserving compatibility with existing file references. Modified GC to track external candidates, retain files with live tablet or scan references, and delete them when unreferenced. Updated Volume replacement and disk usage for external paths. Added unit and integration coverage for metadata, GC, scans, compactions, and volume replacement.

This change is related to #6570. Follow-on changes will modify bulk import to use this feature.

Assisted-By: OpenAI GPT-6 Sol
Assisted-By: OpenAI GPT-6 Luna

[keith-turner]

This change could allow a user to reuse file paths over time. When Accumulo moves files it also selects a unique name that it will never reuse. This is important for caching file data as we use a cache key of <uri>+<block id>. If a filename was ever reused it would cause serious problems as the cache may contain block for the old file. This could result in old and new blocks being read which could cause all kinds of problems.

[dlmarion]

This change could allow a user to reuse file paths over time. When Accumulo moves files it also selects a unique name that it will never reuse. This is important for caching file data as we use a cache key of <uri>+<block id>. If a filename was ever reused it would cause serious problems as the cache may contain block for the old file. This could result in old and new blocks being read which could cause all kinds of problems.

Agreed. I think the pattern here would be for the user to write their rfiles to a directory, bulk import the directory, and never reuse it. The rfiles in that directory would get compacted and removed over time.

Edit: We could also prioritize external files for compaction

[keith-turner]

Requiring the user to remember to never reuse a directory over long periods of time would be very easy overlooked and if it is overlooked the consequences could be catastrophic. Maybe another approach we could take is to add a new API to allocate a bulk dir for a table, then users could write files to this bulk dir and import it. Then there is more assurance that the uri is unique.

[keith-turner]

Even if a file is compacted, its blocks can still live in the caches of tablet and scan servers for an indeterminate amount of time.

[keith-turner]

Edit: We could also prioritize external files for compaction

That still leave a time window for chaos if the file gets in the cache. Also we want to use compaction ratio for compaction selection as much as possible to get the benefit of logarithmic compaction work.

[dlmarion]

Requiring the user to remember to never reuse a directory over long periods of time would be very easy overlooked and if it is overlooked the consequences could be catastrophic. Maybe another approach we could take is to add a new API to allocate a bulk dir for a table, then users could write files to this bulk dir and import it. Then there is more assurance that the uri is unique.

Ok, so I can look at that in the follow-on PR that modifies the bulk import code to take advantage of these external files.

[keith-turner]

Ok, so I can look at that in the follow-on PR that modifies the bulk import code to take advantage of these external files.

We would not necessarily need external files if the new API to allocate a dir creates a dir in the table dir.

[ctubbsii]

The absolute paths feature should make it so that we already support external tablet files outside of our volume path conventions. If there's code that doesn't properly handle this, and makes assumptions about paths, I'd consider that a bug in the absolute path feature. I think that's the whole point of having absolute paths, isn't it?

Scenario 1:

• User creates volume1 and volume2
• Over time, volume1 gets full, but it's fine to continue reading existing files from there
• User creates volume3 and removes volume1 from instance.volumes, and adds volume3 for new files
• The old files in volume1 should still exist in the metadata even though a user no longer has a volume1 configured in instance.volumes

Scenario 2:

• User migrates a table from one network storage to another
• For simplicity of file moves, it's easier to move all the files into a single directory
• User modifies metadata entries to point to the new directory, that doesn't happen to conform to the table/tablets directory structure, instead relying on the absolute paths to just work (because it should)

[dlmarion]

The absolute paths feature should make it so that we already support external tablet files outside of our volume path conventions. If there's code that doesn't properly handle this, and makes assumptions about paths, I'd consider that a bug in the absolute path feature. I think that's the whole point of having absolute paths, isn't it?

Scenario 1:

• User creates volume1 and volume2
• Over time, volume1 gets full, but it's fine to continue reading existing files from there
• User creates volume3 and removes volume1 from instance.volumes, and adds volume3 for new files
• The old files in volume1 should still exist in the metadata even though a user no longer has a volume1 configured in instance.volumes

Scenario 2:

• User migrates a table from one network storage to another
• For simplicity of file moves, it's easier to move all the files into a single directory
• User modifies metadata entries to point to the new directory, that doesn't happen to conform to the table/tablets directory structure, instead relying on the absolute paths to just work (because it should)

I pushed some other changes because I had them locally. But, if I can validate your assumptions that the current code should work with absolute file URLs once they are created, then we can just close this PR.

[dlmarion]

... I can validate your assumptions that the current code should work with absolute file URLs once they are created, then we can just close this PR.

Created PR #6611 to make changes to existing code and add tests to ensure that absolute file URLs should just be honored in the existing code paths.