镜像站点 · 本页由第三方 GitHub 只读镜像提供,非 GitHub 官方站点,不接受任何登录或凭据输入。前往 github.com
Skip to content

query() with release_status="any" fetches every match twice, truncates silently, and loses the sort order #176

Open

Description

@apdavison

Summary

KGClient.query() with release_status="any" (used by KGObject.list(..., release_status="any") and by every existence query in save() and exists()) is implemented by fetching everything and paging in Python:

  1. it runs the query against the released stage with size=100000, ignoring from_index and size;
  2. it runs it again against the in-progress stage, also with size=100000;
  3. it merges the two by id, the in-progress version replacing the released one, and only then applies from_index and size.

This has three consequences:

  • Cost. Asking for one page of results fetches every matching instance, twice. For a broad query on a large type this can mean very large responses, or a timeout, to return a handful of results.
  • Silent truncation. If either stage has more than 100000 matches, the extra ones are dropped without any warning, and total is wrong.
  • Ordering. Each stage's results are sorted by the class's sort_property, but the merge keeps the released results in their order and appends the in-progress-only instances at the end. So with "any", results are not sorted as a whole, and paging through them with from_index gives an order that depends on release status.

The code has a comment acknowledging that this is "simple but very inefficient".

Proposal

  • Page through both stages together (a merge of two sorted streams by sort_property), fetching only as many results from each as are needed for the requested page (at most from_index + size from each). Instances present in both stages need deduplicating, and may sort differently in each if a name-like property has changed since release.
  • At the least: raise or warn instead of silently truncating at 100000, and sort the merged list by sort_property before applying from_index and size.

Activity

  1. added this to the 0.17 milestone on Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingperformanceSomething should be faster

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions