You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
query() with release_status="any" fetches every match twice, truncates silently, and loses the sort order #176
KGClient.query() with release_status="any" (used by KGObject.list(..., release_status="any") and by every existence query in save() and exists()) is implemented by fetching everything and paging in Python:
it runs the query against the released stage with size=100000, ignoring from_index and size;
it runs it again against the in-progress stage, also with size=100000;
it merges the two by id, the in-progress version replacing the released one, and only then applies from_index and size.
This has three consequences:
Cost. Asking for one page of results fetches every matching instance, twice. For a broad query on a large type this can mean very large responses, or a timeout, to return a handful of results.
Silent truncation. If either stage has more than 100000 matches, the extra ones are dropped without any warning, and total is wrong.
Ordering. Each stage's results are sorted by the class's sort_property, but the merge keeps the released results in their order and appends the in-progress-only instances at the end. So with "any", results are not sorted as a whole, and paging through them with from_index gives an order that depends on release status.
The code has a comment acknowledging that this is "simple but very inefficient".
Proposal
Page through both stages together (a merge of two sorted streams by sort_property), fetching only as many results from each as are needed for the requested page (at most from_index + size from each). Instances present in both stages need deduplicating, and may sort differently in each if a name-like property has changed since release.
At the least: raise or warn instead of silently truncating at 100000, and sort the merged list by sort_property before applying from_index and size.
Activity