[akushonkamen]

Fixes #829

Root cause

The pinned lerobot revision (lerobot@0cf8648) keeps every written episode in the in-memory hf_dataset table: LeRobotDataset.save_episode → _save_episode_table appends the new episode into self.hf_dataset via concatenate_datasets on every call (lerobot/common/datasets/lerobot_dataset.py:914 at that revision). examples/libero/convert_libero_data_to_lerobot.py never reads hf_dataset, so the table is pure overhead — RSS grows by ~150 MB per episode for the whole conversion and long conversions eventually OOM (#829).

Fix

examples/libero/convert_libero_data_to_lerobot.py:91-97: after each dataset.save_episode(), reset the in-memory table with dataset.hf_dataset = dataset.create_hf_dataset() — exactly what LeRobotDataset.create initializes it to. The parquet files already written to disk are unaffected, so the on-disk output is unchanged.

Test

Adds scripts/convert_libero_data_to_lerobot_test.py, which drives the real main() against a fake RLDS dataset (with a tfds stub so no GPU / rlds dependency group is needed) and asserts:

1. the in-memory hf_dataset table no longer accumulates across episodes;
2. the converted dataset on disk still round-trips with all frames.

Evidence on this branch:

$ .venv/bin/python -m pytest scripts/convert_libero_data_to_lerobot_test.py -v
scripts/convert_libero_data_to_lerobot_test.py::test_hf_dataset_does_not_accumulate_across_episodes PASSED
scripts/convert_libero_data_to_lerobot_test.py::test_conversion_output_roundtrips PASSED
============================== 2 passed in 6.02s ===============================

Red proof (same test with the fix stashed — it fails on the accumulation assertion, confirming the test catches this bug):

assert [0, 4, 8, 12] == [0, 0, 0, 0]
  index 1: 4 != 0

Lint (ruff 0.8.6, the version pinned in .pre-commit-config.yaml):

$ .venv/bin/ruff check examples/libero/convert_libero_data_to_lerobot.py scripts/convert_libero_data_to_lerobot_test.py
All checks passed!
$ .venv/bin/ruff format --check examples/libero/convert_libero_data_to_lerobot.py scripts/convert_libero_data_to_lerobot_test.py
2 files already formatted

Honest scope note: I did not run the repo-wide gate uv run pytest --strict-markers -m "not manual" locally — the frozen lockfile does not install on macOS arm64 (jax-cuda12-plugin and tensorflow-cpu==2.15.0 publish no macOS arm64 wheels; uv sync failed twice). Local verification used a subset venv pinned to match the repo lock (lerobot@0cf8648, torch 2.7.1, datasets 3.6.0, pyarrow 20.0.0). The new test carries its own tfds stub and needs neither GPU nor the rlds group, so it should run as-is in CI.

AI disclosure

This fix was developed with the assistance of an AI coding agent (Claude / GLM, working as a commit agent under the account owner's direction): the root-cause analysis, patch, and test were drafted by the agent and reviewed/verified by the account owner, with the evidence above reproduced locally. Claim and progress comments on #829 disclose the same.