Analyze and Curate Datasets

Analyze and Curate Datasets#

HandUMI separates dataset analysis from dataset curation:

  1. dataset analyze reads metadata and quality results, computes statistics, and writes an auditable report with automatic review candidates.

  2. A person reviews the candidates, histogram, and duration extremes.

  3. dataset curate requires that person to name the source episode indices to remove, then creates a new validated dataset.

Neither command edits the source dataset or uploads anything. Statistical outliers are not automatically defective, so analysis never makes the deletion decision and curation never infers it from the report.

Analyze a dataset#

Run the episode-level sensor and motion checks first when they are relevant:

handumi validate outputs/hanoi \
  --quality-config configs/quality.yaml

Then compute dataset-level statistics and automatic IQR review candidates:

handumi dataset analyze outputs/hanoi

The default report is written to:

outputs/hanoi/meta/handumi_analysis.json
outputs/hanoi/meta/handumi_analysis.md

It contains:

  • frame and duration statistics;

  • histogram bins and counts;

  • automatic Tukey IQR fences and candidates;

  • the five shortest and five longest episodes for human review;

  • per-episode findings and task labels;

  • task distribution and storage size by modality;

  • global observation.state and action statistics;

  • exact state/action alignment when their shapes are comparable;

  • a payload fingerprint that binds the report to the analyzed dataset.

Use --dry-run to compute and print the summary without writing the report:

handumi dataset analyze outputs/hanoi --dry-run

Review the report#

Inspect candidates_for_review, the histogram, duration extremes, quality findings, and relevant video. A candidate is evidence for review, not an instruction to delete. The episode indices in the report always refer to the source dataset.

The report records source episode and frame totals plus a payload fingerprint. Curation refuses a stale report if the underlying metadata, Parquet, video, or audio changed after analysis.

Curate a new local dataset#

After reviewing the report:

handumi dataset curate outputs/hanoi \
  --analysis outputs/hanoi/meta/handumi_analysis.json \
  --output outputs/hanoi_clean \
  --exclude 6,75 \
  --dry-run

--exclude is mandatory and contains the source indices confirmed by the reviewer. The dry run prints the exact plan. Remove --dry-run to build the derivative:

handumi dataset curate outputs/hanoi \
  --analysis outputs/hanoi/meta/handumi_analysis.json \
  --output outputs/hanoi_clean \
  --exclude 6,75

What curation updates#

The output is built in a temporary sibling directory and moved into place only after validation. The command:

  • filters and reindexes every Parquet row;

  • rebuilds episode indices, frame indices, global indices, and task indices;

  • reconstructs shared MP4 files and preserves their declared codec, pixel format, frame rate, and resolution;

  • reindexes episode-aligned HandUMI audio when enabled;

  • rebuilds episode metadata, video offsets, splits, and aggregate statistics;

  • preserves HandUMI capture metadata and calibration snapshots;

  • regenerates the dataset card;

  • writes meta/handumi_curation.json with removed episodes and the complete source-to-output episode mapping;

  • verifies Parquet counts, statistics, video frame counts, and loading through LeRobotDataset;

  • checks that the source payload did not change during the operation.

The command has no Hub publishing option. Publication remains a separate, explicit step after reviewing the curated dataset.