Record Demonstrations

Record Demonstrations#

Recording uses HandUMI directly and does not require a robot arm. The raw controller poses, camera streams, gripper widths, and calibration metadata remain available for later retargeting to any supported embodiment.

Before Recording#

Confirm that:

Each active gripper width responds correctly from closed to fully open.
Each active controller reports valid tracking.
Camera intrinsics and controller-camera mounts still match the hardware.
The current session/table calibration was created for the same --device and has been visualized in Rerun.
The Controller-to-TCP calibration matches the installed HandUMI tool.

See Setup and Calibration if any check fails.

For a single Panda controlled with the right HandUMI, use --robot franka_emika_panda. The recorder selects right tracking/encoder input automatically. See the Panda workflow for configuration and conversion.

Start with a short pilot:

handumi doctor --device meta
handumi record --output-dir outputs/handumi-demo \
  --task "pick and place" \
  --session-calibration outputs/calibration/session.yaml \
  --rerun \
  --episodes 3 \
  --episode-time-s 30

Each episode begins when you say “start recording” and ends on “stop recording”; see Controls for the full set.

The usual device, cameras, resolution, FPS and target robot belong in the optional recording: section of configs/rig.yaml. CLI values override those defaults. Add --dry-run to resolve the complete plan, probe the encoder and exit before opening any hardware.

Camera keys are user-defined and are preserved as dataset image names. Each logical view chooses its own opencv or zedmini backend. Use role when a name is not one of the recognized aliases (left/left_wrist, right/right_wrist, and top/workspace):

cameras:
  scene:
    role: workspace
    type: zedmini
    index_or_path: 4
    width: 1344
    height: 376
    fps: 30
recording:
  cameras: [left, right, scene]

The ZED Mini capture is stereo 1344×376; only its left 672×376 half is stored. Feature shapes and camera metadata record that output resolution. Use --cameras scene to select that logical view; the command does not need to know which backend it uses.

Use --device pico and a PICO-created --session-calibration for PICO. Add --push-to-hub only after confirming the pilot locally.

Do not connect or configure a robot arm for this step. A target embodiment can be selected later during conversion or replay without modifying the raw recording.

Resume a Recording#

Append more episodes to a finalized local dataset without repeating its recording configuration:

handumi record --output-dir outputs/handumi-demo --resume \
  --episodes 20 \
  --task "pick and place"

--episodes is the number of additional episodes, not the new total. Resume requires an intact dataset from a previous graceful finalization and loads the device, cameras, FPS, image format, tool calibration, Feetech state and robot profile from its meta/info.json snapshot. Explicit incompatible overrides are rejected before hardware starts, including FPS, cameras, image shapes, tracking schemas, calibrations, or target-robot metadata. The task text may change so the same dataset can contain multiple tasks.

The table/session calibration is read from the current file on every run, including resume. Selection order is --session-calibration, recording.session_calibration in the rig, then the last session’s saved path. A missing file is an error; resume never silently reuses an embedded transform. Reusing the same file is valid while the VR origin is unchanged. After a new tracking session, reset, or relocalization, run calibrate spatial ... session and verify before resuming. A new session transform is accepted when its spatial calibration, device, and board remain compatible.

handumi.recording_sessions in meta/info.json preserves calibration snapshots with episode_from (inclusive) and episode_to (exclusive). The top-level spatial_session_calibration remains the original snapshot for compatibility. Each raw row also stores workspace_from_device_pose and original device poses; observation.state already contains controller poses in the common table frame, which replay/conversion uses without applying the session transform again. Legacy datasets receive an initial history entry for their existing episodes; this records their original calibration, not a correction of old measurements.

If the PICO device timestamp stops advancing for one second, recording aborts, discards the active episode and finalizes previously saved episodes. A stationary controller with advancing sample timestamps is valid. Restart tracking and verify/recalibrate the session before recording again.

Image dimensions are stored in each image feature and in the per-camera snapshot. This supports rigs whose cameras have different output resolutions; the recording.width and recording.height values remain fallbacks only for camera entries that do not declare their own dimensions.

Both handumi record and handumi teleop-record require --output-dir. For example, --output-dir outputs/handumi-demo stores the dataset in that directory, and --resume verifies and reads the same finalized dataset before recording additional episodes.

teleop-record has a dedicated real-robot gesture protocol: double-squeeze right while READY to start from home, right again while recording to save, or left while recording to discard. Save and discard synchronously return the robot home and leave the next episode waiting, so the operator can reset the physical task. Double-squeeze both to discard and finish. See the real-robot episode gesture guide for the complete behavior.

Unlike the earlier preview-only behavior, teleop-record stores every selected camera as a LeRobot v3 video feature in the same dataset as the real-robot joint observations and actions. Its dataset writer runs on a dedicated thread; camera devices and MP4 encoders also have their own workers, so disk and codec work does not execute in the robot-control loop. The terminal reports the writer queue alongside the live control timing.

Streaming Video Encoding#

Video is encoded continuously while an episode is recorded. HandUMI probes the local PyAV/FFmpeg encoders with a real MP4 before starting the tracking and camera hardware, then reports the concrete selection, for example:

Encoder: h264_nvenc (hardware, streaming, codec-managed threads).

The default --encoder auto tries a working hardware encoder first (NVIDIA NVENC, Intel/AMD VAAPI or Quick Sync, or macOS VideoToolbox) and falls back to H.264 on CPU. CPU encoding reserves one logical core and limits the threads assigned to each camera so encoding does not starve capture.

Use --encoder cpu to force software encoding or --encoder gpu to require hardware acceleration. --vcodec <codec> remains an advanced explicit override; do not combine an explicit codec with --encoder cpu or --encoder gpu.

Streaming writes frames directly to MP4 and calculates image statistics as frames arrive instead of writing and rereading temporary PNG files. If an encoder crashes, its queue overflows, a video is empty, or its frame count does not match the episode, HandUMI discards the episode before appending its rows to Parquet. --encoder-threads and --encoder-queue-size are advanced diagnostic overrides; increasing the queue does not fix an encoder that is consistently slower than capture.

teleop-record uses this same transactional streaming path automatically. If its bounded background dataset queue fills, the active episode is discarded to preserve exact alignment between video, state, and action instead of slowing robot control.

Controls#

Episodes are driven by voice by default, so the collector never has to put the shells down to reach a keyboard:

“start recording”: begin the episode.
“stop recording”: end and save the current episode.
“restart”: discard the current attempt and record it again.

Recognition is offline and its vocabulary is closed to exactly those three phrases, so nothing else said in the room can trigger a command. The first handumi record downloads a ~40 MB speech model to ~/.cache/handumi/vosk/; after that no network is needed. Audio comes from the system’s default input, which means plugging in a headset moves voice control to its microphone with no flags to change.

Add --clap-control to also accept gripper squeezes. Both controls stay live at once, which is what to use in a room too noisy to be heard reliably:

Right double squeeze: start or save the current episode.
Left double squeeze: discard and restart the current episode.

Always available:

Esc or Ctrl+C: discard an active partial episode and stop.
--episode-time-s: maximum episode length; it still applies as a safety limit while voice or clap control is running.

Pass --no-voice-control to record on the timer alone, or --manual-control to use the PICO buttons (which turns voice off automatically). Since voice is the default control, handumi record stops with an install hint if no microphone or speech model is available — unless --clap-control gives it another hands-free path, in which case it warns and continues. handumi doctor reports the microphone and model state before a session.

The recorder waits for valid controllers and discards an episode after sustained tracking, camera, or encoder failure.

Validate the Pilot#

handumi validate \
  outputs/handumi-demo --strict

Review meta/handumi_quality.json. Fix rejected captures before increasing --episodes.

Hard rejection checks include insufficient duration, excessive tracking loss, unhealthy cameras or encoders, synchronization errors, frozen source timestamps or poses, implausible translation/rotation jumps, and invalid state values. A stationary hand or constant gripper width is only a warning by default. Thresholds live in configs/quality.yaml.

Common additions:

--pico-wifi: stream PICO over Wi-Fi.
--skip-feetech: record without gripper widths.
--dataset-license <id>: set the dataset-card license.
--no-video: store image frames instead of encoded video.
--encoder cpu: force H.264 software encoding for reproducible CPU testing.
--encoder gpu: require hardware encoding instead of falling back to CPU.

Run handumi record --help for the normal interface or handumi record --help-advanced for synchronization, hardware and encoder diagnostic overrides. Physical camera IDs belong only in configs/rig.yaml.