HandUMI Setup and Calibration

HandUMI Setup and Calibration#

Complete this page before recording. No robot arm is required: these steps configure HandUMI, its tracking device, cameras, grippers, and workspace. Some calibrations are permanent for one physical assembly; the table/session alignment must be checked each session.

Calibration

Repeat when

Servo homing and opening width

Servo, linkage, or gripper geometry changes

Camera intrinsics

Camera, resolution, or focus changes

Controller-to-camera mount

A controller or wrist camera mount moves

Controller-to-TCP

The controller/gripper mount or physical tool changes

Table/session frame

Each session, relocalization, or tracking reset

1. Map HandUMI Hardware#

install.sh creates the ignored machine-local configs/rig.yaml. Inspect the connected cameras and Feetech adapters:

handumi setup ports

Reconnect one physical device at a time and assign its port under cameras or feetech in configs/rig.yaml. Robot-arm buses do not belong in this recording setup; configure them only for real-robot teleoperation.

Set new Feetech IDs only when required:

handumi servo set-id --port /dev/ttyACM0 --new-id 0
handumi servo set-id --port /dev/ttyACM1 --new-id 1
Hardware mapping details

Two grippers may share one serial port only when they use different servo_id values. With separate USB adapters, each side normally has its own port.

A USB camera commonly exposes two /dev/video* nodes. Start with the first node reported for each physical camera and confirm the stream. Camera keys are logical dataset names; map each physical camera explicitly in configs/rig.yaml.

Keep these machine-local paths in configs/rig.yaml; do not commit them as portable project configuration.

Camera types and resolutions#

Declare capture settings for the three logical roles: left, right, and workspace. Their keys may be any names. The aliases left/left_wrist, right/right_wrist, and top/workspace are recognized automatically; otherwise add role: left, role: right, or role: workspace. Each view can use the normal opencv backend or zedmini, which expects the ZED Mini side-by-side UVC mode and exposes only its left image:

cameras:
  workspace:
    type: zedmini
    index_or_path: 4
    width: 1344
    height: 376
    fps: 30

For zedmini, width and height describe the captured stereo frame. HandUMI keeps frame[:, :672], so previews and datasets contain one 672×376 RGB image. The supported rates for this mode are 15, 30, 60, and 100 FPS. index_or_path accepts either an OpenCV integer such as 4 or an explicit Linux path such as /dev/video4. Per-camera values in cameras: take precedence over the global camera fallbacks used by older rig files.

For USB cameras whose high-resolution/high-rate modes are available only as Motion-JPEG, add fourcc: MJPG to that camera entry. For example, a camera that exposes 1280x960 @ 30 FPS as MJPG but only 5 FPS as YUYV should use:

cameras:
  left_wrist:
    type: opencv
    index_or_path: /dev/video0
    width: 1280
    height: 960
    fps: 30
    fourcc: MJPG

2. Calibrate the Grippers#

First confirm that both encoders change smoothly while opening and closing:

handumi calibrate grippers monitor

Run the complete guided calibration:

handumi calibrate grippers
handumi calibrate grippers --side right  # one side only

For each side, enter the maximum physical opening in millimeters. The command then asks you to hold the gripper at mid-travel and press Enter, which centers the encoder range to avoid the 0/4095 wrap point. Finally, place the gripper fully open and press Enter, then fully close it and press Enter. The result is stored in ~/.cache/handumi/calibration.yaml. Open and close each gripper again with monitor and confirm that width increases toward fully open without flipping or saturating.

3. Connect Tracking#

Meta Quest#

Enable Developer Mode, connect the headset over USB, authorize adb, and install HandUMI Quest App:

wget https://github.com/murobotics-ai/handumi-quest-app/releases/download/v0.2.1/handumi-quest-app-v0.2.1.apk
adb install -r handumi-quest-app-v0.2.1.apk
adb shell ip route  # find the address after "src"

Set that address as meta_quest.connection.quest_ip in configs/rig.yaml. Launch the app from Library → Unknown Sources and keep it in the foreground.

python -m handumi.tracking.meta_quest --config configs/rig.yaml

A healthy stream reports steady FPS and both controllers tracked.

PICO#

install.sh already installed the XRoboToolkit PC service unless it was run with --skip-xrt; install it from the XRoboToolkit PC Service releases only if /opt/apps/roboticsservice/runService.sh is missing. Follow the current XR Robotics headset instructions for the headset side. Start the PC service in its own terminal and leave it running – xrt.init() aborts with a core dump without it:

bash /opt/apps/roboticsservice/runService.sh

Use 127.0.0.1:63901 for USB or the workstation IP with --pico-wifi.

Smoke-test a short capture before calibration:

handumi record --output-dir outputs/pico-smoke \
  --device pico --skip-feetech --no-voice-control \
  --task "pico smoke" --episodes 1 --episode-time-s 10

Healthy output reports xrobotoolkit_sdk initialised without repeated still waiting for PICO data messages.

4. Calibrate Cameras and Workspace#

Fix the 5 × 7 ChArUco board flat at its marked table position, with IDs 15 and 16 nearest the operator. Its center defines the table origin: +X right, +Y away, and +Z up.

Camera Intrinsics#

handumi calibrate spatial intrinsics --camera left
handumi calibrate spatial intrinsics --camera right
handumi calibrate spatial intrinsics --camera top

Move the board throughout each image and vary distance and inclination. The tool automatically accepts a distinct valid view every two seconds. Repeat after changing camera, resolution, or focus.

Controller-to-Camera Mounts#

Keep the board fixed. Move the complete HandUMI through varied roll, pitch, and yaw poses, pausing briefly for each automatic capture. Keep the controller tracking ring visible to the headset.

Choose the tracking device explicitly. Global options such as --device, --pico-wifi, and --quest-ip come before the subcommand.

Meta Quest:

handumi calibrate spatial --device meta mount --side left
handumi calibrate spatial --device meta mount --side right

PICO:

handumi calibrate spatial --device pico --pico-mode controllers mount --side left
handumi calibrate spatial --device pico --pico-mode controllers mount --side right

PICO calibration relies on live XRoboToolkit snapshots, so hold the HandUMI steady while each view is accepted. Use --pico-wifi for a wireless PICO setup.

Repeat only if a controller or wrist-camera mount moves.

Session/Table Frame#

With the board still at its marked position and the headset fixed as it will be during recording, solve the table frame for the same tracking device.

handumi calibrate spatial --device meta session --side left
handumi calibrate spatial --device meta verify --side left
handumi calibrate spatial --device meta visualize

For PICO:

handumi calibrate spatial --device pico --pico-mode controllers session --side left
handumi calibrate spatial --device pico --pico-mode controllers verify --side left
handumi calibrate spatial --device pico --pico-mode controllers visualize

session reads wrist-camera intrinsics and the controller-to-camera mount from outputs/calibration/spatial.yaml by default. If calibration was saved under a different name, select that same file explicitly for both commands, for example --spatial outputs/calibration/spatial_60mm.yaml. The command stops before hardware capture when the selected spatial file and configs/rig.yaml describe different ChArUco dimensions, and warns when their camera mappings differ.

verify captures fresh views without changing either calibration and rejects a table-frame error above its acceptance limits. Run it while the board is still fixed; recapture the session if it fails.

Inspect the wrist cameras and both TCP trails in Rerun. The table surface must align with z=0.

The fixed workspace camera is not part of the controller-to-table calibration chain. Recording its video does not require a spatial pose. Only calibrate its pose if a downstream workflow needs geometric projection or inpainting:

handumi calibrate spatial workspace

This optional command requires workspace-camera intrinsics. Its absence does not make session, verify, recording, or replay invalid.

Remove the board without moving the table, cameras, or headset. Repeat the session calibration after relocalization or a tracking reset. The saved outputs/calibration/session.yaml records tracking_device and table_from_device; use it only with the same --device.

When continuing a dataset with handumi record --resume, the recorder reads the current session file again and archives its calibration for the new episode range. You may reuse the same session while the VR origin is unchanged; after relocalization or a new day’s tracking setup, regenerate and verify it first. Use --session-calibration PATH if its location changed. See Resume a Recording for selection and history details.

5. Calibrate the HandUMI Tool Tip#

Controller-to-TCP reconstructs the physical tool-tip pose from each tracked controller. It belongs to the tool assembly – the gripper tip screwed onto the HandUMI shells, plus the controller mount – and not to the robot the data is later retargeted to. A different tip needs its own calibration even on the same robot, and one tip serves every robot it is used with.

Each side is captured the same way: wedge the tip into a firm indentation so it cannot slide, then rotate the rest of the assembly around it for 25 seconds, through as many different orientations as the mount allows. The tip staying put is what makes the fit correct; the variety of orientations is what makes it well-conditioned.

Step 1. Capture and fit the left side#

handumi calibrate tcp pivot --side left --time-s 25

This command opens only the VR tracking backend: it does not initialize cameras, Feetech grippers, or robot motors. After the left controller is tracked, fix the tip in the indentation and press ENTER. The raw controller poses are saved to outputs/tcp_pivot_left/poses.csv, and the fit is written to outputs/calibration/controller_tcp_candidate.yaml. The tracking device comes from recording.device in configs/rig.yaml; pass --device to override it.

--output-dir can override the capture directory. The older --dataset, --parquet, and --csv fitting inputs remain supported.

Step 2. Capture and fit the right side#

The same two commands with right in place of left:

handumi calibrate tcp pivot --side right --time-s 25

Both sides write into the same candidate file, including their fit metrics; nothing is applied to the project yet.

Step 3. Check the fit#

handumi calibrate tcp inspect

Metric

Accept

If it fails

RMS

below 0.50 cm

The tip slipped. Find a deeper indentation and recapture.

Maximum error

below 1.00 cm

As above; check for a moment of lost tracking.

Condition

below 500

The capture lacked rotational variety. Recapture covering more orientations.

Recapture that side until it passes. Do not promote a fit that does not.

inspect reports bilateral symmetry as an advisory diagnostic. The nominal mirror axis is x for PICO and y for Meta. This is an assembly assumption, not a measured constraint: seating and mounting rotation can affect every component, including the mirror axis. Differences produce warnings, never automatic averaging or rejection solely for asymmetry. handumi calibrate verify uses the same advisory policy.

Validate each side independently. Repeat a pivot capture with varied rotations and compare its offset with the previous capture of that same side. A low fit residual alone does not prove the correct physical point was held fixed. Use the intended grasp TCP consistently; after promotion, keep that point in a fixed indentation and check that it stays fixed while rotating the tool. Check known-height contacts in the verified table frame as a separate test.

Step 4. Promote it into the project#

Pivot fitting solves translation only, so keep the official quaternions and copy just the two measured position vectors, one per side, into the calibration file for this tool assembly. Record the pivot statistics and the date under metadata so the next reader knows what the numbers are worth.

Update only those two position values in the calibration file for this tool assembly. Those files live in configs/calibration/controller_tcp/ as {device}_{tool}.yaml, and each robot points at its own under controller_tcp_calibrations in configs/robots/<robot>.yaml:

handumi calibrate tcp promote --target pico_piper_beta.yaml

Only the filename is accepted; it is always resolved below configs/calibration/controller_tcp/. Promotion repeats the individual-fit and bilateral-symmetry diagnostics, and copies each side’s measured position without averaging. Both individual fits must pass; positions must be finite and both fits must identify the same tracking device. If the target exists, confirm the Override it? [Y/n] prompt. Existing quaternions and historical metadata are preserved. The top-level tcp_promotion record is authoritative for the current translations: it records promotion time, candidate fingerprint, per-side metrics and positions, and the orientation source. Older metadata may describe older measurements. --force is deprecated and cannot bypass individual fit checks.

For a genuinely new tool file there are no official quaternions to preserve. Name an existing calibration made with the same tracking device and controller mount as the orientation source:

handumi calibrate tcp promote --target pico_new_tip.yaml \
  --quaternion-source pico_piper_beta.yaml

handumi_tool:
  gripper: piper_beta         # the tip physically screwed onto HandUMI
  controller_mount: handumi_v1
controller_tcp_calibrations:
  pico: configs/calibration/controller_tcp/pico_piper_beta.yaml
  meta: configs/calibration/controller_tcp/meta_ARX5_beta.yaml  # Meta rig, ARX5 tip

Fitting a new tip means writing a new file and pointing the robot at it, rather than overwriting the previous tip’s calibration. Existing datasets keep their own recorded assembly identity, so old recordings stay reproducible.

Step 5. Verify#

Continue with the complete verification below. It replaces the removed tests/tracking/test_transforms.py and tests/scripts/test_replay_in_sim.py checks with a supported operator command.

6. Verify All Calibrations#

Run the final preflight after gripper, spatial/session, TCP, and robot/table calibration. --robot and --device may be omitted when they are set under recording in configs/rig.yaml:

handumi calibrate verify --robot piper --device pico

Physical robot/table placement is laboratory-local. Copy configs/calibration/table/local/example.yaml to the ignored conventional per-robot path, then measure it:

cp configs/calibration/table/local/example.yaml \
  configs/calibration/table/local/piper.yaml

Identify the laboratory in configs/rig.yaml:

deployment:
  lab: my_research_lab

Use the same stable lab identifier inside the copied calibration file. This identifier is saved with replay and conversion provenance; a mismatch is rejected instead of silently applying another laboratory’s transform. deployment.table_calibrations.<robot> remains available as an optional path override for laboratories that keep private calibration outside the repository.

The verifier requires scope: physical; it never treats the portable files in configs/calibration/table/sim/ as measurements of the current lab.

The command first checks:

  • both cached gripper calibrations and their encoder spans;

  • all three camera intrinsics, both controller-camera mounts, and their error limits;

  • session identity, spatial hash, tracking device, and table-frame residual;

  • the controller-to-TCP file actually selected for this robot/device, its device-specific mirror invariant, quaternion normalization, and tip-to-controller distance;

  • the robot-to-table file and whether it is marked physically verified.

It then guides the physical check: touch a marked point first with the left tip and then with the right tip, followed by touching the table with both tips. It fails if the calibrated point separation exceeds 15 mm or either table contact is farther than 15 mm from z=0. Use --static-only to inspect files without connecting the tracking device.

The default tip-distance ceiling is 0.35 m. That limit describes the physical tips currently in use, not a universal HandUMI constant. A genuinely longer new tip may use --max-tip-distance-m <meters> after its pivot fit has passed; do not widen the limit to hide a wrong robot/tool selection.

Next: Record Demonstrations.