HandUMI Setup and Calibration

HandUMI Setup and Calibration#

Complete this page before recording. No robot arm is required: these steps configure HandUMI, its tracking device, cameras, grippers, and workspace. Some calibrations are permanent for one physical assembly; the table/session alignment must be checked each session.

Calibration

Repeat when

Servo homing and opening width

Servo, linkage, or gripper geometry changes

Camera intrinsics

Camera, resolution, or focus changes

Controller-to-camera mount

A controller or wrist camera mount moves

Controller-to-TCP

The controller/gripper mount or physical tool changes

Table/session frame

Each session, relocalization, or tracking reset

1. Map HandUMI Hardware#

install.sh creates the ignored machine-local configs/rig.yaml. Inspect the connected cameras and Feetech adapters:

handumi setup ports

Reconnect one physical device at a time and assign its port under cameras or feetech in configs/rig.yaml. Robot-arm buses do not belong in this recording setup; configure them only for real-robot teleoperation.

Set new Feetech IDs only when required:

handumi servo set-id --port /dev/ttyUSB0 --new-id 0
handumi servo set-id --port /dev/ttyUSB0 --new-id 1
Hardware mapping details

Two grippers may share one serial port only when they use different servo_id values. With separate USB adapters, each side normally has its own port.

A USB camera commonly exposes two /dev/video* nodes. Start with the first node reported for each physical camera and confirm the stream. Map left_wrist, right_wrist, and workspace explicitly in configs/rig.yaml.

Keep these machine-local paths in configs/rig.yaml; do not commit them as portable project configuration.

Camera types and resolutions#

Declare capture settings for the three logical views: left_wrist, right_wrist, and workspace. Each view can use the normal opencv backend or zedmini, which expects the ZED Mini side-by-side UVC mode and exposes only its left image:

cameras:
  workspace:
    type: zedmini
    index_or_path: 4
    width: 1344
    height: 376
    fps: 30

For zedmini, width and height describe the captured stereo frame. HandUMI keeps frame[:, :672], so previews and datasets contain one 672×376 RGB image. The supported rates for this mode are 15, 30, 60, and 100 FPS. index_or_path accepts either an OpenCV integer such as 4 or an explicit Linux path such as /dev/video4. Per-camera values in cameras: take precedence over the global camera fallbacks used by older rig files.

2. Calibrate the Grippers#

First confirm that both encoders change smoothly while opening and closing:

handumi calibrate grippers monitor

Run the complete guided calibration:

handumi calibrate grippers
handumi calibrate grippers --side right  # one side only

For each side, enter the maximum physical opening in millimeters. The command then asks you to hold the gripper at mid-travel and press Enter, which centers the encoder range to avoid the 0/4095 wrap point. Finally, place the gripper fully open and press Enter, then fully close it and press Enter. The result is stored in ~/.cache/handumi/calibration.yaml. Open and close each gripper again with monitor and confirm that width increases toward fully open without flipping or saturating.

3. Connect Tracking#

Meta Quest#

Enable Developer Mode, connect the headset over USB, authorize adb, and install HandUMI Quest App:

wget https://github.com/murobotics-ai/handumi-quest-app/releases/download/v0.2.1/handumi-quest-app-v0.2.1.apk
adb install -r handumi-quest-app-v0.2.1.apk
adb shell ip route  # find the address after "src"

Set that address as meta_quest.connection.quest_ip in configs/rig.yaml. Launch the app from Library → Unknown Sources and keep it in the foreground.

python -m handumi.tracking.meta_quest --config configs/rig.yaml

A healthy stream reports steady FPS and both controllers tracked.

PICO#

install.sh already installed the XRoboToolkit PC service unless it was run with --skip-xrt; install it from the XRoboToolkit PC Service releases only if /opt/apps/roboticsservice/runService.sh is missing. Follow the current XR Robotics headset instructions for the headset side. Start the PC service in its own terminal and leave it running – xrt.init() aborts with a core dump without it:

bash /opt/apps/roboticsservice/runService.sh

Use 127.0.0.1:63901 for USB or the workstation IP with --pico-wifi.

Smoke-test a short capture before calibration:

handumi record --output-dir outputs/pico-smoke \
  --device pico --skip-feetech --no-voice-control \
  --task "pico smoke" --episodes 1 --episode-time-s 10

Healthy output reports xrobotoolkit_sdk initialised without repeated still waiting for PICO data messages.

4. Calibrate Cameras and Workspace#

Fix the 5 × 7 ChArUco board flat at its marked table position, with IDs 15 and 16 nearest the operator. Its center defines the table origin: +X right, +Y away, and +Z up.

Camera Intrinsics#

handumi calibrate spatial intrinsics --camera left_wrist
handumi calibrate spatial intrinsics --camera right_wrist
handumi calibrate spatial intrinsics --camera workspace

Move the board throughout each image and vary distance and inclination. The tool automatically accepts a distinct valid view every two seconds. Repeat after changing camera, resolution, or focus.

Controller-to-Camera Mounts#

Keep the board fixed. Move the complete HandUMI through varied roll, pitch, and yaw poses, pausing briefly for each automatic capture. Keep the controller tracking ring visible to the headset.

Choose the tracking device explicitly. Global options such as --device, --pico-wifi, and --quest-ip come before the subcommand.

Meta Quest:

handumi calibrate spatial --device meta mount --side left
handumi calibrate spatial --device meta mount --side right

PICO:

handumi calibrate spatial --device pico --pico-mode mandos mount --side left
handumi calibrate spatial --device pico --pico-mode mandos mount --side right

PICO calibration relies on live XRoboToolkit snapshots, so hold the HandUMI steady while each view is accepted. Use --pico-wifi for a wireless PICO setup.

Repeat only if a controller or wrist-camera mount moves.

Session/Table Frame#

With the board still at its marked position and the headset fixed as it will be during recording, solve the table frame for the same tracking device.

handumi calibrate spatial --device meta session --side left
handumi calibrate spatial --device meta verify --side left
handumi calibrate spatial --device meta visualize

For PICO:

handumi calibrate spatial --device pico --pico-mode mandos session --side left
handumi calibrate spatial --device pico --pico-mode mandos verify --side left
handumi calibrate spatial --device pico --pico-mode mandos visualize

session reads wrist-camera intrinsics and the controller-to-camera mount from outputs/calibration/spatial.yaml by default. If calibration was saved under a different name, select that same file explicitly for both commands, for example --spatial outputs/calibration/spatial_60mm.yaml. The command stops before hardware capture when the selected spatial file and configs/rig.yaml describe different ChArUco dimensions, and warns when their camera mappings differ.

verify captures fresh views without changing either calibration and rejects a table-frame error above its acceptance limits. Run it while the board is still fixed; recapture the session if it fails.

Inspect all cameras and both TCP trails in Rerun. The table surface must align with z=0. If only the workspace-camera stage fails, retry it with:

handumi calibrate spatial workspace

Remove the board without moving the table, cameras, or headset. Repeat the session calibration after relocalization or a tracking reset. The saved outputs/calibration/session.yaml records tracking_device and table_from_device; use it only with the same --device.

5. Calibrate the HandUMI Tool Tip#

Controller-to-TCP reconstructs the physical tool-tip pose from each tracked controller. It belongs to the tool assembly – the gripper tip screwed onto the HandUMI shells, plus the controller mount – and not to the robot the data is later retargeted to. A different tip needs its own calibration even on the same robot, and one tip serves every robot it is used with.

Each side is captured the same way: wedge the tip into a firm indentation so it cannot slide, then rotate the rest of the assembly around it for 25 seconds, through as many different orientations as the mount allows. The tip staying put is what makes the fit correct; the variety of orientations is what makes it well-conditioned.

Step 1. Capture and fit the left side#

LEFT=outputs/tcp_pivot_left
handumi record --output-dir $LEFT --skip-feetech --no-voice-control \
  --cameras left_wrist --task "tcp pivot left" \
  --episodes 1 --episode-time-s 25 --tracking-loss-timeout-s 3 --no-sounds

handumi calibrate tcp pivot --side left --dataset $LEFT

--dataset resolves the recording’s parquet and episode, and the fit is written to outputs/calibration/controller_tcp_candidate.yaml. The tracking device comes from recording.device in configs/rig.yaml; pass --device to override it.

Your hands are busy holding the tool during a pivot capture, so --no-voice-control keeps the episode on the plain ENTER-then-timer flow instead of waiting to be spoken to.

Step 2. Capture and fit the right side#

The same two commands with right in place of left:

RIGHT=outputs/tcp_pivot_right
handumi record --output-dir $RIGHT --skip-feetech --no-voice-control \
  --cameras right_wrist --task "tcp pivot right" \
  --episodes 1 --episode-time-s 25 --tracking-loss-timeout-s 3 --no-sounds

handumi calibrate tcp pivot --side right --dataset $RIGHT

Both sides write into the same candidate file; nothing is applied to the project yet.

Step 3. Check the fit#

handumi calibrate tcp inspect

Metric

Accept

If it fails

RMS

below 0.50 cm

The tip slipped. Find a deeper indentation and recapture.

Maximum error

below 1.00 cm

As above; check for a moment of lost tracking.

Condition

below 500

The capture lacked rotational variety. Recapture covering more orientations.

Recapture that side until it passes. Do not promote a fit that does not.

Then compare the two sides. The mounts are mirror twins, so two of the position components should agree between them and only one flips sign. A mismatch of several millimeters on the components that should agree means one of the captures drifted, not that the tool is asymmetric.

Which component flips sign depends on the tracking device’s own controller frame convention, not on the physical mount: for meta it is y (x/z agree); for pico it is x (y/z agree). Identify it from your own two captures before symmetrizing – don’t assume y just because the formula below is written that way.

Step 4. Promote it into the project#

Pivot fitting solves translation only, so keep the official quaternions and symmetrize just the measured positions. Below, y names whichever component you identified as the sign-flipping one in Step 3 (x for pico, y for meta), and x/z name the two that agree:

x = (left.x + right.x) / 2
y = (left.y - right.y) / 2
z = (left.z + right.z) / 2
left.position  = [x,  y, z]
right.position = [x, -y, z]

Update only those two position values in the calibration file for this tool assembly. Those files live in configs/calibration/controller_tcp/ as {device}_{tool}.yaml, and each robot points at its own under controller_tcp_calibrations in configs/robots/<robot>.yaml:

handumi_tool:
  gripper: ARX5_beta          # the tip physically screwed onto HandUMI
  controller_mount: handumi_v1
controller_tcp_calibrations:
  meta: configs/calibration/controller_tcp/meta_ARX5_beta.yaml

Fitting a new tip means writing a new file and pointing the robot at it, rather than overwriting the previous tip’s calibration. Existing datasets keep their own recorded assembly identity, so old recordings stay reproducible.

Step 5. Verify#

Continue with the complete verification below. It replaces the removed tests/tracking/test_transforms.py and tests/scripts/test_replay_in_sim.py checks with a supported operator command.

6. Verify All Calibrations#

Run the final preflight after gripper, spatial/session, TCP, and robot/table calibration. --robot and --device may be omitted when they are set under recording in configs/rig.yaml:

handumi calibrate verify --robot piper --device pico

Physical robot/table placement is laboratory-local. Copy configs/calibration/table/local/example.yaml to the ignored conventional per-robot path, then measure it:

cp configs/calibration/table/local/example.yaml \
  configs/calibration/table/local/piper.yaml

Identify the laboratory in configs/rig.yaml:

deployment:
  lab: my_research_lab

Use the same stable lab identifier inside the copied calibration file. This identifier is saved with replay and conversion provenance; a mismatch is rejected instead of silently applying another laboratory’s transform. deployment.table_calibrations.<robot> remains available as an optional path override for laboratories that keep private calibration outside the repository.

The verifier requires scope: physical; it never treats the portable files in configs/calibration/table/sim/ as measurements of the current lab.

The command first checks:

  • both cached gripper calibrations and their encoder spans;

  • all three camera intrinsics, both controller-camera mounts, and their error limits;

  • session identity, spatial hash, tracking device, and table-frame residual;

  • the controller-to-TCP file actually selected for this robot/device, its device-specific mirror invariant, quaternion normalization, and tip-to-controller distance;

  • the robot-to-table file and whether it is marked physically verified.

It then guides the physical check: touch a marked point first with the left tip and then with the right tip, followed by touching the table with both tips. It fails if the calibrated point separation exceeds 15 mm or either table contact is farther than 15 mm from z=0. Use --static-only to inspect files without connecting the tracking device.

The default tip-distance ceiling is 0.35 m. That limit describes the physical tips currently in use, not a universal HandUMI constant. A genuinely longer new tip may use --max-tip-distance-m <meters> after its pivot fit has passed; do not widen the limit to hide a wrong robot/tool selection.

Next: Record Demonstrations.