Add a New Robot Embodiment

Add a New Robot Embodiment#

HandUMI recordings are robot agnostic. An embodiment adapts those recordings to a robot for simulation, replay, or real teleoperation.

This guide covers fixed-base robots with one or two arms, with one TCP and one parallel gripper per active side. Start by choosing the scope of the contribution:

Scope	Required
Simulation / replay	URDF, assets, robot YAML, tests, and replay evidence.
Absolute-table replay	Above, plus a `table/<robot>.yaml` placement.
Real hardware	Above, plus a backend, physical calibrations, and hardware safety tests.

Do not add vendor SDKs, local ports, credentials, or robot-specific recording requirements to the HandUMI recorder.

1. Prepare source assets#

Create this layout:

assets/<robot>/
├── README.md                 # vendor URL, commit/release, license, derivations
├── LICENSE.<vendor>          # when redistribution requires it
├── <robot>.urdf
├── <robot>.xml               # optional MuJoCo contact model
└── meshes/

configs/robots/
└── <robot>.yaml

Use official assets when possible. In assets/<robot>/README.md, record the source repository, immutable commit or release, license, and every generated mesh. Do not commit unlicensed assets, local calibration files, datasets, or configs/rig.yaml.

If the vendor supplies two single-arm URDFs, build one combined bimanual URDF:

namespace every copied link and joint as left_ or right_;
preserve the vendor joint origins, axes, limits, and mesh transforms;
add fixed mounts from one shared base to each arm;
add one fixed TCP link per side at the actual grasp point;
keep gripper joints actuated and give all visual mesh paths a resolvable package:// or pkg_root path.

The robot name and joint names are declared in YAML, so no central registry is needed for simulation. Add assets/<robot> to the wheel force-include section of pyproject.toml when the assets must ship with the package.

For one arm, include only that physical arm and declare only arms.right or arms.left. Do not duplicate it or add dummy joints. Recording and simulated teleoperation default to those declared sides; canonical conversion writes only their joints and widths. See assets/franka_emika_panda and configs/robots/franka_emika_panda.yaml for the right-only case. The legacy pose-pair API uses the fixed root as a placeholder for the absent side; IK excludes it and replay does not grade or display it.

2. Add the robot YAML#

Create configs/robots/<robot>.yaml:

kind: myrobot
urdf: assets/myrobot/myrobot.urdf
pkg_root: assets/myrobot
# mjcf: assets/myrobot/myrobot.xml  # only if a MuJoCo model exists

arms:
  left:
    ee_link: left_tcp
    joint_names: [left_joint1, left_joint2]
    gripper_joints:
      - {name: left_finger_joint, closed: 0.0, open: 0.035}
  right:
    ee_link: right_tcp
    joint_names: [right_joint1, right_joint2]
    gripper_joints:
      - {name: right_finger_joint, closed: 0.0, open: 0.035}

# One value for every actuated URDF joint, including fingers.
home_q: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
gripper_max_width_m: 0.07

ik_weights:
  pos: 100.0
  ori: 15.0
  rest: 2.0
  max_joint_delta: 0.06981317
  max_reach: 0.45
  # Optional capsule self-collision penalty; omit for the legacy cost structure.
  # Requires <collision> geometry in the URDF (capsulized automatically, and
  # pairs violating the margin at home_q are auto-ignored as structural).
  # self_collision: 100.0
  # self_collision_margin: 0.015
  # self_collision_pairs: inter-arm  # or "all"; inter-arm is much cheaper
  # collision_activation_distance: 0.05  # farther than this skips the costs

replay:
  max_joint_delta: 0.20
  # Use physical-width when source and target grippers have different strokes.
  # gripper_retarget: physical-width
  # Optional replay-only IK weights (pos/ori/rest). Use when the teleop
  # profile above is tuned for a human in the loop -- a high rest weight
  # holds the arm near home, which is good for teleoperation but cannot
  # follow a recorded trajectory. Real backends keep ik_weights untouched.
  # ik_weights:
  #   ori: 4.5
  #   rest: 4.0

closed and open are the actual URDF joint values for normalized HandUMI opening 0 and 1. Verify them visually; URDF joint signs are not universal. Use gripper_retarget: physical-width to preserve aperture in meters when the target gripper has a different maximum width. The default, normalized, preserves the source opening percentage.

Useful references are configs/robots/piper.yaml, openarmv1.yaml, trlc_dk1.yaml, yam.yaml, metal.yaml and r1lite.yaml. metal.yaml pairs with assets/metal/README.md as a complete worked example of combining a vendor single-arm URDF, including the driven/mimic finger pattern.

3. Add a simulation test#

Add a focused test in tests/robots/test_registry.py that checks:

declared arm joint order and indices;
home pose length and FK sanity (symmetry for paired arms, vendor parity for one arm);
gripper closed/open mapping, if present;
load_urdf(load_meshes=True) resolves every visual mesh.

Run it before attempting replay:

JAX_PLATFORMS=cpu uv run pytest -q tests/robots/test_registry.py
JAX_PLATFORMS=cpu uv run python -c \
  "from handumi.robots.registry import load_embodiment; print(load_embodiment('myrobot').joint_names)"

4. Calibrate only when the scope needs it#

Table placement for absolute replay#

Add configs/calibration/table/sim/<robot>.yaml. This is the canonical simulation transform from the demonstration table frame to the target robot world; it is not a TCP offset or a physical-lab measurement.

schema_version: 1
kind: handumi_robot_table_calibration
robot: myrobot
scope: simulation
source: configured_simulation_layout
verified: false
calibration:
  frame_convention: pose7=[x,y,z,qx,qy,qz,qw], meters, xyzw quaternion
  robot_from_table:
    position: [0.0, 0.0, 0.0]
    quaternion: [0.0, 0.0, 0.0, 1.0]

Simulation placements stay verified: false. For a physical installation, copy configs/calibration/table/local/example.yaml to configs/calibration/table/local/<robot>.yaml, set scope: physical, and measure it. Set that ignored local file to verified: true only after the physical checks pass. Use deployment.table_calibrations.<robot> only as a non-standard path override. Never compensate a wrong TCP calibration by changing either transform.

HandUMI controller-to-TCP calibration#

This calibration belongs to the physical HandUMI assembly:

tracking controller + HandUMI mount + HandUMI gripper/tool + side

It does not automatically change because the target robot changes. A simulation-only PR needs no new pivot capture. Reuse an identity-bound dataset snapshot only when the physical HandUMI assembly is exactly the same; otherwise follow HandUMI Setup and Calibration and reference the new file from controller_tcp_calibrations and handumi_tool in the robot YAML.

5. Replay a validation episode#

For absolute-table replay, use a recorded validation episode and inspect the model in Viser:

JAX_PLATFORMS=cpu uv run handumi replay \
  outputs/myrobot_validation \
  --episode 0 --robot myrobot --retarget-mode absolute-table --strict-ik

Check mesh loading, home pose, TCP placement, gripper direction and aperture, table height, shared workspace, and reported IK errors. A large error usually means a bad TCP, placement, home pose, joint order, or IK limit.

5b. Declare the LeRobot follower layout#

Conversion writes the canonical vector (radians and meters) for every embodiment. To let handumi convert --output-layout <robot_type> write the vector a LeRobot plugin records for this robot, add one ExternalJointLayout to src/handumi/dataset/external_layouts.py:

robot_type: the LeRobot plugin’s name (bi_myrobot_follower), which also names the exported dataset.
arm_names and gripper_name: the plugin’s motor names, in its order.
one JointEncoding per arm joint and one for the gripper, mirroring the plugin’s MotorNormMode: range_m100_100 with the limits and sign the driver normalizes over, degrees, or range_0_100. Read them from the plugin’s source and cite it in source; a dataset never records them.
degrees_option: optional if the plugin exposes use_degrees, always for a bus that only reports degrees, never otherwise.
assumptions: anything the dataset cannot prove, such as the calibration zero coinciding with the URDF zero.

check_layout_limits refuses to export when a ranged encoding’s limits differ from the URDF’s by more than rounding, so a wrong limit fails loudly instead of rescaling every joint. Convert a few episodes, replay the result with handumi replay-joints, and compare the schema against a dataset the plugin recorded itself with handumi dataset export --compare-with.

6. Add real hardware support only when ready#

Replay support does not provide robot control. A hardware PR must implement the TeleopRobotBackend contract in src/handumi/real/base.py: setup, connect, disconnect, read, home, move_home, write, hold, and health check. Register it lazily in make_real_backend (src/handumi/real/registry.py), declare it in the robot YAML as real.backend, keep vendor units and SDK code inside that backend, and add backend tests. configs/rig.yaml should only hold local transport details such as CAN ports. Real control uses radians, meters, normalized openings, and XYZW pose quaternions. A registered backend also serves handumi replay-real, which streams converted datasets through it; declare the backend’s joint speed limit in the robot YAML (real.max_joint_speed_deg_s, or real.control.max_joint_speed_rad_s) and, if its stream limits acceleration, real.max_joint_acceleration_deg_s2, so the replay pre-check can predict the stream’s tracking error.

7. Open the pull request#

Before opening the PR, run:

JAX_PLATFORMS=cpu uv run pytest -q
uv build
.venv-docs/bin/sphinx-build -W -b html docs/source /tmp/handumi-docs
git status --short

The PR description should state:

robot model, vendor source, immutable revision, and license;
scope: simulation, absolute replay, and/or real hardware;
frame convention, TCP and gripper mapping decisions;
replay command, frame count, and position/rotation error summary;
a simulator screenshot or short recording;
physical calibration evidence only when claiming hardware support.