Manipulation

What DimOS offers for arms: motion planning, three ways to teleoperate, and an imitation-learning loop

DimOS manipulation is three things:

1. Classical motion planning - a Drake-backed world model with RRT-Connect planning and Pink IK, driven through the ManipulationModule RPC interface (RoboPlan available as an alternative backend).
2. Teleoperation - three ways to move an arm by hand: zero-gravity teach mode, VR (Quest), and keyboard.
3. An imitation-learning loop - record demonstrations, export a LeRobot dataset, train a policy, and run it back on the arm. Collection and export ship on every arm; running a trained checkpoint on hardware ships today on the Galaxea A1Z. See Collect data & train a policy for the full loop, or the complete A1Z guide for the hand-teach → train → deploy path start to finish.

Where do you want to start?

I want to…	Go to
Move an arm right now, no hardware	Keyboard teleop quick start below
Teach a real arm a task by dragging it	Galaxea A1Z teach mode
Collect demonstrations and train a policy	Collect data & train
Teleoperate with a VR headset	VR teleoperation
Plan collision-free motions programmatically	Motion planning below
Bring my own arm into DimOS	Adding a custom arm

Choosing a teleop method

Ranked by how well they work for demonstrating tasks:

1. Teach mode (kinesthetic) - put the arm in gravity-compensated zero-force mode and drag it through the task. Most natural demonstrations, no extra hardware. Available on the Galaxea A1Z.
2. VR (Quest) - 6-DOF controller tracking with hold-to-engage and per-episode record controls. Best when the operator must stay out of the workspace. VR teleoperation.
3. Keyboard - one key per axis. Good for smoke-testing a stack and demos without hardware; genuinely painful for collecting training data. Do not plan a data-collection session around it.

Quick start: keyboard teleop (no hardware)

Each blueprint launches the full stack - keyboard UI, mock controller, IK solver, and Drake visualization:

dimos run keyboard-teleop-xarm7   # XArm7 7-DOF
dimos run keyboard-teleop-a750    # A-750 6-DOF
dimos run keyboard-teleop-piper   # Piper 6-DOF
dimos run keyboard-teleop-a1z     # Galaxea A1Z 6-DOF

Open the Meshcat URL printed in the terminal (default http://localhost:7000) to see the robot.

Key	Action
W/S	+X/-X (forward/back)
A/D	+Y/-Y (left/right)
Q/E	+Z/-Z (up/down)
R/F	+Roll/-Roll
T/G	+Pitch/-Pitch
Y/H	+Yaw/-Yaw
ESC	Quit

Motion planning

Install the dependencies, then run a planner against a mock coordinator:

uv sync --extra manipulation --inexact

# Terminal 1: mock coordinator
dimos run coordinator-mock

# Terminal 2: planner with Drake visualization
dimos run xarm7-planner-coordinator

Drive it from the IPython client:

python -m dimos.manipulation.planning.examples.manipulation_client

skip

joints()                # Get current joints
plan([0.1] * 7)         # Plan to target
preview()               # Preview in Meshcat
execute()               # Execute via coordinator

Backends

Planning separates the world representation from the planner algorithm and the IK backend:

world_backend	planner_name	kinematics.backend	Status
drake	rrt_connect	pink	Default path
drake	rrt_connect	jacobian	Legacy Jacobian IK
drake	rrt_connect	drake_optimization	Drake-only IK
roboplan	rrt_connect	pink or jacobian	Generic RRT over RoboPlan collision checks
roboplan	roboplan	pink or jacobian	RoboPlan-native planner

Select per run with module options; invalid combinations fail at startup rather than at the first plan request:

dimos run xarm7-planner-coordinator \
  -o manipulationmodule.world_backend=roboplan \
  -o manipulationmodule.planner_name=rrt_connect

Visualization

Planning visualization is configured on ManipulationModuleConfig.visualization, independent of the global Rerun viewer: meshcat (embedded Drake visualizer), viser (in-process web visualizer with preview ghosts and panel controls), or none.

uv run dimos run xarm7-planner-coordinator -o manipulationmodule.visualization.backend=viser

Perception + agent

An arm stack with a camera and an LLM agent on top, in one command:

XARM7_IP=<ip> dimos run coordinator-xarm7 xarm-perception-agent

Architecture

KeyboardTeleopModule ──→ ControlCoordinator ──→ ManipulationModule
  (pygame UI)              (100Hz tick loop)      (WorldSpec backend)
       │                        │                       │
  TwistStamped           EEFTwistTask             RRT planner
  spatial EEF twist      (Pinocchio FK/IK)        Pink / Jacobian IK
                               │                   DrakeWorld
                          JointState ────────────→ (visualization)

• KeyboardTeleopModule - UI publishing spatial end-effector twist intent
• ControlCoordinator - 100 Hz control loop with mock or real hardware adapters
• ManipulationModule - world backend, motion planning, obstacle management, visualization

Supported robots

Robot	DOF	Teleop	Planning	Teach mode	Perception
Galaxea A1Z	6	keyboard	Y	Y	camera recording
A-750	6	keyboard	Y	-	-
OpenArm	2x7	keyboard	Y	-	-
Piper	6	keyboard, VR	Y	-	-
XArm6	6	keyboard, VR	Y	-	-
XArm7	7	keyboard, VR	Y	-	Y

Key files

File	Description
manipulation_module.py	Main module (RPC interface, state machine)
robot/manipulators/common/blueprints.py	Shared coordinator, planner, and task helpers
teleop/keyboard/keyboard_teleop_module.py	Keyboard teleop module
teleop/quest/quest_teleop_module.py	Quest VR teleop module
planning/world/drake_world.py	Drake physics backend
planning/planners/rrt_planner.py	RRT-Connect motion planner
learning/	Data collection + dataset prep