> ## Documentation Index
> Fetch the complete documentation index at: https://dimensionalos.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Manipulation

> What DimOS offers for arms: motion planning, three ways to teleoperate, and an imitation-learning loop

DimOS manipulation is three things:

1. **Classical motion planning** - a Drake-backed world model with RRT-Connect planning and Pink IK, driven through the `ManipulationModule` RPC interface (RoboPlan available as an alternative backend).
2. **Teleoperation** - three ways to move an arm by hand: zero-gravity teach mode, VR (Quest), and keyboard.
3. **An imitation-learning loop** - record demonstrations, export a LeRobot dataset, train a policy, and run it back on the arm. Collection and export ship on every arm; running a trained checkpoint on hardware ships today on the [Galaxea A1Z](/platforms/arms/a1z). See [Collect data & train a policy](/capabilities/manipulation/collect-and-train) for the full loop, or the [complete A1Z guide](/platforms/arms/a1z) for the hand-teach → train → deploy path start to finish.

## Where do you want to start?

| I want to...                                 | Go to                                                                         |
| -------------------------------------------- | ----------------------------------------------------------------------------- |
| Move an arm right now, no hardware           | [Keyboard teleop quick start](#quick-start-keyboard-teleop-no-hardware) below |
| Teach a real arm a task by dragging it       | [Galaxea A1Z teach mode](/platforms/arms/a1z)                                 |
| Collect demonstrations and train a policy    | [Collect data & train](/capabilities/manipulation/collect-and-train)          |
| Teleoperate with a VR headset                | [VR teleoperation](/capabilities/teleoperation/vr-teleoperation)              |
| Plan collision-free motions programmatically | [Motion planning](#motion-planning) below                                     |
| Bring my own arm into DimOS                  | [Adding a custom arm](/platforms/arms/adding_a_custom_arm)                    |

## Choosing a teleop method

Ranked by how well they work for demonstrating tasks:

1. **Teach mode (kinesthetic)** - put the arm in gravity-compensated zero-force mode and drag it through the task. Most natural demonstrations, no extra hardware. Available on the [Galaxea A1Z](/platforms/arms/a1z).
2. **VR (Quest)** - 6-DOF controller tracking with hold-to-engage and per-episode record controls. Best when the operator must stay out of the workspace. [VR teleoperation](/capabilities/teleoperation/vr-teleoperation).
3. **Keyboard** - one key per axis. Good for smoke-testing a stack and demos without hardware; genuinely painful for collecting training data. Do not plan a data-collection session around it.

## Quick start: keyboard teleop (no hardware)

Each blueprint launches the full stack - keyboard UI, mock controller, IK solver, and Drake visualization:

```bash theme={null}
dimos run keyboard-teleop-xarm7   # XArm7 7-DOF
dimos run keyboard-teleop-a750    # A-750 6-DOF
dimos run keyboard-teleop-piper   # Piper 6-DOF
dimos run keyboard-teleop-a1z     # Galaxea A1Z 6-DOF
```

Open the Meshcat URL printed in the terminal (default `http://localhost:7000`) to see the robot.

| Key | Action               |
| --- | -------------------- |
| W/S | +X/-X (forward/back) |
| A/D | +Y/-Y (left/right)   |
| Q/E | +Z/-Z (up/down)      |
| R/F | +Roll/-Roll          |
| T/G | +Pitch/-Pitch        |
| Y/H | +Yaw/-Yaw            |
| ESC | Quit                 |

## Motion planning

Install the dependencies, then run a planner against a mock coordinator:

```bash theme={null}
uv sync --extra manipulation --inexact

# Terminal 1: mock coordinator
dimos run coordinator-mock

# Terminal 2: planner with Drake visualization
dimos run xarm7-planner-coordinator
```

Drive it from the IPython client:

```bash theme={null}
python -m dimos.manipulation.planning.examples.manipulation_client
```

```python skip theme={null}
joints()                # Get current joints
plan([0.1] * 7)         # Plan to target
preview()               # Preview in Meshcat
execute()               # Execute via coordinator
```

### Backends

Planning separates the world representation from the planner algorithm and the IK backend:

| `world_backend` | `planner_name` | `kinematics.backend` | Status                                     |
| --------------- | -------------- | -------------------- | ------------------------------------------ |
| `drake`         | `rrt_connect`  | `pink`               | Default path                               |
| `drake`         | `rrt_connect`  | `jacobian`           | Legacy Jacobian IK                         |
| `drake`         | `rrt_connect`  | `drake_optimization` | Drake-only IK                              |
| `roboplan`      | `rrt_connect`  | `pink` or `jacobian` | Generic RRT over RoboPlan collision checks |
| `roboplan`      | `roboplan`     | `pink` or `jacobian` | RoboPlan-native planner                    |

Select per run with module options; invalid combinations fail at startup rather than at the first plan request:

```bash theme={null}
dimos run xarm7-planner-coordinator \
  -o manipulationmodule.world_backend=roboplan \
  -o manipulationmodule.planner_name=rrt_connect
```

### Visualization

Planning visualization is configured on `ManipulationModuleConfig.visualization`, independent of the global Rerun viewer: `meshcat` (embedded Drake visualizer), `viser` (in-process web visualizer with preview ghosts and panel controls), or `none`.

```bash theme={null}
uv run dimos run xarm7-planner-coordinator -o manipulationmodule.visualization.backend=viser
```

## Perception + agent

An arm stack with a camera and an LLM agent on top, in one command:

```bash theme={null}
XARM7_IP=<ip> dimos run coordinator-xarm7 xarm-perception-agent
```

## Architecture

```
KeyboardTeleopModule ──→ ControlCoordinator ──→ ManipulationModule
  (pygame UI)              (100Hz tick loop)      (WorldSpec backend)
       │                        │                       │
  TwistStamped           EEFTwistTask             RRT planner
  spatial EEF twist      (Pinocchio FK/IK)        Pink / Jacobian IK
                               │                   DrakeWorld
                          JointState ────────────→ (visualization)
```

* **KeyboardTeleopModule** - UI publishing spatial end-effector twist intent
* **ControlCoordinator** - 100 Hz control loop with mock or real hardware adapters
* **ManipulationModule** - world backend, motion planning, obstacle management, visualization

## Supported robots

| Robot                              | DOF | Teleop       | Planning | Teach mode | Perception       |
| ---------------------------------- | --- | ------------ | -------- | ---------- | ---------------- |
| [Galaxea A1Z](/platforms/arms/a1z) | 6   | keyboard     | Y        | **Y**      | camera recording |
| [A-750](/platforms/arms/a750)      | 6   | keyboard     | Y        | -          | -                |
| [OpenArm](/platforms/arms/openarm) | 2x7 | keyboard     | Y        | -          | -                |
| Piper                              | 6   | keyboard, VR | Y        | -          | -                |
| XArm6                              | 6   | keyboard, VR | Y        | -          | -                |
| XArm7                              | 7   | keyboard, VR | Y        | -          | Y                |

## Key files

| File                                                                                            | Description                                   |
| ----------------------------------------------------------------------------------------------- | --------------------------------------------- |
| [`manipulation_module.py`](/dimos/manipulation/manipulation_module.py)                          | Main module (RPC interface, state machine)    |
| [`robot/manipulators/common/blueprints.py`](/dimos/robot/manipulators/common/blueprints.py)     | Shared coordinator, planner, and task helpers |
| [`teleop/keyboard/keyboard_teleop_module.py`](/dimos/teleop/keyboard/keyboard_teleop_module.py) | Keyboard teleop module                        |
| [`teleop/quest/quest_teleop_module.py`](/dimos/teleop/quest/quest_teleop_module.py)             | Quest VR teleop module                        |
| [`planning/world/drake_world.py`](/dimos/manipulation/planning/world/drake_world.py)            | Drake physics backend                         |
| [`planning/planners/rrt_planner.py`](/dimos/manipulation/planning/planners/rrt_planner.py)      | RRT-Connect motion planner                    |
| [`learning/`](/dimos/learning/)                                                                 | Data collection + dataset prep                |