> ## Documentation Index
> Fetch the complete documentation index at: https://dimensionalos.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Perception

> What perception ships in DimOS today: detection, tracking, markers, place memory, and VLM queries

Perception in DimOS is a set of composable modules under `dimos/perception/`. This page lists what actually ships and which blueprints run it - so you know what you can build on versus what is experimental.

## Detection

* **2D detection** - `Detection2DModule` runs any detector implementation over a camera stream. Shipped detectors: YOLO11 (generic objects), YOLO11-pose (people), and **YOLO-E**, an open-vocabulary, promptable segmentation-capable detector.
* **3D detection** - `Detection3DModule` unprojects 2D detections into 3D using depth or pointclouds, and exposes VLM query skills (`ask_vlm`, `nav_vlm`, backed by Qwen VL) so an agent can ask open-vocabulary questions about the scene. Runs in `unitree-go2-detection` and `unitree-g1-detection`.
* **Object database** - `ObjectDBModule` folds 3D detections into a persistent object table with deduplication (pending objects get promoted to permanent as evidence accumulates). Runs in `unitree-g1-detection`.
* **Person tracking** - `PersonTracker` (3D person tracks) feeds person following on the agentic stacks.

Detections flow as typed streams: `Detection2DArray` / `Detection3DArray` messages, consumable by any module.

## Fiducial markers

ArUco/AprilTag detection with pose estimation, published straight into the TF tree (`MarkerDetectionStreamModule` + `MarkerTfModule`). This is load-bearing infrastructure: the Go2 stacks use markers in `unitree-go2-markers`, `unitree-go2-relocalization`, and `unitree-go2-memory`. A standalone desk-webcam demo exists as `desk-marker-tf`.

## Place memory and agent-facing perception

* **SpatialMemory** - the live place map (CLIP embeddings against poses) that powers `navigate_with_text`. Runs in `unitree-go2-spatial` and the G1 perceptive stacks. Details: [Memory](/capabilities/memory).
* **PerceiveLoopSkill** - the agent's "keep looking out for X" tool, wired into `unitree-go2-spatial`.
* **WorldBelief** - on-demand scan-and-recall over recorded perception: DINOv2 appearance galleries for re-identification, CLIP text recall over frames, and present/absent/occluded classification for world points. Runs in `xarm6-worldbelief`.

## Perception for manipulation

`ObjectSceneRegistrationModule` uses YOLO-E to detect and register objects for arm workspaces. Runs in `xarm-perception` (and its agentic and sim variants); try it standalone with `dimos run demo-object-scene-registration`, which exposes open-vocabulary detection as MCP tools.

## Models used

| Model                     | Used for                                                                                                                                   |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| YOLO11 / YOLO11-pose      | 2D object and person detection                                                                                                             |
| YOLO-E                    | Open-vocabulary, promptable detection and segmentation                                                                                     |
| CLIP                      | Place memory embeddings, text-to-frame recall                                                                                              |
| DINOv2                    | Appearance embeddings for re-identification                                                                                                |
| Moondream                 | VLM detection - the default `detection_model`; also used to verify detections in [offline analysis](/capabilities/memory/offline-analysis) |
| Qwen VL                   | Open-vocabulary scene questions (`ask_vlm`)                                                                                                |
| ArUco / AprilTag (OpenCV) | Marker pose estimation                                                                                                                     |

DimOS ships two interchangeable VLM backends behind one interface (`dimos/models/vl/`): **Moondream** (small, fast, the default for detection via the `detection_model` config) and **Qwen VL** (larger, used for open-ended scene questions). Both local; a hosted Moondream variant also exists.

## Status, honestly

* Everything above is composed into registered blueprints and runs today.
* **Experimental:** `TemporalMemory` (video to entity/event graph) lives under `perception/experimental/` and runs only in `unitree-go2-temporal-memory`. Interfaces will change.
* Some modules exist in the registry but are not composed into any shipped stack yet (standalone 2D/3D object trackers, a generic Re-ID module). They work as building blocks but have no polished path; check the code before relying on them.