GHOST in the Robots

What is GHOST?

Two robots beat one: more viewpoints, more reach, and the ability to actually carry things together. The catch is control. One person has to juggle two bases, two arms, and a camera view — while the system keeps multiple video streams flowing over Wi-Fi without falling behind. Most existing systems sidestep this by adding a second operator, leaning on autonomy, or limiting the human to high-level commands.

GHOST takes a different route: an open-source VR system that gives a single operator direct, low-level control of two mobile manipulators, using nothing but the robots' onboard cameras. Live point clouds from both robots are fused into a shared exocentric 3D scene — the operator moves through it like an invisible ghost standing alongside the robots — with a learning-based depth-completion model filling in what the sparse sensors miss. Two VR controllers cover everything through switchable control modes: drive one robot, guide its arm, or lock both robots together and move them as a unit.

And it works. Fifteen first-time users hit 1.6–4× the success rate of the off-the-shelf tablet interface. Across nine hard dual-robot tasks, experts were 1.47× faster on average — and completed two tasks the tablet simply couldn't do.

What's new

A shared exocentric 3D scene for multi-robot teleoperation.
A depth-completion pipeline fast enough to stay in the loop.
A principled control design space for driving two robots with two hands.
A suite of nine challenging tasks for benchmarking dual-robot teleoperation.

How It Works

The robots are two Boston Dynamics Spots — each a quadruped base with a 6-DoF arm and gripper, a LiDAR on its back, and two front-facing RGB-D cameras — talking over Wi-Fi to a ROS 2 server and a VR client. Under the hood, GHOST wires familiar parts (Unity, ROS 2, PromptDA, the Spot SDK) into three tightly coupled pieces:

GHOST system diagram: multi-robot ROS 2 communication, a GPU-accelerated RGB-D perception pipeline, and a Unity-based VR interface.

How the pieces fit together. Small, latency-tolerant data (robot state, commands) flows through ROS 2; the heavy RGB-D streams go straight to the GPU, where they become the point clouds and live robot models the operator sees in the headset.

Client

Unity VR Client

The world the operator lives in. Renders the point-cloud scene and live robot models, and provides the mode-switching controls — fly the camera, drive a robot, steer an arm, or all of the above.

Server

ROS 2 Server

The coordinator. Streams robot state, executes commands, keeps both robots localized in a common frame, and handles synchronized navigation when they move as a pair.

Perception

SpotObserver

The fast path. A GPU-accelerated library that skips ROS entirely: it pulls RGB-D straight off each robot, runs depth completion in real time, and hands Unity dense point clouds with minimal latency.

The Nine Tasks

We stress-tested GHOST on nine tasks that genuinely need two robots — some demand two arms working together, some an extra viewpoint to see around occlusions, and some careful reasoning about where both robots sit in the room.

Bimanual manipulation Perceptual blind spots Spatial context

BibTeX

Placeholder while the paper is under review — real author list and venue to follow.

@article{ghost2026,
  title   = {GHOST in the Robots: Real-Time Exocentric Dual-Robot
             VR Teleoperation from Onboard Cameras},
  author  = {Anonymous Authors},
  journal = {Under Review},
  year    = {2026}
}