GHOST in the Robots

Abstract

Teleoperating multiple robots simultaneously enables additional views and coordinated control. Yet, it poses fundamental challenges: the system must present sensor data cohesively and allow operators to manage multiple robot bases, arms, and cameras while maintaining low latency. Current multi-robot teleoperation systems require multiple operators, rely on autonomous coordination, or restrict operators to high-level commands.

We present GHOST: an open-source VR teleoperation system that enables single-operator control of two mobile manipulators via direct low-level commands using only onboard cameras. GHOST creates an exocentric 3D workspace by integrating real-time point clouds from both robots' body cameras, where point coverage is improved through learning-based completion to aid operator scene understanding. For control, the operator uses a mode-switching control architecture — either for individual robots or to coordinate both robots simultaneously.

Experiments with 15 novice operators demonstrate 1.6–4× the success rate of an off-the-shelf interface. For experts across nine challenging dual-robot tasks, our system enabled task completion on two tasks that were impossible on an off-the-shelf tablet interface, and experts were 1.47× faster on average compared to the tablet.

Contributions

An exocentric 3D scene for multi-robot teleoperation.
A low-latency depth-completion pipeline.
A principled control design space for dual-robot teleoperation.
A set of challenging evaluation tasks for dual-robot teleoperation.

System Overview

A single operator teleoperates two Boston Dynamics Spot mobile manipulators within a unified VR framework. Each Spot has a quadruped base, a 6-DoF arm with gripper, a LiDAR unit, and two front-facing RGB-D sensors. The robots communicate over onboard Wi-Fi with a ROS 2 server and a VR client. GHOST integrates off-the-shelf tools (PromptDA, ROS2, Unity, the Spot SDK) into three tightly coupled components:

GHOST system diagram: multi-robot ROS 2 communication, a GPU-accelerated RGB-D perception pipeline, and a Unity-based VR interface.

The GHOST system. Low-bandwidth state/command data is bridged through ROS 2, while high-bandwidth RGB-D streams are processed on the GPU and rendered as interactive point clouds and virtual robot models for low-latency teleoperation.

Client

Unity VR Client

Renders the 3D scene as dynamic point clouds and live robot models, and gives the operator mode-switching control for independent or coordinated operation.

Server

ROS2 Server

Manages low-bandwidth state streaming, command execution, multi-robot localization, and synchronized navigation between the two robots.

Perception

SpotObserver

A GPU-accelerated library that bypasses ROS messaging to pull RGB-D directly from each robot, runs real-time depth completion, and produces dense point clouds with minimal latency for Unity to render.

Evaluation Tasks

We evaluate GHOST on nine dual-robot tasks spanning perceptual blind spots, bimanual manipulation, and spatial context awareness. All require coordinated control of two mobile manipulators.

Bimanual manipulation Perceptual blind spots Spatial context