GHOST Setup Instructions

Home

Follow the steps below to set up GHOST for use:

Feel free to reach out to us at brown.vr.teleop@gmail.com with any questions!

You will need a VR-ready Windows machine to run the GHOST Unity scene, a Meta/Oculus VR headset (tested on the Meta Quest Pro with Touch controllers), and an Ubuntu machine to run ROS.
The network pipeline for our system is a bi-directional pipeline:
[Spot] |-Spot API Python gRPC Server-> <-SpotRos Python gRPC Client-| [Ubuntu Machine] |-RosBridge Server-> <-RosConnector-| [Unity - Windows Machine] |-Meta VR SDK-| [Meta Quest Pro]
The Ubuntu machine interacts with Spot using the Boston Dynamics Spot API through gRPC, setting up a ROS environment, and broadcasting that environment to the Windows machine through a RosBridge server. The Windows machine subscribes to image, depth, and transformation topics to construct and update the virtual scene, and publishes control messages passed back along the pipeline to control the robot.
For optimal performance, we recommend using a network switch and Ethernet cables to connect the robot, Ubuntu machine, and Windows machine together in a local network. This is advised because onboard depth compression is currently not supported by the Spot API, which can create a significant network bottleneck. On other customizable robots, this issue can be mitigated by compressing depth data on the robot before transmission.
Ensure the Windows machine can ping the Ubuntu machine and the robot, and that the Ubuntu machine can ping the Windows machine and the robot.

git clone https://github.com/h2r/GHOST

In Unity Hub, click Add, select Add project from disk, and choose the GHOST repo folder. Ensure that the editor version selected is 2022.3.40f1 or later.
Open the GHOST repository in Unity. It will automatically import the required packages and build out the environment.
In the Unity Editor, under Assets, navigate to the Scenes folder and double-click the 'ghost' scene to open it. Confirm that the scene loads correctly by checking that a virtual Spot is present.
Once the scene is loaded, configure the RosConnector to communicate with your robot over RosBridge. In the GameObject hierarchy, expand RosReality, then Spots, then SpotRos1, then RosParent to reveal the RosConnector nodes that handle data streaming between Unity and ROS.
Select the five enabled children of RosParent. Within their RosConnector component, set their Ros Bridge Server URL to the IP address and port of the machine running your ROSBridge Server (the ROSBridge Server allows our Unity scene to interact with our ROS environment). By default, the port is 9090.
Follow any additional configuration instructions specific to the Spot robot as provided in the GHOST documentation.

On the same Windows machine, download the Meta Quest Link App (this app allows you to connect your headset to your computer) from Meta.
Set up and pair your Meta Quest Pro or comparable headset.
Enable developer mode on your headset using the Meta Quest Phone app (instructions can be found here).
Ensure the Meta Quest Link app is running in the background with the headset successfully paired and recognized.
Connect the headset to the computer with a USB-C cable. Optionally, you can use AirLink instead, but at the time of testing, this feature wasn't well developed.
Put on the headset and enable Quest Link. Then, on the Windows machine with Unity, play the 'ghost' Unity Scene. You should be able to view a virtual version of Spot, although the point clouds will not be visible until the ROS setup is finished.

Prerequisites - Docker
On an Ubuntu machine, clone the robotdev repo: robotdev.
Follow the instructions in the repo for setting up a Docker container from the Docker image in the repo.
Once your Docker container is set up, start and enter the container.
Change directories into repo/robotdev.
Edit the setup_spot.bash script with the IP, names, and info for your robot(s).
In the ~/.bashrc file in the Docker container, there are helpful command aliases for starting different processes:

sourcespot <Robot Name>: Sources ROS environment and connects to the specified robot name (this name must be configured in the setup_spot.bash file). This must be done before running any of the below commands.
launchspot: Starts RosMaster thread and ROS environment; this is necessary for all other scripts and should be started first and given time to set up.
launchserver: Starts the rosbridge server.
allcams: Starts rospublisher for image data.
alldepths: Starts rospublisher for depth data.
allexts: Starts rospublisher for camera extrinsics.
These commands are all run automatically as part of our tmux environment, initiated by running dev <Robot Name> (Tmux is a terminal multiplexor that lets us run all of these processes in a single terminal window automatically. More info can be found here).

With all of these processes running in a tmux window, you are now able to return to the Unity scene on the Windows machine.
In the Unity Scene, make sure that the active children of RosParent all have the correct Ros Bridge Server URL, as described in the Unity setup above.
Put on the headset and play the Unity scene, where you will be able to see a virtual Spot with a point cloud representation of its environment. For instructions on how to operate Spot, please refer to the README instructions in the GHOST repository.

GHOST is very flexible and is compatible with any robot that has inverse kinematic end-effector movement and cameras with depth data.

To set up GHOST with a custom robot, the setup steps are largely the same as above, but you will need to do a few additional steps:

Transfer over a URDF of your robot from ROS (follow the instructions for Importing Robot Models).
In the 'ghost' scene with our Spot robot, navigate the GameObject hierarchy to /ghost/RosReality/Spots/SpotRos1/spot/body/, where you will find point cloud objects named FL (front left), FR (front right), etc. These should be copied and placed on your custom robot.
In the robot GameObject hierarchy generated from your URDF, find the GameObject that best represents your robot's camera position and drag in our point cloud GameObject as a child of that GameObject.
On the ROS side, either modify our ROS environment for your custom robot or connect your own ROS setup to GHOST's Unity environment using the ROSBridge Server. If you don't use ROS, you can modify our Unity RosConnector code to accept data in various ways, and with some tweaks, it should still work.