This page describes how to use Ein’s stream buffer facility. When saving data, it is useful to be able to stream sensor information from the robot directly to disk, as well as to load sensor streams from disk into the buffers. For example, after taking a spiral scan, the robot can merge the images into a synthetic photograph, but also saves a buffer of the original calibrated images. This page describes how to interact and use the lower-level stream buffer.
Collecting Data in the Stream Buffer
activateSensorStreaming activates sensor streaming, causing the
in-memory buffers to begin filling up with data.
deactivateSensorStreaming turns off streaming.
There are six types of streams: images, poses, labels (marked labels
in the stream), words (words executed by Ein during the event), joint
angles, and range sensor readings. By default, none of these streams
are turned on, so
activateSensorStreaming will not do anything. You
can toggle them with
setSisFlags or individual getters
streamSis*) and setters (
streamSetSis*). A useful shortcut is
streamEnableSisImageAndPoses to stream images and poses.
To stream data, run:
It may also be useful to run
will turn off rendering and other features, so that Ein can stream
data as quickly as possible, minimizing the number of frames that are
To turn off streaming, run
deactivateSensorStreaming. At this
point, you should probably also run
you have shut things down.
After deactivating streaming,
streamImageBufferSize will contain the
number of images saved, while
streamPoseBufferSize will contain the
number of poses.
clearStreamBuffers empties the stream buffer.
Streaming is cumulative, so activiting and deactivating streaming will
append to the stream buffer.
Exercise: Stream some images and poses.
Activate sensor streaming to populate the image and pose stream
buffers. Inspect the size of the two buffers. Which one is larger?
Why might that be? Now run
clearStreamBuffers and observe that the
sizes are back to zero.
Answer (select to see):
Images come in at the camera framerate, approximately 30 Hz from Baxter’s wrist camera. (Your milage may vary when using a Kinect 2, and depending on how many cameras you are using.) Poses come in at approximately 100Hz. So you will receive many more images in a given time period than poses. This is useful because you can find an accurate pose for each image.
Saving and Loading Data
Stream data is saved in the focused class. Stream data itself
consists of the raw data that is saved about an object, most often
pose-annotated images. To save the stream buffers to disk, use
streamWriteBuffersToDisk. Alternatively, you can
enableDiskStreaming to stream as data is loaded into the buffer.
populateStreamBuffer populates the stream buffer with data saved
from the focused class.
initializeAndFocusOnANewClass creates a new
class for saving stream buffer data.
Incrementing and Viewing Data
You can access the current stream buffer index with
streamImageBufferCurrentIdx and set it with
streamSetImageBufferCurrentIdx. The “Stream Viewer” window displays
the contents of the stream buffer. You can run
streamRenderStreamWindow to render the current contents of the
Exercise: Write a word to play back the stream buffer.
Stream some images and poses. You will want to use
streamImageBufferSize to reproduce the rendering command.
Select to see our solution, or also see the word
( streamIncrementImageStreamBuffer streamRenderStreamWindow 0.01 waitForSeconds ) streamImageBufferSize replicateWord
Rendering Synthetic Photographs
To render synthetic photographs from the stream buffer, see
tableStreamBufferTakeScene. This word will render the stream buffer
into the observed map. Note that in general rendering synthetic
photographs is quite complicated and might involve writing custom
words in C++ for maximum speed. There is a very large number of such
words defined for such tasks as glare removal, 3D reconstruction, and
the like, defined in gaussian_map.cpp. The lower-level word
sceneUpdateObservedFromStreamBufferAtZNoRecalcAll updates the
observed map from the stream buffer, using the entire buffer. It
takes two arguments: the stride, and the height z at which to use.
You can compute this height using the table height or else hard-code
to a particular value (or perform a focus sweep by rendering at a
range of values). The word
the entire stream at the table height.