Intel® RealSense™ Cross Platform API
Intel Realsense Cross-platform API
Frame Management

librealsense2 provides flexible model for frame management and synchronization. The document will overview frame memory management, passing frames between threads and synchronization.

API Overview

The core C++ abstraction when dealing is the rs2::frame class and the rs2::device::start method. All other management and synchronization primitives can be derived from those two APIs.

template<class T>
void start(T callback) const;

Once you call start, the library will start dispatching new frames from selected device into the callback you provided. The callback will be invoked from the same thread handling the low-level IO ensuring minimal latency. Any object implementing void operator()(rs2::frame) can be used as a callback. In particular, you can pass an anonymous function (lambda with capture) as the frame callback:

dev.start([](rs::frame f){
std::cout << "This line be printed every frame!" << std::endl;
});

As a side-note, rs2::device::stop will block until all pending callbacks return. This way within callback scope you can be sure the device object is available.

Frame Memory Management

rs2::frame is a smart reference to the underlying frame - as long as you hold ownership of the rs2::frame the underlying memory is exclusively yours and will not be modified or freed.

Frames and Threads

Callbacks are invoked from an internal thread to minimize latency. If you have a lot of processing to do, or simply want to handle the frame in your main event loop, librealsense provides rs2::frame_queue primitive to move frames from one thread to another in a thread-safe fashion:

dev.start([](rs2::frame f){
q.enqueue(std::move(f)); // enqueue any new frames into q
});
while(true)
{
rs2::frame f = q.wait_for_frame(); // wait until new frame is available and dequeue it
// handle frames in the main event loop
}

Since rs2::frame_queue implements operator() you can also pass the queue directly to start:

dev.start(q);

You could also have a separate queue for each stream type:

dev.start(RS2_STREAM_DEPTH, depth_q);
dev.start(RS2_STREAM_INFRARED, ir_q);

This is particularly handy if you want to set-up different processing pipeline for each stream type.

Frame-Drops vs. Latency

There are two common types of applications of the streaming API:

librealsense provides some degree of control over this trade-off using RS2_OPTION_FRAMES_QUEUE_SIZE option. If you increase this number, your application will consume more memory and some frames might potentially wait in line more time, but frame drops will be less likely to happen. On the flip side, if you decrease this number, you will get frames faster, but if new frame will arrive while you are busy it will get dropped.

Frame Syncer

Often the input to an image processing application is not simply a frame, but rather a coherent set of frames, preferably taken at the same time. librealsense provides rs2::syncer primitive to help with this problem:

auto sync = dev.create_syncer(); // syncronization algorithm can be device specific
dev.start(sync);
while(true)
{
auto frameset = sync.wait_for_frames(); // wait for a coherent set of frames
for (auto&& frame : frameset)
{
// handle frame
}
}