Camera

This example is concerned with the conversion of the pixel type of images received from a simulated camera input. The layout of the stream is as below.

The pixel type converter receives image data from two operators. The first one is the camera which constantly sends images to the converter. The second operator BufferArray owns a predefined number of empty image buffers, i.e. pre-allocated, empty images. If BufferArray hands out one of its images to the next operator the image is removed from the array and the BufferArray loses ownership of the image. However, as soon as no other object references the data container which contains the image, the image data is recycled, i.e. it is given back to the BufferArray operator.

In this example the ConvertPixelType operator converts the camera image and writes it to the image buffer it receives from BufferArray. This buffer is then sent to the output of the conversion operator. In other words, the ConvertPixelType operator does not own or allocate any image memory during processing but uses only these images it receives as input data. If the party which obtains the output from ConvertPixelType releases this data soon enough it can be recycled and used again to store the result of the next camera image. The output image of the camera is handled the same way, i.e. as soon as the edge detector finishes processing the camera image is recycled for the acquisition of further images. As a consequence the presented stream can be operated without allocating or deallocating image data.

Note

The camera has an additional output which contains the index of the current image acquired. If the camera runs out of available image buffers, i.e. if all image buffers are referenced by other objects, the camera can not acquire further images. However, the index is still incremented for each image which should have been acquired. Tracking and comparing the index values allows to find out which frames were lost.

The illustrated stream is configured by the XML file below. The parameter settings of the operators a commented in the file.

<?xml version="1.0" encoding="UTF-8" ?>

<Stromx version="0.1.0">
    <Stream>
        <Operator id="0" package="cv::support" type="DummyCamera" version="0.1.0">
            <!-- output image index -->
            <Parameter id="2">
                <Data package="runtime" type="Bool" version="0.1.0">1</Data>
            </Parameter>
            <!-- image for camera simulator -->
            <Parameter id="3">
                <Data type="Image" package="cv::support" version="0.1.0" file="lenna.jpg"/>
            </Parameter>
            <!-- software trigger mode -->
            <Parameter id="4">
                <Data type="Enum" package="runtime" version="0.1.0">1</Data>
            </Parameter>
            <!-- the buffer size is 1 MB -->
            <Parameter id="8">
                <Data type="UInt32" package="runtime" version="0.1.0">1000000</Data>
            </Parameter>
            <!-- one buffer -->
            <Parameter id="9">
                <Data type="UInt32" package="runtime" version="0.1.0">1</Data>
            </Parameter>
            <!-- 8-bit grayscale output -->
            <Parameter id="17">
                <Data type="Enum" package="runtime" version="0.1.0">1</Data>
            </Parameter>
        </Operator>
        <Operator id="1" package="cv::support" type="Buffer" version="0.1.0">
            <!-- one buffer -->
            <Parameter id="1">
                <Data type="UInt32" package="runtime" version="0.1.0">1</Data>
            </Parameter>
            <!-- the buffer size is 1 MB -->
            <Parameter id="2">
                <Data type="UInt32" package="runtime" version="0.1.0">1000000</Data>
            </Parameter>
        </Operator>
        <Operator id="2" package="cv::support" type="ConvertPixelType" version="0.1.0">
            <!-- write result to destination buffer -->
            <Parameter id="3">
                <Data type="Enum" package="runtime" version="0.1.0">0</Data>
            </Parameter>
            <!-- convert to RGB 24-bit image -->
            <Parameter id="4">
                <Data type="Enum" package="runtime" version="0.1.0">3</Data>
            </Parameter>
            <Input id="0" operator="0" output="0"/>
            <Input id="1" operator="1" output="0"/>
        </Operator>
        <Thread>
            <InputConnector operator="2" input="0"/>
            <InputConnector operator="2" input="1"/>
        </Thread> 
    </Stream>
</Stromx>

The program which operates the stream is straightforward. As in the previous examples the program starts by importing all required headers and by registering the runtime and cvsupport libraries.

#include <stromx/runtime/Factory.h>
#include <stromx/runtime/XmlReader.h>
#include <stromx/runtime/Stream.h>
#include <stromx/runtime/Runtime.h>
#include <stromx/runtime/Operator.h>
#include <stromx/runtime/ReadAccess.h>
#include <stromx/runtime/Image.h>

#include <stromx/cvsupport/Cvsupport.h>

#include <iostream>

using namespace stromx;

int main (int, char**)
{
    runtime::Factory factory;
    
    stromxRegisterRuntime(&factory);
    stromxRegisterCvsupport(&factory);

Next the stream is read from the XML file and started.

    runtime::Stream* stream = runtime::XmlReader().readStream("camera.xml", &factory);
    
    stream->start();
    
    runtime::Operator* camera = stream->operators()[0];
    runtime::Operator* convertPixelType = stream->operators()[2];
    
    for(unsigned int i = 0; i < 5; ++i)
    {
        runtime::DataContainer data = convertPixelType->getOutputData(2);
        runtime::ReadAccess image(data);
        std::cout << "Received image " 
                  << image.get<runtime::Image>().height() << "x"
                  << image.get<runtime::Image>().width() << ", "
                  << image.get().variant().title() << std::endl;
        
        convertPixelType->clearOutputData(2);
        camera->clearOutputData(1);
    }

Before starting the iteration explicit references to the DummyCamera and ConvertPixelType operators are acquired. In each step the output image of the edge detector is obtained and its dimension are written to the command line. After that the output of the edge detector and the index output of the camera are cleared. It is important to clear all outputs which are not connected to other operators. Not doing so would eventually stop stream because no further data can be send to the outputs.

    stream->stop();
    stream->join();
    
    delete stream;
}

Finally, the stream is stopped and deleted.

Note

The class runtime::Image does not support saving of images to file because runtime is not linked to any 3rd party image processing library. However, cvsupport::Image can write image files using OpenCV. Thus, the images in the code above can be written by constructing cvsupport::Image objects from the more general runtime::Image data:

#include <cvsupport/Image.h>
...
{
    ...
    for(unsigned int i = 0; i < 5; ++i)
    {
        ...
        cvsupport::Image cvsupportImage(image())
        cvsupportImage.save("image.png");
        ...
    }
    ...
}