Pipeline
Pipeline API allows you to use NNStreamer machine learning inference pipelines in your applications. You can use them to process data with machine learning models and custom callbacks.
Main features of Pipeline API
The main features of the Pipeline API include:
-
Create and dispose machine learning inference pipelines
You can set up pipelines and dispose them when they are no longer needed.
-
Observe pipeline state and respond to its changes
You can register state change listeners or poll the state when needed.
-
Run machine learning inference pipelines
-
Get and set pipeline node properties
-
Input data from the application
-
Read data from pipeline output
-
Change data flow within the pipeline
You can use Switch API to choose the pipeline branch that should receive data.
-
Start and stop data flow to a pipeline branch
You can use Valve API to stop and resume data flow within a pipeline branch.
-
Write custom data filters
You can use CustomFilter API to write custom data processing routines in JS.
-
Use saved models
You can use
tensor_filter
element to read models trained with popular machine learning frameworks.
Prerequisites
To access files, camera or recorder using the Pipeline API (in mobile, wearable, and TV applications), the application has to define proper privileges in its config.xml
:
<!-- for accessing internal storage only -->
<tizen:privilege name="http://tizen.org/privilege/mediastorage"/>
<!-- for accessing external storage only -->
<tizen:privilege name="http://tizen.org/privilege/externalstorage"/>
<!-- for accessing camera -->
<tizen:privilege name="http://tizen.org/privilege/camera"/>
<!-- for accessing recorder -->
<tizen:privilege name="http://tizen.org/privilege/recorder"/>
As these are privacy-related privileges, the application has to request proper permissions using the PPM API (in mobile and wearable applications).
Create and dispose machine learning inference pipelines
NNStreamer is a plugin for GStreamer. It adds new pipeline nodes and data types that allow to run on-device machine learning inference. You can use some of the standard GStreamer and NNStreamer elements (also referred to as nodes throughout this guide) in your applications.
-
NNStreamer pipelines are created from string descriptions, for example:
'appsrc ! other/tensor,dimension=(string)2:1:1:1,type=(string)int8,framerate=(fraction)0/1 ! tensor_filter framework=custom-easy model=my-custom-filter ! tensor_sink'
-
Exclamation marks
!
separate pipeline nodes.
You can learn how to define pipeline descriptions from GStreamer documentation.
To create a machine learning inference pipeline:
-
Describe the pipeline as a string:
var pipelineDescription = 'videotestsrc num-buffers=3 ' + '! video/x-raw,width=20,height=15,format=BGRA ' + '! tensor_converter ' + '! fakesink';
-
Call
tizen.ml.pipeline.createPipeline()
:var pipeline = tizen.ml.pipeline.createPipeline(pipelineDescription);
You can use pipeline
object to manage the pipeline. Machine learning pipelines can be expensive in terms of used system resources. You can dispose pipelines to reclaim resources.
-
To dispose a pipeline, call
dispose()
:pipeline.dispose();
The disposed pipelines enter the NULL
state, and cannot be restarted. Any attempt of calling any method of a disposed pipeline results in NotFoundError
exception.
Observe pipeline state and respond to its changes
Pipeline state reflects what the pipeline is doing right now and defines which methods can be called. To see how pipelines transition between different states, see the API reference (for mobile, wearable, and TV applications).
Pipeline API allows you to register listeners triggered by pipeline state changes and to poll current pipeline state.
To register a state change listener:
-
Define a
PipelineStateChangeListener
, which is called when the pipeline changes its state:function pipelineStateChangeListener(newState) { console.log('New pipeline state: ' + newState); }
-
Call
tizen.ml.pipeline.createPipeline()
, with state change listener as an argument:var pipeline = tizen.ml.pipeline.createPipeline(pipelineDescription, pipelineStateChangeListener);
-
You can also check the current pipeline state by reading the value of its
state
property:console.log(pipeline.state); // 'PLAYING'
Run machine learning inference pipelines
The newly created pipeline transitions through READY
to PAUSED
state, you have to manually start it and set it to PLAYING
state.
-
To start the data flow within a pipeline, call
start()
:pipeline.start();
-
To stop inference, call pipeline’s
stop()
:pipeline.stop();
When pipeline stops, it changes its state to PAUSED
.
Get and set pipeline node properties
Operation of pipeline elements is controlled by properties, which can be read and written with Pipeline API.
You can get the information about nodes using gst-inspect-1.0
command line tool on your Tizen device, for example:
gst-inspect-1.0 videotestsrc
...
Element Properties:
animation-mode : For pattern=ball, which counter defines the position of the ball.
flags: readable, writable
Enum "GstVideoTestSrcAnimationMode" Default: 0, "frames"
(0): frames - frame count
(1): wall-time - wall clock time
(2): running-time - running time
...
You can use the following information to change the pattern
of the test video signal source to ball
:
...
pattern : Type of test pattern to generate
flags: readable, writable
Enum "GstVideoTestSrcPattern" Default: 0, "smpte"
(0): smpte - SMPTE 100% color bars
(1): snow - Random (television snow)
...
(18): ball - Moving ball
...
To control node property with application code:
-
Create a pipeline and define the names for those elements for which you want to get or set a property:
var pipelineDescription = 'videotestsrc name=srcx ! tizenwlsink'; var pipeline = tizen.ml.pipeline.createPipeline(pipelineDescription);
-
Get the
NodeInfo
object associated with the node you want to control:var videotestsrcNode = pipeline.getNodeInfo('srcx');
-
To read the current value of
pattern
property, use the property type defined ingst-inspect-1.0
output:var pattern = videotestsrcNode.getProperty('pattern', 'ENUM'); console.log(videotestsrcNode.name + '\'s pattern: ' + pattern); // 'srcx's pattern: 0';
-
The current
pattern
is0
, which translates toframes
pattern, according togst-inspect-1.0
output. Change it toball
by setting property value to18
:videotestsrcNode.setProperty('pattern', 'ENUM', 18);
Note
You can also set the
pattern
value in pipeline description, for example,videotestsrc pattern=18 ! tizenwlsink
Input data from application
The pipeline input can be generated by specialized nodes, for example videotestsrc
or by the application code feeding the appsrc
nodes.
To input data from application:
-
Create a pipeline with
appsrc
node, define properties of the input tensor in pipeline description, using the GStreamer capsfilter:var pipelineDescription = 'appsrc name=srcx ' + '! other/tensor,dimension=(string)1:1:1:1,type=(string)int8 ' + '! fakesink'; var pipeline = tizen.ml.pipeline.createPipeline(pipelineDescription);
-
Get the
srcx
element from pipeline:var src = pipeline.getSource('srcx');
-
Start the pipeline before providing input:
pipeline.start();
-
Input data is processed by the pipeline. The
Source.inputTensorsInfo
property has theTensorsInfo
expected at input:var inputInfo = src.inputTensorsInfo; var inputData = inputInfo.getTensorsData(); var rawInputData = inputData.getTensorRawData(0); inputData.setTensorRawData(0, [123]); src.inputData(inputData);
Input data is passed to the further stages of the pipeline.
Read data from pipeline output
You can read the tensors output by the pipeline with the application code by registering a callback, triggered by data coming into appsink
or tensor_sink
.
Use gst-inspect-1.0
to learn more about the differences between these two elements.
To get pipeline output:
-
Create a pipeline with a sink node that can pass data to application:
var pipelineDescription = 'videotestsrc num-buffers=3 ' + '! videoconvert ' + '! videoscale ' + '! video/x-raw,format=RGBx,width=16,height=16,framerate=10/1 ' + '! tensor_converter ' + '! tensor_sink name=sinkx'; var pipeline = tizen.ml.pipeline.createPipeline(pipelineDescription);
-
Register a listener triggered by data incoming to
sinkx
:function sinkListener(sinkName, data) { console.log('SinkListener for ' + sinkName + ' sink called. Data dimensions: ' + data.tensorsInfo.getDimensions(0)); // Read and process data according to your needs } pipeline.registerSinkListener('sinkx', sinkListener);
After starting the pipeline, sinkListener
is to be called repeatedly.
Change data flow within pipeline
You may want to set the the source of the data to one of several pipeline branches.
It can be done with input-selector
node and Switch
API.
You can analyze the pipeline from the following figure to see how to use input-selector
to choose between blue and green branch as the source for the white branch:
To choose a source branch with input-selector
:
-
Create a pipeline:
var pipelineDescription = 'input-selector name=ins ! tensor_converter ! tensor_sink name=sinkx ' + // white branch 'videotestsrc is-live=true ! videoconvert ! ins.sink_0 ' + // blue branch 'videotestsrc num-buffers=3 is-live=true ! videoconvert ! ins.sink_1'; // green branch var pipeline = tizen.ml.pipeline.createPipeline(pipelineDescription);
-
Get
insSwitch
:var insSwitch = pipeline.getSwitch('ins');
-
You can get
Switch
pads:console.log(insSwitch.getPadList()); // ["sink_0", "sink_1"]
-
Choose the pad to be used as a source for
input-selector
and the white pipeline branch:insSwitch.select('sink_1'); // green branch used as the source
In the preceding example, an input-selector
is used to choose from alternative sources. Similarly, you can use an output-selector
to direct data to one of a set of sinks.
Start and stop data flow to a pipeline branch
Use valve
element with Valve
API to start and stop data flow in one of a pipeline branches.
To control data flow in a single pipeline branch:
-
Create and start a pipeline with a
valve
element:var pipelineDescription = 'videotestsrc is-live=true ' + '! videoconvert ' + '! videoscale ' + '! video/x-raw,format=RGBx,width=16,height=16,framerate=10/1 ' + '! tensor_converter ' + '! valve name=valve1 ' + '! fakesink'; var pipeline = tizen.ml.pipeline.createPipeline(pipelineDescription); pipeline.start();
-
Get
Valve
element from the pipeline:var valve = pipeline.getValve('valve1');
-
You can check if the valve is opened or closed:
console.log(valve.isOpen); // true
-
Set the desired valve state:
valve.setOpen(false); // stop the data flow
Write custom data filters
The data flowing through the pipeline can be transformed in application using CustomFilter
callbacks.
NoteSending data between
CustomFilter
running in JS application and GStreamer pipeline has large overhead. UsingCustomFilter
with large tensors may result in unsatisfactory performance.
To transform the data within a callback registered in JS application:
-
Define information about
CustomFilter
's input and output tensors:var inputInfo = new tizen.ml.TensorsInfo(); inputInfo.addTensorInfo('3D', 'UINT8', [4, 20, 15, 1]); var outputInfo = new tizen.ml.TensorsInfo(); outputInfo.addTensorInfo('flat', 'UINT8', [1200]);
-
Register
CustomFilter
in the pipeline. Ensure to return a proper status code from the callback. In this case,0
is returned, that indicates success:function flattenFilter(inputData, outputData) { var rawInputData = inputData.getTensorRawData(0); outputData.setTensorRawData(0, rawInputData.data); return 0; }; tizen.ml.pipeline.registerCustomFilter('flattenFilter', flattenFilter, inputInfo, outputInfo);
Note
inputData
andoutputData
passed toflattenFilter
are different than otherTensorsData
objects. These objects cannot be disposed and are only valid within the callback. You have to copy the objects manually such that you can use these objects outside the callback.inputData
is read-only andoutputData
is initialized with random values. -
Create a pipeline with a
custom-easy-filter
element:var pipelineDescription = 'videotestsrc num-buffers=3 ' + '! video/x-raw,width=20,height=15,format=BGRA ' + '! tensor_converter ' + '! tensor_filter framework=custom-easy model=flattenFilter ' + '! fakesink'; var pipeline = tizen.ml.pipeline.createPipeline(pipelineDescription);
When you start the pipeline, the flattenFilter
transforms 3-dimensional tensors into 1-dimensional vector.
Use saved models
Using machine learning models trained with popular frameworks like TensorFlow is one of the main use cases for NNStreamer pipelines.
To read a model from file and use it in a pipeline:
-
You must know the absolute path to the saved model file. It can be obtained from a path relative to a virtual root:
var modelPath = 'documents/mobilenet_v1_1.0_224_quant.tflite'; var URI_PREFIX = 'file://'; var absoluteModelPath = tizen.filesystem.toURI(modelPath).substr(URI_PREFIX.length);
-
Create a pipeline with a
tensor_filter
node withmodel
property set to the location of a model:var pipelineDescription = 'appsrc name=srcx ' + '! other/tensor,dimension=(string)224:224:3:1,type=(string)float ' + '! tensor_filter framework=tflite model=' + absoluteModelPath + ' ' + '! appsink name=sinkx'; var pipeline = tizen.ml.pipeline.createPipeline(pipelineDescription);
-
Set up other pipeline elements, and use custom
Source
andSink
:var source = pipeline.getSource('srcx'); function sinkListener(sinkName, data) { var rawData = data.getTensorRawData(0); console.log(sinkName + ' received the data:'); console.log(rawData); }; pipeline.registerSinkListener('sinkx', sinkListener);
-
Run the pipeline:
pipeline.start(); var inputTensorsData = inputTensorsInfo.getTensorsData(0); var randomInput = []; for (var i = 0; i < 224 * 224 * 3; i++) { randomInput.push(Math.random()); }; inputTensorsData.setTensorRawData(0, randomInput); source.inputData(inputTensorsData);
Related information
- Dependencies
- Tizen 6.5 and Higher for Mobile
- Tizen 6.5 and Higher for Wearable
- Tizen 6.5 and Higher for TV