Machine Learning Training

Machine Learning Training API allows you to construct, control, and train a machine learning model in Tizen devices.

The main features of Machine Learning Training API include:

• Constructing a deep neural network (DNN)
  • You can construct a DNN model using a model description file or by writing code through Machine Learning Training API.
• Training with your own data
  • Machine Learning Training API also allows you to train the model with your own data as a File I/O or by defining a data generator.
• Evaluating the model during training
  • You can validate and test your model during the training process easily by defining the dataset.

Note

Every example code does not handle all error use cases. The error must be handled more extensively compared to the example code written on this page.

Prerequisites

To enable your application to use Machine Learning Training API, go through the following steps:

1. To use the functions and data types of the Machine Learning Inference API, include the <nntrainer.h> header file in your application:

   #include <nntrainer.h>
   

2. To use the Machine Learning Training API, include the following features in your tizen-manifest.xml file:

   <feature name="http://tizen.org/feature/machine_learning">true</feature>
   <feature name="http://tizen.org/feature/machine_learning.training">true</feature>
   

   In case of saving or loading model files from the outside of the application’s own resources, the application has to request permission by adding the following privileges to the tizen-manifest.xml file:

   <privileges>
     <!-- For accessing media storage -->
     <privilege>http://tizen.org/privilege/mediastorage</privilege>
     <!-- For accessing external storage -->
     <privilege>http://tizen.org/privilege/externalstorage</privilege>
   </privileges>
   

Building blocks

Following are the four major components of Machine Learning Training API:

• Model
• Layer
• Optimizer
• Learning rate scheduler
• Dataset

Model

Model is a wrapper component that has the topology of layers, optimizers, and datasets. The model performs training and saves the updated parameters that can later be used for inference. In the following figure, data represents input data or feature and label is the actual value to be tested over prediction:

Note

Until Tizen 6.0, only sequential neural network is supported, and since Tizen 6.5, neural network with directed acyclic graph structure is also supported.

// Create model
ml_train_model_h model;
ml_train_model_construct(&model);

/* Configure model(omitted for brevity) */

// Compile model - This freezes the model. Afterwards the model cannot be modified.
ml_train_model_compile(model, "loss=cross", "batch_size=16", NULL);
// Run model
ml_train_model_run(model, "epochs=2", NULL);

// Save and load below format supported since Tizen 6.5:
ml_train_model_save(model, "model_weights.bin", ML_TRAIN_MODEL_FORMAT_BIN);
ml_train_model_save(model, "model.ini", ML_TRAIN_MODEL_FORMAT_INI);
ml_train_model_save(model, "model_and_weights.ini", ML_TRAIN_MODEL_FORMAT_INI_WITH_BIN);

// Destroy after use.
ml_train_model_destroy(model);

A number of properties can be set at ml_train_model_compile() and ml_train_model_run() phase:

Layer

Layer is a component that does actual computation while managing internal trainable parameters. The following example shows how to create, add and get layer to the model:

// Create layer
ml_train_layer_h layer;
ml_train_layer_h get_layer;
ml_train_layer_create(&layer, ML_TRAIN_LAYER_TYPE_FC);

// Configure layer
ml_train_layer_set_property(layer, "unit=10", "activation=softmax", "bias_initializer=zeros", "name=fc100", NULL);

// After adding the layer to model, you do not need to destroy layer since ownership is transferred to the model.
ml_train_model_add_layer(model, layer);

// Get layer from the model with the given name.
// The returned layer must not be deleted as it is owned by the model.
ml_train_model_get_layer(model, "fc100", &get_layer);

There are two types of layers. One type includes commonly trainable weights and the other type does not include them. The following are the available properties for each layer type which include commonly trainable weights:



The following are the available properties for each layer type which do not include (weight_initializer, bias_initializer, weight_regularizer, weight_regularizer_constant) properties:

Optimizer

Optimizer determines how to update model parameters according to loss from prediction. Currently, Stochastic Gradient Descent optimizer and Adam optimizer are supported:

// Create an optimizer
ml_train_optimizer_h optimizer;
ml_train_optimizer_create(&optimizer, ML_TRAIN_OPTIMIZER_TYPE_SGD);

// Configure an optimizer
ml_train_optimizer_set_property(optimizer, "learning_rate=0.001", NULL);

// Set optimizer to the model
// No need to destroy optimizer after setting optimizer since the ownership is transferred to the model.
ml_train_model_set_optimizer(model, optimizer);

Following are the available properties for each optimizer type:

Learning rate scheduler

Learning rate scheduler determines how to adjust the learning rate. Currently constant, exponential, and step type is supported:

// Create learning rate scheduler
ml_train_lr_scheduler_h scheduler;
ml_train_lr_scheduler_create(&scheduler, ML_TRAIN_LR_SCHEDULER_TYPE_CONSTANT);

// Configure learning rate scheduler
ml_train_lr_scheduler_set_property(scheduler, "learning_rate=0.001", NULL);

// Set learning rate scheduler to optimizer
ml_train_optimizer_h optimizer;
ml_train_optimizer_set_lr_scheduler(optimizer, scheduler);

// Set optimizer to the model
ml_train_model_set_optimizer(model, optimizer);

Following are the available properties for each learning rate scheduler type:

Dataset

Dataset is in charge of feeding data into the model. The dataset can either be created from a callback function or created from a file. For more information, see configure the model section.

The following code is an example of handling dataset:

// Create dataset
ml_train_dataset_h dataset;
ml_train_dataset_construct(&dataset);
ml_train_dataset_add_file(&dataset, train_path, ML_TRAIN_DATASET_MODE_TRAIN);
ml_train_dataset_add_generator(&dataset, get_sample, (void *)val_user_data, ML_TRAIN_DATASET_MODE_VALID);
ml_train_dataset_add_generator(&dataset, get_sample, (void *)test_user_data, ML_TRAIN_DATASET_MODE_TEST);

// configure dataset
ml_train_dataset_set_property_for_mode(dataset, ML_TRAIN_DATASET_MODE_TRAIN, "buffer_size=100", NULL);

// after setting a dataset to model,
// you do not need to destroy dataset since ownership is transferred to the model.
ml_train_model_set_dataset(model, dataset);

Construct a model

A model can be constructed with ml_train_model_construct(). If you have a file that describes the model, the file can be used to construct initially with ml_train_model_construct_with_file(). Even if the model is constructed from a file, switching, modifying, or setting a component is possible until you compile with ml_train_model_compile().

Construct a model from a description file

As of now, only INI formatted files *.ini are supported to construct a model from a file.

Create a model from INI formatted file

Special sections [Model], [Optimizers], [train_set], [valid_set], [test_set], [LearningRateScheduler] are respectively referring to model, optimizer and data provider objects. Rest of INI sections map to a layer. Keys and values from each section set properties of the layer. All keys and values are treated as case-insensitive.

Following is an example of the *.ini file:

[Model] # Special section that describes model itself
Type = NeuralNetwork  # Model type : only NeuralNetwork is supported as of now
batch_size = 9

####  Model run related properties
Epochs = 20     # Epochs

[Optimizer]
Type = adam  # Optimizer : Adaptive Moment Estimation(adam)
beta1 = 0.9     # beta 1 for adam
beta2 = 0.9999  # beta 2 for adam
epsilon = 1e-7  # epsilon for adam

[LearningRateScheduler]
type = constant
Learning_rate = 1e-4

[train_set]
Type = file
BufferSize = 9
path = "trainingSet.dat"

[valid_set]
Type = file
path = "valSet.dat"

# Add [test_set] section if applicable

# Layer Sections, each section name refers to name of the layer
[inputlayer]
Type = input
Input_Shape = 1:1:62720 # Input dimension in channel:height:width
Normalization = true

[outputlayer]
Type = fully_connected
Unit = 2    # Width of output dimension
bias_initializer = zeros
weight_initializer = xavier_uniform
Activation = sigmoid  # activation : sigmoid, softmax
weight_regularizer = l2norm
weight_regularizer_constant = 0.005
input_layers = inputlayer

[loss]
Type = cross # Define loss as a layer

The following restrictions must be adhered to:

• Model file must have a [Model] section.
• Model file must have at least one layer.
• Valid keys must have valid properties. The invalid keys in each section result in an error.

Note

All paths inside the INI file are relative to the INI file path unless the absolute path is stated. Consider setting save_path, train_set, valid_set, and test_set from the code rather than describing inside the model file to avoid this behavior.

Following example constructs model from INI file:

char *res_path = app_get_resource_path();
char model_path[1024];
ml_train_model_h model;

snprintf(model_path, sizeof(model_path), "%s/model.ini", res_path);
free(res_path);

status = ml_train_model_construct_with_conf(model_path, &model);
if(status != ML_ERROR_NONE) {
  // handle error
}

Construct a model on code

An empty model can be constructed with ml_train_model_construct().

Configure the model

After constructing a model, the model can be configured.

Note

Example code written here reproduces the model description from Create Model from INI Formatted File except that following is different:

• Relative path is changed to dynamic app resource and data path.
• Model related properties that can only be set at compile or run phase.
• Demonstration about ml_train_dataset_add_generator(), which cannot be covered in the description file.

First, an empty model needs to be created:

ml_train_model_h model;
ml_train_model_construct(&model);

Add a layer

ml_train_model_add_layer() appends a layer to the end of the graph in the model:

int status = ML_ERROR_NONE;
ml_train_layer_h layers[2];

// create and add input layer
status = ml_train_layer_create(&layers[0], ML_TRAIN_LAYER_TYPE_INPUT);
if(status != ML_ERROR_NONE) {
  // handle error
}

status = ml_train_layer_set_property(layers[0], "name=inputlayer",
                                                "input_shape=1:1:62720",
                                                "normalization=true", NULL);
if(status != ML_ERROR_NONE) {
  //handle error
}
status = ml_train_model_add_layer(model, layers[0]);

// create and add fully connected layer
status = ml_train_layer_create(&layers[1], ML_TRAIN_LAYER_TYPE_FC);
status = ml_train_layer_set_property(layers[1], "name=outputlayer",
                                                "unit=2",
                                                "bias_initializer=zeros",
                                                "weight_initializer=xavier_uniform",
                                                "activation=sigmoid", NULL);
status = ml_train_model_add_layer(model, layers[1]);

Get a layer

ml_train_model_get_layer() gets a layer from the model with the given name:

int status = ML_ERROR_NONE;

ml_train_model_h model;
ml_train_layer_h add_layer;
ml_train_layer_h get_layer;

// constructs the neural network model
status = ml_train_model_construct(&model);
if (status != ML_ERROR_NONE) {
  // handle error
}

// creates and add a neural network layer
status = ml_train_layer_create(&add_layer, ML_TRAIN_LAYER_TYPE_INPUT);
if (status != ML_ERROR_NONE) {
  // handle error
}

// set a name for the layer
status = ml_train_layer_set_property(add_layer, "name=inputlayer", NULL);
if (status != ML_ERROR_NONE) {
  // handle error
}

// adds layer in neural network model
status = ml_train_model_add_layer(model, add_layer);
if (status != ML_ERROR_NONE) {
  // handle error
}

// gets neural network layer from the model with the given name
status = ml_train_model_get_layer(model, "inputlayer", &get_layer);
// get_layer owned by model, Do not delet it
if (status != ML_ERROR_NONE) {
  // handle error
}

Set an optimizer

Creating and setting optimizer to a model can be done in the same manner as layer:

status = ml_train_optimizer_create(&optimizer, ML_TRAIN_OPTIMIZER_TYPE_ADAM);
status = ml_train_optimizer_set_property(optimizer, "beta1=0.002",
                                                    "beta2=0.001",
                                                    "epsilon=1e-7", NULL);
status = ml_train_model_set_optimizer(model, optimizer);

Set a learning rate scheduler

Creating and setting LearningRateScheduler to a optimizer can be done in the same manner as optimizer:

status = ml_train_lr_scheduler_create(&scheduler, ML_TRAIN_LR_SCHEDULER_TYPE_CONSTANT);
status = ml_train_lr_scheduler_set_property(scheduler, "learning_rate=0.001", NULL);
status = ml_train_optimizer_set_lr_scheduler(optimizer, scheduler);

Set a dataset

There are two ways to create a dataset. One is from a file, and the other one is from a callback. In either case, you need to provide streams of tensor data and arrays of values representing the label, usually one-hot-encoded.

Sample, batch and epoch

This section explains the following three concepts: sample, batch and epoch.

Let’s assume a given model requires three inputs and two labels. This has been reflected in the examples below:

1. Sample

   A sample denotes a single pair of inputs/labels. The example can be as follows:

   [input_1][input_2][input_3][label_1][label_2]
   

2. Batch

   A batch is a bundle of samples which is fed to a single iteration. If a batch is of size 5, the batch layout, where /* denotes nth sample will be as follows:

   [[input_1/1][input_1/2][input_1/3][input_1/4][input_1/5]],
   [[input_2/1][input_2/2][input_2/3][input_2/4][input_2/5]],
   [[input_3/1][input_3/2][input_3/3][input_3/4][input_3/5]],
   [[label_1/1][label_1/2][label_1/3][label_1/4][label_1/5]],
   [[label_2/1][label_2/2][label_2/3][label_2/4][label_2/5]],
   

3. Epoch

   An epoch refers to a full, exhaustive visit to a dataset.

   So, if you consider Cifar-100 dataset when using the full training set, a single training epoch will contain 50,000 samples. If batch size is 100, it will be of 500 batches (or iterations).

Construct a dataset on code

An empty dataset handle can be constructed with ml_train_model_create().

The handle abstracts data, providing logics with three dataset usage point: ML_TRAIN_DATASET_MODE_TRAIN, ML_TRAIN_DATASET_MODE_VALID, ML_TRAIN_DATASET_MODE_TEST

A data provider added with ML_TRAIN_DATASET_MODE_TRAIN, ML_TRAIN_DATASET_MODE_VALID, ML_TRAIN_DATASET_MODE_TEST will be used when training, validating, training respectively.

Consider setting property with ml_train_model_set_property_for_mode() when data provider needs certain properties.

Set a dataset from file

To create a dataset from files, each file must contain an array of samples. A single sample consists of an array of raw float array for input and another array or raw float array for labels that contains the same size as model output:

# If a model requires two inputs and a single label for a sample, a single sample would contain...
[[float array for input1][input 2][float array for label1]],
...,
[[float array for input1][input 2][float array for label1]]

As an example, again, consider Cifar-100 dataset. In Cifar-100 data, it contains an image of size 3 * 32 * 32 and a coarse label of 20 classes and a fine label of 100 classes.

If your model requires an image and two labels (coarse, fine) both one-hot encoded for a sample, the file layout would be:

[[3072 pixels][20 floats for coarse label][100 floats for fine label]],
...
...
[[3072 pixels][20 floats for coarse label][100 floats for fine label]]

After preparing the file, create dataset as follows:

int status = ML_ERROR_NONE;
ml_train_dataset_h dataset;
ml_train_dataset_create(&dataset)

char *res_path = app_get_resource_path();
char train_path[1024];

snprintf(train_path, sizeof(train_path), "%s/training.dat", res_path);
free(res_path);

// a file data provider to be used for training
status = ml_train_dataset_add_file(&dataset, train_path, ML_TRAIN_DATASET_MODE_TRAIN);
if(status != ML_ERROR_NONE) {
  // handle error
}

status = ml_train_dataset_set_property_for_mode(dataset, ML_TRAIN_DATASET_MODE_TRAIN, "buffer_size=9", NULL);
status = ml_train_model_set_dataset(model, dataset);

The property buffer_size defines the maximum amount of batches to be queued while training. When it is not given, it is set to 1.

Set a dataset from a generator

Creating a data provider from a generator function and setting it to dataset is also possible.

1. Prepare a callback function:

   /**
   * @brief      fill a single sample
   * @see ml_train_datagen_cb for details
   * @param[in/out] array of allocated input buffers ready to be filled
   * @param[in/out] array of allocated label buffers ready to be filled
   * @param[in/out] last if the data is finished
   * @param[in] user_data private data for the callback
   * @retval status for handling error while training, returning non-zero will cause to abort data fetching.
   */
   int get_train_data(float **inputs, float **labels, bool *last, void *user_data) {
     /* code that fills data, label and last */
     return ML_ERROR_NONE;
   }
   

2. Create a dataset from the callback function:

   int status = ML_ERROR_NONE;
   ml_train_dataset_h dataset;
   ml_train_dataset_create(&dataset)
   
   /// assuming user_data is retrived from `get_user_data()`
   custom_type * user_data = get_user_data();
   
   // a generator based provider to be used for training
   status = ml_train_dataset_add_generator(&dataset, get_train_data, (void *)user_data, ML_TRAIN_DATASET_MODE_TRAIN);
   if(status != ML_ERROR_NONE) {
     // handle error
   }
   
   status = ml_train_dataset_set_property_for_mode(dataset, "buffer_size=9", NULL);
   status = ml_train_model_set_dataset(model, dataset);
   

The property buffer_size defines the maximum amount of batches to be queued while training. When it is not given, it is set to 1.

Compile the model

Compiling a model finalizes the model with loss. Once compiled, any modification to the properties of the model is restricted. Adding layers or changing the optimizer or dataset of the model is not permitted either:

int status = ML_ERROR_NONE;

status = ml_train_model_compile(model, "loss=cross", NULL);

Train the model

Now, the model is ready to train. Train model as follows:

int status = ML_ERROR_NONE;

status = ml_train_model_run(model, "batch_size=9", "epochs=20", NULL);

Destroy the model

After training, the model must be destroyed with ml_train_model_destroy(). ml_train_model_add_layer(), ml_train_set_optimizer(), and ml_train_set_dataset() transfers ownership to the model. layers, optimizers and dataset that belongs to the model are also deleted.

Use the trained model for inference

Note

This feature is supported since Tizen 6.5 only.

The trained model can be used for inference with Machine Learning Inference API. Ensure that the INI file contains the correct weight file in save_path in [Model] section. The valid INI file can be made from ml_train_model_h with ml_train_model_save() if you have constructed the model with the provided API. For example, you can use the trained model with a single API as follows:

  #include <nnstreamer-single.h>

  ml_single_h single;
  ml_tensors_info_h in_info, out_info;

  ...

  ml_single_open (&single, "model_file.ini", in_info, out_info, ML_NNFW_TYPE_NNTR_INF, ML_NNFW_HW_ANY);

You can use the trained model with pipeline API as follows:

  ml_pipeline_h pipe;

  /* framework is 'nntrainer'*/
  const char pipeline[] = "appsrc ! other/tensor,dimension=(string)2:1:1:1,type=(string)  int8,framerate=(fraction)0/1 ! tensor_filter framework=nntrainer model=model.ini  ! tensor_sink";

  status = ml_pipeline_construct (pipeline, NULL, NULL, &pipe);

Related information

• Dependencies
  • Tizen 6.0 and Higher for Mobile
  • Tizen 6.0 and Higher for Wearable
  • Tizen 6.0 and Higher for TV
  • Tizen 6.0 and Higher for IoT