Arduino Machine Learning: Build a Tensorflow lite model to control robot-car

This tutorial covers how to use Machine Learning with Arduino. The aim of this tutorial is to build a voice-controlled car from scratch that uses Tensorflow Machine Learning to recognize voice commands.

7 min readJul 25, 2020

To do it we will use Arduino Nano 33 BLE sense. The availability of the Tensorflow lite for microcontrollers makes it possible to run machine learning algorithms on microcontrollers such as Arduino. In this tutorial, we will build a Tensorflow model that recognizes voice commands. We will use them to control movements. The capability to run Tensorflow models on microcontrollers opens a new perspective really promising that is named TinyML.

The idea of this project starts from the project named micro speech covered in TinyML book. We will extend that example by adding new commands and training the models to recognize them. Moreover, we will build a simple car that is controlled by voice commands.

What you will learn in this tinyML tutorial

To build this project there are at least, two steps:

train a new machine learning model and adapt it to run on Arduino
Build the car that uses the Tensorflow trained in the previous step

As you may already know, we can’t run directly Tensorflow models on an Arduino because this device has limited resources. Therefore, it is necessary, after the model is trained, to shrink it reducing its size. We will describe step by step how to build the model and then how to convert it in a way that is compatible with Arduino.

If you want to skip this part you can download directly the Tensorflow lite model to you in Arduino sketch:

Building the Tensorflow lite model to use with Arduino

As said before, this project uses the work already done in this tutorial. As you may already know, there are several steps to follow to build a machine learning model that runs on Arduino. Below a list of these steps:

Defining the model to use
Training the model
Freezing the model
Quantize the TensorFlow model
Convert the TensorFlow model in a format compatible with Arduino

In this machine learning tutorial, we will use Google Colab to train the model. This is a nice environment where we can have Tensorflow installed and we can train the model.

Brief Tensorflow lite model

To recognize voice commands using Arduino, we need a Tensorflow model that uses CNN to do it. The first step is acquiring the voice using the Arduino Nano 33 built-in microphone and apply to it the FFT. The data extracted using the Fast Fourier Transformation will feed the CNN. The problem to recognize commands is a classification problem. To have more information you have to read the TinyML book.

Training the model

For convenience, this is the link to the colab notebook that we will use in this Tensorflow tutorial. You can use and run directly the code in the colab book. Anyway, this is a brief description:

In the beginning, it is necessary to specify the commands or words we want to recognize. These are the labels. In this example, since we want to control the car using voice commands, we will implement 4 different commands:
Next, it is necessary to install Tensorflow in Colab. You can notice that it will not install the latest version but the 1.15. This version will create a Tensorflow model compatible with Arduino
After it, it is necessary to clone the TensorFlow repository. We will use it to run the training process
Finally, run the model. To run the model, it is necessary to provide some parameters. I suggest you to not change them. The training process requires about 2 hours so take it easy. Anyway, you can make it shorted by changing the TRAINING_STEPS. Be careful when changing these values because it can make the model less accurate

Freezing the Tensorflow model to use with Arduino

When the training task completes, it is necessary to freeze the model. After this step, we can’t modify the Tensorflow model anymore. This will be the model that we will use in the next steps, to create a Tensorflow model for Arduino.

Create the Tensorflow lite model for microcontroller

This is one of the most important step. When we finish freezing the model, we notice that the size of this model is very big and it can’t run on a microcontroller (such as Arduino) with limited resources. To reduce the model size it is necessary to quantize the model or in other words transform everything into an 8bit integer. This reduces the model size so that it can run directly on Arduino.

Converting the Tensorflow lite model in Arduino C array

Even if we could use the quantized model directly, some devices (Arduino for example) don’t have a file system. Therefore it makes it difficult to use this model. To overcome this situation, we can convert the model to a C array that we can copy into the Arduino source code. In this way, we do not need a file system.

How to use Tensorflow model with Arduino

Once we have built our model, we have to use it with Arduino. To do it, we will reuse the micro speech example available with Tensorflow Lite example. Open the Arduino IDE and install the Tensorflow Library for Arduino:

Next in the Tensorflow lite examples, you will find the micro speech project. Open it. You will see that there are several files, do not worry…we have to modify a few of them.

Importing the Tensorflow lite model in Arduino sketch

The first thing to do is importing the Tensorflow lite model into the Arduino sketch. Open the file micro_features_micro_model.cpp and replace the existing model with the one you have created in the steps above. The model to use is the file you have obtained in the last step when you build the C-array.

The line:

const unsigned char g_model[] DATA_ALIGN_ATTRIBUTE = {....}

specifies the model that Arduino will use. You have simply to replace the content of the array. After it, you have to change the model length in the same file:

To know this value read the last line of the file you used to get the new model.

Adjusting the model labels

We want to use four different commands to control our car using Tensorflow. Therefore it is necessary to change the number of labels (aka commands) that the model will handle. To do it, open the file micro_features_micro_model_settings.h and look for:

Replace the value (should be 4) with 6. This value is the sum of the 4 commands (go, stop, left, right) plus two other labels (silence and unknown).

Now, open the file micro_features_micro_model_settings.cpp and replace the values yes and now with go, stop, left and right. The content should be this one:

Using Tensorflow lite model to control motors

In this last step, we will implement the code to control the motors. After Arduino has recognized the voice commands using the Machine Learning, we have to “respond” to this command so that the car moves itself.

Open arduino_command_responder.cpp and in the RespondToCommand method replace the exisisting code with this one:

void RespondToCommand(tflite::ErrorReporter* error_reporter, 
      int32_t current_time, const char* found_command, 
      uint8_t score, bool is_new_command) { 
  
  static bool is_initialized = false; 
  static int32_t last_command_time = 0; 
  static int count = 0; 
  static int certainty = 220; 
  if (is_new_command) { 
       TF_LITE_REPORT_ERROR(error_reporter, "Heard %s (%d) @%dms",
          found_command, score, current_time); 
       if (found_command[0] == 'l') { 
          last_command_time = current_time;   
          TF_LITE_REPORT_ERROR(error_reporter, "---- Left -----"); 
   } 
   if (found_command[0] == 'r') { 
     last_command_time = current_time; 
     TF_LITE_REPORT_ERROR(error_reporter, "---- Right -----"); 
   } 
   if (found_command[0] == 'g') { 
     last_command_time = current_time; 
     TF_LITE_REPORT_ERROR(error_reporter, "---- Go -----"); 
   } 
   if (found_command[0] == 's') { 
     last_command_time = current_time;
     TF_LITE_REPORT_ERROR(error_reporter, "---- Stop -----"); 
   } 
   if (found_command[0] == 'u') { 
     last_command_time = current_time; 
   } 
 } 
}

Then add these lines:

void forward() { 
  digitalWrite(PIN1_M1, HIGH); 
  digitalWrite(PIN2_M1, LOW); 
  digitalWrite(PIN1_M2, HIGH); 
  digitalWrite(PIN2_M2, LOW); 
} 
void stop() { 
  digitalWrite(PIN1_M1, HIGH); 
  digitalWrite(PIN2_M1, LOW); 
  digitalWrite(PIN1_M2, HIGH); 
  digitalWrite(PIN2_M2, LOW); 
}
void left() { 
  digitalWrite(PIN1_M1, HIGH); 
  digitalWrite(PIN2_M1, LOW); 
  digitalWrite(PIN1_M2, LOW); 
  digitalWrite(PIN2_M2, LOW); 
} 
void right() { 
  digitalWrite(PIN1_M1, LOW); 
  digitalWrite(PIN2_M1, LOW); 
  digitalWrite(PIN1_M2, HIGH); 
  digitalWrite(PIN2_M2, LOW); 
}

In this tutorial, Arduino is connected to L289N Dual H to control two motors. I’m not showing here the schematic to connect Arduino 33 Nano BLE to the L289N because it is very simple. The code above uses the pin D2, D3,D4, D5. If you prefer you can use different pins. Below some images showing the connections:

Testing the Tensorflow lite model with Arduino Nano 33 BLE

Now you can test the voice-controlled car-based base on Tensorflow and Arduino Nano that uses Machine Learning to recognize voice commands.

You need an external battery to power Arduino. This battery is not shown in the picture above. Once you have uploaded all the code into Arduino you can start speaking using the following commands:

If everything works the car should move according to your commands.

If you prefer and don’t have a chassis to test this code, you can simply test the voice recognition using Arduino and Tensorflow:

Adjusting some settings

It can happen that the model we have trained can’t recognize perfectly our commands. It can happen because there is a problem in your pronounce or the model is not accurate as we wanted. You could retrain your model using more training steps or you can lower the threshold used in the code to recognize a new command. To do it open the file recognize_commands.h and look for RecognizeCommands method. In this method, you can adjust the threshold value (that is 200). You should lower it anyway be careful because Arduino could recognize false commands.

Wrapping up

At the end of this tutorial, we have explored how to use Tensorflow lite with Arduino. This helps us to use Machine Learning on Arduino devices. We have learned how to train a Tensorflow lite model and how to use this model with Arduino to enable voice recognition. We have built a simple car powered by Arduino that use Machine Learning to recognize commands,

Originally published at https://www.survivingwithandroid.com on July 25, 2020.