Art with ESP32-CAM: Style Transfer — Magenta JS

This tutorial describes how to create art with ESP32-CAM using style transfer and Magenta.js. One field where AI and machine learning can be applied is the art field. In this tutorial, we will explore how to modify the picture taken by ESP32-CAM using machine learning. This example demonstrates how we can mix different technologies to create something new and different. Moreover, this tutorial shows the endless possibilities we can have when using ESP32-CAM.

What is Neural Style Transfer?

Neural Style Transfer is a set of algorithms that manipulate images. It uses two images:

• a content image
• a style image

The aim of this process is to mix these two images together applying the style of one image to the content of the other one. The final effect is one image that holds the content of the original image painted with the style of another image. We will use the picture taken by ESP32-CAM as the content image while we will another image as a style image. We will mix them together. Neural Style transfer is used to create artificial artistic images. In this tutorial, we will experiment on how to apply for Neural style transfer with ESP32-CAM using Magenta.js.

What is Magenta.js?

Magenta.js is an open source javascript library built on TensorFlow. We have already covered how to use Tensorflow with ESP32 in the previous posts. This tutorial extends the concepts covered in the post “ how to classify images with Machine Learning with ESP32-CAM “ using the same process described previously. Magenta is an effort to explore how Machine Learning can be applied to art. As we did in the previous post, we can’t run this machine learning model directly on the ESP32 because it would require too much power. Therefore, we use the client power. In more detail we will use the browser because Magenta.js runs inside it.

How to apply Style transfer to image using ESP32-CAM

To achieve our goal, we will create a simple HTML Web interface using the ESP32. This interface has three different tasks:

• Stream the video and capture the image
• Let the user to select the style image
• Apply Style Transfer using Magenta.js to the image taken by the ESP32

The UI interface is very simple. You can find more details about how to create it in the previous post. Moreover, we will use the same ESP32-CAM code developed previously:

UI preview

This is the UI preview:

Installing the code into the ESP32-CAM

Clone the repository on Github and install it in your ESP32-CAM using the IDE you prefer.

Notice that the HTML page is in the hex format. This is a simple trick to avoid using a filesystem in the ESP32-CAM to provide the HTML page.

Applying Style transfer using Magenta.js and ESP32-CAM

Once you have installed the code into your ESP32-CAM device, we can test it connecting to your device using a browser.

Below some examples:

Below another example:

As you can notice from the image above, the content image, that is the ESP32-CAM image, is transformed using the style of the another image.

Wrapping up

At the end of this post, we have learned how to use ESP32-CAM with Magenta.js. This is a javascript library to create art using Machine Learning. We have discovered how to apply Style Transfer to the image captured by the ESP32-CAM. We can modify these images by applying the style from another image we select.

If you liked this post you can visit my blog or follow me on Twitter.

Originally published at https://www.survivingwithandroid.com on July 5, 2020.