6 min readFeb 28, 2019

This article describes how to apply Machine Learning to Android using Fritz.ai. Before diving into the details about how to develop a Machine learning Android app, it is useful to describe briefly what is Fritz.ai platform. As you may know, Machine Learning is an interesting topic that is gaining importance and promises to transform several areas including the way we interact with Android apps.

To experiment how to apply Machine Learning to Android using Fritz.ai we will develop an Android app that uses Image Classification.

Machine Learning is an application of AI that gives to a system the capability to accomplish tasks without using explicit instructions but learning from the data and improving from experience.

What is Fritz.ai Machine Learning platform?

Fritz.ai is a platform for Android and iOS that simplifies the development of an Android Machine Learning app. When developing a an Android app, usually it is necessary to create a ML-model that will be used by the Android app. The process to build a ML-model is tedious and difficult. Fritz.ai provides a set of built-in models ready to use that speeds up the development of a Machine Learning Android app.

The built-in ML-models provided by Fritz.ai are:

Image Labeling
Object detection
Style transfer
Image segmentation
Pose extimation

Moreover, it is possible to upload a custom model if we want to use a specific model. You can download the source code here:

How to configure Fritz.ai

The first step to use Fritz.ai Machine Learning platform is creating a free account and then create a new Machine Learning Android project, as shown in the picture below:

Now, it is possible to add a new project:

You have to provide the Project name. Once the project is created, it is necessary to provide the project package:

How to set up an Android project using Fritz.ai Image classification

Once the project is correctly configured in Fritz.ai console, we can set up the Android project to include Fritz.ai. At the project-level, it is necessary to modify build.gradle in this way:

... allprojects { 
     repositories { 
       google() 
       jcenter()
       maven { 
         url "https://raw.github.com/fritzlabs/fritz- repository/master" 
       } 
     } 
 } ...

and in the build.gradle at app-level:

dependencies { 
   .... 
   implementation 'ai.fritz:core:3.0.1' 
   implementation 'ai.fritz:vision:3.0.1' 
}

The last step is registering a service so that the Android app can receive model updates. To do it in the Manifest.xml it is necessary to add:

<service
   android:name="ai.fritz.core.FritzCustomModelService"   
   android:exported="true" 
   android:permission="android.permission.BIND_JOB_SERVICE" />

That’s all. We are now ready to start developing our image classification Android app.

Implementing Machine Learning image classification in Android

To implement an Android app that uses image classification we have to follow these steps:

Configure Fritz.ai SDK in the Android app
Label the image detected

Configure Fritz.ai SDK in Android app

The first step is configuring the Fritz.ai SDK in our Android app so that we can use it later when we need to label the captured image. Let us create an activity and in onCreate() method add the following lines:

protected void onCreate(Bundle savedInstanceState) {
   super.onCreate(savedInstanceState);
   setContentView(R.layout.activity_main); 
   Fritz.configure(this, "your_fritz.ai_key"); 
   downloadModel(); 
}

In line 4, it is necessary to configure Fritz using the API_KEY we get during the project set up in the console. The next step, it is downloading the model from the cloud:

private void downloadModel() { 
   FritzManagedModel model = new ImageLabelManagedModel(); 
   Log.d(TAG, "Download model..."); 
   FritzVision.ImageLabeling.loadPredictor(model, 
         new PredictorStatusListener<FritzVisionLabelPredictor>() { 
           @Override 
           public void onPredictorReady(FritzVisionLabelPredictor 
                         predictor) {
             Log.d(TAG, "Model downloaded");
             MainActivity.this.predictor = predictor; 
           } 
    }); 
}

In this case, the Android app uses the ImageLabelManageModel because we want to label the image using Machine Learning.

<?xml version="1.0" encoding="utf-8"?> 
<RelativeLayout   
   xmlns:android="http://schemas.android.com/apk/res/android" 
   xmlns:app="http://schemas.android.com/apk/res-auto" 
   xmlns:tools="http://schemas.android.com/tools" 
   android:layout_width="match_parent" 
   android:layout_height="match_parent"    
   tools:context=".MainActivity"> 
     <TextureView 
      android:layout_width="wrap_content"  
      android:layout_height="wrap_content" 
      android:layout_alignParentTop="true" 
      android:id="@+id/preview"/>
       <Button 
          android:layout_width="match_parent" 
          android:layout_height="wrap_content" 
          android:id="@+id/btnLabel" 
          android:text="Label it" 
          android:layout_alignParentBottom="true"/> </RelativeLayout>

The TextureView will hold the image stream coming from the camera, while the button will be used to label the image, as we will see later.

How to use Fritz.ai to classify an image

To classify the image, once we have an image from the camera, it is necessary to create an instance of FritzVisionImage. Let us suppose, by now, that we have an instance of ImageReader that holds the image captured from the camera. It is necessary to convert it into a bitmap and then use the predictor to label the image:

Image image = reader.acquireLatestImage(); 
ByteBuffer buffer = image.getPlanes()[0].getBuffer();
byte[] bytes = new byte[buffer.capacity()];
buffer.get(bytes);  
Bitmap bitmapImage = 
     BitmapFactory.decodeByteArray(bytes, 0, bytes.length, null); FritzVisionImage fvi = FritzVisionImage.fromBitmap(bitmapImage);

Finally, at line 8, the predictor, that will classify the image, is ready to be used. Now, the Android app can retrieve a list of labels with the relative accuracy:

FritzVisionLabelResult labels = predictor.predict(fvi); List<FritzVisionLabel> labelList = labels.getVisionLabels(); 
if (labelList != null && labelList.size() > 0) { 
    FritzVisionLabel label = labels.getVisionLabels().get(0);
    System.out.println("Label [" + label.getText() + "]");
}

The app gets the first element of the list, that has higher accuracy. Now we can describe how to use the camera in Android to capture an image.

Handle the Camera in Android

Once the Machine Learning engine is ready, we have to focus our attention on how to capture the image. As stated before, TextureView hold the image stream coming from the camera. It is necessary to add a listener to the TextureView to know when we can use the camera:

TextureView.SurfaceTextureListener surfaceListener =
   new TextureView.SurfaceTextureListener() {
      @Override
       public void onSurfaceTextureAvailable(
                 SurfaceTexture surface, 
                 int width, int height) { 
           Log.d(TAG, "Surface available"); 
           openCamera(); 
      } 
      @Override 
      public void onSurfaceTextureSizeChanged(
                SurfaceTexture surface, 
                int width, int height) {} 
      @Override public boolean onSurfaceTextureDestroyed(
                 SurfaceTexture surface) { 
             return false; 
      } 
      @Override public void onSurfaceTextureUpdated(
           SurfaceTexture surface) { } 
 };

When the surface is ready, the app can open the camera and start using it. Please notice that:

preview = (TextureView) findViewById(R.id.preview); preview.setSurfaceTextureListener(surfaceListener);

Now it is possible to open the camera:

private void openCamera() { 
   // We get the first available camera  
   CameraManager manager = (CameraManager)
           getSystemService(Context.CAMERA_SERVICE);
    try {
      cameraId = manager.getCameraIdList()[0]; 
      CameraCharacteristics characteristics = 
                    manager.getCameraCharacteristics(cameraId);   
       StreamConfigurationMap streamConfMap =      
        characteristics.get(
             CameraCharacteristics.SCALER_STREAM_CONFIGURATION_MAP
        );
        imageDimension = 
          streamConfMap.getOutputSizes(SurfaceTexture.class)[0];
       // We can open the camera now 
       manager.openCamera(cameraId, 
             new cameraDevice.StateCallback() { 
                 @Override public void onOpened(
                    @NonNull CameraDevice camera) { 
                     Log.i(TAG, "Camera opened"); 
                     cameraDevice = camera; 
                     createPreview(); 
                } 
                @Override 
                public void onDisconnected(
                   @NonNull CameraDevice camera) { } 
                @Override 
                public void onError(
                  @NonNull CameraDevice camera, 
                  int error) { 
                    Log.e(TAG, "Error opening the camera"); 
               }
      }, null); 
  } 
  catch(CameraAccessException cae) { // Let us handle the error }  
  catch(SecurityException se) { } 
}

When the camera is opened and ready to use, the Android app can starts streaming images:

private void createPreview() { 
  SurfaceTexture surfaceTexture = preview.getSurfaceTexture();  
  surfaceTexture.setDefaultBufferSize(
      imageDimension.getWidth(), 
      imageDimension.getHeight()
  ); 
  Surface surface = new Surface(surfaceTexture); 
  try { 
   previewRequestBuilder = cameraDevice
        .createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW); 
   previewRequestBuilder.addTarget(surface);   
   cameraDevice.createCaptureSession(
           Arrays.asList(surface), 
           new CameraCaptureSession.StateCallback() { 
               public void onConfigured(@NonNull 
                  CameraCaptureSession cameraCaptureSession) { 
                 cameraSession = cameraCaptureSession; 
                 previewRequestBuilder.set( 
                        CaptureRequest.CONTROL_AF_MODE,                     CaptureRequest.CONTROL_AF_MODE_CONTINUOUS_PICTURE); 
// Flash is automatically enabled when necessary. 
           previewRequestBuilder.set( 
                  CaptureRequest.CONTROL_AE_MODE, 
                  CaptureRequest.CONTROL_AE_MODE_ON_AUTO_FLASH); 
// Finally, we start displaying the camera preview. CaptureRequest 
          previewRequest = previewRequestBuilder.build(); 
          try { 
            cameraSession.setRepeatingRequest( 
             previewRequest, 
             new CameraCaptureSession.CaptureCallback() { 
               @Override 
               public void onCaptureProgressed( 
                  final CameraCaptureSession session, 
                  final CaptureRequest request, 
                  final CaptureResult partialResult) { }
              @Override public void onCaptureCompleted( 
                  final CameraCaptureSession session, 
                  final CaptureRequest request, 
                  final TotalCaptureResult result) { } 
             }, 
             backgroundHandler); 
          } 
          catch(CameraAccessException cae) {} 
   }
   @Override 
   public void onConfigureFailed(
           @NonNull CameraCaptureSession cameraCaptureSession) { 
              Log.e(TAG, "Configuration failed"); 
           }
   }, 
   null); 
 } 
 catch (CameraAccessException cae) {} 
}

Classifying the image captured by the camera

In this last step, it is necessary to capture the image and classify it using Machine Learning. To this purpose, the Android app uses the button defined in the previous layout:

labelBtn = (Button) findViewById(R.id.btnLabel); labelBtn.setOnClickListener(new View.OnClickListener() { 
   @Override 
   public void onClick(View v) { 
     takePicture(); 
   }
 });

In the takePicture method, the Android app takes the picture and label it (as shown above). If you want to have the full code you can go to Github and download it.

Summary

At the end of this article, you hopefully gained the knowledge about how to apply Machine Learning to Android using Fritz.ai. You have discovered how to use Machine Learning image classification to label an image captured by the camera.

Originally published at https://www.survivingwithandroid.com on February 28, 2019.