tensorflow_lite_flutter

Description:

tensorflow lite flutter

tensorflow_lite_flutter #
A Flutter plugin for accessing TensorFlow Lite API. Supports image classification, object detection (SSD and YOLO), Pix2Pix and Deeplab and PoseNet on both iOS and Android.
Table of Contents #

Installation
Usage

Image Classification
Object Detection

SSD MobileNet
YOLO


Pix2Pix
Deeplab
PoseNet


Example

Prediction in Static Images
Real-time Detection



Breaking changes #
Since 1.1.0:

iOS TensorFlow Lite library is upgraded from TensorFlowLite　1.x to TensorFlowLiteObjC　2.x. Changes to native code are denoted with TFLITE2.

Since 1.0.0:

Updated to TensorFlow Lite API v1.12.0.
No longer accepts parameter inputSize and numChannels. They will be retrieved from input tensor.
numThreads is moved to Tflite.loadModel.

Installation #
Add tflite as a dependency in your pubspec.yaml file.
Android #
In android/app/build.gradle, add the following setting in android block.
aaptOptions {
noCompress 'tflite'
noCompress 'lite'
}
copied to clipboard
iOS #
Solutions to build errors on iOS:


'vector' file not found"
Open ios/Runner.xcworkspace in Xcode, click Runner > Tagets > Runner > Build Settings, search Compile Sources As, change the value to Objective-C++


'tensorflow/lite/kernels/register.h' file not found
The plugin assumes the tensorflow header files are located in path "tensorflow/lite/kernels".
However, for early versions of tensorflow the header path is "tensorflow/contrib/lite/kernels".
Use CONTRIB_PATH to toggle the path. Uncomment //#define CONTRIB_PATH from here:


Usage #

Create a assets folder and place your label file and model file in it. In pubspec.yaml add:

assets:
- assets/labels.txt
- assets/mobilenet_v1_1.0_224.tflite
copied to clipboard

Import the library:

import 'package:tensorflow_lite_flutter/tensorflow_lite_flutter.dart';
copied to clipboard

Load the model and labels:

String res = await Tflite.loadModel(
model: "assets/mobilenet_v1_1.0_224.tflite",
labels: "assets/labels.txt",
numThreads: 1, // defaults to 1
isAsset: true, // defaults to true, set to false to load resources outside assets
useGpuDelegate: false // defaults to false, set to true to use GPU delegate
);
copied to clipboard


See the section for the respective model below.


Release resources:


await Tflite.close();
copied to clipboard
GPU Delegate #
When using GPU delegate, refer to this step for release mode setting to get better performance.
Image Classification #

Output format:

{
index: 0,
label: "person",
confidence: 0.629
}
copied to clipboard

Run on image:

var recognitions = await Tflite.runModelOnImage(
path: filepath, // required
imageMean: 0.0, // defaults to 117.0
imageStd: 255.0, // defaults to 1.0
numResults: 2, // defaults to 5
threshold: 0.2, // defaults to 0.1
asynch: true // defaults to true
);
copied to clipboard

Run on binary:

var recognitions = await Tflite.runModelOnBinary(
binary: imageToByteListFloat32(image, 224, 127.5, 127.5),// required
numResults: 6, // defaults to 5
threshold: 0.05, // defaults to 0.1
asynch: true // defaults to true
);

Uint8List imageToByteListFloat32(
img.Image image, int inputSize, double mean, double std) {
var convertedBytes = Float32List(1 * inputSize * inputSize * 3);
var buffer = Float32List.view(convertedBytes.buffer);
int pixelIndex = 0;
for (var i = 0; i < inputSize; i++) {
for (var j = 0; j < inputSize; j++) {
var pixel = image.getPixel(j, i);
buffer[pixelIndex++] = (img.getRed(pixel) - mean) / std;
buffer[pixelIndex++] = (img.getGreen(pixel) - mean) / std;
buffer[pixelIndex++] = (img.getBlue(pixel) - mean) / std;
}
}
return convertedBytes.buffer.asUint8List();
}

Uint8List imageToByteListUint8(img.Image image, int inputSize) {
var convertedBytes = Uint8List(1 * inputSize * inputSize * 3);
var buffer = Uint8List.view(convertedBytes.buffer);
int pixelIndex = 0;
for (var i = 0; i < inputSize; i++) {
for (var j = 0; j < inputSize; j++) {
var pixel = image.getPixel(j, i);
buffer[pixelIndex++] = img.getRed(pixel);
buffer[pixelIndex++] = img.getGreen(pixel);
buffer[pixelIndex++] = img.getBlue(pixel);
}
}
return convertedBytes.buffer.asUint8List();
}
copied to clipboard

Run on image stream (video frame):


Works with camera plugin 4.0.0. Video format: (iOS) kCVPixelFormatType_32BGRA, (Android) YUV_420_888.

var recognitions = await Tflite.runModelOnFrame(
bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required
imageHeight: img.height,
imageWidth: img.width,
imageMean: 127.5, // defaults to 127.5
imageStd: 127.5, // defaults to 127.5
rotation: 90, // defaults to 90, Android only
numResults: 2, // defaults to 5
threshold: 0.1, // defaults to 0.1
asynch: true // defaults to true
);
copied to clipboard
Object Detection #

Output format:

x, y, w, h are between [0, 1]. You can scale x, w by the width and y, h by the height of the image.
{
detectedClass: "hot dog",
confidenceInClass: 0.123,
rect: {
x: 0.15,
y: 0.33,
w: 0.80,
h: 0.27
}
}
copied to clipboard
SSD MobileNet:

Run on image:

var recognitions = await Tflite.detectObjectOnImage(
path: filepath, // required
model: "SSDMobileNet",
imageMean: 127.5,
imageStd: 127.5,
threshold: 0.4, // defaults to 0.1
numResultsPerClass: 2,// defaults to 5
asynch: true // defaults to true
);
copied to clipboard

Run on binary:

var recognitions = await Tflite.detectObjectOnBinary(
binary: imageToByteListUint8(resizedImage, 300), // required
model: "SSDMobileNet",
threshold: 0.4, // defaults to 0.1
numResultsPerClass: 2, // defaults to 5
asynch: true // defaults to true
);
copied to clipboard

Run on image stream (video frame):


Works with camera plugin 4.0.0. Video format: (iOS) kCVPixelFormatType_32BGRA, (Android) YUV_420_888.

var recognitions = await Tflite.detectObjectOnFrame(
bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required
model: "SSDMobileNet",
imageHeight: img.height,
imageWidth: img.width,
imageMean: 127.5, // defaults to 127.5
imageStd: 127.5, // defaults to 127.5
rotation: 90, // defaults to 90, Android only
numResults: 2, // defaults to 5
threshold: 0.1, // defaults to 0.1
asynch: true // defaults to true
);
copied to clipboard
Tiny YOLOv2:

Run on image:

var recognitions = await Tflite.detectObjectOnImage(
path: filepath, // required
model: "YOLO",
imageMean: 0.0,
imageStd: 255.0,
threshold: 0.3, // defaults to 0.1
numResultsPerClass: 2,// defaults to 5
anchors: anchors, // defaults to [0.57273,0.677385,1.87446,2.06253,3.33843,5.47434,7.88282,3.52778,9.77052,9.16828]
blockSize: 32, // defaults to 32
numBoxesPerBlock: 5, // defaults to 5
asynch: true // defaults to true
);
copied to clipboard

Run on binary:

var recognitions = await Tflite.detectObjectOnBinary(
binary: imageToByteListFloat32(resizedImage, 416, 0.0, 255.0), // required
model: "YOLO",
threshold: 0.3, // defaults to 0.1
numResultsPerClass: 2,// defaults to 5
anchors: anchors, // defaults to [0.57273,0.677385,1.87446,2.06253,3.33843,5.47434,7.88282,3.52778,9.77052,9.16828]
blockSize: 32, // defaults to 32
numBoxesPerBlock: 5, // defaults to 5
asynch: true // defaults to true
);
copied to clipboard

Run on image stream (video frame):


Works with camera plugin 4.0.0. Video format: (iOS) kCVPixelFormatType_32BGRA, (Android) YUV_420_888.

var recognitions = await Tflite.detectObjectOnFrame(
bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required
model: "YOLO",
imageHeight: img.height,
imageWidth: img.width,
imageMean: 0, // defaults to 127.5
imageStd: 255.0, // defaults to 127.5
numResults: 2, // defaults to 5
threshold: 0.1, // defaults to 0.1
numResultsPerClass: 2,// defaults to 5
anchors: anchors, // defaults to [0.57273,0.677385,1.87446,2.06253,3.33843,5.47434,7.88282,3.52778,9.77052,9.16828]
blockSize: 32, // defaults to 32
numBoxesPerBlock: 5, // defaults to 5
asynch: true // defaults to true
);
copied to clipboard
Pix2Pix #

Thanks to RP from Green Appers



Output format:
The output of Pix2Pix inference is Uint8List type. Depending on the outputType used, the output is:


(if outputType is png) byte array of a png image


(otherwise) byte array of the raw output




Run on image:


var result = await runPix2PixOnImage(
path: filepath, // required
imageMean: 0.0, // defaults to 0.0
imageStd: 255.0, // defaults to 255.0
asynch: true // defaults to true
);
copied to clipboard

Run on binary:

var result = await runPix2PixOnBinary(
binary: binary, // required
asynch: true // defaults to true
);
copied to clipboard

Run on image stream (video frame):

var result = await runPix2PixOnFrame(
bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required
imageHeight: img.height, // defaults to 1280
imageWidth: img.width, // defaults to 720
imageMean: 127.5, // defaults to 0.0
imageStd: 127.5, // defaults to 255.0
rotation: 90, // defaults to 90, Android only
asynch: true // defaults to true
);
copied to clipboard
Deeplab #

Thanks to RP from see-- for Android implementation.



Output format:
The output of Deeplab inference is Uint8List type. Depending on the outputType used, the output is:


(if outputType is png) byte array of a png image


(otherwise) byte array of r, g, b, a values of the pixels




Run on image:


var result = await runSegmentationOnImage(
path: filepath, // required
imageMean: 0.0, // defaults to 0.0
imageStd: 255.0, // defaults to 255.0
labelColors: [...], // defaults to https://github.com/shaqian/flutter_tflite/blob/master/lib/tflite.dart#L219
outputType: "png", // defaults to "png"
asynch: true // defaults to true
);
copied to clipboard

Run on binary:

var result = await runSegmentationOnBinary(
binary: binary, // required
labelColors: [...], // defaults to https://github.com/shaqian/flutter_tflite/blob/master/lib/tflite.dart#L219
outputType: "png", // defaults to "png"
asynch: true // defaults to true
);
copied to clipboard

Run on image stream (video frame):

var result = await runSegmentationOnFrame(
bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required
imageHeight: img.height, // defaults to 1280
imageWidth: img.width, // defaults to 720
imageMean: 127.5, // defaults to 0.0
imageStd: 127.5, // defaults to 255.0
rotation: 90, // defaults to 90, Android only
labelColors: [...], // defaults to https://github.com/shaqian/flutter_tflite/blob/master/lib/tflite.dart#L219
outputType: "png", // defaults to "png"
asynch: true // defaults to true
);
copied to clipboard
PoseNet #

Model is from StackOverflow thread.


Output format:

x, y are between [0, 1]. You can scale x by the width and y by the height of the image.
[ // array of poses/persons
{ // pose #1
score: 0.6324902,
keypoints: {
0: {
x: 0.250,
y: 0.125,
part: nose,
score: 0.9971070
},
1: {
x: 0.230,
y: 0.105,
part: leftEye,
score: 0.9978438
}
......
}
},
{ // pose #2
score: 0.32534285,
keypoints: {
0: {
x: 0.402,
y: 0.538,
part: nose,
score: 0.8798978
},
1: {
x: 0.380,
y: 0.513,
part: leftEye,
score: 0.7090239
}
......
}
},
......
]
copied to clipboard

Run on image:

var result = await runPoseNetOnImage(
path: filepath, // required
imageMean: 125.0, // defaults to 125.0
imageStd: 125.0, // defaults to 125.0
numResults: 2, // defaults to 5
threshold: 0.7, // defaults to 0.5
nmsRadius: 10, // defaults to 20
asynch: true // defaults to true
);
copied to clipboard

Run on binary:

var result = await runPoseNetOnBinary(
binary: binary, // required
numResults: 2, // defaults to 5
threshold: 0.7, // defaults to 0.5
nmsRadius: 10, // defaults to 20
asynch: true // defaults to true
);
copied to clipboard

Run on image stream (video frame):

var result = await runPoseNetOnFrame(
bytesList: img.planes.map((plane) {return plane.bytes;}).toList(),// required
imageHeight: img.height, // defaults to 1280
imageWidth: img.width, // defaults to 720
imageMean: 125.0, // defaults to 125.0
imageStd: 125.0, // defaults to 125.0
rotation: 90, // defaults to 90, Android only
numResults: 2, // defaults to 5
threshold: 0.7, // defaults to 0.5
nmsRadius: 10, // defaults to 20
asynch: true // defaults to true
);
copied to clipboard
Example #
Prediction in Static Images #
Refer to the example.
Run test cases #
flutter test test/tflite_test.dart