flutter_mediapipe

Description:

flutter mediapipe

flutter_mediapipe #
Flutter plugin with mediapipe facemesh.
Devices #
Currently, it runs on below devices with "OK".

[OK] Android
[NG] iOS

Android #
There are Mediapipe Manual Build for Android flutter plugin.
There are mobile_calculators list to run on Mobile.
Choose one. This plugin choose face_mesh.

face_detection
face_mesh
object_detection
hand_tracking

Docker #
git clone https://github.com/google/mediapipe.git
cd mediapipe
docker build --tag=mediapipe .
docker run -it --name mediapipe mediapipe:latest
copied to clipboard
When enter docker again:
docker start [your container ID]
docker exec -i -t mediapipe bash
copied to clipboard
Run below commands on the container.
Setup #
apt update
apt install vim
apt install zip
bash ./setup_android_sdk_and_ndk.sh
copied to clipboard
Prepare #
mkdir mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe
vi mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe/BUILD
copied to clipboard

"BUILD" file content.

load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar")

mediapipe_aar(
name = "flutter_mediapipe",
calculators = ["//mediapipe/graphs/face_mesh:mobile_calculators"],
)
copied to clipboard
Build #

jniLibs

bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a --strip=ALWAYS //mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe:BUILD --linkopt="-s"
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a //mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe:flutter_mediapipe --linkopt="-s"
copied to clipboard

binary graph

bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a --strip=ALWAYS //mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu:BUILD
bazel build -c opt --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --fat_apk_cpu=arm64-v8a,armeabi-v7a //mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu:facemeshgpu
copied to clipboard
mkdir #
mkdir flutter_mediapipe/
mkdir flutter_mediapipe/android
mkdir flutter_mediapipe/android/libs
mkdir flutter_mediapipe/android/src
mkdir flutter_mediapipe/android/src/main
mkdir flutter_mediapipe/android/src/main/assets
mkdir flutter_mediapipe/android/src/main/jniLibs
mkdir flutter_mediapipe/protos
copied to clipboard
libs #
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe/libflutter_mediapipe_android_lib.jar flutter_mediapipe/android/libs
copied to clipboard
assets #
cp mediapipe/modules/face_detection/face_detection_front.tflite flutter_mediapipe/android/src/main/assets
cp mediapipe/modules/face_landmark/face_landmark.tflite flutter_mediapipe/android/src/main/assets
cp bazel-out/k8-opt/bin/mediapipe/graphs/face_mesh/face_mesh_mobile_gpu.binarypb flutter_mediapipe/android/src/main/assets

copied to clipboard
jniLibs #
mkdir work
cp bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/flutter_mediapipe/flutter_mediapipe.aar work/aar.zip
cd work/
unzip aar.zip
cd ..
cp -r work/jni/* flutter_mediapipe/android/src/main/jniLibs/
copied to clipboard
protos #
cp mediapipe/framework/formats/landmark.proto flutter_mediapipe/protos/
copied to clipboard
See regenerate.md
Zip #
zip -r flutter_mediapipe.zip flutter_mediapipe
copied to clipboard
And then, exit from the container.
Copy #
Change "3730fdc4d319" to your container ID.
docker cp 3730fdc4d319:/mediapipe/flutter_mediapipe.zip .
unzip flutter_mediapipe.zip
copied to clipboard
Then copy "flutter_mediapipe" to flutter plugin projects.
APK #
flutter build apk --split-per-abi
copied to clipboard
Refferences #

sample plugin is useful.
sample android app without flutter
Flutter don't support local AAR
jniLibs size down

iOS #
Not implemented. Help me to develop.
Custormize mediapipe #
Graph #
When editing graphs or subgraphs, Build again.

graphs: binary graph
subgraphs: jniLibs

vi mediapipe/graphs/face_mesh/subgraphs/face_renderer_gpu.pbtxt
copied to clipboard
It is a sample to comment out of lines from 92 to 94 for hiding rectangles and landmarks.
# Draws annotations and overlays them on top of the input images.
node {
calculator: "AnnotationOverlayCalculator"
input_stream: "IMAGE_GPU:input_image"
# input_stream: "detections_render_data"
# input_stream: "VECTOR:0:multi_face_landmarks_render_data"
# input_stream: "rects_render_data"
output_stream: "IMAGE_GPU:output_image"
}
copied to clipboard
Calculator #
When editing c++ calculator source, Build again.

c++ source: jniLibs

vi mediapipe/graphs/face_mesh/calculators/face_landmarks_to_render_data_calculator.cc
copied to clipboard
It is a sample to comment out the lines(31 and 45) for hiding Left eyebrow connections.
constexpr int kNumFaceLandmarkConnections = 116; // (124 - (16/2)) c.f. l.93
// Pairs of landmark indices to be rendered with connections.
constexpr int kFaceLandmarkConnections[] = {
:
133,
// Left eyebrow.
// 46, 53, 53, 52, 52, 65, 65, 55, 70, 63, 63, 105, 105, 66, 66, 107,
// Right eye.
:
for (int i = 0; i < kNumFaceLandmarkConnections; ++i) {
landmark_connections_.push_back(kFaceLandmarkConnections[i * 2]);
landmark_connections_.push_back(kFaceLandmarkConnections[i * 2 + 1]);
}
copied to clipboard