cassdlcm/mediapipe
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Project import generated by Copybara.

Browse Source 
GitOrigin-RevId: 5b23708185311ae39a8605b0c2eff721e7b4939f
This commit is contained in:
MediaPipe Team 2020-08-05 01:34:32 -04:00 committed by chuoling
parent bdfdaef305
commit 2f86a459b6
115 changed files with 5242 additions and 251 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
BUILD → BUILD.bazel
View File

@ -12,6 +12,6 @@
# See the License for the specific language governing permissions and
# limitations under the License.
licenses(["notice"]) # Apache 2.0
licenses(["notice"])
exports_files(["LICENSE"])

17
README.md
View File

@ -22,13 +22,13 @@ desktop/cloud, web and IoT devices.
## ML solutions in MediaPipe
Face Detection | Face Mesh | Hands | Hair Segmentation
:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :---------------:
[![face_detection](docs/images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](docs/images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![hand](docs/images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands) | [![hair_segmentation](docs/images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation)
Face Detection | Face Mesh | Iris | Hands
:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :---:
[![face_detection](docs/images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](docs/images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](docs/images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](docs/images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands)
Object Detection | Box Tracking | Objectron | KNIFT
:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
[![object_detection](docs/images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](docs/images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![objectron](docs/images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](docs/images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift)
Hair Segmentation | Object Detection | Box Tracking | Objectron | KNIFT
:-------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
[![hair_segmentation](docs/images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation) | [![object_detection](docs/images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](docs/images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![objectron](docs/images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](docs/images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift)
<!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. -->
<!-- Whenever this table is updated, paste a copy to solutions/solutions.md. -->
@ -37,6 +37,7 @@ Object Detection
:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :-: | :---:
[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | ✅ | ✅
[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | |
[Iris](https://google.github.io/mediapipe/solutions/iris) | ✅ | ✅ | ✅ | ✅ |
[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | ✅ |
[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | ✅ |
[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | ✅
@ -63,6 +64,8 @@ never leaves your device.
![visualizer_runner](docs/images/visualizer_runner.png)
* [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection)
* [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
* [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
* [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
* [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
* [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
@ -83,6 +86,8 @@ run code search using
## Publications
* [MediaPipe Iris: Real-time Eye Tracking and Depth Estimation from a Single
Image](https://mediapipe.page.link/iris-blog) in Google AI Blog
* [MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html)
in Google Developers Blog
* [Alfred Camera: Smart camera features using MediaPipe](https://developers.googleblog.com/2020/03/alfred-camera-smart-camera-features-using-mediapipe.html)

6
docs/framework_concepts/calculators.md
View File

@ -405,8 +405,4 @@ packets (bottom) based on its series of input packets (top).
| ![Graph using |
: PacketClonerCalculator](../images/packet_cloner_calculator.png) :
| :--------------------------------------------------------------------------: |
| *Each time it receives a packet on its TICK input stream, the |
: PacketClonerCalculator outputs the most recent packet from each of its input :
: streams. The sequence of output packets (bottom) is determined by the :
: sequence of input packets (top) and their timestamps. The timestamps are :
: shown along the right side of the diagram.* :
| *Each time it receives a packet on its TICK input stream, the PacketClonerCalculator outputs the most recent packet from each of its input streams. The sequence of output packets (bottom) is determined by the sequence of input packets (top) and their timestamps. The timestamps are shown along the right side of the diagram.* |

20
docs/getting_started/building_examples.md
View File

@ -280,16 +280,16 @@ are two options:
2. In the project navigator in the left sidebar, select the "Mediapipe"
project.
3. Select the "Signing & Capabilities" tab.
3. Select one of the application targets, e.g. HandTrackingGpuApp.
4. Select one of the application targets, e.g. HandTrackingGpuApp.
4. Select the "Signing & Capabilities" tab.
5. Check "Automatically manage signing", and confirm the dialog box.
6. Select "_Your Name_ (Personal Team)" in the Team pop-up menu.
7. This set-up needs to be done once for each application you want to install.
Repeat steps 4-6 as needed.
Repeat steps 3-6 as needed.
This generates provisioning profiles for each app you have selected. Now we need
to tell Bazel to use them. We have provided a script to make this easier.
@ -390,9 +390,6 @@ developer (yourself) is trusted.
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu
```
This will open up your webcam as long as it is connected and on. Any errors
is likely due to your webcam being not accessible.
2. To run the application:
```bash
@ -400,6 +397,9 @@ developer (yourself) is trusted.
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt
```
This will open up your webcam as long as it is connected and on. Any errors
is likely due to your webcam being not accessible.
### Option 2: Running on GPU
Note: This currently works only on Linux, and please first follow
@ -412,13 +412,13 @@ Note: This currently works only on Linux, and please first follow
mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu
```
This will open up your webcam as long as it is connected and on. Any errors
is likely due to your webcam being not accessible, or GPU drivers not setup
properly.
2. To run the application:
```bash
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu \
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt
```
This will open up your webcam as long as it is connected and on. Any errors
is likely due to your webcam being not accessible, or GPU drivers not setup
properly.

BIN
docs/images/mobile/iris_tracking_android_gpu.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.4 MiB

BIN
docs/images/mobile/iris_tracking_android_gpu_small.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1019 KiB

BIN
docs/images/mobile/iris_tracking_depth_from_iris.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 10 MiB

BIN
docs/images/mobile/iris_tracking_example.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.8 MiB

BIN
docs/images/mobile/iris_tracking_eye_and_iris_landmarks.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 182 KiB

17
docs/index.md
View File

@ -22,13 +22,13 @@ desktop/cloud, web and IoT devices.
## ML solutions in MediaPipe
Face Detection | Face Mesh | Hands | Hair Segmentation
:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :---------------:
[![face_detection](images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![hand](images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands) | [![hair_segmentation](images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation)
Face Detection | Face Mesh | Iris | Hands
:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :---:
[![face_detection](images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands)
Object Detection | Box Tracking | Objectron | KNIFT
:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
[![object_detection](images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![objectron](images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift)
Hair Segmentation | Object Detection | Box Tracking | Objectron | KNIFT
:-------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---:
[![hair_segmentation](images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation) | [![object_detection](images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![objectron](images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift)
<!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. -->
<!-- Whenever this table is updated, paste a copy to solutions/solutions.md. -->
@ -37,6 +37,7 @@ Object Detection
:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :-: | :---:
[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | ✅ | ✅
[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | |
[Iris](https://google.github.io/mediapipe/solutions/iris) | ✅ | ✅ | ✅ | ✅ |
[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | ✅ |
[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | ✅ |
[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | ✅
@ -63,6 +64,8 @@ never leaves your device.
![visualizer_runner](images/visualizer_runner.png)
* [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection)
* [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
* [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
* [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
* [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
* [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
@ -83,6 +86,8 @@ run code search using
## Publications
* [MediaPipe Iris: Real-time Eye Tracking and Depth Estimation from a Single
Image](https://mediapipe.page.link/iris-blog) in Google AI Blog
* [MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html)
in Google Developers Blog
* [Alfred Camera: Smart camera features using MediaPipe](https://developers.googleblog.com/2020/03/alfred-camera-smart-camera-features-using-mediapipe.html)

2
docs/solutions/autoflip.md
View File

@ -2,7 +2,7 @@
layout: default
title: AutoFlip (Saliency-aware Video Cropping)
parent: Solutions
nav_order: 9
nav_order: 10
---
# AutoFlip: Saliency-aware Video Cropping

2
docs/solutions/box_tracking.md
View File

@ -2,7 +2,7 @@
layout: default
title: Box Tracking
parent: Solutions
nav_order: 6
nav_order: 7
---
# MediaPipe Box Tracking

6
docs/solutions/face_mesh.md
View File

@ -153,8 +153,8 @@ it, in the graph file modify the option of `ConstantSidePacketCalculator`.
[Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs](https://arxiv.org/abs/1907.06724)
([poster](https://docs.google.com/presentation/d/1-LWwOMO9TzEVdrZ1CS1ndJzciRHfYDJfbSxH_ke_JRg/present?slide=id.g5986dd4b4c_4_212))
* Face detection model:
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_front.tflite)
* Face landmark mode:
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_landmark.tflite),
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite)
* Face landmark model:
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark.tflite),
[TF.js model](https://tfhub.dev/mediapipe/facemesh/1)
* [Model card](https://drive.google.com/file/d/1VFC_wIpw4O7xBOiTgUldl79d9LA-LsnA/view)

2
docs/solutions/hair_segmentation.md
View File

@ -2,7 +2,7 @@
layout: default
title: Hair Segmentation
parent: Solutions
nav_order: 4
nav_order: 5
---
# MediaPipe Hair Segmentation

2
docs/solutions/hands.md
View File

@ -2,7 +2,7 @@
layout: default
title: Hands
parent: Solutions
nav_order: 3
nav_order: 4
---
# MediaPipe Hands

204
docs/solutions/iris.md Normal file
View File

@ -0,0 +1,204 @@
---
layout: default
title: Iris
parent: Solutions
nav_order: 3
---
# MediaPipe Iris
{: .no_toc }
1. TOC
{:toc}
---
## Overview
A wide range of real-world applications, including computational photography
(glint reflection) and augmented reality effects (virtual avatars) rely on
accurately tracking the iris within an eye. This is a challenging task to solve
on mobile devices, due to the limited computing resources, variable light
conditions and the presence of occlusions, such as hair or people squinting.
Iris tracking can also be utilized to determine the metric distance of the
camera to the user. This can improve a variety of use cases, ranging from
virtual try-on of properly sized glasses and hats to accessibility features that
adopt the font size depending on the viewer’s distance. Often, sophisticated
specialized hardware is employed to compute the metric distance, limiting the
range of devices on which the solution could be applied.
MediaPipe Iris is a ML solution for accurate iris estimation, able to track
landmarks involving the iris, pupil and the eye contours using a single RGB
camera, in real-time, without the need for specialized hardware. Through use of
iris landmarks, the solution is also able to determine the metric distance
between the subject and the camera with relative error less than 10%. Note that
iris tracking does not infer the location at which people are looking, nor does
it provide any form of identity recognition. With the cross-platfrom capability
of the MediaPipe framework, MediaPipe Iris can run on most modern
[mobile phones](#mobile), [desktops/laptops](#desktop) and even on the
[web](#web).
![iris_tracking_example.gif](../images/mobile/iris_tracking_example.gif) |
:------------------------------------------------------------------------: |
*Fig 1. Example of MediaPipe Iris: eyelid (red) and iris (blue) contours.* |
## ML Pipeline
The first step in the pipeline leverages [MediaPipe Face Mesh](./face_mesh.md),
which generates a mesh of the approximate face geometry. From this mesh, we
isolate the eye region in the original image for use in the subsequent iris
tracking step.
The pipeline is implemented as a MediaPipe
[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt)
that uses a
[face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
from the
[face landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark),
an
[iris landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_tracking/iris_landmark_left_and_right_gpu.pbtxt)
from the
[iris landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark),
and renders using a dedicated
[iris-and-depth renderer subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/iris_tracking/subgraphs/iris_and_depth_renderer_gpu.pbtxt).
The
[face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
internally uses a
[face detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front_gpu.pbtxt)
from the
[face detection module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection).
Note: To visualize a graph, copy the graph and paste it into
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
to visualize its associated subgraphs, please see
[visualizer documentation](../tools/visualizer.md).
## Models
### Face Detection Model
The face detector is the same [bazelFace](https://arxiv.org/abs/1907.05047)
model used in [MediaPipe Face Detection](./face_detection.md).
### Face Landmark Model
The face landmark model is the same as in [MediaPipe Face Mesh](./face_mesh.md).
You can also find more details in this
[paper](https://arxiv.org/abs/1907.06724).
### Iris Landmark Model
The iris model takes an image patch of the eye region and estimates both the eye
landmarks (along the eyelid) and iris landmarks (along ths iris contour). You
can find more details in this [paper](https://arxiv.org/abs/2006.11341).
![iris_tracking_eye_and_iris_landmarks.png](../images/mobile/iris_tracking_eye_and_iris_landmarks.png) |
:----------------------------------------------------------------------------------------------------: |
*Fig 2. Eye landmarks (red) and iris landmarks (green).* |
## Depth-from-Iris
MediaPipe Iris is able to determine the metric distance of a subject to the
camera with less than 10% error, without requiring any specialized hardware.
This is done by relying on the fact that the horizontal iris diameter of the
human eye remains roughly constant at 11.7±0.5 mm across a wide population,
along with some simple geometric arguments. For more details please refer to our
[Google AI Blog post](https://mediapipe.page.link/iris-blog).
![iris_tracking_depth_from_iris.gif](../images/mobile/iris_tracking_depth_from_iris.gif) |
:--------------------------------------------------------------------------------------------: |
*Fig 3. (Left) MediaPipe Iris predicting metric distance in cm on a Pixel 2 from iris tracking without use of a depth sensor. (Right) Ground-truth depth.* |
## Example Apps
Please first see general instructions for
[Android](../getting_started/building_examples.md#android),
[iOS](../getting_started/building_examples.md#ios) and
[desktop](../getting_started/building_examples.md#desktop) on how to build
MediaPipe examples.
Note: To visualize a graph, copy the graph and paste it into
[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how
to visualize its associated subgraphs, please see
[visualizer documentation](../tools/visualizer.md).
### Mobile
* Graph:
[`mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt)
* Android target:
[(or download prebuilt ARM64 APK)](https://drive.google.com/file/d/1cywcNtqk764TlZf1lvSTV4F3NGB2aL1R/view?usp=sharing)
[`mediapipe/examples/android/src/java/com/google/mediapipe/apps/iristrackinggpu:iristrackinggpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/iristrackinggpu/BUILD)
* iOS target:
[`mediapipe/examples/ios/iristrackinggpu:IrisTrackingGpuApp`](http:/mediapipe/examples/ios/iristrackinggpu/BUILD)
### Desktop
#### Live Camera Input
Please first see general instructions for
[desktop](../getting_started/building_examples.md#desktop) on how to build
MediaPipe examples.
* Running on CPU
* Graph:
[`mediapipe/graphs/iris_tracking/iris_tracking_cpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/iris_tracking/iris_tracking_cpu.pbtxt)
* Target:
[`mediapipe/examples/desktop/iris_tracking:iris_tracking_cpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/iris_tracking/BUILD)
* Running on GPU
* Graph:
[`mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt)
* Target:
[`mediapipe/examples/desktop/iris_tracking:iris_tracking_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/iris_tracking/BUILD)
#### Video File Input
1. To build the application, run:
```bash
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/iris_tracking:iris_tracking_cpu_video_input
```
2. To run the application, replace `<input video path>` and `<output video
path>` in the command below with your own paths:
```
bazel-bin/mediapipe/examples/desktop/iris_tracking/iris_tracking_cpu_video_input \
--calculator_graph_config_file=mediapipe/graphs/iris_tracking/iris_tracking_cpu_video_input.pbtxt \
--input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
```
#### Single-image Depth Estimation
1. To build the application, run:
```bash
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/iris_tracking:iris_depth_from_image_desktop
```
2. To run the application, replace `<input image path>` and `<output image
path>` in the command below with your own paths:
```bash
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/iris_tracking/iris_depth_from_image_desktop \
--input_image_path=<input image path> --output_image_path=<output image path>
```
### Web
Please refer to [these instructions](../index.md#mediapipe-on-the-web).
## Resources
* Google AI Blog: [MediaPipe Iris: Real-time Eye Tracking and Depth Estimation
from a Single Image](https://mediapipe.page.link/iris-blog)
* Paper:
[Real-time Pupil Tracking from Monocular Video for Digital Puppetry](https://arxiv.org/abs/2006.11341)
([presentation](https://youtu.be/cIhXkiiapQI))
* Face detection model:
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite)
* Face landmark model:
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark.tflite),
[TF.js model](https://tfhub.dev/mediapipe/facemesh/1)
* Iris landmark model:
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark/iris_landmark.tflite)
* [Model card](https://mediapipe.page.link/iris-mc)

2
docs/solutions/knift.md
View File

@ -2,7 +2,7 @@
layout: default
title: KNIFT (Template-based Feature Matching)
parent: Solutions
nav_order: 8
nav_order: 9
---
# MediaPipe KNIFT

2
docs/solutions/media_sequence.md
View File

@ -2,7 +2,7 @@
layout: default
title: Dataset Preparation with MediaSequence
parent: Solutions
nav_order: 10
nav_order: 11
---
# Dataset Preparation with MediaSequence

10
docs/solutions/object_detection.md
View File

@ -2,7 +2,7 @@
layout: default
title: Object Detection
parent: Solutions
nav_order: 5
nav_order: 6
---
# MediaPipe Object Detection
@ -95,8 +95,8 @@ Please first see general instructions for
```
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tflite \
--calculator_graph_config_file=mediapipe/graphs/object_detection/object_detection_desktop_tflite_graph.pbtxt \
--input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
--calculator_graph_config_file=mediapipe/graphs/object_detection/object_detection_desktop_tflite_graph.pbtxt \
--input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
```
* With a TensorFlow Model
@ -131,8 +131,8 @@ Please first see general instructions for
```bash
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tflite \
--calculator_graph_config_file=mediapipe/graphs/object_detection/object_detection_desktop_tensorflow_graph.pbtxt \
--input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
--calculator_graph_config_file=mediapipe/graphs/object_detection/object_detection_desktop_tensorflow_graph.pbtxt \
--input_side_packets=input_video_path=<input video path>,output_video_path=<output video path>
```
### Coral

2
docs/solutions/objectron.md
View File

@ -2,7 +2,7 @@
layout: default
title: Objectron (3D Object Detection)
parent: Solutions
nav_order: 7
nav_order: 8
---
# MediaPipe Objectron

3
docs/solutions/solutions.md
View File

@ -14,12 +14,13 @@ has_toc: false
---
<!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. -->
<!-- Whenever this table is updated, paste a copy to ../index.md. -->
<!-- Whenever this table is updated, paste a copy to ../external_index.md. -->
[]() | Android | iOS | Desktop | Web | Coral
:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :-: | :---:
[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | ✅ | ✅
[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | |
[Iris](https://google.github.io/mediapipe/solutions/iris) | ✅ | ✅ | ✅ | ✅ |
[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | ✅ |
[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | ✅ |
[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | ✅

2
docs/solutions/youtube_8m.md
View File

@ -2,7 +2,7 @@
layout: default
title: YouTube-8M Feature Extraction and Model Inference
parent: Solutions
nav_order: 11
nav_order: 12
---
# YouTube-8M Feature Extraction and Model Inference

5
mediapipe/MediaPipe.tulsiproj/Configs/MediaPipe.tulsigen
View File

@ -10,6 +10,7 @@
"mediapipe/examples/ios/facemeshgpu/BUILD",
"mediapipe/examples/ios/handdetectiongpu/BUILD",
"mediapipe/examples/ios/handtrackinggpu/BUILD",
"mediapipe/examples/ios/iristrackinggpu/BUILD",
"mediapipe/examples/ios/multihandtrackinggpu/BUILD",
"mediapipe/examples/ios/objectdetectioncpu/BUILD",
"mediapipe/examples/ios/objectdetectiongpu/BUILD"
@ -21,6 +22,7 @@
"//mediapipe/examples/ios/facemeshgpu:FaceMeshGpuApp",
"//mediapipe/examples/ios/handdetectiongpu:HandDetectionGpuApp",
"//mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp",
"//mediapipe/examples/ios/iristrackinggpu:IrisTrackingGpuApp",
"//mediapipe/examples/ios/multihandtrackinggpu:MultiHandTrackingGpuApp",
"//mediapipe/examples/ios/objectdetectioncpu:ObjectDetectionCpuApp",
"//mediapipe/examples/ios/objectdetectiongpu:ObjectDetectionGpuApp",
@ -88,6 +90,8 @@
"mediapipe/examples/ios/handdetectiongpu/Base.lproj",
"mediapipe/examples/ios/handtrackinggpu",
"mediapipe/examples/ios/handtrackinggpu/Base.lproj",
"mediapipe/examples/ios/iristrackinggpu",
"mediapipe/examples/ios/iristrackinggpu/Base.lproj",
"mediapipe/examples/ios/multihandtrackinggpu",
"mediapipe/examples/ios/multihandtrackinggpu/Base.lproj",
"mediapipe/examples/ios/objectdetectioncpu",
@ -110,6 +114,7 @@
"mediapipe/graphs/hand_tracking",
"mediapipe/graphs/object_detection",
"mediapipe/models",
"mediapipe/modules",
"mediapipe/objc",
"mediapipe/util",
"mediapipe/util/android",

1
mediapipe/MediaPipe.tulsiproj/project.tulsiconf
View File

@ -17,6 +17,7 @@
"mediapipe/examples/ios/facemeshgpu",
"mediapipe/examples/ios/handdetectiongpu",
"mediapipe/examples/ios/handtrackinggpu",
"mediapipe/examples/ios/iristrackinggpu",
"mediapipe/examples/ios/multihandtrackinggpu",
"mediapipe/examples/ios/objectdetectioncpu",
"mediapipe/examples/ios/objectdetectiongpu"

31
mediapipe/calculators/core/BUILD
View File

@ -316,6 +316,37 @@ cc_library(
alwayslink = 1,
)
cc_library(
name = "concatenate_normalized_landmark_list_calculator",
srcs = ["concatenate_normalized_landmark_list_calculator.cc"],
visibility = ["//visibility:public"],
deps = [
":concatenate_vector_calculator_cc_proto",
"//mediapipe/framework:calculator_framework",
"//mediapipe/framework/formats:landmark_cc_proto",
"//mediapipe/framework/port:ret_check",
"//mediapipe/framework/port:status",
],
alwayslink = 1,
)
cc_test(
name = "concatenate_normalized_landmark_list_calculator_test",
srcs = ["concatenate_normalized_landmark_list_calculator_test.cc"],
deps = [
":concatenate_normalized_landmark_list_calculator",
":concatenate_vector_calculator_cc_proto",
"//mediapipe/framework:calculator_framework",
"//mediapipe/framework:calculator_runner",
"//mediapipe/framework:timestamp",
"//mediapipe/framework/formats:landmark_cc_proto",
"//mediapipe/framework/port:gtest_main",
"//mediapipe/framework/port:parse_text_proto",
"//mediapipe/framework/port:status",
"@com_google_absl//absl/strings",
],
)
cc_test(
name = "concatenate_vector_calculator_test",
srcs = ["concatenate_vector_calculator_test.cc"],

84
mediapipe/calculators/core/concatenate_normalized_landmark_list_calculator.cc Normal file
View File

@ -0,0 +1,84 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#ifndef MEDIAPIPE_CALCULATORS_CORE_CONCATENATE_NORMALIZED_LIST_CALCULATOR_H_ // NOLINT
#define MEDIAPIPE_CALCULATORS_CORE_CONCATENATE_NORMALIZED_LIST_CALCULATOR_H_ // NOLINT
#include "mediapipe/calculators/core/concatenate_vector_calculator.pb.h"
#include "mediapipe/framework/calculator_framework.h"
#include "mediapipe/framework/formats/landmark.pb.h"
#include "mediapipe/framework/port/canonical_errors.h"
#include "mediapipe/framework/port/ret_check.h"
#include "mediapipe/framework/port/status.h"
namespace mediapipe {
// Concatenates several NormalizedLandmarkList protos following stream index
// order. This class assumes that every input stream contains a
// NormalizedLandmarkList proto object.
class ConcatenateNormalizedLandmarkListCalculator : public CalculatorBase {
public:
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
RET_CHECK(cc->Inputs().NumEntries() != 0);
RET_CHECK(cc->Outputs().NumEntries() == 1);
for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
cc->Inputs().Index(i).Set<NormalizedLandmarkList>();
}
cc->Outputs().Index(0).Set<NormalizedLandmarkList>();
return ::mediapipe::OkStatus();
}
::mediapipe::Status Open(CalculatorContext* cc) override {
cc->SetOffset(TimestampDiff(0));
only_emit_if_all_present_ =
cc->Options<::mediapipe::ConcatenateVectorCalculatorOptions>()
.only_emit_if_all_present();
return ::mediapipe::OkStatus();
}
::mediapipe::Status Process(CalculatorContext* cc) override {
if (only_emit_if_all_present_) {
for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
if (cc->Inputs().Index(i).IsEmpty()) return ::mediapipe::OkStatus();
}
}
NormalizedLandmarkList output;
for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
if (cc->Inputs().Index(i).IsEmpty()) continue;
const NormalizedLandmarkList& input =
cc->Inputs().Index(i).Get<NormalizedLandmarkList>();
for (int j = 0; j < input.landmark_size(); ++j) {
const NormalizedLandmark& input_landmark = input.landmark(j);
*output.add_landmark() = input_landmark;
}
}
cc->Outputs().Index(0).AddPacket(
MakePacket<NormalizedLandmarkList>(output).At(cc->InputTimestamp()));
return ::mediapipe::OkStatus();
}
private:
bool only_emit_if_all_present_;
};
REGISTER_CALCULATOR(ConcatenateNormalizedLandmarkListCalculator);
} // namespace mediapipe
// NOLINTNEXTLINE
#endif // MEDIAPIPE_CALCULATORS_CORE_CONCATENATE_NORMALIZED_LIST_CALCULATOR_H_

184
mediapipe/calculators/core/concatenate_normalized_landmark_list_calculator_test.cc Normal file
View File

@ -0,0 +1,184 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include <memory>
#include <string>
#include <vector>
#include "mediapipe/framework/calculator_framework.h"
#include "mediapipe/framework/calculator_runner.h"
#include "mediapipe/framework/formats/landmark.pb.h"
#include "mediapipe/framework/port/gmock.h"
#include "mediapipe/framework/port/gtest.h"
#include "mediapipe/framework/port/parse_text_proto.h"
#include "mediapipe/framework/port/status_matchers.h" // NOLINT
namespace mediapipe {
constexpr float kLocationValue = 3;
NormalizedLandmarkList GenerateLandmarks(int landmarks_size,
int value_multiplier) {
NormalizedLandmarkList landmarks;
for (int i = 0; i < landmarks_size; ++i) {
NormalizedLandmark* landmark = landmarks.add_landmark();
landmark->set_x(value_multiplier * kLocationValue);
landmark->set_y(value_multiplier * kLocationValue);
landmark->set_z(value_multiplier * kLocationValue);
}
return landmarks;
}
void ValidateCombinedLandmarks(
const std::vector<NormalizedLandmarkList>& inputs,
const NormalizedLandmarkList& result) {
int element_id = 0;
int expected_size = 0;
for (int i = 0; i < inputs.size(); ++i) {
const NormalizedLandmarkList& landmarks_i = inputs[i];
expected_size += landmarks_i.landmark_size();
for (int j = 0; j < landmarks_i.landmark_size(); ++j) {
const NormalizedLandmark& expected = landmarks_i.landmark(j);
const NormalizedLandmark& got = result.landmark(element_id);
EXPECT_FLOAT_EQ(expected.x(), got.x());
EXPECT_FLOAT_EQ(expected.y(), got.y());
EXPECT_FLOAT_EQ(expected.z(), got.z());
++element_id;
}
}
EXPECT_EQ(expected_size, result.landmark_size());
}
void AddInputLandmarkLists(
const std::vector<NormalizedLandmarkList>& input_landmarks_vec,
int64 timestamp, CalculatorRunner* runner) {
for (int i = 0; i < input_landmarks_vec.size(); ++i) {
runner->MutableInputs()->Index(i).packets.push_back(
MakePacket<NormalizedLandmarkList>(input_landmarks_vec[i])
.At(Timestamp(timestamp)));
}
}
TEST(ConcatenateNormalizedLandmarkListCalculatorTest, EmptyVectorInputs) {
CalculatorRunner runner("ConcatenateNormalizedLandmarkListCalculator",
/*options_string=*/"", /*num_inputs=*/3,
/*num_outputs=*/1, /*num_side_packets=*/0);
NormalizedLandmarkList empty_list;
std::vector<NormalizedLandmarkList> inputs = {empty_list, empty_list,
empty_list};
AddInputLandmarkLists(inputs, /*timestamp=*/1, &runner);
MP_ASSERT_OK(runner.Run());
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
EXPECT_EQ(1, outputs.size());
EXPECT_EQ(0, outputs[0].Get<NormalizedLandmarkList>().landmark_size());
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
}
TEST(ConcatenateNormalizedLandmarkListCalculatorTest, OneTimestamp) {
CalculatorRunner runner("ConcatenateNormalizedLandmarkListCalculator",
/*options_string=*/"", /*num_inputs=*/3,
/*num_outputs=*/1, /*num_side_packets=*/0);
NormalizedLandmarkList input_0 =
GenerateLandmarks(/*landmarks_size=*/3, /*value_multiplier=*/0);
NormalizedLandmarkList input_1 =
GenerateLandmarks(/*landmarks_size=*/1, /*value_multiplier=*/1);
NormalizedLandmarkList input_2 =
GenerateLandmarks(/*landmarks_size=*/2, /*value_multiplier=*/2);
std::vector<NormalizedLandmarkList> inputs = {input_0, input_1, input_2};
AddInputLandmarkLists(inputs, /*timestamp=*/1, &runner);
MP_ASSERT_OK(runner.Run());
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
EXPECT_EQ(1, outputs.size());
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
const NormalizedLandmarkList& result =
outputs[0].Get<NormalizedLandmarkList>();
ValidateCombinedLandmarks(inputs, result);
}
TEST(ConcatenateNormalizedLandmarkListCalculatorTest,
TwoInputsAtTwoTimestamps) {
CalculatorRunner runner("ConcatenateNormalizedLandmarkListCalculator",
/*options_string=*/"", /*num_inputs=*/3,
/*num_outputs=*/1, /*num_side_packets=*/0);
NormalizedLandmarkList input_0 =
GenerateLandmarks(/*landmarks_size=*/3, /*value_multiplier=*/0);
NormalizedLandmarkList input_1 =
GenerateLandmarks(/*landmarks_size=*/1, /*value_multiplier=*/1);
NormalizedLandmarkList input_2 =
GenerateLandmarks(/*landmarks_size=*/2, /*value_multiplier=*/2);
std::vector<NormalizedLandmarkList> inputs = {input_0, input_1, input_2};
{ AddInputLandmarkLists(inputs, /*timestamp=*/1, &runner); }
{ AddInputLandmarkLists(inputs, /*timestamp=*/2, &runner); }
MP_ASSERT_OK(runner.Run());
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
EXPECT_EQ(2, outputs.size());
{
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
const NormalizedLandmarkList& result =
outputs[0].Get<NormalizedLandmarkList>();
ValidateCombinedLandmarks(inputs, result);
}
{
EXPECT_EQ(Timestamp(2), outputs[1].Timestamp());
const NormalizedLandmarkList& result =
outputs[1].Get<NormalizedLandmarkList>();
ValidateCombinedLandmarks(inputs, result);
}
}
TEST(ConcatenateNormalizedLandmarkListCalculatorTest,
OneEmptyStreamStillOutput) {
CalculatorRunner runner("ConcatenateNormalizedLandmarkListCalculator",
/*options_string=*/"", /*num_inputs=*/2,
/*num_outputs=*/1, /*num_side_packets=*/0);
NormalizedLandmarkList input_0 =
GenerateLandmarks(/*landmarks_size=*/3, /*value_multiplier=*/0);
std::vector<NormalizedLandmarkList> inputs = {input_0};
AddInputLandmarkLists(inputs, /*timestamp=*/1, &runner);
MP_ASSERT_OK(runner.Run());
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
EXPECT_EQ(1, outputs.size());
EXPECT_EQ(Timestamp(1), outputs[0].Timestamp());
const NormalizedLandmarkList& result =
outputs[0].Get<NormalizedLandmarkList>();
ValidateCombinedLandmarks(inputs, result);
}
TEST(ConcatenateNormalizedLandmarkListCalculatorTest, OneEmptyStreamNoOutput) {
CalculatorRunner runner("ConcatenateNormalizedLandmarkListCalculator",
/*options_string=*/
"[mediapipe.ConcatenateVectorCalculatorOptions.ext]: "
"{only_emit_if_all_present: true}",
/*num_inputs=*/2,
/*num_outputs=*/1, /*num_side_packets=*/0);
NormalizedLandmarkList input_0 =
GenerateLandmarks(/*landmarks_size=*/3, /*value_multiplier=*/0);
std::vector<NormalizedLandmarkList> inputs = {input_0};
AddInputLandmarkLists(inputs, /*timestamp=*/1, &runner);
MP_ASSERT_OK(runner.Run());
const std::vector<Packet>& outputs = runner.Outputs().Index(0).packets;
EXPECT_EQ(0, outputs.size());
}
} // namespace mediapipe

31
mediapipe/calculators/image/BUILD
View File

@ -630,3 +630,34 @@ cc_library(
],
alwayslink = 1,
)
cc_library(
name = "image_file_properties_calculator",
srcs = ["image_file_properties_calculator.cc"],
visibility = ["//visibility:public"],
deps = [
"//mediapipe/framework:calculator_framework",
"//mediapipe/framework/formats:image_file_properties_cc_proto",
"//mediapipe/framework/port:ret_check",
"//mediapipe/framework/port:status",
"@easyexif",
],
alwayslink = 1,
)
cc_test(
name = "image_file_properties_calculator_test",
srcs = ["image_file_properties_calculator_test.cc"],
data = ["//mediapipe/calculators/image/testdata:test_images"],
deps = [
":image_file_properties_calculator",
"//mediapipe/framework:calculator_framework",
"//mediapipe/framework:calculator_runner",
"//mediapipe/framework/deps:file_path",
"//mediapipe/framework/formats:image_file_properties_cc_proto",
"//mediapipe/framework/port:file_helpers",
"//mediapipe/framework/port:gtest_main",
"//mediapipe/framework/port:parse_text_proto",
"//mediapipe/framework/port:status",
],
)

195
mediapipe/calculators/image/image_file_properties_calculator.cc Normal file
View File

@ -0,0 +1,195 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include <memory>
#include "exif.h"
#include "mediapipe/framework/calculator_framework.h"
#include "mediapipe/framework/formats/image_file_properties.pb.h"
#include "mediapipe/framework/port/canonical_errors.h"
#include "mediapipe/framework/port/status.h"
namespace mediapipe {
namespace {
// 35 MM sensor has dimensions 36 mm x 24 mm, so diagonal length is
// sqrt(36^2 + 24^2).
static const double SENSOR_DIAGONAL_35MM = std::sqrt(1872.0);
::mediapipe::StatusOr<double> ComputeFocalLengthInPixels(
int image_width, int image_height, double focal_length_35mm,
double focal_length_mm) {
// TODO: Allow returning image file properties even when focal length
// computation is not possible.
if (image_width == 0 || image_height == 0) {
return ::mediapipe::InternalError(
"Image dimensions should be non-zero to compute focal length in "
"pixels.");
}
if (focal_length_mm == 0) {
return ::mediapipe::InternalError(
"Focal length in mm should be non-zero to compute focal length in "
"pixels.");
}
if (focal_length_35mm == 0) {
return ::mediapipe::InternalError(
"Focal length in 35 mm should be non-zero to compute focal length in "
"pixels.");
}
// Derived from
// https://en.wikipedia.org/wiki/35_mm_equivalent_focal_length#Calculation.
/// Using focal_length_35mm = focal_length_mm * SENSOR_DIAGONAL_35MM /
/// sensor_diagonal_mm, we can calculate the diagonal length of the sensor in
/// millimeters i.e. sensor_diagonal_mm.
double sensor_diagonal_mm =
SENSOR_DIAGONAL_35MM / focal_length_35mm * focal_length_mm;
// Note that for the following computations, the longer dimension is treated
// as image width and the shorter dimension is treated as image height.
int width = image_width;
int height = image_height;
if (image_height > image_width) {
width = image_height;
height = image_width;
}
double inv_aspect_ratio = (double)height / width;
// Compute sensor width.
/// Using Pythagoras theorem, sensor_width^2 + sensor_height^2 =
/// sensor_diagonal_mm^2. We can substitute sensor_width / sensor_height with
/// the aspect ratio calculated in pixels to compute the sensor width.
double sensor_width = std::sqrt((sensor_diagonal_mm * sensor_diagonal_mm) /
(1.0 + inv_aspect_ratio * inv_aspect_ratio));
// Compute focal length in pixels.
double focal_length_pixels = width * focal_length_mm / sensor_width;
return focal_length_pixels;
}
::mediapipe::StatusOr<ImageFileProperties> GetImageFileProperites(
const std::string& image_bytes) {
easyexif::EXIFInfo result;
int code = result.parseFrom(image_bytes);
if (code) {
return ::mediapipe::InternalError("Error parsing EXIF, code: " +
std::to_string(code));
}
ImageFileProperties properties;
properties.set_image_width(result.ImageWidth);
properties.set_image_height(result.ImageHeight);
properties.set_focal_length_mm(result.FocalLength);
properties.set_focal_length_35mm(result.FocalLengthIn35mm);
ASSIGN_OR_RETURN(auto focal_length_pixels,
ComputeFocalLengthInPixels(properties.image_width(),
properties.image_height(),
properties.focal_length_35mm(),
properties.focal_length_mm()));
properties.set_focal_length_pixels(focal_length_pixels);
return properties;
}
} // namespace
// Calculator to extract EXIF information from an image file. The input is
// a std::string containing raw byte data from a file, and the output is an
// ImageFileProperties proto object with the relevant fields filled in.
// The calculator accepts the input as a stream or a side packet, and can output
// the result as a stream or a side packet. The calculator checks that if an
// output stream is present, it outputs to that stream, and if not, it checks if
// it can output to a side packet.
//
// Example config with input and output streams:
// node {
// calculator: "ImageFilePropertiesCalculator"
// input_stream: "image_bytes"
// output_stream: "image_properties"
// }
// Example config with input and output side packets:
// node {
// calculator: "ImageFilePropertiesCalculator"
// input_side_packet: "image_bytes"
// output_side_packet: "image_properties"
// }
class ImageFilePropertiesCalculator : public CalculatorBase {
public:
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
if (cc->Inputs().NumEntries() != 0) {
RET_CHECK(cc->Inputs().NumEntries() == 1);
cc->Inputs().Index(0).Set<std::string>();
} else {
RET_CHECK(cc->InputSidePackets().NumEntries() == 1);
cc->InputSidePackets().Index(0).Set<std::string>();
}
if (cc->Outputs().NumEntries() != 0) {
RET_CHECK(cc->Outputs().NumEntries() == 1);
cc->Outputs().Index(0).Set<::mediapipe::ImageFileProperties>();
} else {
RET_CHECK(cc->OutputSidePackets().NumEntries() == 1);
cc->OutputSidePackets().Index(0).Set<::mediapipe::ImageFileProperties>();
}
return ::mediapipe::OkStatus();
}
::mediapipe::Status Open(CalculatorContext* cc) override {
cc->SetOffset(TimestampDiff(0));
if (cc->InputSidePackets().NumEntries() == 1) {
const std::string& image_bytes =
cc->InputSidePackets().Index(0).Get<std::string>();
ASSIGN_OR_RETURN(properties_, GetImageFileProperites(image_bytes));
read_properties_ = true;
}
if (read_properties_ && cc->OutputSidePackets().NumEntries() == 1) {
cc->OutputSidePackets().Index(0).Set(
MakePacket<ImageFileProperties>(properties_));
}
return ::mediapipe::OkStatus();
}
::mediapipe::Status Process(CalculatorContext* cc) override {
if (cc->Inputs().NumEntries() == 1) {
if (cc->Inputs().Index(0).IsEmpty()) {
return ::mediapipe::OkStatus();
}
const std::string& image_bytes = cc->Inputs().Index(0).Get<std::string>();
ASSIGN_OR_RETURN(properties_, GetImageFileProperites(image_bytes));
read_properties_ = true;
}
if (read_properties_) {
if (cc->Outputs().NumEntries() == 1) {
cc->Outputs().Index(0).AddPacket(
MakePacket<ImageFileProperties>(properties_)
.At(cc->InputTimestamp()));
} else {
cc->OutputSidePackets().Index(0).Set(
MakePacket<ImageFileProperties>(properties_)
.At(::mediapipe::Timestamp::Unset()));
}
}
return ::mediapipe::OkStatus();
}
private:
ImageFileProperties properties_;
bool read_properties_ = false;
};
REGISTER_CALCULATOR(ImageFilePropertiesCalculator);
} // namespace mediapipe

134
mediapipe/calculators/image/image_file_properties_calculator_test.cc Normal file
View File

@ -0,0 +1,134 @@
// Copyright 2018 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include <math.h>
#include <cmath>
#include <limits>
#include "mediapipe/framework/calculator_framework.h"
#include "mediapipe/framework/calculator_runner.h"
#include "mediapipe/framework/deps/file_path.h"
#include "mediapipe/framework/formats/image_file_properties.pb.h"
#include "mediapipe/framework/port/file_helpers.h"
#include "mediapipe/framework/port/gmock.h"
#include "mediapipe/framework/port/gtest.h"
#include "mediapipe/framework/port/parse_text_proto.h"
#include "mediapipe/framework/port/status_matchers.h"
namespace mediapipe {
namespace {
constexpr char kImageFilePath[] =
"/mediapipe/calculators/image/testdata/"
"front_camera_pixel2.jpg";
constexpr int kExpectedWidth = 2448;
constexpr int kExpectedHeight = 3264;
constexpr double kExpectedFocalLengthMm = 3.38;
constexpr double kExpectedFocalLengthIn35Mm = 25;
constexpr double kExpectedFocalLengthPixels = 2357.48;
double RoundToNDecimals(double value, int n) {
return std::round(value * pow(10.0, n)) / pow(10.0, n);
}
TEST(ImageFilePropertiesCalculatorTest, ReadsFocalLengthFromJpegInStreams) {
std::string image_filepath = file::JoinPath("./", kImageFilePath);
std::string image_contents;
MP_ASSERT_OK(file::GetContents(image_filepath, &image_contents));
CalculatorGraphConfig::Node node_config =
ParseTextProtoOrDie<CalculatorGraphConfig::Node>(R"(
calculator: "ImageFilePropertiesCalculator"
input_stream: "image_bytes"
output_stream: "properties"
)");
CalculatorRunner runner(node_config);
runner.MutableInputs()->Index(0).packets.push_back(
MakePacket<std::string>(image_contents).At(Timestamp(0)));
MP_ASSERT_OK(runner.Run());
const auto& outputs = runner.Outputs();
ASSERT_EQ(1, outputs.NumEntries());
const std::vector<Packet>& packets = outputs.Index(0).packets;
ASSERT_EQ(1, packets.size());
const auto& result = packets[0].Get<::mediapipe::ImageFileProperties>();
EXPECT_EQ(kExpectedWidth, result.image_width());
EXPECT_EQ(kExpectedHeight, result.image_height());
EXPECT_DOUBLE_EQ(kExpectedFocalLengthMm, result.focal_length_mm());
EXPECT_DOUBLE_EQ(kExpectedFocalLengthIn35Mm, result.focal_length_35mm());
EXPECT_DOUBLE_EQ(kExpectedFocalLengthPixels,
RoundToNDecimals(result.focal_length_pixels(), /*n=*/2));
}
TEST(ImageFilePropertiesCalculatorTest, ReadsFocalLengthFromJpegInSidePackets) {
std::string image_filepath = file::JoinPath("./", kImageFilePath);
std::string image_contents;
MP_ASSERT_OK(file::GetContents(image_filepath, &image_contents));
CalculatorGraphConfig::Node node_config =
ParseTextProtoOrDie<CalculatorGraphConfig::Node>(R"(
calculator: "ImageFilePropertiesCalculator"
input_side_packet: "image_bytes"
output_side_packet: "properties"
)");
CalculatorRunner runner(node_config);
runner.MutableSidePackets()->Index(0) =
MakePacket<std::string>(image_contents).At(Timestamp(0));
MP_ASSERT_OK(runner.Run());
const auto& outputs = runner.OutputSidePackets();
EXPECT_EQ(1, outputs.NumEntries());
const auto& packet = outputs.Index(0);
const auto& result = packet.Get<::mediapipe::ImageFileProperties>();
EXPECT_EQ(kExpectedWidth, result.image_width());
EXPECT_EQ(kExpectedHeight, result.image_height());
EXPECT_DOUBLE_EQ(kExpectedFocalLengthMm, result.focal_length_mm());
EXPECT_DOUBLE_EQ(kExpectedFocalLengthIn35Mm, result.focal_length_35mm());
EXPECT_DOUBLE_EQ(kExpectedFocalLengthPixels,
RoundToNDecimals(result.focal_length_pixels(), /*n=*/2));
}
TEST(ImageFilePropertiesCalculatorTest,
ReadsFocalLengthFromJpegStreamToSidePacket) {
std::string image_filepath = file::JoinPath("./", kImageFilePath);
std::string image_contents;
MP_ASSERT_OK(file::GetContents(image_filepath, &image_contents));
CalculatorGraphConfig::Node node_config =
ParseTextProtoOrDie<CalculatorGraphConfig::Node>(R"(
calculator: "ImageFilePropertiesCalculator"
input_stream: "image_bytes"
output_side_packet: "properties"
)");
CalculatorRunner runner(node_config);
runner.MutableInputs()->Index(0).packets.push_back(
MakePacket<std::string>(image_contents).At(Timestamp(0)));
MP_ASSERT_OK(runner.Run());
const auto& outputs = runner.OutputSidePackets();
EXPECT_EQ(1, outputs.NumEntries());
const auto& packet = outputs.Index(0);
const auto& result = packet.Get<::mediapipe::ImageFileProperties>();
EXPECT_EQ(kExpectedWidth, result.image_width());
EXPECT_EQ(kExpectedHeight, result.image_height());
EXPECT_DOUBLE_EQ(kExpectedFocalLengthMm, result.focal_length_mm());
EXPECT_DOUBLE_EQ(kExpectedFocalLengthIn35Mm, result.focal_length_35mm());
EXPECT_DOUBLE_EQ(kExpectedFocalLengthPixels,
RoundToNDecimals(result.focal_length_pixels(), /*n=*/2));
}
} // namespace
} // namespace mediapipe

24
mediapipe/calculators/util/annotation_overlay_calculator.cc
View File

@ -160,8 +160,8 @@ class AnnotationOverlayCalculator : public CalculatorBase {
GLuint image_mat_tex_ = 0; // Overlay drawing image for GPU.
int width_ = 0;
int height_ = 0;
int width_gpu_ = 0; // Size of overlay drawing texture.
int height_gpu_ = 0;
int width_canvas_ = 0; // Size of overlay drawing texture canvas.
int height_canvas_ = 0;
#endif // MEDIAPIPE_DISABLE_GPU
};
REGISTER_CALCULATOR(AnnotationOverlayCalculator);
@ -250,6 +250,7 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
// Initialize the helper renderer library.
renderer_ = absl::make_unique<AnnotationRenderer>();
renderer_->SetFlipTextVertically(options_.flip_text_vertically());
if (use_gpu_) renderer_->SetScaleFactor(options_.gpu_scale_factor());
// Set the output header based on the input header (if present).
const char* input_tag = use_gpu_ ? kInputFrameTagGpu : kInputFrameTag;
@ -391,8 +392,8 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glBindTexture(GL_TEXTURE_2D, image_mat_tex_);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, width_gpu_, height_gpu_, GL_RGB,
GL_UNSIGNED_BYTE, overlay_image);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, width_canvas_, height_canvas_,
GL_RGB, GL_UNSIGNED_BYTE, overlay_image);
glBindTexture(GL_TEXTURE_2D, 0);
}
@ -494,12 +495,13 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
if (format != mediapipe::ImageFormat::SRGBA &&
format != mediapipe::ImageFormat::SRGB)
RET_CHECK_FAIL() << "Unsupported GPU input format: " << format;
image_mat = absl::make_unique<cv::Mat>(height_gpu_, width_gpu_, CV_8UC3);
image_mat =
absl::make_unique<cv::Mat>(height_canvas_, width_canvas_, CV_8UC3);
memset(image_mat->data, kAnnotationBackgroundColor,
height_gpu_ * width_gpu_ * image_mat->elemSize());
height_canvas_ * width_canvas_ * image_mat->elemSize());
} else {
image_mat = absl::make_unique<cv::Mat>(
height_gpu_, width_gpu_, CV_8UC3,
height_canvas_, width_canvas_, CV_8UC3,
cv::Scalar(options_.canvas_color().r(), options_.canvas_color().g(),
options_.canvas_color().b()));
}
@ -646,8 +648,8 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
width_ = RoundUp(options_.canvas_width_px(), alignment);
height_ = RoundUp(options_.canvas_height_px(), alignment);
}
width_gpu_ = RoundUp(width_ * scale_factor, alignment);
height_gpu_ = RoundUp(height_ * scale_factor, alignment);
width_canvas_ = RoundUp(width_ * scale_factor, alignment);
height_canvas_ = RoundUp(height_ * scale_factor, alignment);
// Init texture for opencv rendered frame.
{
@ -655,8 +657,8 @@ REGISTER_CALCULATOR(AnnotationOverlayCalculator);
glBindTexture(GL_TEXTURE_2D, image_mat_tex_);
// TODO
// OpenCV only renders to RGB images, not RGBA. Ideally this should be RGBA.
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB8, width_gpu_, height_gpu_, 0, GL_RGB,
GL_UNSIGNED_BYTE, nullptr);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB8, width_canvas_, height_canvas_, 0,
GL_RGB, GL_UNSIGNED_BYTE, nullptr);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);

2
mediapipe/calculators/util/annotation_overlay_calculator.proto
View File

@ -50,7 +50,5 @@ message AnnotationOverlayCalculatorOptions {
// This can be used to speed up annotation by drawing the annotation on an
// intermediate image with a reduced scale, e.g. 0.5 (of the input image width
// and height), before resizing and overlaying it on top of the input image.
// Should only be used if *all* render data uses normalized coordinates
// (or absolute coordinates are updated to scale accordingly).
optional float gpu_scale_factor = 7 [default = 1.0];
}

1
mediapipe/calculators/video/BUILD
View File

@ -316,6 +316,7 @@ cc_library(
"//mediapipe/util/tracking",
"//mediapipe/util/tracking:box_tracker",
"//mediapipe/util/tracking:tracking_visualization_utilities",
"@com_google_absl//absl/container:flat_hash_set",
"@com_google_absl//absl/container:node_hash_set",
"@com_google_absl//absl/strings",
],

13
mediapipe/calculators/video/box_tracker_calculator.cc
View File

@ -18,6 +18,7 @@
#include <unordered_map>
#include <unordered_set>
#include "absl/container/flat_hash_set.h"
#include "absl/container/node_hash_set.h"
#include "absl/strings/numbers.h"
#include "mediapipe/calculators/video/box_tracker_calculator.pb.h"
@ -238,6 +239,11 @@ class BoxTrackerCalculator : public CalculatorBase {
// Queued track time requests.
std::vector<Timestamp> queued_track_requests_;
// Stores the tracked ids that have been discarded actively, from continuous
// tracking data. It may accumulate across multiple frames. Once consumed, it
// should be cleared immediately.
absl::flat_hash_set<int> actively_discarded_tracked_ids_;
// Add smooth transition between re-acquisition and previous tracked boxes.
// `result_box` is the tracking result of one specific timestamp. The smoothed
// result will be updated in place.
@ -1144,9 +1150,16 @@ void BoxTrackerCalculator::StreamTrack(const TrackingData& data,
CHECK(box_map);
CHECK(failed_ids);
// Cache the actively discarded tracked ids from the new tracking data.
for (const int discarded_id :
data.motion_data().actively_discarded_tracked_ids()) {
actively_discarded_tracked_ids_.insert(discarded_id);
}
// Track all existing boxes by one frame.
MotionVectorFrame mvf; // Holds motion from current to previous frame.
MotionVectorFrameFromTrackingData(data, &mvf);
mvf.actively_discarded_tracked_ids = &actively_discarded_tracked_ids_;
if (forward) {
MotionVectorFrame mvf_inverted;

62
mediapipe/examples/android/src/java/com/google/mediapipe/apps/iristrackinggpu/BUILD Normal file
View File

@ -0,0 +1,62 @@
# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
licenses(["notice"]) # Apache 2.0
package(default_visibility = ["//visibility:private"])
cc_binary(
name = "libmediapipe_jni.so",
linkshared = 1,
linkstatic = 1,
deps = [
"//mediapipe/graphs/iris_tracking:iris_tracking_gpu_deps",
"//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
],
)
cc_library(
name = "mediapipe_jni_lib",
srcs = [":libmediapipe_jni.so"],
alwayslink = 1,
)
android_binary(
name = "iristrackinggpu",
srcs = glob(["*.java"]),
assets = [
"//mediapipe/graphs/iris_tracking:iris_tracking_gpu.binarypb",
"//mediapipe/modules/face_landmark:face_landmark.tflite",
"//mediapipe/modules/iris_landmark:iris_landmark.tflite",
"//mediapipe/modules/face_detection:face_detection_front.tflite",
],
assets_dir = "",
manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
manifest_values = {
"applicationId": "com.google.mediapipe.apps.iristrackinggpu",
"appName": "Iris Tracking",
"mainActivity": ".MainActivity",
"cameraFacingFront": "True",
"binaryGraphName": "iris_tracking_gpu.binarypb",
"inputVideoStreamName": "input_video",
"outputVideoStreamName": "output_video",
"flipFramesVertically": "True",
},
multidex = "native",
deps = [
":mediapipe_jni_lib",
"//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:basic_lib",
"//mediapipe/java/com/google/mediapipe/framework:android_framework",
],
)

40
mediapipe/examples/android/src/java/com/google/mediapipe/apps/iristrackinggpu/MainActivity.java Normal file
View File

@ -0,0 +1,40 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package com.google.mediapipe.apps.iristrackinggpu;
import android.graphics.SurfaceTexture;
import com.google.mediapipe.framework.Packet;
import java.util.HashMap;
import java.util.Map;
/** Main activity of MediaPipe iris tracking app. */
public class MainActivity extends com.google.mediapipe.apps.basic.MainActivity {
private static final String TAG = "MainActivity";
private static final String FOCAL_LENGTH_STREAM_NAME = "focal_length_pixel";
@Override
protected void onCameraStarted(SurfaceTexture surfaceTexture) {
super.onCameraStarted(surfaceTexture);
float focalLength = cameraHelper.getFocalLengthPixels();
if (focalLength != Float.MIN_VALUE) {
Packet focalLengthSidePacket = processor.getPacketCreator().createFloat32(focalLength);
Map<String, Packet> inputSidePackets = new HashMap<>();
inputSidePackets.put(FOCAL_LENGTH_STREAM_NAME, focalLengthSidePacket);
processor.setInputSidePackets(inputSidePackets);
}
}
}

53
mediapipe/examples/desktop/autoflip/calculators/content_zooming_calculator.cc
View File

@ -165,28 +165,53 @@ REGISTER_CALCULATOR(ContentZoomingCalculator);
}
namespace {
::mediapipe::Status UpdateRanges(const SalientRegion& region, float* xmin,
mediapipe::LocationData::RelativeBoundingBox ShiftDetection(
const mediapipe::LocationData::RelativeBoundingBox& relative_bounding_box,
const float y_offset_percent, const float x_offset_percent) {
auto shifted_bb = relative_bounding_box;
shifted_bb.set_ymin(relative_bounding_box.ymin() +
relative_bounding_box.height() * y_offset_percent);
shifted_bb.set_xmin(relative_bounding_box.xmin() +
relative_bounding_box.width() * x_offset_percent);
return shifted_bb;
}
mediapipe::autoflip::RectF ShiftDetection(
const mediapipe::autoflip::RectF& relative_bounding_box,
const float y_offset_percent, const float x_offset_percent) {
auto shifted_bb = relative_bounding_box;
shifted_bb.set_y(relative_bounding_box.y() +
relative_bounding_box.height() * y_offset_percent);
shifted_bb.set_x(relative_bounding_box.x() +
relative_bounding_box.width() * x_offset_percent);
return shifted_bb;
}
::mediapipe::Status UpdateRanges(const SalientRegion& region,
const float shift_vertical,
const float shift_horizontal, float* xmin,
float* xmax, float* ymin, float* ymax) {
if (!region.has_location_normalized()) {
return ::mediapipe::UnknownErrorBuilder(MEDIAPIPE_LOC)
<< "SalientRegion did not have location normalized set.";
}
*xmin = fmin(*xmin, region.location_normalized().x());
*xmax = fmax(*xmax, region.location_normalized().x() +
region.location_normalized().width());
*ymin = fmin(*ymin, region.location_normalized().y());
*ymax = fmax(*ymax, region.location_normalized().y() +
region.location_normalized().height());
auto location = ShiftDetection(region.location_normalized(), shift_vertical,
shift_horizontal);
*xmin = fmin(*xmin, location.x());
*xmax = fmax(*xmax, location.x() + location.width());
*ymin = fmin(*ymin, location.y());
*ymax = fmax(*ymax, location.y() + location.height());
return ::mediapipe::OkStatus();
}
::mediapipe::Status UpdateRanges(const mediapipe::Detection& detection,
float* xmin, float* xmax, float* ymin,
float* ymax) {
const float shift_vertical,
const float shift_horizontal, float* xmin,
float* xmax, float* ymin, float* ymax) {
RET_CHECK(detection.location_data().format() ==
mediapipe::LocationData::RELATIVE_BOUNDING_BOX)
<< "Face detection input is lacking required relative_bounding_box()";
const auto& location = detection.location_data().relative_bounding_box();
const auto& location =
ShiftDetection(detection.location_data().relative_bounding_box(),
shift_vertical, shift_horizontal);
*xmin = fmin(*xmin, location.xmin());
*xmax = fmax(*xmax, location.xmin() + location.width());
*ymin = fmin(*ymin, location.ymin());
@ -270,7 +295,9 @@ void MakeStaticFeatures(const int top_border, const int bottom_border,
continue;
}
only_required_found = true;
MP_RETURN_IF_ERROR(UpdateRanges(region, &xmin, &xmax, &ymin, &ymax));
MP_RETURN_IF_ERROR(UpdateRanges(
region, options_.detection_shift_vertical(),
options_.detection_shift_horizontal(), &xmin, &xmax, &ymin, &ymax));
}
}
@ -279,7 +306,9 @@ void MakeStaticFeatures(const int top_border, const int bottom_border,
cc->Inputs().Tag(kDetections).Get<std::vector<mediapipe::Detection>>();
for (const auto& detection : raw_detections) {
only_required_found = true;
MP_RETURN_IF_ERROR(UpdateRanges(detection, &xmin, &xmax, &ymin, &ymax));
MP_RETURN_IF_ERROR(UpdateRanges(
detection, options_.detection_shift_vertical(),
options_.detection_shift_horizontal(), &xmin, &xmax, &ymin, &ymax));
}
}

7
mediapipe/examples/desktop/autoflip/calculators/content_zooming_calculator.proto
View File

@ -19,6 +19,7 @@ package mediapipe.autoflip;
import "mediapipe/examples/desktop/autoflip/quality/kinematic_path_solver.proto";
import "mediapipe/framework/calculator.proto";
// NextTag: 13
message ContentZoomingCalculatorOptions {
extend mediapipe.CalculatorOptions {
optional ContentZoomingCalculatorOptions ext = 313091992;
@ -44,6 +45,12 @@ message ContentZoomingCalculatorOptions {
optional int64 height = 2;
}
optional Size target_size = 8;
// Amount to shift an input detection as a ratio of the size (positive:
// down/right, negative: up/left). Use a negative value to increase padding
// above/left of an object, positive to increase padding below/right of an
// object.
optional float detection_shift_vertical = 11 [default = 0.0];
optional float detection_shift_horizontal = 12 [default = 0.0];
// Deprecated parameters
optional KinematicOptions kinematic_options = 2 [deprecated = true];

39
mediapipe/examples/desktop/autoflip/calculators/content_zooming_calculator_test.cc
View File

@ -366,6 +366,45 @@ TEST(ContentZoomingCalculatorTest, ZoomTestNearInsideBorder) {
CheckCropRect(42, 42, 83, 83, 1, runner->Outputs().Tag("CROP_RECT").packets);
}
TEST(ContentZoomingCalculatorTest, VerticalShift) {
auto config = ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigD);
auto* options = config.mutable_options()->MutableExtension(
ContentZoomingCalculatorOptions::ext);
options->set_detection_shift_vertical(0.2);
auto runner = ::absl::make_unique<CalculatorRunner>(config);
AddDetection(cv::Rect_<float>(.1, .1, .1, .1), 0, runner.get());
MP_ASSERT_OK(runner->Run());
// 1000px * .1 offset + 1000*.1*.1 shift = 170
CheckCropRect(150, 170, 111, 111, 0,
runner->Outputs().Tag("CROP_RECT").packets);
}
TEST(ContentZoomingCalculatorTest, HorizontalShift) {
auto config = ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigD);
auto* options = config.mutable_options()->MutableExtension(
ContentZoomingCalculatorOptions::ext);
options->set_detection_shift_horizontal(0.2);
auto runner = ::absl::make_unique<CalculatorRunner>(config);
AddDetection(cv::Rect_<float>(.1, .1, .1, .1), 0, runner.get());
MP_ASSERT_OK(runner->Run());
// 1000px * .1 offset + 1000*.1*.1 shift = 170
CheckCropRect(170, 150, 111, 111, 0,
runner->Outputs().Tag("CROP_RECT").packets);
}
TEST(ContentZoomingCalculatorTest, ShiftOutsideBounds) {
auto config = ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigD);
auto* options = config.mutable_options()->MutableExtension(
ContentZoomingCalculatorOptions::ext);
options->set_detection_shift_vertical(-0.2);
options->set_detection_shift_horizontal(0.2);
auto runner = ::absl::make_unique<CalculatorRunner>(config);
AddDetection(cv::Rect_<float>(.9, 0, .1, .1), 0, runner.get());
MP_ASSERT_OK(runner->Run());
CheckCropRect(944, 56, 111, 111, 0,
runner->Outputs().Tag("CROP_RECT").packets);
}
} // namespace
} // namespace autoflip

60
mediapipe/examples/desktop/iris_tracking/BUILD Normal file
View File

@ -0,0 +1,60 @@
# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
licenses(["notice"]) # Apache 2.0
package(default_visibility = ["//mediapipe/examples:__subpackages__"])
cc_binary(
name = "iris_depth_from_image_desktop",
srcs = ["iris_depth_from_image_desktop.cc"],
deps = [
"//mediapipe/framework:calculator_framework",
"//mediapipe/framework/formats:image_frame",
"//mediapipe/framework/formats:image_frame_opencv",
"//mediapipe/framework/port:commandlineflags",
"//mediapipe/framework/port:file_helpers",
"//mediapipe/framework/port:opencv_highgui",
"//mediapipe/framework/port:opencv_imgproc",
"//mediapipe/framework/port:opencv_video",
"//mediapipe/framework/port:parse_text_proto",
"//mediapipe/framework/port:status",
"//mediapipe/graphs/iris_tracking:iris_depth_cpu_deps",
],
)
cc_binary(
name = "iris_tracking_cpu_video_input",
deps = [
"//mediapipe/examples/desktop:simple_run_graph_main",
"//mediapipe/graphs/iris_tracking:iris_tracking_cpu_video_input_deps",
],
)
cc_binary(
name = "iris_tracking_cpu",
deps = [
"//mediapipe/examples/desktop:demo_run_graph_main",
"//mediapipe/graphs/iris_tracking:iris_tracking_cpu_deps",
],
)
# Linux only
cc_binary(
name = "iris_tracking_gpu",
deps = [
"//mediapipe/examples/desktop:demo_run_graph_main_gpu",
"//mediapipe/graphs/iris_tracking:iris_tracking_gpu_deps",
],
)

162
mediapipe/examples/desktop/iris_tracking/iris_depth_from_image_desktop.cc Normal file
View File

@ -0,0 +1,162 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// A utility to extract iris depth from a single image of face using the graph
// mediapipe/graphs/iris_tracking/iris_depth_cpu.pbtxt.
#include <cstdlib>
#include <memory>
#include "mediapipe/framework/calculator_framework.h"
#include "mediapipe/framework/formats/image_frame.h"
#include "mediapipe/framework/formats/image_frame_opencv.h"
#include "mediapipe/framework/port/canonical_errors.h"
#include "mediapipe/framework/port/commandlineflags.h"
#include "mediapipe/framework/port/file_helpers.h"
#include "mediapipe/framework/port/opencv_highgui_inc.h"
#include "mediapipe/framework/port/opencv_imgproc_inc.h"
#include "mediapipe/framework/port/opencv_video_inc.h"
#include "mediapipe/framework/port/parse_text_proto.h"
#include "mediapipe/framework/port/status.h"
constexpr char kInputStream[] = "input_image_bytes";
constexpr char kOutputImageStream[] = "output_image";
constexpr char kLeftIrisDepthMmStream[] = "left_iris_depth_mm";
constexpr char kRightIrisDepthMmStream[] = "right_iris_depth_mm";
constexpr char kWindowName[] = "MediaPipe";
constexpr char kCalculatorGraphConfigFile[] =
"mediapipe/graphs/iris_tracking/iris_depth_cpu.pbtxt";
constexpr float kMicrosPerSecond = 1e6;
DEFINE_string(input_image_path, "",
"Full path of image to load. "
"If not provided, nothing will run.");
DEFINE_string(output_image_path, "",
"Full path of where to save image result (.jpg only). "
"If not provided, show result in a window.");
namespace {
::mediapipe::StatusOr<std::string> ReadFileToString(
const std::string& file_path) {
std::string contents;
MP_RETURN_IF_ERROR(::mediapipe::file::GetContents(file_path, &contents));
return contents;
}
::mediapipe::Status ProcessImage(
std::unique_ptr<::mediapipe::CalculatorGraph> graph) {
LOG(INFO) << "Load the image.";
ASSIGN_OR_RETURN(const std::string raw_image,
ReadFileToString(FLAGS_input_image_path));
LOG(INFO) << "Start running the calculator graph.";
ASSIGN_OR_RETURN(::mediapipe::OutputStreamPoller output_image_poller,
graph->AddOutputStreamPoller(kOutputImageStream));
ASSIGN_OR_RETURN(::mediapipe::OutputStreamPoller left_iris_depth_poller,
graph->AddOutputStreamPoller(kLeftIrisDepthMmStream));
ASSIGN_OR_RETURN(::mediapipe::OutputStreamPoller right_iris_depth_poller,
graph->AddOutputStreamPoller(kRightIrisDepthMmStream));
MP_RETURN_IF_ERROR(graph->StartRun({}));
// Send image packet into the graph.
const size_t fake_timestamp_us = (double)cv::getTickCount() /
(double)cv::getTickFrequency() *
kMicrosPerSecond;
MP_RETURN_IF_ERROR(graph->AddPacketToInputStream(
kInputStream, ::mediapipe::MakePacket<std::string>(raw_image).At(
::mediapipe::Timestamp(fake_timestamp_us))));
// Get the graph result packets, or stop if that fails.
::mediapipe::Packet left_iris_depth_packet;
if (!left_iris_depth_poller.Next(&left_iris_depth_packet)) {
return ::mediapipe::UnknownError(
"Failed to get packet from output stream 'left_iris_depth_mm'.");
}
const auto& left_iris_depth_mm = left_iris_depth_packet.Get<float>();
const int left_iris_depth_cm = std::round(left_iris_depth_mm / 10);
std::cout << "Left Iris Depth: " << left_iris_depth_cm << " cm." << std::endl;
::mediapipe::Packet right_iris_depth_packet;
if (!right_iris_depth_poller.Next(&right_iris_depth_packet)) {
return ::mediapipe::UnknownError(
"Failed to get packet from output stream 'right_iris_depth_mm'.");
}
const auto& right_iris_depth_mm = right_iris_depth_packet.Get<float>();
const int right_iris_depth_cm = std::round(right_iris_depth_mm / 10);
std::cout << "Right Iris Depth: " << right_iris_depth_cm << " cm."
<< std::endl;
::mediapipe::Packet output_image_packet;
if (!output_image_poller.Next(&output_image_packet)) {
return ::mediapipe::UnknownError(
"Failed to get packet from output stream 'output_image'.");
}
auto& output_frame = output_image_packet.Get<::mediapipe::ImageFrame>();
// Convert back to opencv for display or saving.
cv::Mat output_frame_mat = ::mediapipe::formats::MatView(&output_frame);
cv::cvtColor(output_frame_mat, output_frame_mat, cv::COLOR_RGB2BGR);
const bool save_image = !FLAGS_output_image_path.empty();
if (save_image) {
LOG(INFO) << "Saving image to file...";
cv::imwrite(FLAGS_output_image_path, output_frame_mat);
} else {
cv::namedWindow(kWindowName, /*flags=WINDOW_AUTOSIZE*/ 1);
cv::imshow(kWindowName, output_frame_mat);
// Press any key to exit.
cv::waitKey(0);
}
LOG(INFO) << "Shutting down.";
MP_RETURN_IF_ERROR(graph->CloseInputStream(kInputStream));
return graph->WaitUntilDone();
}
::mediapipe::Status RunMPPGraph() {
std::string calculator_graph_config_contents;
MP_RETURN_IF_ERROR(::mediapipe::file::GetContents(
kCalculatorGraphConfigFile, &calculator_graph_config_contents));
LOG(INFO) << "Get calculator graph config contents: "
<< calculator_graph_config_contents;
::mediapipe::CalculatorGraphConfig config =
::mediapipe::ParseTextProtoOrDie<::mediapipe::CalculatorGraphConfig>(
calculator_graph_config_contents);
LOG(INFO) << "Initialize the calculator graph.";
std::unique_ptr<::mediapipe::CalculatorGraph> graph =
absl::make_unique<::mediapipe::CalculatorGraph>();
MP_RETURN_IF_ERROR(graph->Initialize(config));
const bool load_image = !FLAGS_input_image_path.empty();
if (load_image) {
return ProcessImage(std::move(graph));
} else {
return ::mediapipe::InvalidArgumentError("Missing image file.");
}
}
} // namespace
int main(int argc, char** argv) {
google::InitGoogleLogging(argv[0]);
gflags::ParseCommandLineFlags(&argc, &argv, true);
::mediapipe::Status run_status = RunMPPGraph();
if (!run_status.ok()) {
LOG(ERROR) << "Failed to run the graph: " << run_status.message();
return EXIT_FAILURE;
} else {
LOG(INFO) << "Success!";
}
return EXIT_SUCCESS;
}

12
mediapipe/examples/ios/edgedetectiongpu/Base.lproj/Main.storyboard
View File

@ -1,10 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait">
<adaptation id="fullscreen"/>
</device>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="16097.2" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait" appearance="light"/>
<dependencies>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="16087"/>
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
</dependencies>
@ -18,11 +16,11 @@
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
<rect key="frame" x="0.0" y="20" width="375" height="647"/>
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="emf-N5-sEd">
<rect key="frame" x="57" y="248" width="260" height="151"/>
<rect key="frame" x="57" y="258" width="260" height="151"/>
<autoresizingMask key="autoresizingMask" flexibleMinX="YES" flexibleMaxX="YES" flexibleMinY="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="17"/>
<color key="textColor" white="1" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>

18
mediapipe/examples/ios/facedetectioncpu/Base.lproj/Main.storyboard
View File

@ -1,10 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait">
<adaptation id="fullscreen"/>
</device>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="16097.2" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait" appearance="light"/>
<dependencies>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="16087"/>
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
</dependencies>
@ -18,11 +16,11 @@
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
<rect key="frame" x="0.0" y="20" width="375" height="647"/>
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="GOf-1M-7j5">
<rect key="frame" x="57.478260869565474" y="248.17580340264635" width="260" height="151.00000000000014"/>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="emf-N5-sEd">
<rect key="frame" x="57" y="258" width="260" height="151"/>
<autoresizingMask key="autoresizingMask" flexibleMinX="YES" flexibleMaxX="YES" flexibleMinY="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="17"/>
<color key="textColor" white="1" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>
@ -39,8 +37,8 @@
<viewLayoutGuide key="safeArea" id="6Tk-OE-BBY"/>
</view>
<connections>
<outlet property="_liveView" destination="EfB-xq-knP" id="wac-VF-etz"/>
<outlet property="_noCameraLabel" destination="GOf-1M-7j5" id="gL5-Oy-Gkr"/>
<outlet property="_liveView" destination="EfB-xq-knP" id="JQp-2n-q9q"/>
<outlet property="_noCameraLabel" destination="emf-N5-sEd" id="91G-3Z-cU3"/>
</connections>
</viewController>
<placeholder placeholderIdentifier="IBFirstResponder" id="dkx-z0-nzr" sceneMemberID="firstResponder"/>

4
mediapipe/examples/ios/facedetectioncpu/ViewController.mm
View File

@ -92,8 +92,6 @@ static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
_renderer.layer.frame = _liveView.layer.bounds;
[_liveView.layer addSublayer:_renderer.layer];
_renderer.frameScaleMode = MPPFrameScaleModeFillAndCrop;
// When using the front camera, mirror the input for a more natural look.
_renderer.mirrored = YES;
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
@ -105,6 +103,8 @@ static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
_cameraSource.cameraPosition = AVCaptureDevicePositionFront;
// The frame's native format is rotated with respect to the portrait orientation.
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
// When using the front camera, mirror the input for a more natural look.
_cameraSource.videoMirrored = YES;
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
self.mediapipeGraph.delegate = self;

12
mediapipe/examples/ios/facedetectiongpu/Base.lproj/Main.storyboard
View File

@ -1,10 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait">
<adaptation id="fullscreen"/>
</device>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="16097.2" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait" appearance="light"/>
<dependencies>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="16087"/>
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
</dependencies>
@ -18,11 +16,11 @@
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
<rect key="frame" x="0.0" y="20" width="375" height="647"/>
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="emf-N5-sEd">
<rect key="frame" x="57" y="248" width="260" height="151"/>
<rect key="frame" x="57" y="258" width="260" height="151"/>
<autoresizingMask key="autoresizingMask" flexibleMinX="YES" flexibleMaxX="YES" flexibleMinY="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="17"/>
<color key="textColor" white="1" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>

4
mediapipe/examples/ios/facedetectiongpu/ViewController.mm
View File

@ -92,8 +92,6 @@ static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
_renderer.layer.frame = _liveView.layer.bounds;
[_liveView.layer addSublayer:_renderer.layer];
_renderer.frameScaleMode = MPPFrameScaleModeFillAndCrop;
// When using the front camera, mirror the input for a more natural look.
_renderer.mirrored = YES;
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
@ -105,6 +103,8 @@ static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
_cameraSource.cameraPosition = AVCaptureDevicePositionFront;
// The frame's native format is rotated with respect to the portrait orientation.
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
// When using the front camera, mirror the input for a more natural look.
_cameraSource.videoMirrored = YES;
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
self.mediapipeGraph.delegate = self;

12
mediapipe/examples/ios/facemeshgpu/Base.lproj/Main.storyboard
View File

@ -1,10 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait">
<adaptation id="fullscreen"/>
</device>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="16097.2" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait" appearance="light"/>
<dependencies>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="16087"/>
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
</dependencies>
@ -18,11 +16,11 @@
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
<rect key="frame" x="0.0" y="20" width="375" height="647"/>
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="emf-N5-sEd">
<rect key="frame" x="57" y="248" width="260" height="151"/>
<rect key="frame" x="57" y="258" width="260" height="151"/>
<autoresizingMask key="autoresizingMask" flexibleMinX="YES" flexibleMaxX="YES" flexibleMinY="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="17"/>
<color key="textColor" white="1" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>

4
mediapipe/examples/ios/facemeshgpu/ViewController.mm
View File

@ -101,8 +101,6 @@ static const int kNumFaces = 1;
_renderer.layer.frame = _liveView.layer.bounds;
[_liveView.layer addSublayer:_renderer.layer];
_renderer.frameScaleMode = MPPFrameScaleModeFillAndCrop;
// When using the front camera, mirror the input for a more natural look.
_renderer.mirrored = YES;
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
@ -114,6 +112,8 @@ static const int kNumFaces = 1;
_cameraSource.cameraPosition = AVCaptureDevicePositionFront;
// The frame's native format is rotated with respect to the portrait orientation.
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
// When using the front camera, mirror the input for a more natural look.
_cameraSource.videoMirrored = YES;
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
self.mediapipeGraph.delegate = self;

12
mediapipe/examples/ios/handdetectiongpu/Base.lproj/Main.storyboard
View File

@ -1,10 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait">
<adaptation id="fullscreen"/>
</device>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="16097.2" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait" appearance="light"/>
<dependencies>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="16087"/>
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
</dependencies>
@ -18,11 +16,11 @@
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
<rect key="frame" x="0.0" y="20" width="375" height="647"/>
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="emf-N5-sEd">
<rect key="frame" x="57" y="248" width="260" height="151"/>
<rect key="frame" x="57" y="258" width="260" height="151"/>
<autoresizingMask key="autoresizingMask" flexibleMinX="YES" flexibleMaxX="YES" flexibleMinY="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="17"/>
<color key="textColor" white="1" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>

4
mediapipe/examples/ios/handdetectiongpu/ViewController.mm
View File

@ -92,8 +92,6 @@ static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
_renderer.layer.frame = _liveView.layer.bounds;
[_liveView.layer addSublayer:_renderer.layer];
_renderer.frameScaleMode = MPPFrameScaleModeFillAndCrop;
// When using the front camera, mirror the input for a more natural look.
_renderer.mirrored = YES;
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
@ -105,6 +103,8 @@ static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
_cameraSource.cameraPosition = AVCaptureDevicePositionFront;
// The frame's native format is rotated with respect to the portrait orientation.
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
// When using the front camera, mirror the input for a more natural look.
_cameraSource.videoMirrored = YES;
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
self.mediapipeGraph.delegate = self;

12
mediapipe/examples/ios/handtrackinggpu/Base.lproj/Main.storyboard
View File

@ -1,10 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait">
<adaptation id="fullscreen"/>
</device>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="16097.2" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait" appearance="light"/>
<dependencies>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="16087"/>
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
</dependencies>
@ -18,11 +16,11 @@
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
<rect key="frame" x="0.0" y="20" width="375" height="647"/>
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="emf-N5-sEd">
<rect key="frame" x="57" y="248" width="260" height="151"/>
<rect key="frame" x="57" y="258" width="260" height="151"/>
<autoresizingMask key="autoresizingMask" flexibleMinX="YES" flexibleMaxX="YES" flexibleMinY="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="17"/>
<color key="textColor" white="1" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>

4
mediapipe/examples/ios/handtrackinggpu/ViewController.mm
View File

@ -96,8 +96,6 @@ static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
_renderer.layer.frame = _liveView.layer.bounds;
[_liveView.layer addSublayer:_renderer.layer];
_renderer.frameScaleMode = MPPFrameScaleModeFillAndCrop;
// When using the front camera, mirror the input for a more natural look.
_renderer.mirrored = YES;
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
@ -109,6 +107,8 @@ static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
_cameraSource.cameraPosition = AVCaptureDevicePositionFront;
// The frame's native format is rotated with respect to the portrait orientation.
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
// When using the front camera, mirror the input for a more natural look.
_cameraSource.videoMirrored = YES;
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
self.mediapipeGraph.delegate = self;

21
mediapipe/examples/ios/iristrackinggpu/AppDelegate.h Normal file
View File

@ -0,0 +1,21 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#import <UIKit/UIKit.h>
@interface AppDelegate : UIResponder <UIApplicationDelegate>
@property(strong, nonatomic) UIWindow *window;
@end

59
mediapipe/examples/ios/iristrackinggpu/AppDelegate.m Normal file
View File

@ -0,0 +1,59 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#import "AppDelegate.h"
@interface AppDelegate ()
@end
@implementation AppDelegate
- (BOOL)application:(UIApplication *)application
didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
// Override point for customization after application launch.
return YES;
}
- (void)applicationWillResignActive:(UIApplication *)application {
// Sent when the application is about to move from active to inactive state. This can occur for
// certain types of temporary interruptions (such as an incoming phone call or SMS message) or
// when the user quits the application and it begins the transition to the background state. Use
// this method to pause ongoing tasks, disable timers, and invalidate graphics rendering
// callbacks. Games should use this method to pause the game.
}
- (void)applicationDidEnterBackground:(UIApplication *)application {
// Use this method to release shared resources, save user data, invalidate timers, and store
// enough application state information to restore your application to its current state in case
// it is terminated later. If your application supports background execution, this method is
// called instead of applicationWillTerminate: when the user quits.
}
- (void)applicationWillEnterForeground:(UIApplication *)application {
// Called as part of the transition from the background to the active state; here you can undo
// many of the changes made on entering the background.
}
- (void)applicationDidBecomeActive:(UIApplication *)application {
// Restart any tasks that were paused (or not yet started) while the application was inactive. If
// the application was previously in the background, optionally refresh the user interface.
}
- (void)applicationWillTerminate:(UIApplication *)application {
// Called when the application is about to terminate. Save data if appropriate. See also
// applicationDidEnterBackground:.
}
@end

99
mediapipe/examples/ios/iristrackinggpu/Assets.xcassets/AppIcon.appiconset/Contents.json Normal file
View File

@ -0,0 +1,99 @@
{
"images" : [
{
"idiom" : "iphone",
"size" : "20x20",
"scale" : "2x"
},
{
"idiom" : "iphone",
"size" : "20x20",
"scale" : "3x"
},
{
"idiom" : "iphone",
"size" : "29x29",
"scale" : "2x"
},
{
"idiom" : "iphone",
"size" : "29x29",
"scale" : "3x"
},
{
"idiom" : "iphone",
"size" : "40x40",
"scale" : "2x"
},
{
"idiom" : "iphone",
"size" : "40x40",
"scale" : "3x"
},
{
"idiom" : "iphone",
"size" : "60x60",
"scale" : "2x"
},
{
"idiom" : "iphone",
"size" : "60x60",
"scale" : "3x"
},
{
"idiom" : "ipad",
"size" : "20x20",
"scale" : "1x"
},
{
"idiom" : "ipad",
"size" : "20x20",
"scale" : "2x"
},
{
"idiom" : "ipad",
"size" : "29x29",
"scale" : "1x"
},
{
"idiom" : "ipad",
"size" : "29x29",
"scale" : "2x"
},
{
"idiom" : "ipad",
"size" : "40x40",
"scale" : "1x"
},
{
"idiom" : "ipad",
"size" : "40x40",
"scale" : "2x"
},
{
"idiom" : "ipad",
"size" : "76x76",
"scale" : "1x"
},
{
"idiom" : "ipad",
"size" : "76x76",
"scale" : "2x"
},
{
"idiom" : "ipad",
"size" : "83.5x83.5",
"scale" : "2x"
},
{
"idiom" : "ios-marketing",
"size" : "1024x1024",
"scale" : "1x"
}
],
"info" : {
"version" : 1,
"author" : "xcode"
}
}

7
mediapipe/examples/ios/iristrackinggpu/Assets.xcassets/Contents.json Normal file
View File

@ -0,0 +1,7 @@
{
"info" : {
"version" : 1,
"author" : "xcode"
}
}

87
mediapipe/examples/ios/iristrackinggpu/BUILD Normal file
View File

@ -0,0 +1,87 @@
# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
load(
"@build_bazel_rules_apple//apple:ios.bzl",
"ios_application",
)
load(
"//mediapipe/examples/ios:bundle_id.bzl",
"BUNDLE_ID_PREFIX",
"example_provisioning",
)
licenses(["notice"]) # Apache 2.0
MIN_IOS_VERSION = "10.0"
alias(
name = "iristrackinggpu",
actual = "IrisTrackingGpuApp",
)
ios_application(
name = "IrisTrackingGpuApp",
bundle_id = BUNDLE_ID_PREFIX + ".IrisTrackingGpu",
families = [
"iphone",
"ipad",
],
infoplists = ["Info.plist"],
minimum_os_version = MIN_IOS_VERSION,
provisioning_profile = example_provisioning(),
deps = [
":IrisTrackingGpuAppLibrary",
"@ios_opencv//:OpencvFramework",
],
)
objc_library(
name = "IrisTrackingGpuAppLibrary",
srcs = [
"AppDelegate.m",
"ViewController.mm",
"main.m",
],
hdrs = [
"AppDelegate.h",
"ViewController.h",
],
data = [
"Base.lproj/LaunchScreen.storyboard",
"Base.lproj/Main.storyboard",
"//mediapipe/graphs/iris_tracking:iris_tracking_gpu.binarypb",
"//mediapipe/modules/face_detection:face_detection_front.tflite",
"//mediapipe/modules/face_landmark:face_landmark.tflite",
"//mediapipe/modules/iris_landmark:iris_landmark.tflite",
],
sdk_frameworks = [
"AVFoundation",
"CoreGraphics",
"CoreMedia",
"UIKit",
],
deps = [
"//mediapipe/objc:mediapipe_framework_ios",
"//mediapipe/objc:mediapipe_input_sources_ios",
"//mediapipe/objc:mediapipe_layer_renderer",
] + select({
"//mediapipe:ios_i386": [],
"//mediapipe:ios_x86_64": [],
"//conditions:default": [
"//mediapipe/graphs/iris_tracking:iris_tracking_gpu_deps",
"//mediapipe/framework/formats:landmark_cc_proto",
],
}),
)

25
mediapipe/examples/ios/iristrackinggpu/Base.lproj/LaunchScreen.storyboard Normal file
View File

@ -0,0 +1,25 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="13122.16" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" launchScreen="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="01J-lp-oVM">
<dependencies>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="13104.12"/>
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
</dependencies>
<scenes>
<!--View Controller-->
<scene sceneID="EHf-IW-A2E">
<objects>
<viewController id="01J-lp-oVM" sceneMemberID="viewController">
<view key="view" contentMode="scaleToFill" id="Ze5-6b-2t3">
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<color key="backgroundColor" red="1" green="1" blue="1" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
<viewLayoutGuide key="safeArea" id="6Tk-OE-BBY"/>
</view>
</viewController>
<placeholder placeholderIdentifier="IBFirstResponder" id="iYj-Kq-Ea1" userLabel="First Responder" sceneMemberID="firstResponder"/>
</objects>
<point key="canvasLocation" x="53" y="375"/>
</scene>
</scenes>
</document>

49
mediapipe/examples/ios/iristrackinggpu/Base.lproj/Main.storyboard Normal file
View File

@ -0,0 +1,49 @@
<?xml version="1.0" encoding="UTF-8"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="16097.2" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait" appearance="light"/>
<dependencies>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="16087"/>
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
</dependencies>
<scenes>
<!--View Controller-->
<scene sceneID="tne-QT-ifu">
<objects>
<viewController id="BYZ-38-t0r" customClass="ViewController" sceneMemberID="viewController">
<view key="view" contentMode="scaleToFill" id="8bC-Xf-vdC">
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="emf-N5-sEd">
<rect key="frame" x="57" y="258" width="260" height="151"/>
<autoresizingMask key="autoresizingMask" flexibleMinX="YES" flexibleMaxX="YES" flexibleMinY="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="17"/>
<color key="textColor" white="1" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>
<nil key="highlightedColor"/>
</label>
</subviews>
<color key="backgroundColor" white="0.0" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>
<accessibility key="accessibilityConfiguration" label="PreviewDisplayView">
<bool key="isElement" value="YES"/>
</accessibility>
</view>
</subviews>
<color key="backgroundColor" red="1" green="1" blue="1" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
<viewLayoutGuide key="safeArea" id="6Tk-OE-BBY"/>
</view>
<connections>
<outlet property="_liveView" destination="EfB-xq-knP" id="JQp-2n-q9q"/>
<outlet property="_noCameraLabel" destination="emf-N5-sEd" id="91G-3Z-cU3"/>
</connections>
</viewController>
<placeholder placeholderIdentifier="IBFirstResponder" id="dkx-z0-nzr" sceneMemberID="firstResponder"/>
</objects>
<point key="canvasLocation" x="48.799999999999997" y="20.239880059970016"/>
</scene>
</scenes>
</document>

42
mediapipe/examples/ios/iristrackinggpu/Info.plist Normal file
View File

@ -0,0 +1,42 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>NSCameraUsageDescription</key>
<string>This app uses the camera to demonstrate live video processing.</string>
<key>CFBundleDevelopmentRegion</key>
<string>en</string>
<key>CFBundleExecutable</key>
<string>$(EXECUTABLE_NAME)</string>
<key>CFBundleIdentifier</key>
<string>$(PRODUCT_BUNDLE_IDENTIFIER)</string>
<key>CFBundleInfoDictionaryVersion</key>
<string>6.0</string>
<key>CFBundleName</key>
<string>$(PRODUCT_NAME)</string>
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
<string>1.0</string>
<key>CFBundleVersion</key>
<string>1</string>
<key>LSRequiresIPhoneOS</key>
<true/>
<key>UILaunchStoryboardName</key>
<string>LaunchScreen</string>
<key>UIMainStoryboardFile</key>
<string>Main</string>
<key>UIRequiredDeviceCapabilities</key>
<array>
<string>armv7</string>
</array>
<key>UISupportedInterfaceOrientations</key>
<array>
<string>UIInterfaceOrientationPortrait</string>
</array>
<key>UISupportedInterfaceOrientations~ipad</key>
<array>
<string>UIInterfaceOrientationPortrait</string>
</array>
</dict>
</plist>

19
mediapipe/examples/ios/iristrackinggpu/ViewController.h Normal file
View File

@ -0,0 +1,19 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#import <UIKit/UIKit.h>
@interface ViewController : UIViewController
@end

216
mediapipe/examples/ios/iristrackinggpu/ViewController.mm Normal file
View File

@ -0,0 +1,216 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#import "ViewController.h"
#import "mediapipe/objc/MPPCameraInputSource.h"
#import "mediapipe/objc/MPPGraph.h"
#import "mediapipe/objc/MPPLayerRenderer.h"
#include "mediapipe/framework/formats/landmark.pb.h"
static NSString* const kGraphName = @"iris_tracking_gpu";
static const char* kInputStream = "input_video";
static const char* kOutputStream = "output_video";
static const char* kLandmarksOutputStream = "iris_landmarks";
static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
@interface ViewController () <MPPGraphDelegate, MPPInputSourceDelegate>
// The MediaPipe graph currently in use. Initialized in viewDidLoad, started in viewWillAppear: and
// sent video frames on _videoQueue.
@property(nonatomic) MPPGraph* mediapipeGraph;
@end
@implementation ViewController {
/// Handles camera access via AVCaptureSession library.
MPPCameraInputSource* _cameraSource;
/// Input side packet for focal length parameter.
std::map<std::string, mediapipe::Packet> _input_side_packets;
mediapipe::Packet _focal_length_side_packet;
/// Inform the user when camera is unavailable.
IBOutlet UILabel* _noCameraLabel;
/// Display the camera preview frames.
IBOutlet UIView* _liveView;
/// Render frames in a layer.
MPPLayerRenderer* _renderer;
/// Process camera frames on this queue.
dispatch_queue_t _videoQueue;
}
#pragma mark - Cleanup methods
- (void)dealloc {
self.mediapipeGraph.delegate = nil;
[self.mediapipeGraph cancel];
// Ignore errors since we're cleaning up.
[self.mediapipeGraph closeAllInputStreamsWithError:nil];
[self.mediapipeGraph waitUntilDoneWithError:nil];
}
#pragma mark - MediaPipe graph methods
+ (MPPGraph*)loadGraphFromResource:(NSString*)resource {
// Load the graph config resource.
NSError* configLoadError = nil;
NSBundle* bundle = [NSBundle bundleForClass:[self class]];
if (!resource || resource.length == 0) {
return nil;
}
NSURL* graphURL = [bundle URLForResource:resource withExtension:@"binarypb"];
NSData* data = [NSData dataWithContentsOfURL:graphURL options:0 error:&configLoadError];
if (!data) {
NSLog(@"Failed to load MediaPipe graph config: %@", configLoadError);
return nil;
}
// Parse the graph config resource into mediapipe::CalculatorGraphConfig proto object.
mediapipe::CalculatorGraphConfig config;
config.ParseFromArray(data.bytes, data.length);
// Create MediaPipe graph with mediapipe::CalculatorGraphConfig proto object.
MPPGraph* newGraph = [[MPPGraph alloc] initWithGraphConfig:config];
[newGraph addFrameOutputStream:kOutputStream outputPacketType:MPPPacketTypePixelBuffer];
[newGraph addFrameOutputStream:kLandmarksOutputStream outputPacketType:MPPPacketTypeRaw];
return newGraph;
}
#pragma mark - UIViewController methods
- (void)viewDidLoad {
[super viewDidLoad];
_renderer = [[MPPLayerRenderer alloc] init];
_renderer.layer.frame = _liveView.layer.bounds;
[_liveView.layer addSublayer:_renderer.layer];
_renderer.frameScaleMode = MPPFrameScaleModeFillAndCrop;
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
_videoQueue = dispatch_queue_create(kVideoQueueLabel, qosAttribute);
_cameraSource = [[MPPCameraInputSource alloc] init];
[_cameraSource setDelegate:self queue:_videoQueue];
_cameraSource.sessionPreset = AVCaptureSessionPresetHigh;
_cameraSource.cameraPosition = AVCaptureDevicePositionFront;
// The frame's native format is rotated with respect to the portrait orientation.
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
// When using the front camera, mirror the input for a more natural look.
_cameraSource.videoMirrored = YES;
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
self.mediapipeGraph.delegate = self;
// Set maxFramesInFlight to a small value to avoid memory contention for real-time processing.
self.mediapipeGraph.maxFramesInFlight = 2;
_focal_length_side_packet =
mediapipe::MakePacket<std::unique_ptr<float>>(absl::make_unique<float>(0.0));
_input_side_packets = {
{"focal_length_pixel", _focal_length_side_packet},
};
[self.mediapipeGraph addSidePackets:_input_side_packets];
}
// In this application, there is only one ViewController which has no navigation to other view
// controllers, and there is only one View with live display showing the result of running the
// MediaPipe graph on the live video feed. If more view controllers are needed later, the graph
// setup/teardown and camera start/stop logic should be updated appropriately in response to the
// appearance/disappearance of this ViewController, as viewWillAppear: can be invoked multiple times
// depending on the application navigation flow in that case.
- (void)viewWillAppear:(BOOL)animated {
[super viewWillAppear:animated];
[_cameraSource requestCameraAccessWithCompletionHandler:^void(BOOL granted) {
if (granted) {
[self startGraphAndCamera];
dispatch_async(dispatch_get_main_queue(), ^{
_noCameraLabel.hidden = YES;
});
}
}];
}
- (void)startGraphAndCamera {
// Start running self.mediapipeGraph.
NSError* error;
if (![self.mediapipeGraph startWithError:&error]) {
NSLog(@"Failed to start graph: %@", error);
}
// Start fetching frames from the camera.
dispatch_async(_videoQueue, ^{
[_cameraSource start];
});
}
#pragma mark - MPPGraphDelegate methods
// Receives CVPixelBufferRef from the MediaPipe graph. Invoked on a MediaPipe worker thread.
- (void)mediapipeGraph:(MPPGraph*)graph
didOutputPixelBuffer:(CVPixelBufferRef)pixelBuffer
fromStream:(const std::string&)streamName {
if (streamName == kOutputStream) {
// Display the captured image on the screen.
CVPixelBufferRetain(pixelBuffer);
dispatch_async(dispatch_get_main_queue(), ^{
[_renderer renderPixelBuffer:pixelBuffer];
CVPixelBufferRelease(pixelBuffer);
});
}
}
// Receives a raw packet from the MediaPipe graph. Invoked on a MediaPipe worker thread.
- (void)mediapipeGraph:(MPPGraph*)graph
didOutputPacket:(const ::mediapipe::Packet&)packet
fromStream:(const std::string&)streamName {
if (streamName == kLandmarksOutputStream) {
if (packet.IsEmpty()) {
NSLog(@"[TS:%lld] No iris landmarks", packet.Timestamp().Value());
return;
}
const auto& landmarks = packet.Get<::mediapipe::NormalizedLandmarkList>();
NSLog(@"[TS:%lld] Number of landmarks on iris: %d", packet.Timestamp().Value(),
landmarks.landmark_size());
for (int i = 0; i < landmarks.landmark_size(); ++i) {
NSLog(@"\tLandmark[%d]: (%f, %f, %f)", i, landmarks.landmark(i).x(),
landmarks.landmark(i).y(), landmarks.landmark(i).z());
}
}
}
#pragma mark - MPPInputSourceDelegate methods
// Must be invoked on _videoQueue.
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
timestamp:(CMTime)timestamp
fromSource:(MPPInputSource*)source {
if (source != _cameraSource) {
NSLog(@"Unknown source: %@", source);
return;
}
// TODO: This is a temporary solution. Need to verify whether the focal length is
// constant. In that case, we need to use input stream instead of using side packet.
*(_input_side_packets["focal_length_pixel"].Get<std::unique_ptr<float>>()) =
_cameraSource.cameraIntrinsicMatrix.columns[0][0];
[self.mediapipeGraph sendPixelBuffer:imageBuffer
intoStream:kInputStream
packetType:MPPPacketTypePixelBuffer];
}
@end

22
mediapipe/examples/ios/iristrackinggpu/main.m Normal file
View File

@ -0,0 +1,22 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#import <UIKit/UIKit.h>
#import "AppDelegate.h"
int main(int argc, char * argv[]) {
@autoreleasepool {
return UIApplicationMain(argc, argv, nil, NSStringFromClass([AppDelegate class]));
}
}

12
mediapipe/examples/ios/multihandtrackinggpu/Base.lproj/Main.storyboard
View File

@ -1,10 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait">
<adaptation id="fullscreen"/>
</device>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="16097.2" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait" appearance="light"/>
<dependencies>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="16087"/>
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
</dependencies>
@ -18,11 +16,11 @@
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
<rect key="frame" x="0.0" y="20" width="375" height="647"/>
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="emf-N5-sEd">
<rect key="frame" x="57" y="248" width="260" height="151"/>
<rect key="frame" x="57" y="258" width="260" height="151"/>
<autoresizingMask key="autoresizingMask" flexibleMinX="YES" flexibleMaxX="YES" flexibleMinY="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="17"/>
<color key="textColor" white="1" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>

4
mediapipe/examples/ios/multihandtrackinggpu/ViewController.mm
View File

@ -96,8 +96,6 @@ static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
_renderer.layer.frame = _liveView.layer.bounds;
[_liveView.layer addSublayer:_renderer.layer];
_renderer.frameScaleMode = MPPFrameScaleModeFillAndCrop;
// When using the front camera, mirror the input for a more natural look.
_renderer.mirrored = YES;
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
@ -109,6 +107,8 @@ static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
_cameraSource.cameraPosition = AVCaptureDevicePositionFront;
// The frame's native format is rotated with respect to the portrait orientation.
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
// When using the front camera, mirror the input for a more natural look.
_cameraSource.videoMirrored = YES;
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
self.mediapipeGraph.delegate = self;

30
mediapipe/examples/ios/objectdetectioncpu/Base.lproj/Main.storyboard
View File

@ -1,10 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait">
<adaptation id="fullscreen"/>
</device>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="16097.2" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait" appearance="light"/>
<dependencies>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="16087"/>
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
</dependencies>
@ -18,27 +16,29 @@
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
<rect key="frame" x="0.0" y="20" width="375" height="647"/>
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="emf-N5-sEd">
<rect key="frame" x="57" y="258" width="260" height="151"/>
<autoresizingMask key="autoresizingMask" flexibleMinX="YES" flexibleMaxX="YES" flexibleMinY="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="17"/>
<color key="textColor" white="1" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>
<nil key="highlightedColor"/>
</label>
</subviews>
<color key="backgroundColor" white="0.0" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>
<accessibility key="accessibilityConfiguration" label="PreviewDisplayView">
<bool key="isElement" value="YES"/>
</accessibility>
</view>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="9za-p3-JZ4">
<rect key="frame" x="54" y="285" width="266" height="97"/>
<autoresizingMask key="autoresizingMask" flexibleMaxX="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="17"/>
<color key="textColor" white="1" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>
<nil key="highlightedColor"/>
</label>
</subviews>
<color key="backgroundColor" red="1" green="1" blue="1" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
<viewLayoutGuide key="safeArea" id="6Tk-OE-BBY"/>
</view>
<connections>
<outlet property="_liveView" destination="EfB-xq-knP" id="wac-VF-etz"/>
<outlet property="_noCameraLabel" destination="9za-p3-JZ4" id="K1L-Ut-TfH"/>
<outlet property="_liveView" destination="EfB-xq-knP" id="JQp-2n-q9q"/>
<outlet property="_noCameraLabel" destination="emf-N5-sEd" id="91G-3Z-cU3"/>
</connections>
</viewController>
<placeholder placeholderIdentifier="IBFirstResponder" id="dkx-z0-nzr" sceneMemberID="firstResponder"/>

12
mediapipe/examples/ios/objectdetectiongpu/Base.lproj/Main.storyboard
View File

@ -1,10 +1,8 @@
<?xml version="1.0" encoding="UTF-8"?>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait">
<adaptation id="fullscreen"/>
</device>
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="16097.2" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
<device id="retina4_7" orientation="portrait" appearance="light"/>
<dependencies>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="16087"/>
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
</dependencies>
@ -18,11 +16,11 @@
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
<rect key="frame" x="0.0" y="20" width="375" height="647"/>
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
<subviews>
<label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" fixedFrame="YES" text="Camera access needed for this demo. Please enable camera access in the Settings app." textAlignment="center" lineBreakMode="tailTruncation" numberOfLines="0" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="emf-N5-sEd">
<rect key="frame" x="57" y="248" width="260" height="151"/>
<rect key="frame" x="57" y="258" width="260" height="151"/>
<autoresizingMask key="autoresizingMask" flexibleMinX="YES" flexibleMaxX="YES" flexibleMinY="YES" flexibleMaxY="YES"/>
<fontDescription key="fontDescription" type="system" pointSize="17"/>
<color key="textColor" white="1" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>

2
mediapipe/framework/calculator_graph.cc
View File

@ -575,7 +575,7 @@ CalculatorGraph::PrepareGpu(const std::map<std::string, Packet>& side_packets) {
// Set up executors.
for (auto& node : *nodes_) {
if (node.UsesGpu()) {
gpu_resources->PrepareGpuNode(&node);
MP_RETURN_IF_ERROR(gpu_resources->PrepareGpuNode(&node));
}
}
for (const auto& name_executor : gpu_resources->GetGpuExecutors()) {

29
mediapipe/framework/calculator_node.cc
View File

@ -473,6 +473,19 @@ bool CalculatorNode::OutputsAreConstant(CalculatorContext* cc) {
"Calculator::Open() for node \"$0\" failed: ", DebugName());
needs_to_close_ = true;
bool offset_enabled = false;
for (auto& stream : output_stream_handler_->OutputStreams()) {
offset_enabled = offset_enabled || stream->Spec()->offset_enabled;
}
if (offset_enabled && input_stream_handler_->SyncSetCount() > 1) {
LOG(WARNING) << absl::Substitute(
"Calculator node \"$0\" is configured with multiple input sync-sets "
"and an output timestamp-offset, which will often conflict due to "
"the order of packet arrival. With multiple input sync-sets, use "
"SetProcessTimestampBounds in place of SetTimestampOffset.",
DebugName());
}
output_stream_handler_->Open(outputs);
{
@ -737,21 +750,7 @@ std::string CalculatorNode::DebugInputStreamNames() const {
std::string CalculatorNode::DebugName() const {
DCHECK(calculator_state_);
const std::string first_output_stream_name =
output_stream_handler_->FirstStreamName();
if (!first_output_stream_name.empty()) {
// A calculator is unique by its output streams (one of them is
// sufficient) unless it is a sink. For readability, its type name is
// included.
return absl::Substitute(
"[$0, $1 with output stream: $2]", calculator_state_->NodeName(),
calculator_state_->CalculatorType(), first_output_stream_name);
}
// If it is a sink, its full node spec is returned.
return absl::Substitute(
"[$0, $1 with node ID: $2 and $3]", calculator_state_->NodeName(),
calculator_state_->CalculatorType(), node_id_, DebugInputStreamNames());
return calculator_state_->NodeName();
}
// TODO: Split this function.

4
mediapipe/framework/calculator_node_test.cc
View File

@ -259,9 +259,7 @@ class CalculatorNodeTest : public ::testing::Test {
TEST_F(CalculatorNodeTest, Initialize) {
InitializeEnvironment(/*use_tags=*/false);
EXPECT_EQ(2, node_->Id());
EXPECT_THAT(node_->DebugName(),
::testing::AllOf(::testing::HasSubstr("CountCalculator"),
::testing::HasSubstr("stream_b")));
EXPECT_THAT(node_->DebugName(), ::testing::HasSubstr("CountCalculator"));
EXPECT_FALSE(node_->Prepared());
EXPECT_FALSE(node_->Opened());

6
mediapipe/framework/formats/BUILD
View File

@ -63,6 +63,12 @@ mediapipe_proto_library(
visibility = ["//visibility:public"],
)
mediapipe_proto_library(
name = "image_file_properties_proto",
srcs = ["image_file_properties.proto"],
visibility = ["//visibility:public"],
)
cc_library(
name = "deleting_file",
srcs = ["deleting_file.cc"],

30
mediapipe/framework/formats/image_file_properties.proto Normal file
View File

@ -0,0 +1,30 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
syntax = "proto2";
package mediapipe;
// A list of properties extracted from EXIF metadata from an image file.
message ImageFileProperties {
// Image dimensions.
optional uint32 image_width = 1;
optional uint32 image_height = 2;
// Focal length of camera lens in millimeters.
optional double focal_length_mm = 3;
// Focal length of camera lens in 35 mm equivalent.
optional double focal_length_35mm = 4;
// Focal length in pixels.
optional double focal_length_pixels = 5;
}

3
mediapipe/framework/input_stream_handler.h
View File

@ -185,6 +185,9 @@ class InputStreamHandler {
// When true, Calculator::Process is called for every input timestamp bound.
bool ProcessTimestampBounds() { return process_timestamps_; }
// Returns the number of sync-sets populated by this input stream handler.
virtual int SyncSetCount() { return 1; }
// A helper class to build input packet sets for a certain set of streams.
//
// ReadyForProcess requires all of the streams to be fully determined

2
mediapipe/framework/port/BUILD
View File

@ -404,7 +404,7 @@ cc_library(
visibility = ["//mediapipe/framework/port:__pkg__"],
deps = [
":status",
"//mediapipe/framework/deps:status_matchers",
"@com_google_googletest//:gtest",
],
)

8
mediapipe/framework/stream_handler/immediate_input_stream_handler.cc
View File

@ -56,6 +56,9 @@ class ImmediateInputStreamHandler : public InputStreamHandler {
void FillInputSet(Timestamp input_timestamp,
InputStreamShardSet* input_set) override;
// Returns the number of sync-sets maintained by this input-handler.
int SyncSetCount() override;
absl::Mutex mutex_;
// The packet-set builder for each input stream.
std::vector<SyncSet> sync_sets_ ABSL_GUARDED_BY(mutex_);
@ -169,4 +172,9 @@ void ImmediateInputStreamHandler::FillInputSet(Timestamp input_timestamp,
}
}
int ImmediateInputStreamHandler::SyncSetCount() {
absl::MutexLock lock(&mutex_);
return sync_sets_.size();
}
} // namespace mediapipe

8
mediapipe/framework/stream_handler/sync_set_input_stream_handler.cc
View File

@ -67,6 +67,9 @@ class SyncSetInputStreamHandler : public InputStreamHandler {
InputStreamShardSet* input_set)
ABSL_EXCLUSIVE_LOCKS_REQUIRED(mutex_);
// Returns the number of sync-sets maintained by this input-handler.
int SyncSetCount() override;
private:
absl::Mutex mutex_;
// The ids of each set of inputs.
@ -194,4 +197,9 @@ void SyncSetInputStreamHandler::FillInputSet(Timestamp input_timestamp,
ready_timestamp_ = Timestamp::Done();
}
int SyncSetInputStreamHandler::SyncSetCount() {
absl::MutexLock lock(&mutex_);
return sync_sets_.size();
}
} // namespace mediapipe

37
mediapipe/gpu/gpu_shared_data_internal.cc
View File

@ -16,6 +16,7 @@
#include "mediapipe/framework/deps/no_destructor.h"
#include "mediapipe/framework/port/ret_check.h"
#include "mediapipe/gpu/gl_context.h"
#include "mediapipe/gpu/gl_context_options.pb.h"
#include "mediapipe/gpu/graph_support.h"
@ -103,7 +104,7 @@ GpuResources::~GpuResources() {
#endif
}
void GpuResources::PrepareGpuNode(CalculatorNode* node) {
::mediapipe::Status GpuResources::PrepareGpuNode(CalculatorNode* node) {
CHECK(node->UsesGpu());
std::string node_id = node->GetCalculatorState().NodeName();
std::string node_type = node->GetCalculatorState().CalculatorType();
@ -127,38 +128,46 @@ void GpuResources::PrepareGpuNode(CalculatorNode* node) {
}
node_key_[node_id] = context_key;
ASSIGN_OR_RETURN(std::shared_ptr<GlContext> context,
GetOrCreateGlContext(context_key));
if (kGlContextUseDedicatedThread) {
std::string executor_name =
absl::StrCat(kGpuExecutorName, "_", context_key);
node->SetExecutor(executor_name);
if (!ContainsKey(named_executors_, executor_name)) {
named_executors_.emplace(
executor_name,
std::make_shared<GlContextExecutor>(gl_context(context_key).get()));
executor_name, std::make_shared<GlContextExecutor>(context.get()));
}
}
gl_context(context_key)
->SetProfilingContext(
node->GetCalculatorState().GetSharedProfilingContext());
context->SetProfilingContext(
node->GetCalculatorState().GetSharedProfilingContext());
return OkStatus();
}
// TODO: expose and use an actual ID instead of using the
// canonicalized name.
const std::shared_ptr<GlContext>& GpuResources::gl_context(
CalculatorContext* cc) {
return gl_context(cc ? node_key_[cc->NodeName()] : SharedContextKey());
if (cc) {
auto it = gl_key_context_.find(node_key_[cc->NodeName()]);
if (it != gl_key_context_.end()) {
return it->second;
}
}
return gl_key_context_[SharedContextKey()];
}
const std::shared_ptr<GlContext>& GpuResources::gl_context(
GlContext::StatusOrGlContext GpuResources::GetOrCreateGlContext(
const std::string& key) {
auto it = gl_key_context_.find(key);
if (it == gl_key_context_.end()) {
it = gl_key_context_
.emplace(key,
GlContext::Create(*gl_key_context_[SharedContextKey()],
kGlContextUseDedicatedThread)
.ValueOrDie())
.first;
ASSIGN_OR_RETURN(std::shared_ptr<GlContext> new_context,
GlContext::Create(*gl_key_context_[SharedContextKey()],
kGlContextUseDedicatedThread));
it = gl_key_context_.emplace(key, new_context).first;
#if __APPLE__
gpu_buffer_pool_.RegisterTextureCache(it->second->cv_texture_cache());
#endif

6
mediapipe/gpu/gpu_shared_data_internal.h
View File

@ -67,9 +67,9 @@ class GpuResources {
#ifdef __APPLE__
MPPGraphGPUData* ios_gpu_data();
#endif // defined(__APPLE__)
#endif // defined(__APPLE__)§
void PrepareGpuNode(CalculatorNode* node);
::mediapipe::Status PrepareGpuNode(CalculatorNode* node);
// If the node requires custom GPU executors in the current configuration,
// returns the executor's names and the executors themselves.
@ -81,7 +81,7 @@ class GpuResources {
GpuResources() = delete;
explicit GpuResources(std::shared_ptr<GlContext> gl_context);
const std::shared_ptr<GlContext>& gl_context(const std::string& key);
GlContext::StatusOrGlContext GetOrCreateGlContext(const std::string& key);
const std::string& ContextKey(const std::string& canonical_node_name);
std::map<std::string, std::string> node_key_;

82
mediapipe/graphs/iris_tracking/BUILD Normal file
View File

@ -0,0 +1,82 @@
# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
load(
"//mediapipe/framework/tool:mediapipe_graph.bzl",
"mediapipe_binary_graph",
)
licenses(["notice"]) # Apache 2.0
package(default_visibility = ["//visibility:public"])
cc_library(
name = "iris_depth_cpu_deps",
deps = [
"//mediapipe/calculators/core:constant_side_packet_calculator",
"//mediapipe/calculators/core:flow_limiter_calculator",
"//mediapipe/calculators/core:split_vector_calculator",
"//mediapipe/calculators/image:image_file_properties_calculator",
"//mediapipe/calculators/image:opencv_encoded_image_to_image_frame_calculator",
"//mediapipe/calculators/image:opencv_image_encoder_calculator",
"//mediapipe/graphs/iris_tracking/subgraphs:iris_and_depth_renderer_cpu",
"//mediapipe/modules/face_landmark:face_landmark_front_cpu",
"//mediapipe/modules/iris_landmark:iris_landmark_left_and_right_cpu",
],
)
cc_library(
name = "iris_tracking_cpu_deps",
deps = [
"//mediapipe/calculators/core:constant_side_packet_calculator",
"//mediapipe/calculators/core:flow_limiter_calculator",
"//mediapipe/calculators/core:split_vector_calculator",
"//mediapipe/graphs/iris_tracking/subgraphs:iris_renderer_cpu",
"//mediapipe/modules/face_landmark:face_landmark_front_cpu",
"//mediapipe/modules/iris_landmark:iris_landmark_left_and_right_cpu",
],
)
cc_library(
name = "iris_tracking_cpu_video_input_deps",
deps = [
"//mediapipe/calculators/core:constant_side_packet_calculator",
"//mediapipe/calculators/core:flow_limiter_calculator",
"//mediapipe/calculators/core:split_vector_calculator",
"//mediapipe/calculators/video:opencv_video_decoder_calculator",
"//mediapipe/calculators/video:opencv_video_encoder_calculator",
"//mediapipe/graphs/iris_tracking/subgraphs:iris_renderer_cpu",
"//mediapipe/modules/face_landmark:face_landmark_front_cpu",
"//mediapipe/modules/iris_landmark:iris_landmark_left_and_right_cpu",
],
)
cc_library(
name = "iris_tracking_gpu_deps",
deps = [
"//mediapipe/calculators/core:constant_side_packet_calculator",
"//mediapipe/calculators/core:flow_limiter_calculator",
"//mediapipe/calculators/core:split_vector_calculator",
"//mediapipe/graphs/iris_tracking/subgraphs:iris_and_depth_renderer_gpu",
"//mediapipe/modules/face_landmark:face_landmark_front_gpu",
"//mediapipe/modules/iris_landmark:iris_landmark_left_and_right_gpu",
],
)
mediapipe_binary_graph(
name = "iris_tracking_gpu_binary_graph",
graph = "iris_tracking_gpu.pbtxt",
output_name = "iris_tracking_gpu.binarypb",
deps = [":iris_tracking_gpu_deps"],
)

92
mediapipe/graphs/iris_tracking/calculators/BUILD Normal file
View File

@ -0,0 +1,92 @@
# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
load("//mediapipe/framework/port:build_config.bzl", "mediapipe_cc_proto_library")
licenses(["notice"]) # Apache 2.0
proto_library(
name = "iris_to_render_data_calculator_proto",
srcs = ["iris_to_render_data_calculator.proto"],
visibility = ["//visibility:public"],
deps = [
"//mediapipe/framework:calculator_proto",
"//mediapipe/util:color_proto",
"//mediapipe/util:render_data_proto",
],
)
mediapipe_cc_proto_library(
name = "iris_to_render_data_calculator_cc_proto",
srcs = ["iris_to_render_data_calculator.proto"],
cc_deps = [
"//mediapipe/framework:calculator_cc_proto",
"//mediapipe/util:color_cc_proto",
"//mediapipe/util:render_data_cc_proto",
],
visibility = ["//visibility:public"],
deps = [":iris_to_render_data_calculator_proto"],
)
cc_library(
name = "iris_to_render_data_calculator",
srcs = ["iris_to_render_data_calculator.cc"],
visibility = ["//visibility:public"],
deps = [
":iris_to_render_data_calculator_cc_proto",
"//mediapipe/framework:calculator_framework",
"//mediapipe/framework/formats:landmark_cc_proto",
"//mediapipe/framework/port:ret_check",
"//mediapipe/framework/port:status",
"//mediapipe/util:color_cc_proto",
"//mediapipe/util:render_data_cc_proto",
"@com_google_absl//absl/strings",
],
alwayslink = 1,
)
proto_library(
name = "iris_to_depth_calculator_proto",
srcs = ["iris_to_depth_calculator.proto"],
visibility = ["//visibility:public"],
deps = [
"//mediapipe/framework:calculator_proto",
],
)
mediapipe_cc_proto_library(
name = "iris_to_depth_calculator_cc_proto",
srcs = ["iris_to_depth_calculator.proto"],
cc_deps = [
"//mediapipe/framework:calculator_cc_proto",
],
visibility = ["//visibility:public"],
deps = [":iris_to_depth_calculator_proto"],
)
cc_library(
name = "iris_to_depth_calculator",
srcs = ["iris_to_depth_calculator.cc"],
visibility = ["//visibility:public"],
deps = [
":iris_to_depth_calculator_cc_proto",
"//mediapipe/framework:calculator_framework",
"//mediapipe/framework/formats:image_file_properties_cc_proto",
"//mediapipe/framework/formats:landmark_cc_proto",
"//mediapipe/framework/port:ret_check",
"//mediapipe/framework/port:status",
"@com_google_absl//absl/strings",
],
alwayslink = 1,
)

245
mediapipe/graphs/iris_tracking/calculators/iris_to_depth_calculator.cc Normal file
View File

@ -0,0 +1,245 @@
// Copyright 2020 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include <cmath>
#include <memory>
#include "absl/strings/str_cat.h"
#include "mediapipe/framework/calculator_framework.h"
#include "mediapipe/framework/formats/image_file_properties.pb.h"
#include "mediapipe/framework/formats/landmark.pb.h"
#include "mediapipe/framework/port/ret_check.h"
#include "mediapipe/framework/port/status.h"
#include "mediapipe/graphs/iris_tracking/calculators/iris_to_depth_calculator.pb.h"
namespace mediapipe {
namespace {
constexpr char kIrisTag[] = "IRIS";
constexpr char kImageSizeTag[] = "IMAGE_SIZE";
constexpr char kFocalLengthPixelTag[] = "FOCAL_LENGTH";
constexpr char kImageFilePropertiesTag[] = "IMAGE_FILE_PROPERTIES";
constexpr char kLeftIrisDepthTag[] = "LEFT_IRIS_DEPTH_MM";
constexpr char kRightIrisDepthTag[] = "RIGHT_IRIS_DEPTH_MM";
constexpr int kNumIrisLandmarksPerEye = 5;
constexpr float kDepthWeightUpdate = 0.1;
// Avergae fixed iris size across human beings.
constexpr float kIrisSizeInMM = 11.8;
inline float GetDepth(float x0, float y0, float x1, float y1) {
return std::sqrt((x0 - x1) * (x0 - x1) + (y0 - y1) * (y0 - y1));
}
inline float GetLandmarkDepth(const NormalizedLandmark& ld0,
const NormalizedLandmark& ld1,
const std::pair<int, int>& image_size) {
return GetDepth(ld0.x() * image_size.first, ld0.y() * image_size.second,
ld1.x() * image_size.first, ld1.y() * image_size.second);
}
float CalculateIrisDiameter(const NormalizedLandmarkList& landmarks,
const std::pair<int, int>& image_size) {
const float dist_vert = GetLandmarkDepth(landmarks.landmark(1),
landmarks.landmark(2), image_size);
const float dist_hori = GetLandmarkDepth(landmarks.landmark(3),
landmarks.landmark(4), image_size);
return (dist_hori + dist_vert) / 2.0f;
}
float CalculateDepth(const NormalizedLandmark& center, float focal_length,
float iris_size, float img_w, float img_h) {
std::pair<float, float> origin{img_w / 2.f, img_h / 2.f};
const auto y = GetDepth(origin.first, origin.second, center.x() * img_w,
center.y() * img_h);
const auto x = std::sqrt(focal_length * focal_length + y * y);
const auto depth = kIrisSizeInMM * x / iris_size;
return depth;
}
} // namespace
// Estimates depth from iris to camera given focal length and image size.
//
// Usage example:
// node {
// calculator: "IrisToDepthCalculator"
// # A NormalizedLandmarkList contains landmarks for both iris.
// input_stream: "IRIS:iris_landmarks"
// input_stream: "IMAGE_SIZE:image_size"
// # Note: Only one of FOCAL_LENGTH or IMAGE_FILE_PROPERTIES is necessary
// # to get focal length in pixels. Sending focal length in pixels to
// # this calculator is optional.
// input_side_packet: "FOCAL_LENGTH:focal_length_pixel"
// # OR
// input_side_packet: "IMAGE_FILE_PROPERTIES:image_file_properties"
// output_stream: "LEFT_IRIS_DEPTH_MM:left_iris_depth_mm"
// output_stream: "RIGHT_IRIS_DEPTH_MM:right_iris_depth_mm"
// }
class IrisToDepthCalculator : public CalculatorBase {
public:
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
cc->Inputs().Tag(kIrisTag).Set<NormalizedLandmarkList>();
cc->Inputs().Tag(kImageSizeTag).Set<std::pair<int, int>>();
// Only one of kFocalLengthPixelTag or kImageFilePropertiesTag must exist
// if they are present.
RET_CHECK(!(cc->InputSidePackets().HasTag(kFocalLengthPixelTag) &&
cc->InputSidePackets().HasTag(kImageFilePropertiesTag)));
if (cc->InputSidePackets().HasTag(kFocalLengthPixelTag)) {
cc->InputSidePackets().Tag(kFocalLengthPixelTag).SetAny();
}
if (cc->InputSidePackets().HasTag(kImageFilePropertiesTag)) {
cc->InputSidePackets()
.Tag(kImageFilePropertiesTag)
.Set<ImageFileProperties>();
}
if (cc->Outputs().HasTag(kLeftIrisDepthTag)) {
cc->Outputs().Tag(kLeftIrisDepthTag).Set<float>();
}
if (cc->Outputs().HasTag(kRightIrisDepthTag)) {
cc->Outputs().Tag(kRightIrisDepthTag).Set<float>();
}
return ::mediapipe::OkStatus();
}
::mediapipe::Status Open(CalculatorContext* cc) override;
::mediapipe::Status Process(CalculatorContext* cc) override;
private:
float focal_length_pixels_ = -1.f;
// TODO: Consolidate the logic when switching to input stream for
// focal length.
bool compute_depth_from_iris_ = false;
float smoothed_left_depth_mm_ = -1.f;
float smoothed_right_depth_mm_ = -1.f;
void GetLeftIris(const NormalizedLandmarkList& lds,
NormalizedLandmarkList* iris);
void GetRightIris(const NormalizedLandmarkList& lds,
NormalizedLandmarkList* iris);
::mediapipe::IrisToDepthCalculatorOptions options_;
};
REGISTER_CALCULATOR(IrisToDepthCalculator);
::mediapipe::Status IrisToDepthCalculator::Open(CalculatorContext* cc) {
cc->SetOffset(TimestampDiff(0));
if (cc->InputSidePackets().HasTag(kFocalLengthPixelTag)) {
#if defined(__APPLE__)
focal_length_pixels_ = *cc->InputSidePackets()
.Tag(kFocalLengthPixelTag)
.Get<std::unique_ptr<float>>();
#else
focal_length_pixels_ =
cc->InputSidePackets().Tag(kFocalLengthPixelTag).Get<float>();
#endif
compute_depth_from_iris_ = true;
} else if (cc->InputSidePackets().HasTag(kImageFilePropertiesTag)) {
const auto& properties = cc->InputSidePackets()
.Tag(kImageFilePropertiesTag)
.Get<ImageFileProperties>();
focal_length_pixels_ = properties.focal_length_pixels();
compute_depth_from_iris_ = true;
}
options_ = cc->Options<::mediapipe::IrisToDepthCalculatorOptions>();
return ::mediapipe::OkStatus();
}
::mediapipe::Status IrisToDepthCalculator::Process(CalculatorContext* cc) {
// Only process if there's input landmarks.
if (cc->Inputs().Tag(kIrisTag).IsEmpty()) {
return ::mediapipe::OkStatus();
}
const auto& iris_landmarks =
cc->Inputs().Tag(kIrisTag).Get<NormalizedLandmarkList>();
RET_CHECK_EQ(iris_landmarks.landmark_size(), kNumIrisLandmarksPerEye * 2)
<< "Wrong number of iris landmarks";
std::pair<int, int> image_size;
RET_CHECK(!cc->Inputs().Tag(kImageSizeTag).IsEmpty());
image_size = cc->Inputs().Tag(kImageSizeTag).Get<std::pair<int, int>>();
auto left_iris = absl::make_unique<NormalizedLandmarkList>();
auto right_iris = absl::make_unique<NormalizedLandmarkList>();
GetLeftIris(iris_landmarks, left_iris.get());
GetRightIris(iris_landmarks, right_iris.get());
const auto left_iris_size = CalculateIrisDiameter(*left_iris, image_size);
const auto right_iris_size = CalculateIrisDiameter(*right_iris, image_size);
#if defined(__APPLE__)
if (cc->InputSidePackets().HasTag(kFocalLengthPixelTag)) {
focal_length_pixels_ = *cc->InputSidePackets()
.Tag(kFocalLengthPixelTag)
.Get<std::unique_ptr<float>>();
}
#endif
if (compute_depth_from_iris_ && focal_length_pixels_ > 0) {
const auto left_depth =
CalculateDepth(left_iris->landmark(0), focal_length_pixels_,
left_iris_size, image_size.first, image_size.second);
const auto right_depth =
CalculateDepth(right_iris->landmark(0), focal_length_pixels_,
right_iris_size, image_size.first, image_size.second);
smoothed_left_depth_mm_ =
smoothed_left_depth_mm_ < 0 || std::isinf(smoothed_left_depth_mm_)
? left_depth
: smoothed_left_depth_mm_ * (1 - kDepthWeightUpdate) +
left_depth * kDepthWeightUpdate;
smoothed_right_depth_mm_ =
smoothed_right_depth_mm_ < 0 || std::isinf(smoothed_right_depth_mm_)
? right_depth
: smoothed_right_depth_mm_ * (1 - kDepthWeightUpdate) +
right_depth * kDepthWeightUpdate;
if (cc->Outputs().HasTag(kLeftIrisDepthTag)) {
cc->Outputs()
.Tag(kLeftIrisDepthTag)
.AddPacket(MakePacket<float>(smoothed_left_depth_mm_)
.At(cc->InputTimestamp()));
}
if (cc->Outputs().HasTag(kRightIrisDepthTag)) {
cc->Outputs()
.Tag(kRightIrisDepthTag)
.AddPacket(MakePacket<float>(smoothed_right_depth_mm_)
.At(cc->InputTimestamp()));
}
}
return ::mediapipe::OkStatus();
}
void IrisToDepthCalculator::GetLeftIris(const NormalizedLandmarkList& lds,
NormalizedLandmarkList* iris) {
// Center, top, bottom, left, right
*iris->add_landmark() = lds.landmark(options_.left_iris_center_index());
*iris->add_landmark() = lds.landmark(options_.left_iris_top_index());
*iris->add_landmark() = lds.landmark(options_.left_iris_bottom_index());
*iris->add_landmark() = lds.landmark(options_.left_iris_left_index());
*iris->add_landmark() = lds.landmark(options_.left_iris_right_index());
}
void IrisToDepthCalculator::GetRightIris(const NormalizedLandmarkList& lds,
NormalizedLandmarkList* iris) {
// Center, top, bottom, left, right
*iris->add_landmark() = lds.landmark(options_.right_iris_center_index());
*iris->add_landmark() = lds.landmark(options_.right_iris_top_index());
*iris->add_landmark() = lds.landmark(options_.right_iris_bottom_index());
*iris->add_landmark() = lds.landmark(options_.right_iris_left_index());
*iris->add_landmark() = lds.landmark(options_.right_iris_right_index());
}
} // namespace mediapipe

39
mediapipe/graphs/iris_tracking/calculators/iris_to_depth_calculator.proto Normal file
View File

@ -0,0 +1,39 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
syntax = "proto2";
package mediapipe;
import "mediapipe/framework/calculator.proto";
message IrisToDepthCalculatorOptions {
extend CalculatorOptions {
optional IrisToDepthCalculatorOptions ext = 303429002;
}
// Indices of correspondent left iris landmarks in input stream.
optional int32 left_iris_center_index = 1 [default = 0];
optional int32 left_iris_top_index = 2 [default = 2];
optional int32 left_iris_bottom_index = 3 [default = 4];
optional int32 left_iris_left_index = 4 [default = 3];
optional int32 left_iris_right_index = 5 [default = 1];
// Indices of correspondent right iris landmarks in input stream.
optional int32 right_iris_center_index = 6 [default = 5];
optional int32 right_iris_top_index = 7 [default = 7];
optional int32 right_iris_bottom_index = 8 [default = 9];
optional int32 right_iris_left_index = 9 [default = 6];
optional int32 right_iris_right_index = 10 [default = 8];
}

318
mediapipe/graphs/iris_tracking/calculators/iris_to_render_data_calculator.cc Normal file
View File

@ -0,0 +1,318 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include <cmath>
#include <memory>
#include "absl/strings/str_cat.h"
#include "mediapipe/framework/calculator_framework.h"
#include "mediapipe/framework/formats/landmark.pb.h"
#include "mediapipe/framework/port/ret_check.h"
#include "mediapipe/framework/port/status.h"
#include "mediapipe/graphs/iris_tracking/calculators/iris_to_render_data_calculator.pb.h"
#include "mediapipe/util/color.pb.h"
#include "mediapipe/util/render_data.pb.h"
namespace mediapipe {
namespace {
constexpr char kIrisTag[] = "IRIS";
constexpr char kRenderDataTag[] = "RENDER_DATA";
constexpr char kImageSizeTag[] = "IMAGE_SIZE";
constexpr char kLeftIrisDepthTag[] = "LEFT_IRIS_DEPTH_MM";
constexpr char kRightIrisDepthTag[] = "RIGHT_IRIS_DEPTH_MM";
constexpr char kOvalLabel[] = "OVAL";
constexpr float kFontHeightScale = 1.5f;
constexpr int kNumIrisLandmarksPerEye = 5;
// TODO: Source.
constexpr float kIrisSizeInMM = 11.8;
inline void SetColor(RenderAnnotation* annotation, const Color& color) {
annotation->mutable_color()->set_r(color.r());
annotation->mutable_color()->set_g(color.g());
annotation->mutable_color()->set_b(color.b());
}
inline float GetDepth(float x0, float y0, float x1, float y1) {
return std::sqrt((x0 - x1) * (x0 - x1) + (y0 - y1) * (y0 - y1));
}
inline float GetLandmarkDepth(const NormalizedLandmark& ld0,
const NormalizedLandmark& ld1,
const std::pair<int, int>& image_size) {
return GetDepth(ld0.x() * image_size.first, ld0.y() * image_size.second,
ld1.x() * image_size.first, ld1.y() * image_size.second);
}
float CalculateIrisDiameter(const NormalizedLandmarkList& landmarks,
const std::pair<int, int>& image_size) {
const float dist_vert = GetLandmarkDepth(landmarks.landmark(1),
landmarks.landmark(2), image_size);
const float dist_hori = GetLandmarkDepth(landmarks.landmark(3),
landmarks.landmark(4), image_size);
return (dist_hori + dist_vert) / 2.0f;
}
float CalculateDepth(const NormalizedLandmark& center, float focal_length,
float iris_size, float img_w, float img_h) {
std::pair<float, float> origin{img_w / 2.f, img_h / 2.f};
const auto y = GetDepth(origin.first, origin.second, center.x() * img_w,
center.y() * img_h);
const auto x = std::sqrt(focal_length * focal_length + y * y);
const auto depth = kIrisSizeInMM * x / iris_size;
return depth;
}
} // namespace
// Converts iris landmarks to render data and estimates depth from the camera if
// focal length and image size. The depth will be rendered as part of the render
// data on the frame.
//
// Usage example:
// node {
// calculator: "IrisToRenderDataCalculator"
// input_stream: "IRIS:iris_landmarks"
// input_stream: "IMAGE_SIZE:image_size"
// # Note: Only one of FOCAL_LENGTH or IMAGE_FILE_PROPERTIES is necessary
// # to get focal length in pixels. Sending focal length in pixels to
// # this calculator is optional.
// input_side_packet: "FOCAL_LENGTH:focal_length_pixel"
// # OR
// input_side_packet: "IMAGE_FILE_PROPERTIES:image_file_properties"
// output_stream: "RENDER_DATA:iris_render_data"
// output_stream: "LEFT_IRIS_DEPTH_MM:left_iris_depth_mm"
// output_stream: "RIGHT_IRIS_DEPTH_MM:right_iris_depth_mm"
// node_options: {
// [type.googleapis.com/mediapipe.IrisToRenderDataCalculatorOptions] {
// color { r: 255 g: 255 b: 255 }
// thickness: 2.0
// font_height_px: 50
// horizontal_offset_px: 200
// vertical_offset_px: 200
// location: TOP_LEFT
// }
// }
// }
class IrisToRenderDataCalculator : public CalculatorBase {
public:
static ::mediapipe::Status GetContract(CalculatorContract* cc) {
cc->Inputs().Tag(kIrisTag).Set<NormalizedLandmarkList>();
cc->Outputs().Tag(kRenderDataTag).Set<RenderData>();
cc->Inputs().Tag(kImageSizeTag).Set<std::pair<int, int>>();
if (cc->Inputs().HasTag(kLeftIrisDepthTag)) {
cc->Inputs().Tag(kLeftIrisDepthTag).Set<float>();
}
if (cc->Inputs().HasTag(kRightIrisDepthTag)) {
cc->Inputs().Tag(kRightIrisDepthTag).Set<float>();
}
return ::mediapipe::OkStatus();
}
::mediapipe::Status Open(CalculatorContext* cc) override;
::mediapipe::Status Process(CalculatorContext* cc) override;
private:
void RenderIris(const NormalizedLandmarkList& iris_landmarks,
const IrisToRenderDataCalculatorOptions& options,
const std::pair<int, int>& image_size, float iris_size,
RenderData* render_data);
void GetLeftIris(const NormalizedLandmarkList& lds,
NormalizedLandmarkList* iris);
void GetRightIris(const NormalizedLandmarkList& lds,
NormalizedLandmarkList* iris);
void AddTextRenderData(const IrisToRenderDataCalculatorOptions& options,
const std::pair<int, int>& image_size,
const std::vector<std::string>& lines,
RenderData* render_data);
static RenderAnnotation* AddOvalRenderData(
const IrisToRenderDataCalculatorOptions& options,
RenderData* render_data);
static RenderAnnotation* AddPointRenderData(
const IrisToRenderDataCalculatorOptions& options,
RenderData* render_data);
};
REGISTER_CALCULATOR(IrisToRenderDataCalculator);
::mediapipe::Status IrisToRenderDataCalculator::Open(CalculatorContext* cc) {
cc->SetOffset(TimestampDiff(0));
return ::mediapipe::OkStatus();
}
::mediapipe::Status IrisToRenderDataCalculator::Process(CalculatorContext* cc) {
// Only process if there's input landmarks.
if (cc->Inputs().Tag(kIrisTag).IsEmpty()) {
return ::mediapipe::OkStatus();
}
const auto& options =
cc->Options<::mediapipe::IrisToRenderDataCalculatorOptions>();
const auto& iris_landmarks =
cc->Inputs().Tag(kIrisTag).Get<NormalizedLandmarkList>();
RET_CHECK_EQ(iris_landmarks.landmark_size(), kNumIrisLandmarksPerEye * 2)
<< "Wrong number of iris landmarks";
std::pair<int, int> image_size;
RET_CHECK(!cc->Inputs().Tag(kImageSizeTag).IsEmpty());
image_size = cc->Inputs().Tag(kImageSizeTag).Get<std::pair<int, int>>();
auto render_data = absl::make_unique<RenderData>();
auto left_iris = absl::make_unique<NormalizedLandmarkList>();
auto right_iris = absl::make_unique<NormalizedLandmarkList>();
GetLeftIris(iris_landmarks, left_iris.get());
GetRightIris(iris_landmarks, right_iris.get());
const auto left_iris_size = CalculateIrisDiameter(*left_iris, image_size);
const auto right_iris_size = CalculateIrisDiameter(*right_iris, image_size);
RenderIris(*left_iris, options, image_size, left_iris_size,
render_data.get());
RenderIris(*right_iris, options, image_size, right_iris_size,
render_data.get());
std::vector<std::string> lines;
std::string line;
if (cc->Inputs().HasTag(kLeftIrisDepthTag) &&
!cc->Inputs().Tag(kLeftIrisDepthTag).IsEmpty()) {
const float left_iris_depth =
cc->Inputs().Tag(kLeftIrisDepthTag).Get<float>();
if (!std::isinf(left_iris_depth)) {
line = "Left : ";
absl::StrAppend(&line, ":", std::round(left_iris_depth / 10), " cm");
lines.emplace_back(line);
}
}
if (cc->Inputs().HasTag(kRightIrisDepthTag) &&
!cc->Inputs().Tag(kRightIrisDepthTag).IsEmpty()) {
const float right_iris_depth =
cc->Inputs().Tag(kRightIrisDepthTag).Get<float>();
if (!std::isinf(right_iris_depth)) {
line = "Right : ";
absl::StrAppend(&line, ":", std::round(right_iris_depth / 10), " cm");
lines.emplace_back(line);
}
}
AddTextRenderData(options, image_size, lines, render_data.get());
cc->Outputs()
.Tag(kRenderDataTag)
.Add(render_data.release(), cc->InputTimestamp());
return ::mediapipe::OkStatus();
}
void IrisToRenderDataCalculator::AddTextRenderData(
const IrisToRenderDataCalculatorOptions& options,
const std::pair<int, int>& image_size,
const std::vector<std::string>& lines, RenderData* render_data) {
int label_baseline_px = options.vertical_offset_px();
float label_height_px =
std::ceil(options.font_height_px() * kFontHeightScale);
if (options.location() == IrisToRenderDataCalculatorOptions::TOP_LEFT) {
label_baseline_px += label_height_px;
} else if (options.location() ==
IrisToRenderDataCalculatorOptions::BOTTOM_LEFT) {
label_baseline_px += image_size.second - label_height_px * lines.size();
}
const auto label_left_px = options.horizontal_offset_px();
for (int i = 0; i < lines.size(); ++i) {
auto* label_annotation = render_data->add_render_annotations();
label_annotation->set_thickness(5);
label_annotation->mutable_color()->set_r(255);
label_annotation->mutable_color()->set_g(0);
label_annotation->mutable_color()->set_b(0);
//
auto* text = label_annotation->mutable_text();
text->set_display_text(lines[i]);
text->set_font_height(options.font_height_px());
text->set_left(label_left_px);
text->set_baseline(label_baseline_px + i * label_height_px);
text->set_font_face(options.font_face());
}
}
void IrisToRenderDataCalculator::RenderIris(
const NormalizedLandmarkList& iris_landmarks,
const IrisToRenderDataCalculatorOptions& options,
const std::pair<int, int>& image_size, float iris_size,
RenderData* render_data) {
auto* oval_data_render = AddOvalRenderData(options, render_data);
auto* oval_data = oval_data_render->mutable_oval();
const float iris_radius = iris_size / 2.f;
const auto& iris_center = iris_landmarks.landmark(0);
oval_data->mutable_rectangle()->set_top(iris_center.y() -
iris_radius / image_size.second);
oval_data->mutable_rectangle()->set_bottom(iris_center.y() +
iris_radius / image_size.second);
oval_data->mutable_rectangle()->set_left(iris_center.x() -
iris_radius / image_size.first);
oval_data->mutable_rectangle()->set_right(iris_center.x() +
iris_radius / image_size.first);
oval_data->mutable_rectangle()->set_normalized(true);
for (int i = 0; i < iris_landmarks.landmark_size(); ++i) {
const NormalizedLandmark& landmark = iris_landmarks.landmark(i);
auto* landmark_data_render = AddPointRenderData(options, render_data);
auto* landmark_data = landmark_data_render->mutable_point();
landmark_data->set_normalized(true);
landmark_data->set_x(landmark.x());
landmark_data->set_y(landmark.y());
}
}
void IrisToRenderDataCalculator::GetLeftIris(const NormalizedLandmarkList& lds,
NormalizedLandmarkList* iris) {
// Center, top, bottom, left, right
*iris->add_landmark() = lds.landmark(0);
*iris->add_landmark() = lds.landmark(2);
*iris->add_landmark() = lds.landmark(4);
*iris->add_landmark() = lds.landmark(3);
*iris->add_landmark() = lds.landmark(1);
}
void IrisToRenderDataCalculator::GetRightIris(const NormalizedLandmarkList& lds,
NormalizedLandmarkList* iris) {
// Center, top, bottom, left, right
*iris->add_landmark() = lds.landmark(5);
*iris->add_landmark() = lds.landmark(7);
*iris->add_landmark() = lds.landmark(9);
*iris->add_landmark() = lds.landmark(6);
*iris->add_landmark() = lds.landmark(8);
}
RenderAnnotation* IrisToRenderDataCalculator::AddOvalRenderData(
const IrisToRenderDataCalculatorOptions& options, RenderData* render_data) {
auto* oval_data_annotation = render_data->add_render_annotations();
oval_data_annotation->set_scene_tag(kOvalLabel);
SetColor(oval_data_annotation, options.oval_color());
oval_data_annotation->set_thickness(options.oval_thickness());
return oval_data_annotation;
}
RenderAnnotation* IrisToRenderDataCalculator::AddPointRenderData(
const IrisToRenderDataCalculatorOptions& options, RenderData* render_data) {
auto* landmark_data_annotation = render_data->add_render_annotations();
SetColor(landmark_data_annotation, options.landmark_color());
landmark_data_annotation->set_thickness(options.landmark_thickness());
return landmark_data_annotation;
}
} // namespace mediapipe

62
mediapipe/graphs/iris_tracking/calculators/iris_to_render_data_calculator.proto Normal file
View File

@ -0,0 +1,62 @@
// Copyright 2019 The MediaPipe Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
syntax = "proto2";
package mediapipe;
import "mediapipe/framework/calculator.proto";
import "mediapipe/util/color.proto";
message IrisToRenderDataCalculatorOptions {
extend CalculatorOptions {
optional IrisToRenderDataCalculatorOptions ext = 289530040;
}
// Color of the oval.
optional Color oval_color = 1;
// Color of the landmarks.
optional Color landmark_color = 9;
// Thickness of the drawing of landmarks and iris oval.
optional double oval_thickness = 2 [default = 1.0];
optional double landmark_thickness = 10 [default = 1.0];
// The font height in absolute pixels.
optional int32 font_height_px = 3 [default = 50];
// The offset of the starting text in horizontal direction in absolute pixels.
optional int32 horizontal_offset_px = 7 [default = 0];
// The offset of the starting text in vertical direction in absolute pixels.
optional int32 vertical_offset_px = 8 [default = 0];
// Specifies the font for the text. Font must be one of the following from
// OpenCV:
// cv::FONT_HERSHEY_SIMPLEX (0)
// cv::FONT_HERSHEY_PLAIN (1)
// cv::FONT_HERSHEY_DUPLEX (2)
// cv::FONT_HERSHEY_COMPLEX (3)
// cv::FONT_HERSHEY_TRIPLEX (4)
// cv::FONT_HERSHEY_COMPLEX_SMALL (5)
// cv::FONT_HERSHEY_SCRIPT_SIMPLEX (6)
// cv::FONT_HERSHEY_SCRIPT_COMPLEX (7)
optional int32 font_face = 5 [default = 0];
// Label location.
enum Location {
TOP_LEFT = 0;
BOTTOM_LEFT = 1;
}
optional Location location = 6 [default = TOP_LEFT];
}

144
mediapipe/graphs/iris_tracking/iris_depth_cpu.pbtxt Normal file
View File

@ -0,0 +1,144 @@
# MediaPipe graph that performs iris distance computation on desktop with
# TensorFlow Lite on CPU.
# Used in the example in
# mediapipie/examples/desktop/iris_tracking:iris_depth_from_image_desktop.
# Raw image bytes. (std::string)
input_stream: "input_image_bytes"
# Image with all the detections rendered. (ImageFrame)
output_stream: "output_image"
# Estimated depth in mm from the camera to the left iris of the face (if any) in
# the image. (float)
output_stream: "left_iris_depth_mm"
# Estimated depth in mm from the camera to the right iris of the face (if any)
# in the image. (float)
output_stream: "right_iris_depth_mm"
# Computes the focal length in pixels based on EXIF information stored in the
# image file. The output is an ImageFileProperties object containing relevant
# image EXIF information along with focal length in pixels.
node {
calculator: "ImageFilePropertiesCalculator"
input_stream: "input_image_bytes"
output_side_packet: "image_file_properties"
}
# Converts a raw string with encoded image bytes into an ImageFrame object
# via OpenCV so that it can be processed by downstream calculators.
node {
calculator: "OpenCvEncodedImageToImageFrameCalculator"
input_stream: "input_image_bytes"
output_stream: "input_image"
}
# Defines how many faces to detect. Iris tracking currently only handles one
# face (left and right eye), and therefore this should always be set to 1.
node {
calculator: "ConstantSidePacketCalculator"
output_side_packet: "PACKET:0:num_faces"
node_options: {
[type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
packet { int_value: 1 }
}
}
}
# Detects faces and corresponding landmarks.
node {
calculator: "FaceLandmarkFrontCpu"
input_stream: "IMAGE:input_image"
input_side_packet: "NUM_FACES:num_faces"
output_stream: "LANDMARKS:multi_face_landmarks"
output_stream: "ROIS_FROM_LANDMARKS:face_rects_from_landmarks"
output_stream: "DETECTIONS:face_detections"
output_stream: "ROIS_FROM_DETECTIONS:face_rects_from_detections"
}
# Gets the very first and only face from "multi_face_landmarks" vector.
node {
calculator: "SplitNormalizedLandmarkListVectorCalculator"
input_stream: "multi_face_landmarks"
output_stream: "face_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 1 }
element_only: true
}
}
}
# Gets the very first and only face rect from "face_rects_from_landmarks"
# vector.
node {
calculator: "SplitNormalizedRectVectorCalculator"
input_stream: "face_rects_from_landmarks"
output_stream: "face_rect"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 1 }
element_only: true
}
}
}
# Gets two landmarks which define left eye boundary.
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "face_landmarks"
output_stream: "left_eye_boundary_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 33 end: 34 }
ranges: { begin: 133 end: 134 }
combine_outputs: true
}
}
}
# Gets two landmarks which define right eye boundary.
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "face_landmarks"
output_stream: "right_eye_boundary_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 362 end: 363 }
ranges: { begin: 263 end: 264 }
combine_outputs: true
}
}
}
# Detects iris landmarks, eye contour landmarks, and corresponding rect (ROI).
node {
calculator: "IrisLandmarkLeftAndRightCpu"
input_stream: "IMAGE:input_image"
input_stream: "LEFT_EYE_BOUNDARY_LANDMARKS:left_eye_boundary_landmarks"
input_stream: "RIGHT_EYE_BOUNDARY_LANDMARKS:right_eye_boundary_landmarks"
output_stream: "LEFT_EYE_CONTOUR_LANDMARKS:left_eye_contour_landmarks"
output_stream: "LEFT_EYE_IRIS_LANDMARKS:left_iris_landmarks"
output_stream: "LEFT_EYE_ROI:left_eye_rect_from_landmarks"
output_stream: "RIGHT_EYE_CONTOUR_LANDMARKS:right_eye_contour_landmarks"
output_stream: "RIGHT_EYE_IRIS_LANDMARKS:right_iris_landmarks"
output_stream: "RIGHT_EYE_ROI:right_eye_rect_from_landmarks"
}
# Renders annotations and overlays them on top of the input images.
node {
calculator: "IrisAndDepthRendererCpu"
input_stream: "IMAGE:input_image"
input_stream: "FACE_LANDMARKS:face_landmarks"
input_stream: "EYE_LANDMARKS_LEFT:left_eye_contour_landmarks"
input_stream: "EYE_LANDMARKS_RIGHT:right_eye_contour_landmarks"
input_stream: "IRIS_LANDMARKS_LEFT:left_iris_landmarks"
input_stream: "IRIS_LANDMARKS_RIGHT:right_iris_landmarks"
input_stream: "NORM_RECT:face_rect"
input_stream: "LEFT_EYE_RECT:left_eye_rect_from_landmarks"
input_stream: "RIGHT_EYE_RECT:right_eye_rect_from_landmarks"
input_stream: "DETECTIONS:face_detections"
input_side_packet: "IMAGE_FILE_PROPERTIES:image_file_properties"
output_stream: "IMAGE:output_image"
output_stream: "LEFT_IRIS_DEPTH_MM:left_iris_depth_mm"
output_stream: "RIGHT_IRIS_DEPTH_MM:right_iris_depth_mm"
}

118
mediapipe/graphs/iris_tracking/iris_tracking_cpu.pbtxt Normal file
View File

@ -0,0 +1,118 @@
# MediaPipe graph that performs iris tracking on desktop with TensorFlow Lite
# on CPU.
# Used in the example in
# mediapipie/examples/desktop/iris_tracking:iris_tracking_cpu.
# CPU image. (ImageFrame)
input_stream: "input_video"
# CPU image. (ImageFrame)
output_stream: "output_video"
# Defines how many faces to detect. Iris tracking currently only handles one
# face (left and right eye), and therefore this should always be set to 1.
node {
calculator: "ConstantSidePacketCalculator"
output_side_packet: "PACKET:0:num_faces"
node_options: {
[type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
packet { int_value: 1 }
}
}
}
# Detects faces and corresponding landmarks.
node {
calculator: "FaceLandmarkFrontCpu"
input_stream: "IMAGE:input_video"
input_side_packet: "NUM_FACES:num_faces"
output_stream: "LANDMARKS:multi_face_landmarks"
output_stream: "ROIS_FROM_LANDMARKS:face_rects_from_landmarks"
output_stream: "DETECTIONS:face_detections"
output_stream: "ROIS_FROM_DETECTIONS:face_rects_from_detections"
}
# Gets the very first and only face from "multi_face_landmarks" vector.
node {
calculator: "SplitNormalizedLandmarkListVectorCalculator"
input_stream: "multi_face_landmarks"
output_stream: "face_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 1 }
element_only: true
}
}
}
# Gets the very first and only face rect from "face_rects_from_landmarks"
# vector.
node {
calculator: "SplitNormalizedRectVectorCalculator"
input_stream: "face_rects_from_landmarks"
output_stream: "face_rect"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 1 }
element_only: true
}
}
}
# Gets two landmarks which define left eye boundary.
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "face_landmarks"
output_stream: "left_eye_boundary_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 33 end: 34 }
ranges: { begin: 133 end: 134 }
combine_outputs: true
}
}
}
# Gets two landmarks which define right eye boundary.
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "face_landmarks"
output_stream: "right_eye_boundary_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 362 end: 363 }
ranges: { begin: 263 end: 264 }
combine_outputs: true
}
}
}
# Detects iris landmarks, eye contour landmarks, and corresponding rect (ROI).
node {
calculator: "IrisLandmarkLeftAndRightCpu"
input_stream: "IMAGE:input_video"
input_stream: "LEFT_EYE_BOUNDARY_LANDMARKS:left_eye_boundary_landmarks"
input_stream: "RIGHT_EYE_BOUNDARY_LANDMARKS:right_eye_boundary_landmarks"
output_stream: "LEFT_EYE_CONTOUR_LANDMARKS:left_eye_contour_landmarks"
output_stream: "LEFT_EYE_IRIS_LANDMARKS:left_iris_landmarks"
output_stream: "LEFT_EYE_ROI:left_eye_rect_from_landmarks"
output_stream: "RIGHT_EYE_CONTOUR_LANDMARKS:right_eye_contour_landmarks"
output_stream: "RIGHT_EYE_IRIS_LANDMARKS:right_iris_landmarks"
output_stream: "RIGHT_EYE_ROI:right_eye_rect_from_landmarks"
}
# Renders annotations and overlays them on top of the input images.
node {
calculator: "IrisRendererCpu"
input_stream: "IMAGE:input_video"
input_stream: "FACE_LANDMARKS:face_landmarks"
input_stream: "EYE_LANDMARKS_LEFT:left_eye_contour_landmarks"
input_stream: "EYE_LANDMARKS_RIGHT:right_eye_contour_landmarks"
input_stream: "IRIS_LANDMARKS_LEFT:left_iris_landmarks"
input_stream: "IRIS_LANDMARKS_RIGHT:right_iris_landmarks"
input_stream: "NORM_RECT:face_rect"
input_stream: "LEFT_EYE_RECT:left_eye_rect_from_landmarks"
input_stream: "RIGHT_EYE_RECT:right_eye_rect_from_landmarks"
input_stream: "DETECTIONS:face_detections"
output_stream: "IMAGE:output_video"
}

138
mediapipe/graphs/iris_tracking/iris_tracking_cpu_video_input.pbtxt Normal file
View File

@ -0,0 +1,138 @@
# MediaPipe graph that performs iris tracking on desktop with TensorFlow Lite
# on CPU.
# max_queue_size limits the number of packets enqueued on any input stream
# by throttling inputs to the graph. This makes the graph only process one
# frame per time.
max_queue_size: 1
# Decodes an input video file into images and a video header.
node {
calculator: "OpenCvVideoDecoderCalculator"
input_side_packet: "INPUT_FILE_PATH:input_video_path"
output_stream: "VIDEO:input_video"
output_stream: "VIDEO_PRESTREAM:input_video_header"
}
# Defines how many faces to detect. Iris tracking currently only handles one
# face (left and right eye), and therefore this should always be set to 1.
node {
calculator: "ConstantSidePacketCalculator"
output_side_packet: "PACKET:0:num_faces"
node_options: {
[type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
packet { int_value: 1 }
}
}
}
# Detects faces and corresponding landmarks.
node {
calculator: "FaceLandmarkFrontCpu"
input_stream: "IMAGE:input_video"
input_side_packet: "NUM_FACES:num_faces"
output_stream: "LANDMARKS:multi_face_landmarks"
output_stream: "ROIS_FROM_LANDMARKS:face_rects_from_landmarks"
output_stream: "DETECTIONS:face_detections"
output_stream: "ROIS_FROM_DETECTIONS:face_rects_from_detections"
}
# Gets the very first and only face from "multi_face_landmarks" vector.
node {
calculator: "SplitNormalizedLandmarkListVectorCalculator"
input_stream: "multi_face_landmarks"
output_stream: "face_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 1 }
element_only: true
}
}
}
# Gets the very first and only face rect from "face_rects_from_landmarks"
# vector.
node {
calculator: "SplitNormalizedRectVectorCalculator"
input_stream: "face_rects_from_landmarks"
output_stream: "face_rect"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 1 }
element_only: true
}
}
}
# Gets two landmarks which define left eye boundary.
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "face_landmarks"
output_stream: "left_eye_boundary_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 33 end: 34 }
ranges: { begin: 133 end: 134 }
combine_outputs: true
}
}
}
# Gets two landmarks which define right eye boundary.
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "face_landmarks"
output_stream: "right_eye_boundary_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 362 end: 363 }
ranges: { begin: 263 end: 264 }
combine_outputs: true
}
}
}
# Detects iris landmarks, eye contour landmarks, and corresponding rect (ROI).
node {
calculator: "IrisLandmarkLeftAndRightCpu"
input_stream: "IMAGE:input_video"
input_stream: "LEFT_EYE_BOUNDARY_LANDMARKS:left_eye_boundary_landmarks"
input_stream: "RIGHT_EYE_BOUNDARY_LANDMARKS:right_eye_boundary_landmarks"
output_stream: "LEFT_EYE_CONTOUR_LANDMARKS:left_eye_contour_landmarks"
output_stream: "LEFT_EYE_IRIS_LANDMARKS:left_iris_landmarks"
output_stream: "LEFT_EYE_ROI:left_eye_rect_from_landmarks"
output_stream: "RIGHT_EYE_CONTOUR_LANDMARKS:right_eye_contour_landmarks"
output_stream: "RIGHT_EYE_IRIS_LANDMARKS:right_iris_landmarks"
output_stream: "RIGHT_EYE_ROI:right_eye_rect_from_landmarks"
}
# Renders annotations and overlays them on top of the input images.
node {
calculator: "IrisRendererCpu"
input_stream: "IMAGE:input_video"
input_stream: "FACE_LANDMARKS:face_landmarks"
input_stream: "EYE_LANDMARKS_LEFT:left_eye_contour_landmarks"
input_stream: "EYE_LANDMARKS_RIGHT:right_eye_contour_landmarks"
input_stream: "IRIS_LANDMARKS_LEFT:left_iris_landmarks"
input_stream: "IRIS_LANDMARKS_RIGHT:right_iris_landmarks"
input_stream: "NORM_RECT:face_rect"
input_stream: "LEFT_EYE_RECT:left_eye_rect_from_landmarks"
input_stream: "RIGHT_EYE_RECT:right_eye_rect_from_landmarks"
input_stream: "DETECTIONS:face_detections"
output_stream: "IMAGE:output_video"
}
# Encodes the annotated images into a video file, adopting properties specified
# in the input video header, e.g., video framerate.
node {
calculator: "OpenCvVideoEncoderCalculator"
input_stream: "VIDEO:output_video"
input_stream: "VIDEO_PRESTREAM:input_video_header"
input_side_packet: "OUTPUT_FILE_PATH:output_video_path"
node_options: {
[type.googleapis.com/mediapipe.OpenCvVideoEncoderCalculatorOptions]: {
codec: "avc1"
video_format: "mp4"
}
}
}

140
mediapipe/graphs/iris_tracking/iris_tracking_gpu.pbtxt Normal file
View File

@ -0,0 +1,140 @@
# MediaPipe graph that performs iris tracking with TensorFlow Lite on GPU.
# Used in the examples in
# mediapipie/examples/android/src/java/com/mediapipe/apps/iristrackinggpu and
# GPU buffer. (GpuBuffer)
input_stream: "input_video"
# GPU buffer. (GpuBuffer)
output_stream: "output_video"
# Throttles the images flowing downstream for flow control. It passes through
# the very first incoming image unaltered, and waits for downstream nodes
# (calculators and subgraphs) in the graph to finish their tasks before it
# passes through another image. All images that come in while waiting are
# dropped, limiting the number of in-flight images in most part of the graph to
# 1. This prevents the downstream nodes from queuing up incoming images and data
# excessively, which leads to increased latency and memory usage, unwanted in
# real-time mobile applications. It also eliminates unnecessarily computation,
# e.g., the output produced by a node may get dropped downstream if the
# subsequent nodes are still busy processing previous inputs.
node {
calculator: "FlowLimiterCalculator"
input_stream: "input_video"
input_stream: "FINISHED:output_video"
input_stream_info: {
tag_index: "FINISHED"
back_edge: true
}
output_stream: "throttled_input_video"
}
# Defines how many faces to detect. Iris tracking currently only handles one
# face (left and right eye), and therefore this should always be set to 1.
node {
calculator: "ConstantSidePacketCalculator"
output_side_packet: "PACKET:num_faces"
node_options: {
[type.googleapis.com/mediapipe.ConstantSidePacketCalculatorOptions]: {
packet { int_value: 1 }
}
}
}
# Detects faces and corresponding landmarks.
node {
calculator: "FaceLandmarkFrontGpu"
input_stream: "IMAGE:throttled_input_video"
input_side_packet: "NUM_FACES:num_faces"
output_stream: "LANDMARKS:multi_face_landmarks"
output_stream: "ROIS_FROM_LANDMARKS:face_rects_from_landmarks"
output_stream: "DETECTIONS:face_detections"
output_stream: "ROIS_FROM_DETECTIONS:face_rects_from_detections"
}
# Gets the very first and only face from "multi_face_landmarks" vector.
node {
calculator: "SplitNormalizedLandmarkListVectorCalculator"
input_stream: "multi_face_landmarks"
output_stream: "face_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 1 }
element_only: true
}
}
}
# Gets the very first and only face rect from "face_rects_from_landmarks"
# vector.
node {
calculator: "SplitNormalizedRectVectorCalculator"
input_stream: "face_rects_from_landmarks"
output_stream: "face_rect"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 1 }
element_only: true
}
}
}
# Gets two landmarks which define left eye boundary.
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "face_landmarks"
output_stream: "left_eye_boundary_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 33 end: 34 }
ranges: { begin: 133 end: 134 }
combine_outputs: true
}
}
}
# Gets two landmarks which define right eye boundary.
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "face_landmarks"
output_stream: "right_eye_boundary_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 362 end: 363 }
ranges: { begin: 263 end: 264 }
combine_outputs: true
}
}
}
# Detects iris landmarks, eye contour landmarks, and corresponding rect (ROI).
node {
calculator: "IrisLandmarkLeftAndRightGpu"
input_stream: "IMAGE:throttled_input_video"
input_stream: "LEFT_EYE_BOUNDARY_LANDMARKS:left_eye_boundary_landmarks"
input_stream: "RIGHT_EYE_BOUNDARY_LANDMARKS:right_eye_boundary_landmarks"
output_stream: "LEFT_EYE_CONTOUR_LANDMARKS:left_eye_contour_landmarks"
output_stream: "LEFT_EYE_IRIS_LANDMARKS:left_iris_landmarks"
output_stream: "LEFT_EYE_ROI:left_eye_rect_from_landmarks"
output_stream: "RIGHT_EYE_CONTOUR_LANDMARKS:right_eye_contour_landmarks"
output_stream: "RIGHT_EYE_IRIS_LANDMARKS:right_iris_landmarks"
output_stream: "RIGHT_EYE_ROI:right_eye_rect_from_landmarks"
}
# Renders annotations and overlays them on top of the input images.
node {
calculator: "IrisAndDepthRendererGpu"
input_stream: "IMAGE:throttled_input_video"
input_stream: "FACE_LANDMARKS:face_landmarks"
input_stream: "EYE_LANDMARKS_LEFT:left_eye_contour_landmarks"
input_stream: "EYE_LANDMARKS_RIGHT:right_eye_contour_landmarks"
input_stream: "IRIS_LANDMARKS_LEFT:left_iris_landmarks"
input_stream: "IRIS_LANDMARKS_RIGHT:right_iris_landmarks"
input_stream: "NORM_RECT:face_rect"
input_stream: "LEFT_EYE_RECT:left_eye_rect_from_landmarks"
input_stream: "RIGHT_EYE_RECT:right_eye_rect_from_landmarks"
input_stream: "DETECTIONS:face_detections"
input_side_packet: "FOCAL_LENGTH:focal_length_pixel"
output_stream: "IRIS_LANDMARKS:iris_landmarks"
output_stream: "IMAGE:output_video"
}

68
mediapipe/graphs/iris_tracking/subgraphs/BUILD Normal file
View File

@ -0,0 +1,68 @@
# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
load(
"//mediapipe/framework/tool:mediapipe_graph.bzl",
"mediapipe_simple_subgraph",
)
licenses(["notice"]) # Apache 2.0
package(default_visibility = ["//visibility:public"])
cc_library(
name = "renderer_calculators",
deps = [
"//mediapipe/calculators/core:concatenate_normalized_landmark_list_calculator",
"//mediapipe/calculators/core:concatenate_vector_calculator",
"//mediapipe/calculators/core:split_normalized_landmark_list_calculator",
"//mediapipe/calculators/util:annotation_overlay_calculator",
"//mediapipe/calculators/util:detection_label_id_to_text_calculator",
"//mediapipe/calculators/util:detections_to_render_data_calculator",
"//mediapipe/calculators/util:landmarks_to_render_data_calculator",
"//mediapipe/calculators/util:rect_to_render_data_calculator",
],
)
mediapipe_simple_subgraph(
name = "iris_and_depth_renderer_gpu",
graph = "iris_and_depth_renderer_gpu.pbtxt",
register_as = "IrisAndDepthRendererGpu",
deps = [
":renderer_calculators",
"//mediapipe/graphs/iris_tracking/calculators:iris_to_depth_calculator",
"//mediapipe/graphs/iris_tracking/calculators:iris_to_render_data_calculator",
],
)
mediapipe_simple_subgraph(
name = "iris_renderer_cpu",
graph = "iris_renderer_cpu.pbtxt",
register_as = "IrisRendererCpu",
deps = [
":renderer_calculators",
"//mediapipe/graphs/iris_tracking/calculators:iris_to_render_data_calculator",
],
)
mediapipe_simple_subgraph(
name = "iris_and_depth_renderer_cpu",
graph = "iris_and_depth_renderer_cpu.pbtxt",
register_as = "IrisAndDepthRendererCpu",
deps = [
":renderer_calculators",
"//mediapipe/graphs/iris_tracking/calculators:iris_to_depth_calculator",
"//mediapipe/graphs/iris_tracking/calculators:iris_to_render_data_calculator",
],
)

259
mediapipe/graphs/iris_tracking/subgraphs/iris_and_depth_renderer_cpu.pbtxt Normal file
View File

@ -0,0 +1,259 @@
# MediaPipe iris tracking rendering subgraph.
type: "IrisAndDepthRendererCpu"
input_stream: "IMAGE:input_image"
input_stream: "DETECTIONS:detections"
input_stream: "FACE_LANDMARKS:face_landmarks"
input_stream: "EYE_LANDMARKS_LEFT:all_left_eye_contour_landmarks"
input_stream: "EYE_LANDMARKS_RIGHT:all_right_eye_contour_landmarks"
input_stream: "IRIS_LANDMARKS_LEFT:left_iris_landmarks"
input_stream: "IRIS_LANDMARKS_RIGHT:right_iris_landmarks"
input_stream: "NORM_RECT:rect"
input_stream: "LEFT_EYE_RECT:left_eye_rect_from_landmarks"
input_stream: "RIGHT_EYE_RECT:right_eye_rect_from_landmarks"
input_side_packet: "IMAGE_FILE_PROPERTIES:image_file_properties"
output_stream: "IMAGE:output_image"
output_stream: "LEFT_IRIS_DEPTH_MM:left_iris_depth_mm"
output_stream: "RIGHT_IRIS_DEPTH_MM:right_iris_depth_mm"
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "all_left_eye_contour_landmarks"
output_stream: "left_eye_contour_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 15 }
}
}
}
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "all_right_eye_contour_landmarks"
output_stream: "right_eye_contour_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 15 }
}
}
}
# Concatenate iris landmarks from both eyes.
node {
calculator: "ConcatenateNormalizedLandmarkListCalculator"
input_stream: "left_iris_landmarks"
input_stream: "right_iris_landmarks"
output_stream: "iris_landmarks"
}
# Concatenate iris landmarks from both eyes and face landmarks.
node {
calculator: "ConcatenateNormalizedLandmarkListCalculator"
input_stream: "left_iris_landmarks"
input_stream: "right_iris_landmarks"
input_stream: "face_landmarks"
output_stream: "face_iris_landmarks"
}
node {
calculator: "ImagePropertiesCalculator"
input_stream: "IMAGE:input_image"
output_stream: "SIZE:image_size"
}
# Maps detection label IDs to the corresponding label text ("Face").
node {
calculator: "DetectionLabelIdToTextCalculator"
input_stream: "detections"
output_stream: "labeled_detections"
node_options: {
[type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
label: "Face"
}
}
}
# Converts detections to drawing primitives for annotation overlay.
node {
calculator: "DetectionsToRenderDataCalculator"
input_stream: "DETECTIONS:labeled_detections"
output_stream: "RENDER_DATA:detection_render_data"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
thickness: 4.0
color { r: 0 g: 255 b: 0 }
}
}
}
# Converts landmarks to drawing primitives for annotation overlay.
node {
calculator: "LandmarksToRenderDataCalculator"
input_stream: "NORM_LANDMARKS:left_eye_contour_landmarks"
output_stream: "RENDER_DATA:left_eye_contour_landmarks_render_data"
node_options: {
[type.googleapis.com/mediapipe.LandmarksToRenderDataCalculatorOptions] {
landmark_connections: 0
landmark_connections: 1
landmark_connections: 1
landmark_connections: 2
landmark_connections: 2
landmark_connections: 3
landmark_connections: 3
landmark_connections: 4
landmark_connections: 4
landmark_connections: 5
landmark_connections: 5
landmark_connections: 6
landmark_connections: 6
landmark_connections: 7
landmark_connections: 7
landmark_connections: 8
landmark_connections: 9
landmark_connections: 10
landmark_connections: 10
landmark_connections: 11
landmark_connections: 11
landmark_connections: 12
landmark_connections: 12
landmark_connections: 13
landmark_connections: 13
landmark_connections: 14
landmark_connections: 0
landmark_connections: 9
landmark_connections: 8
landmark_connections: 14
landmark_color { r: 255 g: 0 b: 0 }
connection_color { r: 255 g: 0 b: 0 }
visualize_landmark_depth: false
thickness: 1.0
}
}
}
# Converts landmarks to drawing primitives for annotation overlay.
node {
calculator: "LandmarksToRenderDataCalculator"
input_stream: "NORM_LANDMARKS:right_eye_contour_landmarks"
output_stream: "RENDER_DATA:right_eye_contour_landmarks_render_data"
node_options: {
[type.googleapis.com/mediapipe.LandmarksToRenderDataCalculatorOptions] {
landmark_connections: 0
landmark_connections: 1
landmark_connections: 1
landmark_connections: 2
landmark_connections: 2
landmark_connections: 3
landmark_connections: 3
landmark_connections: 4
landmark_connections: 4
landmark_connections: 5
landmark_connections: 5
landmark_connections: 6
landmark_connections: 6
landmark_connections: 7
landmark_connections: 7
landmark_connections: 8
landmark_connections: 9
landmark_connections: 10
landmark_connections: 10
landmark_connections: 11
landmark_connections: 11
landmark_connections: 12
landmark_connections: 12
landmark_connections: 13
landmark_connections: 13
landmark_connections: 14
landmark_connections: 0
landmark_connections: 9
landmark_connections: 8
landmark_connections: 14
landmark_color { r: 255 g: 0 b: 0 }
connection_color { r: 255 g: 0 b: 0 }
visualize_landmark_depth: false
thickness: 1.0
}
}
}
# Converts normalized rects to drawing primitives for annotation overlay.
node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:rect"
output_stream: "RENDER_DATA:rect_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}
node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:right_eye_rect_from_landmarks"
output_stream: "RENDER_DATA:right_eye_rect_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}
node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:left_eye_rect_from_landmarks"
output_stream: "RENDER_DATA:left_eye_rect_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}
node {
calculator: "IrisToDepthCalculator"
input_stream: "IRIS:iris_landmarks"
input_stream: "IMAGE_SIZE:image_size"
input_side_packet: "IMAGE_FILE_PROPERTIES:image_file_properties"
output_stream: "LEFT_IRIS_DEPTH_MM:left_iris_depth_mm"
output_stream: "RIGHT_IRIS_DEPTH_MM:right_iris_depth_mm"
}
node {
calculator: "IrisToRenderDataCalculator"
input_stream: "IRIS:iris_landmarks"
input_stream: "IMAGE_SIZE:image_size"
input_stream: "LEFT_IRIS_DEPTH_MM:left_iris_depth_mm"
input_stream: "RIGHT_IRIS_DEPTH_MM:right_iris_depth_mm"
output_stream: "RENDER_DATA:iris_render_data"
node_options: {
[type.googleapis.com/mediapipe.IrisToRenderDataCalculatorOptions] {
oval_color { r: 0 g: 0 b: 255 }
landmark_color { r: 0 g: 255 b: 0 }
oval_thickness: 2.0
landmark_thickness: 1.0
font_height_px: 50
horizontal_offset_px: 200
vertical_offset_px: 200
location: TOP_LEFT
}
}
}
# Draws annotations and overlays them on top of the input images.
node {
calculator: "AnnotationOverlayCalculator"
input_stream: "IMAGE:input_image"
input_stream: "detection_render_data"
input_stream: "right_eye_contour_landmarks_render_data"
input_stream: "left_eye_contour_landmarks_render_data"
input_stream: "iris_render_data"
output_stream: "IMAGE:output_image"
}

263
mediapipe/graphs/iris_tracking/subgraphs/iris_and_depth_renderer_gpu.pbtxt Normal file
View File

@ -0,0 +1,263 @@
# MediaPipe iris tracking rendering subgraph.
type: "IrisAndDepthRendererGpu"
input_stream: "IMAGE:input_image"
input_stream: "DETECTIONS:detections"
input_stream: "FACE_LANDMARKS:face_landmarks"
input_stream: "EYE_LANDMARKS_LEFT:all_left_eye_contour_landmarks"
input_stream: "EYE_LANDMARKS_RIGHT:all_right_eye_contour_landmarks"
input_stream: "IRIS_LANDMARKS_LEFT:left_iris_landmarks"
input_stream: "IRIS_LANDMARKS_RIGHT:right_iris_landmarks"
input_stream: "NORM_RECT:rect"
input_stream: "LEFT_EYE_RECT:left_eye_rect_from_landmarks"
input_stream: "RIGHT_EYE_RECT:right_eye_rect_from_landmarks"
input_side_packet: "FOCAL_LENGTH:focal_length_pixel"
output_stream: "IRIS_LANDMARKS:iris_landmarks"
output_stream: "IMAGE:output_image"
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "all_left_eye_contour_landmarks"
output_stream: "left_eye_contour_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 15 }
}
}
}
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "all_right_eye_contour_landmarks"
output_stream: "right_eye_contour_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 15 }
}
}
}
# Concatenate iris landmarks from both eyes.
node {
calculator: "ConcatenateNormalizedLandmarkListCalculator"
input_stream: "left_iris_landmarks"
input_stream: "right_iris_landmarks"
output_stream: "iris_landmarks"
}
# Concatenate iris landmarks from both eyes and face landmarks.
node {
calculator: "ConcatenateNormalizedLandmarkListCalculator"
input_stream: "left_iris_landmarks"
input_stream: "right_iris_landmarks"
input_stream: "face_landmarks"
output_stream: "face_iris_landmarks"
}
node {
calculator: "ImagePropertiesCalculator"
input_stream: "IMAGE_GPU:input_image"
output_stream: "SIZE:image_size"
}
# Maps detection label IDs to the corresponding label text ("Face").
node {
calculator: "DetectionLabelIdToTextCalculator"
input_stream: "detections"
output_stream: "labeled_detections"
node_options: {
[type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
label: "Face"
}
}
}
# Converts detections to drawing primitives for annotation overlay.
node {
calculator: "DetectionsToRenderDataCalculator"
input_stream: "DETECTIONS:labeled_detections"
output_stream: "RENDER_DATA:detection_render_data"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
thickness: 4.0
color { r: 0 g: 255 b: 0 }
}
}
}
# Converts landmarks to drawing primitives for annotation overlay.
node {
calculator: "LandmarksToRenderDataCalculator"
input_stream: "NORM_LANDMARKS:left_eye_contour_landmarks"
output_stream: "RENDER_DATA:left_eye_contour_landmarks_render_data"
node_options: {
[type.googleapis.com/mediapipe.LandmarksToRenderDataCalculatorOptions] {
landmark_connections: 0
landmark_connections: 1
landmark_connections: 1
landmark_connections: 2
landmark_connections: 2
landmark_connections: 3
landmark_connections: 3
landmark_connections: 4
landmark_connections: 4
landmark_connections: 5
landmark_connections: 5
landmark_connections: 6
landmark_connections: 6
landmark_connections: 7
landmark_connections: 7
landmark_connections: 8
landmark_connections: 9
landmark_connections: 10
landmark_connections: 10
landmark_connections: 11
landmark_connections: 11
landmark_connections: 12
landmark_connections: 12
landmark_connections: 13
landmark_connections: 13
landmark_connections: 14
landmark_connections: 0
landmark_connections: 9
landmark_connections: 8
landmark_connections: 14
landmark_color { r: 255 g: 0 b: 0 }
connection_color { r: 255 g: 0 b: 0 }
visualize_landmark_depth: false
thickness: 2.0
}
}
}
# Converts landmarks to drawing primitives for annotation overlay.
node {
calculator: "LandmarksToRenderDataCalculator"
input_stream: "NORM_LANDMARKS:right_eye_contour_landmarks"
output_stream: "RENDER_DATA:right_eye_contour_landmarks_render_data"
node_options: {
[type.googleapis.com/mediapipe.LandmarksToRenderDataCalculatorOptions] {
landmark_connections: 0
landmark_connections: 1
landmark_connections: 1
landmark_connections: 2
landmark_connections: 2
landmark_connections: 3
landmark_connections: 3
landmark_connections: 4
landmark_connections: 4
landmark_connections: 5
landmark_connections: 5
landmark_connections: 6
landmark_connections: 6
landmark_connections: 7
landmark_connections: 7
landmark_connections: 8
landmark_connections: 9
landmark_connections: 10
landmark_connections: 10
landmark_connections: 11
landmark_connections: 11
landmark_connections: 12
landmark_connections: 12
landmark_connections: 13
landmark_connections: 13
landmark_connections: 14
landmark_connections: 0
landmark_connections: 9
landmark_connections: 8
landmark_connections: 14
landmark_color { r: 255 g: 0 b: 0 }
connection_color { r: 255 g: 0 b: 0 }
visualize_landmark_depth: false
thickness: 2.0
}
}
}
# Converts normalized rects to drawing primitives for annotation overlay.
node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:rect"
output_stream: "RENDER_DATA:rect_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}
node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:right_eye_rect_from_landmarks"
output_stream: "RENDER_DATA:right_eye_rect_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}
node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:left_eye_rect_from_landmarks"
output_stream: "RENDER_DATA:left_eye_rect_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}
node {
calculator: "IrisToDepthCalculator"
input_stream: "IRIS:iris_landmarks"
input_stream: "IMAGE_SIZE:image_size"
input_side_packet: "FOCAL_LENGTH:focal_length_pixel"
output_stream: "LEFT_IRIS_DEPTH_MM:left_iris_depth_mm"
output_stream: "RIGHT_IRIS_DEPTH_MM:right_iris_depth_mm"
}
node {
calculator: "IrisToRenderDataCalculator"
input_stream: "IRIS:iris_landmarks"
input_stream: "IMAGE_SIZE:image_size"
input_stream: "LEFT_IRIS_DEPTH_MM:left_iris_depth_mm"
input_stream: "RIGHT_IRIS_DEPTH_MM:right_iris_depth_mm"
output_stream: "RENDER_DATA:iris_render_data"
node_options: {
[type.googleapis.com/mediapipe.IrisToRenderDataCalculatorOptions] {
oval_color { r: 0 g: 0 b: 255 }
landmark_color { r: 0 g: 255 b: 0 }
oval_thickness: 4.0
landmark_thickness: 2.0
font_height_px: 50
horizontal_offset_px: 200
vertical_offset_px: 200
location: TOP_LEFT
}
}
}
# Draws annotations and overlays them on top of the input images.
node {
calculator: "AnnotationOverlayCalculator"
input_stream: "IMAGE_GPU:input_image"
input_stream: "detection_render_data"
input_stream: "right_eye_contour_landmarks_render_data"
input_stream: "left_eye_contour_landmarks_render_data"
input_stream: "iris_render_data"
output_stream: "IMAGE_GPU:output_image"
node_options: {
[type.googleapis.com/mediapipe.AnnotationOverlayCalculatorOptions] {
gpu_scale_factor: 0.5
}
}
}

245
mediapipe/graphs/iris_tracking/subgraphs/iris_renderer_cpu.pbtxt Normal file
View File

@ -0,0 +1,245 @@
# MediaPipe iris tracking rendering subgraph.
type: "IrisRendererCpu"
input_stream: "IMAGE:input_image"
input_stream: "DETECTIONS:detections"
input_stream: "FACE_LANDMARKS:face_landmarks"
input_stream: "EYE_LANDMARKS_LEFT:all_left_eye_contour_landmarks"
input_stream: "EYE_LANDMARKS_RIGHT:all_right_eye_contour_landmarks"
input_stream: "IRIS_LANDMARKS_LEFT:left_iris_landmarks"
input_stream: "IRIS_LANDMARKS_RIGHT:right_iris_landmarks"
input_stream: "NORM_RECT:rect"
input_stream: "LEFT_EYE_RECT:left_eye_rect_from_landmarks"
input_stream: "RIGHT_EYE_RECT:right_eye_rect_from_landmarks"
output_stream: "IMAGE:output_image"
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "all_left_eye_contour_landmarks"
output_stream: "left_eye_contour_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 15 }
}
}
}
node {
calculator: "SplitNormalizedLandmarkListCalculator"
input_stream: "all_right_eye_contour_landmarks"
output_stream: "right_eye_contour_landmarks"
node_options: {
[type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
ranges: { begin: 0 end: 15 }
}
}
}
# Concatenate iris landmarks from both eyes.
node {
calculator: "ConcatenateNormalizedLandmarkListCalculator"
input_stream: "left_iris_landmarks"
input_stream: "right_iris_landmarks"
output_stream: "iris_landmarks"
}
# Concatenate iris landmarks from both eyes and face landmarks.
node {
calculator: "ConcatenateNormalizedLandmarkListCalculator"
input_stream: "left_iris_landmarks"
input_stream: "right_iris_landmarks"
input_stream: "face_landmarks"
output_stream: "face_iris_landmarks"
}
node {
calculator: "ImagePropertiesCalculator"
input_stream: "IMAGE:input_image"
output_stream: "SIZE:image_size"
}
# Maps detection label IDs to the corresponding label text ("Face").
node {
calculator: "DetectionLabelIdToTextCalculator"
input_stream: "detections"
output_stream: "labeled_detections"
node_options: {
[type.googleapis.com/mediapipe.DetectionLabelIdToTextCalculatorOptions] {
label: "Face"
}
}
}
# Converts detections to drawing primitives for annotation overlay.
node {
calculator: "DetectionsToRenderDataCalculator"
input_stream: "DETECTIONS:labeled_detections"
output_stream: "RENDER_DATA:detection_render_data"
node_options: {
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
thickness: 4.0
color { r: 0 g: 255 b: 0 }
}
}
}
# Converts landmarks to drawing primitives for annotation overlay.
node {
calculator: "LandmarksToRenderDataCalculator"
input_stream: "NORM_LANDMARKS:left_eye_contour_landmarks"
output_stream: "RENDER_DATA:left_eye_contour_landmarks_render_data"
node_options: {
[type.googleapis.com/mediapipe.LandmarksToRenderDataCalculatorOptions] {
landmark_connections: 0
landmark_connections: 1
landmark_connections: 1
landmark_connections: 2
landmark_connections: 2
landmark_connections: 3
landmark_connections: 3
landmark_connections: 4
landmark_connections: 4
landmark_connections: 5
landmark_connections: 5
landmark_connections: 6
landmark_connections: 6
landmark_connections: 7
landmark_connections: 7
landmark_connections: 8
landmark_connections: 9
landmark_connections: 10
landmark_connections: 10
landmark_connections: 11
landmark_connections: 11
landmark_connections: 12
landmark_connections: 12
landmark_connections: 13
landmark_connections: 13
landmark_connections: 14
landmark_connections: 0
landmark_connections: 9
landmark_connections: 8
landmark_connections: 14
landmark_color { r: 255 g: 0 b: 0 }
connection_color { r: 255 g: 0 b: 0 }
visualize_landmark_depth: false
thickness: 1.0
}
}
}
# Converts landmarks to drawing primitives for annotation overlay.
node {
calculator: "LandmarksToRenderDataCalculator"
input_stream: "NORM_LANDMARKS:right_eye_contour_landmarks"
output_stream: "RENDER_DATA:right_eye_contour_landmarks_render_data"
node_options: {
[type.googleapis.com/mediapipe.LandmarksToRenderDataCalculatorOptions] {
landmark_connections: 0
landmark_connections: 1
landmark_connections: 1
landmark_connections: 2
landmark_connections: 2
landmark_connections: 3
landmark_connections: 3
landmark_connections: 4
landmark_connections: 4
landmark_connections: 5
landmark_connections: 5
landmark_connections: 6
landmark_connections: 6
landmark_connections: 7
landmark_connections: 7
landmark_connections: 8
landmark_connections: 9
landmark_connections: 10
landmark_connections: 10
landmark_connections: 11
landmark_connections: 11
landmark_connections: 12
landmark_connections: 12
landmark_connections: 13
landmark_connections: 13
landmark_connections: 14
landmark_connections: 0
landmark_connections: 9
landmark_connections: 8
landmark_connections: 14
landmark_color { r: 255 g: 0 b: 0 }
connection_color { r: 255 g: 0 b: 0 }
visualize_landmark_depth: false
thickness: 1.0
}
}
}
# Converts normalized rects to drawing primitives for annotation overlay.
node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:rect"
output_stream: "RENDER_DATA:rect_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}
node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:right_eye_rect_from_landmarks"
output_stream: "RENDER_DATA:right_eye_rect_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}
node {
calculator: "RectToRenderDataCalculator"
input_stream: "NORM_RECT:left_eye_rect_from_landmarks"
output_stream: "RENDER_DATA:left_eye_rect_render_data"
node_options: {
[type.googleapis.com/mediapipe.RectToRenderDataCalculatorOptions] {
filled: false
color { r: 255 g: 0 b: 0 }
thickness: 4.0
}
}
}
node {
calculator: "IrisToRenderDataCalculator"
input_stream: "IRIS:iris_landmarks"
input_stream: "IMAGE_SIZE:image_size"
output_stream: "RENDER_DATA:iris_render_data"
node_options: {
[type.googleapis.com/mediapipe.IrisToRenderDataCalculatorOptions] {
oval_color { r: 0 g: 0 b: 255 }
landmark_color { r: 0 g: 255 b: 0 }
oval_thickness: 4.0
landmark_thickness: 2.0
font_height_px: 50
horizontal_offset_px: 200
vertical_offset_px: 200
location: TOP_LEFT
}
}
}
# Draws annotations and overlays them on top of the input images.
node {
calculator: "AnnotationOverlayCalculator"
input_stream: "IMAGE:input_image"
input_stream: "detection_render_data"
input_stream: "right_eye_contour_landmarks_render_data"
input_stream: "left_eye_contour_landmarks_render_data"
input_stream: "iris_render_data"
output_stream: "IMAGE:output_image"
}

5
mediapipe/java/com/google/mediapipe/framework/BUILD
View File

@ -136,6 +136,9 @@ android_library(
# Expose the java source files for building mediapipe AAR.
filegroup(
name = "java_src",
srcs = glob(["*.java"]),
srcs = glob(
["*.java"],
exclude = ["TypeNameRegistryFull.java"],
),
visibility = ["//mediapipe:__subpackages__"],
)

3
mediapipe/java/com/google/mediapipe/framework/jni/register_natives.cc
View File

@ -181,6 +181,9 @@ void RegisterPacketCreatorNatives(JNIEnv *env) {
&packet_creator_methods, packet_creator, "nativeCreateFloatImageFrame",
"(JLjava/nio/ByteBuffer;II)J",
(void *)&PACKET_CREATOR_METHOD(nativeCreateFloatImageFrame));
AddJNINativeMethod(&packet_creator_methods, packet_creator,
"nativeCreateInt32", "(JI)J",
(void *)&PACKET_CREATOR_METHOD(nativeCreateInt32));
RegisterNativesVector(env, packet_creator_class, packet_creator_methods);
}

BIN
mediapipe/models/iris_landmark.tflite Normal file
View File

Binary file not shown.

1
mediapipe/modules/README.md
View File

@ -8,4 +8,5 @@ Each module (represented as a subfolder) provides subgraphs and corresponding re
| :--- | :--- |
| [`face_detection`](face_detection/README.md) | Subgraphs to detect faces. |
| [`face_landmark`](face_landmark/README.md) | Subgraphs to detect and track face landmarks. |
| [`iris_landmark`](iris_landmark/README.md) | Subgraphs to detect iris landmarks. |

1
mediapipe/modules/face_detection/face_detection_front_cpu.pbtxt
View File

@ -60,6 +60,7 @@ node {
options: {
[mediapipe.TfLiteInferenceCalculatorOptions.ext] {
model_path: "mediapipe/modules/face_detection/face_detection_front.tflite"
delegate { xnnpack {} }
}
}
}

1
mediapipe/modules/face_landmark/face_landmark_cpu.pbtxt
View File

@ -74,6 +74,7 @@ node {
options: {
[mediapipe.TfLiteInferenceCalculatorOptions.ext] {
model_path: "mediapipe/modules/face_landmark/face_landmark.tflite"
delegate { xnnpack {} }
}
}
}

103
mediapipe/modules/iris_landmark/BUILD Normal file
View File

@ -0,0 +1,103 @@
# Copyright 2020 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
load(
"//mediapipe/framework/tool:mediapipe_graph.bzl",
"mediapipe_simple_subgraph",
)
licenses(["notice"]) # Apache 2.0
package(default_visibility = ["//visibility:public"])
mediapipe_simple_subgraph(
name = "iris_landmark_cpu",
graph = "iris_landmark_cpu.pbtxt",
register_as = "IrisLandmarkCpu",
deps = [
"//mediapipe/calculators/core:clip_vector_size_calculator",
"//mediapipe/calculators/core:split_vector_calculator",
"//mediapipe/calculators/image:image_cropping_calculator",
"//mediapipe/calculators/image:image_properties_calculator",
"//mediapipe/calculators/image:image_transformation_calculator",
"//mediapipe/calculators/tflite:tflite_converter_calculator",
"//mediapipe/calculators/tflite:tflite_inference_calculator",
"//mediapipe/calculators/tflite:tflite_tensors_to_floats_calculator",
"//mediapipe/calculators/tflite:tflite_tensors_to_landmarks_calculator",
"//mediapipe/calculators/util:landmark_letterbox_removal_calculator",
"//mediapipe/calculators/util:landmark_projection_calculator",
],
)
mediapipe_simple_subgraph(
name = "iris_landmark_gpu",
graph = "iris_landmark_gpu.pbtxt",
register_as = "IrisLandmarkGpu",
deps = [
"//mediapipe/calculators/core:clip_vector_size_calculator",
"//mediapipe/calculators/core:split_vector_calculator",
"//mediapipe/calculators/image:image_cropping_calculator",
"//mediapipe/calculators/image:image_properties_calculator",
"//mediapipe/calculators/image:image_transformation_calculator",
"//mediapipe/calculators/tflite:tflite_converter_calculator",
"//mediapipe/calculators/tflite:tflite_inference_calculator",
"//mediapipe/calculators/tflite:tflite_tensors_to_floats_calculator",
"//mediapipe/calculators/tflite:tflite_tensors_to_landmarks_calculator",
"//mediapipe/calculators/util:landmark_letterbox_removal_calculator",
"//mediapipe/calculators/util:landmark_projection_calculator",
],
)
mediapipe_simple_subgraph(
name = "iris_landmark_left_and_right_gpu",
graph = "iris_landmark_left_and_right_gpu.pbtxt",
register_as = "IrisLandmarkLeftAndRightGpu",
deps = [
":iris_landmark_gpu",
":iris_landmark_landmarks_to_roi",
"//mediapipe/calculators/core:constant_side_packet_calculator",
"//mediapipe/calculators/core:side_packet_to_stream_calculator",
"//mediapipe/calculators/image:image_properties_calculator",
],
)
mediapipe_simple_subgraph(
name = "iris_landmark_left_and_right_cpu",
graph = "iris_landmark_left_and_right_cpu.pbtxt",
register_as = "IrisLandmarkLeftAndRightCpu",
deps = [
":iris_landmark_cpu",
":iris_landmark_landmarks_to_roi",
"//mediapipe/calculators/core:constant_side_packet_calculator",
"//mediapipe/calculators/core:side_packet_to_stream_calculator",
"//mediapipe/calculators/image:image_properties_calculator",
],
)
exports_files(
srcs = [
"iris_landmark.tflite",
],
)
mediapipe_simple_subgraph(
name = "iris_landmark_landmarks_to_roi",
graph = "iris_landmark_landmarks_to_roi.pbtxt",
register_as = "IrisLandmarkLandmarksToRoi",
deps = [
"//mediapipe/calculators/util:detections_to_rects_calculator",
"//mediapipe/calculators/util:landmarks_to_detection_calculator",
"//mediapipe/calculators/util:rect_transformation_calculator",
],
)

8
mediapipe/modules/iris_landmark/README.md Normal file
View File

@ -0,0 +1,8 @@
# iris_landmark
Subgraphs|Details
:--- | :---
[`IrisLandmarkCpu`](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark/iris_landmark_cpu.pbtxt)| Detects iris landmarks for left or right eye. (CPU input, and inference is executed on CPU.)
[`IrisLandmarkGpu`](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark/iris_landmark_gpu.pbtxt)| Detects iris landmarks for left or right eye. (GPU input, and inference is executed on GPU)
[`IrisLandmarkLeftAndRightCpu`](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark/iris_landmark_left_and_right_cpu.pbtxt)| Detects iris landmarks for both left and right eyes. (CPU input, and inference is executed on CPU)
[`IrisLandmarkLeftAndRightGpu`](https://github.com/google/mediapipe/tree/master/mediapipe/modules/iris_landmark/iris_landmark_left_and_right_gpu.pbtxt)| Detects iris landmarks for both left and right eyes. (GPU input, and inference is executed on GPU.)

Some files were not shown because too many files have changed in this diff Show More