Project import generated by Copybara.
PiperOrigin-RevId: 263982686
|
@ -20,7 +20,7 @@ Check out the [Examples page](https://mediapipe.readthedocs.io/en/latest/example
|
||||||
A web-based visualizer is hosted on [viz.mediapipe.dev](https://viz.mediapipe.dev/). Please also see instructions [here](mediapipe/docs/visualizer.md).
|
A web-based visualizer is hosted on [viz.mediapipe.dev](https://viz.mediapipe.dev/). Please also see instructions [here](mediapipe/docs/visualizer.md).
|
||||||
|
|
||||||
## Community forum
|
## Community forum
|
||||||
* [discuss](https://groups.google.com/forum/#!forum/mediapipe) - General community discussion around MediaPipe
|
* [Discuss](https://groups.google.com/forum/#!forum/mediapipe) - General community discussion around MediaPipe
|
||||||
|
|
||||||
## Publications
|
## Publications
|
||||||
* [MediaPipe: A Framework for Building Perception Pipelines](https://arxiv.org/abs/1906.08172)
|
* [MediaPipe: A Framework for Building Perception Pipelines](https://arxiv.org/abs/1906.08172)
|
||||||
|
@ -29,7 +29,7 @@ A web-based visualizer is hosted on [viz.mediapipe.dev](https://viz.mediapipe.de
|
||||||
[Open sourced at CVPR 2019](https://sites.google.com/corp/view/perception-cv4arvr/mediapipe) on June 17~20 in Long Beach, CA
|
[Open sourced at CVPR 2019](https://sites.google.com/corp/view/perception-cv4arvr/mediapipe) on June 17~20 in Long Beach, CA
|
||||||
|
|
||||||
## Alpha Disclaimer
|
## Alpha Disclaimer
|
||||||
MediaPipe is currently in alpha for v0.5. We are still making breaking API changes and expect to get to stable API by v1.0.
|
MediaPipe is currently in alpha for v0.6. We are still making breaking API changes and expect to get to stable API by v1.0.
|
||||||
|
|
||||||
## Contributing
|
## Contributing
|
||||||
We welcome contributions. Please follow these [guidelines](./CONTRIBUTING.md).
|
We welcome contributions. Please follow these [guidelines](./CONTRIBUTING.md).
|
||||||
|
|
|
@ -1 +0,0 @@
|
||||||
theme: jekyll-theme-minimal
|
|
|
@ -22,7 +22,7 @@ Android example users go through in detail. It teaches the following:
|
||||||
### Hello World! on iOS
|
### Hello World! on iOS
|
||||||
|
|
||||||
[Hello World! on iOS](./hello_world_ios.md) is the iOS version of Sobel edge
|
[Hello World! on iOS](./hello_world_ios.md) is the iOS version of Sobel edge
|
||||||
detection example
|
detection example.
|
||||||
|
|
||||||
### Object Detection with GPU
|
### Object Detection with GPU
|
||||||
|
|
||||||
|
@ -44,8 +44,9 @@ graphs can be easily adapted to run on CPU v.s. GPU.
|
||||||
[Face Detection with GPU](./face_detection_mobile_gpu.md) illustrates how to use
|
[Face Detection with GPU](./face_detection_mobile_gpu.md) illustrates how to use
|
||||||
MediaPipe with a TFLite model for face detection in a GPU-accelerated pipeline.
|
MediaPipe with a TFLite model for face detection in a GPU-accelerated pipeline.
|
||||||
The selfie face detection TFLite model is based on
|
The selfie face detection TFLite model is based on
|
||||||
["BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/blazeface).
|
["BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/blazeface),
|
||||||
[Model card](https://sites.google.com/corp/view/perception-cv4arvr/blazeface#h.p_21ojPZDx3cqq).
|
and model details are described in the
|
||||||
|
[model card](https://sites.google.com/corp/view/perception-cv4arvr/blazeface#h.p_21ojPZDx3cqq).
|
||||||
|
|
||||||
* [Android](./face_detection_mobile_gpu.md#android)
|
* [Android](./face_detection_mobile_gpu.md#android)
|
||||||
* [iOS](./face_detection_mobile_gpu.md#ios)
|
* [iOS](./face_detection_mobile_gpu.md#ios)
|
||||||
|
@ -71,8 +72,9 @@ MediaPipe with a TFLite model for hand tracking in a GPU-accelerated pipeline.
|
||||||
[Hair Segmentation on GPU](./hair_segmentation_mobile_gpu.md) illustrates how to
|
[Hair Segmentation on GPU](./hair_segmentation_mobile_gpu.md) illustrates how to
|
||||||
use MediaPipe with a TFLite model for hair segmentation in a GPU-accelerated
|
use MediaPipe with a TFLite model for hair segmentation in a GPU-accelerated
|
||||||
pipeline. The selfie hair segmentation TFLite model is based on
|
pipeline. The selfie hair segmentation TFLite model is based on
|
||||||
["Real-time Hair segmentation and recoloring on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/hair-segmentation).
|
["Real-time Hair segmentation and recoloring on Mobile GPUs"](https://sites.google.com/view/perception-cv4arvr/hair-segmentation),
|
||||||
[Model card](https://sites.google.com/corp/view/perception-cv4arvr/hair-segmentation#h.p_NimuO7PgHxlY).
|
and model details are described in the
|
||||||
|
[model card](https://sites.google.com/corp/view/perception-cv4arvr/hair-segmentation#h.p_NimuO7PgHxlY).
|
||||||
|
|
||||||
* [Android](./hair_segmentation_mobile_gpu.md#android)
|
* [Android](./hair_segmentation_mobile_gpu.md#android)
|
||||||
|
|
||||||
|
|
|
@ -4,22 +4,22 @@ This doc focuses on the
|
||||||
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt)
|
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt)
|
||||||
that performs face detection with TensorFlow Lite on GPU.
|
that performs face detection with TensorFlow Lite on GPU.
|
||||||
|
|
||||||
![face_detection_android_gpu_gif](images/mobile/face_detection_android_gpu.gif){width="300"}
|
![face_detection_android_gpu_gif](images/mobile/face_detection_android_gpu.gif)
|
||||||
|
|
||||||
## Android
|
## Android
|
||||||
|
|
||||||
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
|
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
|
||||||
general instructions to develop an Android application that uses MediaPipe.
|
general instructions to develop an Android application that uses MediaPipe.
|
||||||
|
|
||||||
The graph is used in the
|
The graph below is used in the
|
||||||
[Face Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu)
|
[Face Detection GPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu).
|
||||||
example app. To build the app, run:
|
To build the app, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu
|
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu
|
||||||
```
|
```
|
||||||
|
|
||||||
To further install the app on android device, run:
|
To further install the app on an Android device, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/facedetectiongpu.apk
|
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/facedetectiongpu.apk
|
||||||
|
@ -28,13 +28,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
|
||||||
## iOS
|
## iOS
|
||||||
|
|
||||||
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
|
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
|
||||||
instructions to develop an iOS application that uses MediaPipe. The graph below
|
instructions to develop an iOS application that uses MediaPipe.
|
||||||
is used in the
|
|
||||||
[Face Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facedetectiongpu).
|
|
||||||
|
|
||||||
To build the iOS app, please see the general
|
The graph below is used in the
|
||||||
|
[Face Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facedetectiongpu).
|
||||||
|
To build the app, please see the general
|
||||||
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
|
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
|
||||||
Specifically, run:
|
Specific to this example, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/facedetectiongpu:FaceDetectionGpuApp
|
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/facedetectiongpu:FaceDetectionGpuApp
|
||||||
|
@ -42,11 +42,13 @@ bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/facedetectiongpu:Fa
|
||||||
|
|
||||||
## Graph
|
## Graph
|
||||||
|
|
||||||
![face_detection_mobile_gpu_graph](images/mobile/face_detection_mobile_gpu.png){width="400"}
|
![face_detection_mobile_gpu_graph](images/mobile/face_detection_mobile_gpu.png)
|
||||||
|
|
||||||
To visualize the graph as shown above, copy the text specification of the graph
|
To visualize the graph as shown above, copy the text specification of the graph
|
||||||
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
|
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
|
||||||
|
|
||||||
|
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# MediaPipe graph that performs face detection with TensorFlow Lite on GPU.
|
# MediaPipe graph that performs face detection with TensorFlow Lite on GPU.
|
||||||
# Used in the example in
|
# Used in the example in
|
||||||
|
|
|
@ -1,25 +1,25 @@
|
||||||
# Hair Segmentation (GPU)
|
# Hair Segmentation (GPU)
|
||||||
|
|
||||||
This doc focuses on the
|
This doc focuses on the
|
||||||
[below example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_android_gpu.pbtxt)
|
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt)
|
||||||
that performs hair segmentation with TensorFlow Lite on GPU.
|
that performs hair segmentation with TensorFlow Lite on GPU.
|
||||||
|
|
||||||
![hair_segmentation_android_gpu_gif](images/mobile/hair_segmentation_android_gpu.gif){width="300"}
|
![hair_segmentation_android_gpu_gif](images/mobile/hair_segmentation_android_gpu.gif)
|
||||||
|
|
||||||
## Android
|
## Android
|
||||||
|
|
||||||
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
|
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
|
||||||
general instructions to develop an Android application that uses MediaPipe.
|
general instructions to develop an Android application that uses MediaPipe.
|
||||||
|
|
||||||
The graph is used in the
|
The graph below is used in the
|
||||||
[Hair Segmentation GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu)
|
[Hair Segmentation GPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu).
|
||||||
example app. To build the app, run:
|
To build the app, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu
|
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu
|
||||||
```
|
```
|
||||||
|
|
||||||
To further install the app on android device, run:
|
To further install the app on an Android device, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu/hairsegmentationgpu.apk
|
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu/hairsegmentationgpu.apk
|
||||||
|
@ -27,11 +27,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
|
||||||
|
|
||||||
## Graph
|
## Graph
|
||||||
|
|
||||||
![hair_segmentation_mobile_gpu_graph](images/mobile/hair_segmentation_mobile_gpu.png){width="600"}
|
![hair_segmentation_mobile_gpu_graph](images/mobile/hair_segmentation_mobile_gpu.png)
|
||||||
|
|
||||||
To visualize the graph as shown above, copy the text specification of the graph
|
To visualize the graph as shown above, copy the text specification of the graph
|
||||||
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
|
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
|
||||||
|
|
||||||
|
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# MediaPipe graph that performs hair segmentation with TensorFlow Lite on GPU.
|
# MediaPipe graph that performs hair segmentation with TensorFlow Lite on GPU.
|
||||||
# Used in the example in
|
# Used in the example in
|
||||||
|
|
|
@ -2,30 +2,36 @@
|
||||||
|
|
||||||
This doc focuses on the
|
This doc focuses on the
|
||||||
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
|
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
|
||||||
that performs hand detection with TensorFlow Lite on GPU. This hand detection
|
that performs hand detection with TensorFlow Lite on GPU. It is related to the
|
||||||
example is related to
|
[hand tracking example](./hand_tracking_mobile_gpu.md).
|
||||||
[hand tracking GPU example](./hand_tracking_mobile_gpu.md). Here is the
|
|
||||||
[model card](https://mediapipe.page.link/handmc) for hand detection.
|
|
||||||
|
|
||||||
For overall context on hand detection and hand tracking, please read
|
For overall context on hand detection and hand tracking, please read this
|
||||||
[this Google AI blog post](https://mediapipe.page.link/handgoogleaiblog).
|
[Google AI Blog post](https://mediapipe.page.link/handgoogleaiblog).
|
||||||
|
|
||||||
![hand_detection_android_gpu_gif](images/mobile/hand_detection_android_gpu.gif){width="300"}
|
![hand_detection_android_gpu_gif](images/mobile/hand_detection_android_gpu.gif)
|
||||||
|
|
||||||
|
In the visualization above, green boxes represent the results of palm detection,
|
||||||
|
and the red box represents the extended hand rectangle designed to cover the
|
||||||
|
entire hand. The palm detection ML model (see also
|
||||||
|
[model card](https://mediapipe.page.link/handmc)) supports detection of multiple
|
||||||
|
palms, and this example selects only the one with the highest detection
|
||||||
|
confidence score to generate the hand rectangle, to be further utilized in the
|
||||||
|
[hand tracking example](./hand_tracking_mobile_gpu.md).
|
||||||
|
|
||||||
## Android
|
## Android
|
||||||
|
|
||||||
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
|
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
|
||||||
general instructions to develop an Android application that uses MediaPipe.
|
general instructions to develop an Android application that uses MediaPipe.
|
||||||
|
|
||||||
The graph is used in the
|
The graph below is used in the
|
||||||
[Hand Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu)
|
[Hand Detection GPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu).
|
||||||
example app. To build the app, run:
|
To build the app, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu
|
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu
|
||||||
```
|
```
|
||||||
|
|
||||||
To further install the app on android device, run:
|
To further install the app on an Android device, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu/handdetectiongpu.apk
|
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu/handdetectiongpu.apk
|
||||||
|
@ -34,13 +40,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
|
||||||
## iOS
|
## iOS
|
||||||
|
|
||||||
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
|
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
|
||||||
instructions to develop an iOS application that uses MediaPipe. The graph below
|
instructions to develop an iOS application that uses MediaPipe.
|
||||||
is used in the
|
|
||||||
[Hand Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handdetectiongpu)
|
|
||||||
|
|
||||||
To build the iOS app, please see the general
|
The graph below is used in the
|
||||||
|
[Hand Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handdetectiongpu).
|
||||||
|
To build the app, please see the general
|
||||||
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
|
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
|
||||||
Specifically, run:
|
Specific to this example, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handdetectiongpu:HandDetectionGpuApp
|
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handdetectiongpu:HandDetectionGpuApp
|
||||||
|
@ -48,17 +54,18 @@ bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handdetectiongpu:Ha
|
||||||
|
|
||||||
## Graph
|
## Graph
|
||||||
|
|
||||||
The hand detection graph is
|
The hand detection [main graph](#main-graph) internally utilizes a
|
||||||
[hand_detection_mobile.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_mobile.pbtxt)
|
[hand detection subgraph](#hand-detection-subgraph). The subgraph shows up in
|
||||||
and it includes a [HandDetectionSubgraph](./framework_concepts.md#subgraph) with
|
the main graph visualization as the `HandDetection` node colored in purple, and
|
||||||
filename
|
the subgraph itself can also be visualized just like a regular graph. For more
|
||||||
[hand_detection_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
|
information on how to visualize a graph that includes subgraphs, see
|
||||||
shown as a box called `HandDetection` in purple
|
[visualizing subgraphs](./visualizer.md#visualizing-subgraphs).
|
||||||
|
|
||||||
For more information on how to visualize a graph that includes subgraphs, see
|
### Main Graph
|
||||||
[subgraph documentation](./visualizer.md#visualizing-subgraphs) for Visualizer.
|
|
||||||
|
|
||||||
![hand_detection_mobile_graph](images/mobile/hand_detection_mobile.png){width="500"}
|
![hand_detection_mobile_graph](images/mobile/hand_detection_mobile.png)
|
||||||
|
|
||||||
|
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_mobile.pbtxt)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# MediaPipe graph that performs hand detection with TensorFlow Lite on GPU.
|
# MediaPipe graph that performs hand detection with TensorFlow Lite on GPU.
|
||||||
|
@ -125,9 +132,15 @@ node {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
![hand_detection_gpu_subgraph](images/mobile/hand_detection_gpu_subgraph.png){width="500"}
|
### Hand Detection Subgraph
|
||||||
|
|
||||||
|
![hand_detection_gpu_subgraph](images/mobile/hand_detection_gpu_subgraph.png)
|
||||||
|
|
||||||
|
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# MediaPipe hand detection subgraph.
|
||||||
|
|
||||||
type: "HandDetectionSubgraph"
|
type: "HandDetectionSubgraph"
|
||||||
|
|
||||||
input_stream: "input_video"
|
input_stream: "input_video"
|
||||||
|
|
|
@ -1,32 +1,41 @@
|
||||||
# Hand Tracking (GPU)
|
# Hand Tracking (GPU)
|
||||||
|
|
||||||
This doc focuses on the
|
This doc focuses on the
|
||||||
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_android_gpu.pbtxt)
|
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt)
|
||||||
that performs hand tracking with TensorFlow Lite on GPU. This hand tracking
|
that performs hand tracking with TensorFlow Lite on GPU. It is related to the
|
||||||
example is related to
|
[hand detection example](./hand_detection_mobile_gpu.md), and we recommend users
|
||||||
[hand detection GPU example](./hand_detection_mobile_gpu.md). We recommend users
|
to review the hand detection example first.
|
||||||
to review the hand detection GPU example first. Here is the
|
|
||||||
[model card](https://mediapipe.page.link/handmc) for hand tracking.
|
|
||||||
|
|
||||||
For overall context on hand detection and hand tracking, please read
|
For overall context on hand detection and hand tracking, please read this
|
||||||
[this Google AI blog post](https://mediapipe.page.link/handgoogleaiblog).
|
[Google AI Blog post](https://mediapipe.page.link/handgoogleaiblog).
|
||||||
|
|
||||||
![hand_tracking_android_gpu.gif](images/mobile/hand_tracking_android_gpu.gif){width="300"}
|
![hand_tracking_android_gpu.gif](images/mobile/hand_tracking_android_gpu.gif)
|
||||||
|
|
||||||
|
In the visualization above, the red dots represent the localized hand landmarks,
|
||||||
|
and the green lines are simply connections between selected landmark pairs for
|
||||||
|
visualization of the hand skeleton. The red box represents a hand rectangle that
|
||||||
|
covers the entire hand, derived either from hand detection (see
|
||||||
|
[hand detection example](./hand_detection_mobile_gpu.md)) or from the pervious
|
||||||
|
round of hand landmark localization using an ML model (see also
|
||||||
|
[model card](https://mediapipe.page.link/handmc)). Hand landmark localization is
|
||||||
|
performed only within the hand rectangle for computational efficiency and
|
||||||
|
accuracy, and hand detection is only invoked when landmark localization could
|
||||||
|
not identify hand presence in the previous iteration.
|
||||||
|
|
||||||
## Android
|
## Android
|
||||||
|
|
||||||
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
|
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
|
||||||
general instructions to develop an Android application that uses MediaPipe.
|
general instructions to develop an Android application that uses MediaPipe.
|
||||||
|
|
||||||
The graph is used in the
|
The graph below is used in the
|
||||||
[Hand Tracking GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu)
|
[Hand Tracking GPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu).
|
||||||
example app. To build the app, run:
|
To build the app, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu
|
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu
|
||||||
```
|
```
|
||||||
|
|
||||||
To further install the app on android device, run:
|
To further install the app on an Android device, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/handtrackinggpu.apk
|
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/handtrackinggpu.apk
|
||||||
|
@ -35,13 +44,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
|
||||||
## iOS
|
## iOS
|
||||||
|
|
||||||
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
|
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
|
||||||
instructions to develop an iOS application that uses MediaPipe. The graph below
|
instructions to develop an iOS application that uses MediaPipe.
|
||||||
is used in the
|
|
||||||
[Hand Tracking GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handtrackinggpu)
|
|
||||||
|
|
||||||
To build the iOS app, please see the general
|
The graph below is used in the
|
||||||
|
[Hand Tracking GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handtrackinggpu).
|
||||||
|
To build the app, please see the general
|
||||||
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
|
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
|
||||||
Specifically, run:
|
Specific to this example, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp
|
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp
|
||||||
|
@ -49,20 +58,21 @@ bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handtrackinggpu:Han
|
||||||
|
|
||||||
## Graph
|
## Graph
|
||||||
|
|
||||||
For more information on how to visualize a graph that includes subgraphs, see
|
The hand tracking [main graph](#main-graph) internally utilizes a
|
||||||
[subgraph documentation](./visualizer.md#visualizing-subgraphs) for Visualizer.
|
[hand detection subgraph](#hand-detection-subgraph), a
|
||||||
|
[hand landmark subgraph](#hand-landmark-subgraph) and a
|
||||||
|
[renderer subgraph](#renderer-subgraph).
|
||||||
|
|
||||||
The hand tracking graph is
|
The subgraphs show up in the main graph visualization as nodes colored in
|
||||||
[hand_tracking_mobile.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt)
|
purple, and the subgraph itself can also be visualized just like a regular
|
||||||
and it includes 3 [subgraphs](./framework_concepts.md#subgraph):
|
graph. For more information on how to visualize a graph that includes subgraphs,
|
||||||
|
see [visualizing subgraphs](./visualizer.md#visualizing-subgraphs).
|
||||||
|
|
||||||
* [HandDetectionSubgraph - hand_detection_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
|
### Main Graph
|
||||||
|
|
||||||
* [HandLandmarkSubgraph - hand_landmark_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_landmark_gpu.pbtxt)
|
![hand_tracking_mobile_graph](images/mobile/hand_tracking_mobile.png)
|
||||||
|
|
||||||
* [RendererSubgraph - renderer_gpu.pbtxt](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/renderer_gpu.pbtxt)
|
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt)
|
||||||
|
|
||||||
![hand_tracking_mobile_graph](images/mobile/hand_tracking_mobile.png){width="400"}
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# MediaPipe graph that performs hand tracking with TensorFlow Lite on GPU.
|
# MediaPipe graph that performs hand tracking with TensorFlow Lite on GPU.
|
||||||
|
@ -152,9 +162,15 @@ node {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
![hand_detection_gpu_subgraph](images/mobile/hand_detection_gpu_subgraph.png){width="500"}
|
### Hand Detection Subgraph
|
||||||
|
|
||||||
|
![hand_detection_gpu_subgraph](images/mobile/hand_detection_gpu_subgraph.png)
|
||||||
|
|
||||||
|
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_gpu.pbtxt)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# MediaPipe hand detection subgraph.
|
||||||
|
|
||||||
type: "HandDetectionSubgraph"
|
type: "HandDetectionSubgraph"
|
||||||
|
|
||||||
input_stream: "input_video"
|
input_stream: "input_video"
|
||||||
|
@ -352,7 +368,11 @@ node {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
![hand_landmark_gpu_subgraph.pbtxt](images/mobile/hand_landmark_gpu_subgraph.png){width="400"}
|
### Hand Landmark Subgraph
|
||||||
|
|
||||||
|
![hand_landmark_gpu_subgraph.pbtxt](images/mobile/hand_landmark_gpu_subgraph.png)
|
||||||
|
|
||||||
|
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_landmark_gpu.pbtxt)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# MediaPipe hand landmark localization subgraph.
|
# MediaPipe hand landmark localization subgraph.
|
||||||
|
@ -532,7 +552,11 @@ node {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
![hand_renderer_gpu_subgraph.pbtxt](images/mobile/hand_renderer_gpu_subgraph.png){width="500"}
|
### Renderer Subgraph
|
||||||
|
|
||||||
|
![hand_renderer_gpu_subgraph.pbtxt](images/mobile/hand_renderer_gpu_subgraph.png)
|
||||||
|
|
||||||
|
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/renderer_gpu.pbtxt)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# MediaPipe hand tracking rendering subgraph.
|
# MediaPipe hand tracking rendering subgraph.
|
||||||
|
|
|
@ -14,7 +14,7 @@ graph on Android.
|
||||||
A simple camera app for real-time Sobel edge detection applied to a live video
|
A simple camera app for real-time Sobel edge detection applied to a live video
|
||||||
stream on an Android device.
|
stream on an Android device.
|
||||||
|
|
||||||
![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif){width="300"}
|
![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif)
|
||||||
|
|
||||||
## Setup
|
## Setup
|
||||||
|
|
||||||
|
@ -56,7 +56,7 @@ node: {
|
||||||
|
|
||||||
A visualization of the graph is shown below:
|
A visualization of the graph is shown below:
|
||||||
|
|
||||||
![edge_detection_mobile_gpu_graph](images/mobile/edge_detection_mobile_graph_gpu.png){width="200"}
|
![edge_detection_mobile_gpu](images/mobile/edge_detection_mobile_gpu.png)
|
||||||
|
|
||||||
This graph has a single input stream named `input_video` for all incoming frames
|
This graph has a single input stream named `input_video` for all incoming frames
|
||||||
that will be provided by your device's camera.
|
that will be provided by your device's camera.
|
||||||
|
@ -252,7 +252,7 @@ adb install bazel-bin/$APPLICATION_PATH/edgedetectiongpu.apk
|
||||||
Open the application on your device. It should display a screen with the text
|
Open the application on your device. It should display a screen with the text
|
||||||
`Hello World!`.
|
`Hello World!`.
|
||||||
|
|
||||||
![bazel_hello_world_android](images/mobile/bazel_hello_world_android.png){width="300"}
|
![bazel_hello_world_android](images/mobile/bazel_hello_world_android.png)
|
||||||
|
|
||||||
## Using the camera via `CameraX`
|
## Using the camera via `CameraX`
|
||||||
|
|
||||||
|
@ -369,7 +369,7 @@ Add the following line in the `$APPLICATION_PATH/res/values/strings.xml` file:
|
||||||
When the user doesn't grant camera permission, the screen will now look like
|
When the user doesn't grant camera permission, the screen will now look like
|
||||||
this:
|
this:
|
||||||
|
|
||||||
![missing_camera_permission_android](images/mobile/missing_camera_permission_android.png){width="300"}
|
![missing_camera_permission_android](images/mobile/missing_camera_permission_android.png)
|
||||||
|
|
||||||
Now, we will add the [`SurfaceTexture`] and [`SurfaceView`] objects to
|
Now, we will add the [`SurfaceTexture`] and [`SurfaceView`] objects to
|
||||||
`MainActivity`:
|
`MainActivity`:
|
||||||
|
@ -709,7 +709,7 @@ And that's it! You should now be able to successfully build and run the
|
||||||
application on the device and see Sobel edge detection running on a live camera
|
application on the device and see Sobel edge detection running on a live camera
|
||||||
feed! Congrats!
|
feed! Congrats!
|
||||||
|
|
||||||
![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif){width="300"}
|
![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif)
|
||||||
|
|
||||||
If you ran into any issues, please see the full code of the tutorial
|
If you ran into any issues, please see the full code of the tutorial
|
||||||
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/edgedetectiongpu).
|
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/edgedetectiongpu).
|
||||||
|
|
|
@ -72,7 +72,7 @@
|
||||||
This graph consists of 1 graph input stream (`in`) and 1 graph output stream
|
This graph consists of 1 graph input stream (`in`) and 1 graph output stream
|
||||||
(`out`), and 2 [`PassThroughCalculator`]s connected serially.
|
(`out`), and 2 [`PassThroughCalculator`]s connected serially.
|
||||||
|
|
||||||
![hello_world.cc graph](./images/hello_world_graph.png){width="200"}
|
![hello_world graph](./images/hello_world.png)
|
||||||
|
|
||||||
4. Before running the graph, an `OutputStreamPoller` object is connected to the
|
4. Before running the graph, an `OutputStreamPoller` object is connected to the
|
||||||
output stream in order to later retrieve the graph output, and a graph run
|
output stream in order to later retrieve the graph output, and a graph run
|
||||||
|
|
|
@ -14,7 +14,7 @@ graph on iOS.
|
||||||
A simple camera app for real-time Sobel edge detection applied to a live video
|
A simple camera app for real-time Sobel edge detection applied to a live video
|
||||||
stream on an iOS device.
|
stream on an iOS device.
|
||||||
|
|
||||||
![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif){width="300"}
|
![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif)
|
||||||
|
|
||||||
## Setup
|
## Setup
|
||||||
|
|
||||||
|
@ -54,7 +54,7 @@ node: {
|
||||||
|
|
||||||
A visualization of the graph is shown below:
|
A visualization of the graph is shown below:
|
||||||
|
|
||||||
![edge_detection_mobile_gpu_graph](images/mobile/edge_detection_mobile_graph_gpu.png){width="200"}
|
![edge_detection_mobile_gpu](images/mobile/edge_detection_mobile_gpu.png)
|
||||||
|
|
||||||
This graph has a single input stream named `input_video` for all incoming frames
|
This graph has a single input stream named `input_video` for all incoming frames
|
||||||
that will be provided by your device's camera.
|
that will be provided by your device's camera.
|
||||||
|
@ -174,10 +174,11 @@ bazel build -c opt --config=ios_arm64 <$APPLICATION_PATH>:EdgeDetectionGpuApp'
|
||||||
```
|
```
|
||||||
|
|
||||||
For example, to build the `EdgeDetectionGpuApp` application in
|
For example, to build the `EdgeDetectionGpuApp` application in
|
||||||
`mediapipe/examples/ios/edgedetection`, use the following command:
|
`mediapipe/examples/ios/edgedetectiongpu`, use the following
|
||||||
|
command:
|
||||||
|
|
||||||
```
|
```
|
||||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/edgedetection:EdgeDetectionGpuApp
|
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/edgedetectiongpu:EdgeDetectionGpuApp
|
||||||
```
|
```
|
||||||
|
|
||||||
Then, go back to XCode, open Window > Devices and Simulators, select your
|
Then, go back to XCode, open Window > Devices and Simulators, select your
|
||||||
|
@ -188,9 +189,9 @@ blank white screen.
|
||||||
|
|
||||||
## Use the camera for the live view feed
|
## Use the camera for the live view feed
|
||||||
|
|
||||||
In this tutorial, we will use the `MediaPipeCameraInputSource` class to access
|
In this tutorial, we will use the `MPPCameraInputSource` class to access and
|
||||||
and grab frames from the camera. This class uses the `AVCaptureSession` API to
|
grab frames from the camera. This class uses the `AVCaptureSession` API to get
|
||||||
get the frames from the camera.
|
the frames from the camera.
|
||||||
|
|
||||||
But before using this class, change the `Info.plist` file to support camera
|
But before using this class, change the `Info.plist` file to support camera
|
||||||
usage in the app.
|
usage in the app.
|
||||||
|
@ -198,7 +199,7 @@ usage in the app.
|
||||||
In `ViewController.m`, add the following import line:
|
In `ViewController.m`, add the following import line:
|
||||||
|
|
||||||
```
|
```
|
||||||
#import "mediapipe/objc/MediaPipeCameraInputSource.h"
|
#import "mediapipe/objc/MPPCameraInputSource.h"
|
||||||
```
|
```
|
||||||
|
|
||||||
Add the following to its implementation block to create an object
|
Add the following to its implementation block to create an object
|
||||||
|
@ -207,7 +208,7 @@ Add the following to its implementation block to create an object
|
||||||
```
|
```
|
||||||
@implementation ViewController {
|
@implementation ViewController {
|
||||||
// Handles camera access via AVCaptureSession library.
|
// Handles camera access via AVCaptureSession library.
|
||||||
MediaPipeCameraInputSource* _cameraSource;
|
MPPCameraInputSource* _cameraSource;
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
@ -217,7 +218,7 @@ Add the following code to `viewDidLoad()`:
|
||||||
-(void)viewDidLoad {
|
-(void)viewDidLoad {
|
||||||
[super viewDidLoad];
|
[super viewDidLoad];
|
||||||
|
|
||||||
_cameraSource = [[MediaPipeCameraInputSource alloc] init];
|
_cameraSource = [[MPPCameraInputSource alloc] init];
|
||||||
_cameraSource.sessionPreset = AVCaptureSessionPresetHigh;
|
_cameraSource.sessionPreset = AVCaptureSessionPresetHigh;
|
||||||
_cameraSource.cameraPosition = AVCaptureDevicePositionBack;
|
_cameraSource.cameraPosition = AVCaptureDevicePositionBack;
|
||||||
// The frame's native format is rotated with respect to the portrait orientation.
|
// The frame's native format is rotated with respect to the portrait orientation.
|
||||||
|
@ -229,10 +230,10 @@ The code initializes `_cameraSource`, sets the capture session preset, and which
|
||||||
camera to use.
|
camera to use.
|
||||||
|
|
||||||
We need to get frames from the `_cameraSource` into our application
|
We need to get frames from the `_cameraSource` into our application
|
||||||
`ViewController` to display them. `MediaPipeCameraInputSource` is a subclass of
|
`ViewController` to display them. `MPPCameraInputSource` is a subclass of
|
||||||
`MediaPipeInputSource`, which provides a protocol for its delegates, namely the
|
`MPPInputSource`, which provides a protocol for its delegates, namely the
|
||||||
`MediaPipeInputSourceDelegate`. So our application `ViewController` can be a
|
`MPPInputSourceDelegate`. So our application `ViewController` can be a delegate
|
||||||
delegate of `_cameraSource`.
|
of `_cameraSource`.
|
||||||
|
|
||||||
To handle camera setup and process incoming frames, we should use a queue
|
To handle camera setup and process incoming frames, we should use a queue
|
||||||
different from the main queue. Add the following to the implementation block of
|
different from the main queue. Add the following to the implementation block of
|
||||||
|
@ -269,11 +270,11 @@ the interface/implementation of the `ViewController`:
|
||||||
static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
|
static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
|
||||||
```
|
```
|
||||||
|
|
||||||
Before implementing any method from `MediaPipeInputSourceDelegate` protocol, we
|
Before implementing any method from `MPPInputSourceDelegate` protocol, we must
|
||||||
must first set up a way to display the camera frames. MediaPipe provides another
|
first set up a way to display the camera frames. MediaPipe provides another
|
||||||
utility called `MediaPipeLayerRenderer` to display images on the screen. This
|
utility called `MPPLayerRenderer` to display images on the screen. This utility
|
||||||
utility can be used to display `CVPixelBufferRef` objects, which is the type of
|
can be used to display `CVPixelBufferRef` objects, which is the type of the
|
||||||
the images provided by `MediaPipeCameraInputSource` to its delegates.
|
images provided by `MPPCameraInputSource` to its delegates.
|
||||||
|
|
||||||
To display images of the screen, we need to add a new `UIView` object called
|
To display images of the screen, we need to add a new `UIView` object called
|
||||||
`_liveView` to the `ViewController`.
|
`_liveView` to the `ViewController`.
|
||||||
|
@ -284,7 +285,7 @@ Add the following lines to the implementation block of the `ViewController`:
|
||||||
// Display the camera preview frames.
|
// Display the camera preview frames.
|
||||||
IBOutlet UIView* _liveView;
|
IBOutlet UIView* _liveView;
|
||||||
// Render frames in a layer.
|
// Render frames in a layer.
|
||||||
MediaPipeLayerRenderer* _renderer;
|
MPPLayerRenderer* _renderer;
|
||||||
```
|
```
|
||||||
|
|
||||||
Go to `Main.storyboard`, add a `UIView` object from the object library to the
|
Go to `Main.storyboard`, add a `UIView` object from the object library to the
|
||||||
|
@ -296,7 +297,7 @@ Go back to `ViewController.m` and add the following code to `viewDidLoad()` to
|
||||||
initialize the `_renderer` object:
|
initialize the `_renderer` object:
|
||||||
|
|
||||||
```
|
```
|
||||||
_renderer = [[MediaPipeLayerRenderer alloc] init];
|
_renderer = [[MPPLayerRenderer alloc] init];
|
||||||
_renderer.layer.frame = _liveView.layer.bounds;
|
_renderer.layer.frame = _liveView.layer.bounds;
|
||||||
[_liveView.layer addSublayer:_renderer.layer];
|
[_liveView.layer addSublayer:_renderer.layer];
|
||||||
_renderer.frameScaleMode = MediaPipeFrameScaleFillAndCrop;
|
_renderer.frameScaleMode = MediaPipeFrameScaleFillAndCrop;
|
||||||
|
@ -308,7 +309,7 @@ To get frames from the camera, we will implement the following method:
|
||||||
// Must be invoked on _videoQueue.
|
// Must be invoked on _videoQueue.
|
||||||
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
|
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
|
||||||
timestamp:(CMTime)timestamp
|
timestamp:(CMTime)timestamp
|
||||||
fromSource:(MediaPipeInputSource*)source {
|
fromSource:(MPPInputSource*)source {
|
||||||
if (source != _cameraSource) {
|
if (source != _cameraSource) {
|
||||||
NSLog(@"Unknown source: %@", source);
|
NSLog(@"Unknown source: %@", source);
|
||||||
return;
|
return;
|
||||||
|
@ -322,7 +323,7 @@ To get frames from the camera, we will implement the following method:
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
This is a delegate method of `MediaPipeInputSource`. We first check that we are
|
This is a delegate method of `MPPInputSource`. We first check that we are
|
||||||
getting frames from the right source, i.e. the `_cameraSource`. Then we display
|
getting frames from the right source, i.e. the `_cameraSource`. Then we display
|
||||||
the frame received from the camera via `_renderer` on the main queue.
|
the frame received from the camera via `_renderer` on the main queue.
|
||||||
|
|
||||||
|
@ -337,7 +338,7 @@ about to appear. To do this, we will implement the
|
||||||
```
|
```
|
||||||
|
|
||||||
Before we start running the camera, we need the user's permission to access it.
|
Before we start running the camera, we need the user's permission to access it.
|
||||||
`MediaPipeCameraInputSource` provides a function
|
`MPPCameraInputSource` provides a function
|
||||||
`requestCameraAccessWithCompletionHandler:(void (^_Nullable)(BOOL
|
`requestCameraAccessWithCompletionHandler:(void (^_Nullable)(BOOL
|
||||||
granted))handler` to request camera access and do some work when the user has
|
granted))handler` to request camera access and do some work when the user has
|
||||||
responded. Add the following code to `viewWillAppear:animated`:
|
responded. Add the following code to `viewWillAppear:animated`:
|
||||||
|
@ -413,7 +414,7 @@ Add the following property to the interface of the `ViewController`:
|
||||||
```
|
```
|
||||||
// The MediaPipe graph currently in use. Initialized in viewDidLoad, started in viewWillAppear: and
|
// The MediaPipe graph currently in use. Initialized in viewDidLoad, started in viewWillAppear: and
|
||||||
// sent video frames on _videoQueue.
|
// sent video frames on _videoQueue.
|
||||||
@property(nonatomic) MediaPipeGraph* mediapipeGraph;
|
@property(nonatomic) MPPGraph* mediapipeGraph;
|
||||||
```
|
```
|
||||||
|
|
||||||
As explained in the comment above, we will initialize this graph in
|
As explained in the comment above, we will initialize this graph in
|
||||||
|
@ -421,7 +422,7 @@ As explained in the comment above, we will initialize this graph in
|
||||||
using the following function:
|
using the following function:
|
||||||
|
|
||||||
```
|
```
|
||||||
+ (MediaPipeGraph*)loadGraphFromResource:(NSString*)resource {
|
+ (MPPGraph*)loadGraphFromResource:(NSString*)resource {
|
||||||
// Load the graph config resource.
|
// Load the graph config resource.
|
||||||
NSError* configLoadError = nil;
|
NSError* configLoadError = nil;
|
||||||
NSBundle* bundle = [NSBundle bundleForClass:[self class]];
|
NSBundle* bundle = [NSBundle bundleForClass:[self class]];
|
||||||
|
@ -440,7 +441,7 @@ using the following function:
|
||||||
config.ParseFromArray(data.bytes, data.length);
|
config.ParseFromArray(data.bytes, data.length);
|
||||||
|
|
||||||
// Create MediaPipe graph with mediapipe::CalculatorGraphConfig proto object.
|
// Create MediaPipe graph with mediapipe::CalculatorGraphConfig proto object.
|
||||||
MediaPipeGraph* newGraph = [[MediaPipeGraph alloc] initWithGraphConfig:config];
|
MPPGraph* newGraph = [[MPPGraph alloc] initWithGraphConfig:config];
|
||||||
[newGraph addFrameOutputStream:kOutputStream outputPacketType:MediaPipePacketPixelBuffer];
|
[newGraph addFrameOutputStream:kOutputStream outputPacketType:MediaPipePacketPixelBuffer];
|
||||||
return newGraph;
|
return newGraph;
|
||||||
}
|
}
|
||||||
|
@ -498,7 +499,7 @@ this function's implementation to do the following:
|
||||||
```
|
```
|
||||||
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
|
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
|
||||||
timestamp:(CMTime)timestamp
|
timestamp:(CMTime)timestamp
|
||||||
fromSource:(MediaPipeInputSource*)source {
|
fromSource:(MPPInputSource*)source {
|
||||||
if (source != _cameraSource) {
|
if (source != _cameraSource) {
|
||||||
NSLog(@"Unknown source: %@", source);
|
NSLog(@"Unknown source: %@", source);
|
||||||
return;
|
return;
|
||||||
|
@ -518,7 +519,7 @@ The graph will run with this input packet and output a result in
|
||||||
method to receive packets on this output stream and display them on the screen:
|
method to receive packets on this output stream and display them on the screen:
|
||||||
|
|
||||||
```
|
```
|
||||||
- (void)mediapipeGraph:(MediaPipeGraph*)graph
|
- (void)mediapipeGraph:(MPPGraph*)graph
|
||||||
didOutputPixelBuffer:(CVPixelBufferRef)pixelBuffer
|
didOutputPixelBuffer:(CVPixelBufferRef)pixelBuffer
|
||||||
fromStream:(const std::string&)streamName {
|
fromStream:(const std::string&)streamName {
|
||||||
if (streamName == kOutputStream) {
|
if (streamName == kOutputStream) {
|
||||||
|
@ -535,7 +536,7 @@ method to receive packets on this output stream and display them on the screen:
|
||||||
And that is all! Build and run the app on your iOS device. You should see the
|
And that is all! Build and run the app on your iOS device. You should see the
|
||||||
results of running the edge detection graph on a live video feed. Congrats!
|
results of running the edge detection graph on a live video feed. Congrats!
|
||||||
|
|
||||||
![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif){width="300"}
|
![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif)
|
||||||
|
|
||||||
If you ran into any issues, please see the full code of the tutorial
|
If you ran into any issues, please see the full code of the tutorial
|
||||||
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/edgedetectiongpu).
|
[here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/edgedetectiongpu).
|
||||||
|
|
BIN
mediapipe/docs/images/hello_world.png
Normal file
After Width: | Height: | Size: 9.6 KiB |
Before Width: | Height: | Size: 39 KiB |
Before Width: | Height: | Size: 24 KiB After Width: | Height: | Size: 18 KiB |
Before Width: | Height: | Size: 1.4 MiB After Width: | Height: | Size: 666 KiB |
Before Width: | Height: | Size: 964 KiB After Width: | Height: | Size: 529 KiB |
BIN
mediapipe/docs/images/mobile/edge_detection_mobile_gpu.png
Normal file
After Width: | Height: | Size: 12 KiB |
Before Width: | Height: | Size: 37 KiB |
Before Width: | Height: | Size: 1.8 MiB After Width: | Height: | Size: 1.8 MiB |
Before Width: | Height: | Size: 80 KiB After Width: | Height: | Size: 64 KiB |
Before Width: | Height: | Size: 105 KiB After Width: | Height: | Size: 84 KiB |
Before Width: | Height: | Size: 65 KiB After Width: | Height: | Size: 49 KiB |
Before Width: | Height: | Size: 28 KiB After Width: | Height: | Size: 20 KiB |
Before Width: | Height: | Size: 94 KiB |
Before Width: | Height: | Size: 80 KiB |
Before Width: | Height: | Size: 220 KiB After Width: | Height: | Size: 94 KiB |
Before Width: | Height: | Size: 156 KiB After Width: | Height: | Size: 64 KiB |
|
@ -8,9 +8,9 @@ machine learning pipeline can be built as a graph of modular components,
|
||||||
including, for instance, inference models and media processing functions. Sensory
|
including, for instance, inference models and media processing functions. Sensory
|
||||||
data such as audio and video streams enter the graph, and perceived descriptions
|
data such as audio and video streams enter the graph, and perceived descriptions
|
||||||
such as object-localization and face-landmark streams exit the graph. An example
|
such as object-localization and face-landmark streams exit the graph. An example
|
||||||
graph that performs real-time hair segmentation on mobile GPU is shown below.
|
graph that performs real-time hand tracking on mobile GPU is shown below.
|
||||||
|
|
||||||
.. image:: images/mobile/hair_segmentation_android_gpu.png
|
.. image:: images/mobile/hand_tracking_mobile.png
|
||||||
:width: 400
|
:width: 400
|
||||||
:alt: Example MediaPipe graph
|
:alt: Example MediaPipe graph
|
||||||
|
|
||||||
|
@ -29,11 +29,11 @@ APIs for MediaPipe
|
||||||
* (Coming Soon) Graph Construction API in C++
|
* (Coming Soon) Graph Construction API in C++
|
||||||
* Graph Execution API in C++
|
* Graph Execution API in C++
|
||||||
* Graph Execution API in Java (Android)
|
* Graph Execution API in Java (Android)
|
||||||
* (Coming Soon) Graph Execution API in Objective-C (iOS)
|
* Graph Execution API in Objective-C (iOS)
|
||||||
|
|
||||||
Alpha Disclaimer
|
Alpha Disclaimer
|
||||||
==================
|
==================
|
||||||
MediaPipe is currently in alpha for v0.5. We are still making breaking API changes and expect to get to stable API by v1.0. We recommend that you target a specific version of MediaPipe, and periodically bump to the latest release. That way you have control over when a breaking change affects you.
|
MediaPipe is currently in alpha for v0.6. We are still making breaking API changes and expect to get to stable API by v1.0. We recommend that you target a specific version of MediaPipe, and periodically bump to the latest release. That way you have control over when a breaking change affects you.
|
||||||
|
|
||||||
User Documentation
|
User Documentation
|
||||||
==================
|
==================
|
||||||
|
|
|
@ -44,7 +44,7 @@ $ bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tensorf
|
||||||
|
|
||||||
#### Graph
|
#### Graph
|
||||||
|
|
||||||
![graph visualization](images/object_detection_desktop_tensorflow.png){width="800"}
|
![graph visualization](images/object_detection_desktop_tensorflow.png)
|
||||||
|
|
||||||
To visualize the graph as shown above, copy the text specification of the graph
|
To visualize the graph as shown above, copy the text specification of the graph
|
||||||
below and paste it into
|
below and paste it into
|
||||||
|
@ -209,7 +209,7 @@ $ bazel-bin/mediapipe/examples/desktop/object_detection/object_detection_tflite
|
||||||
|
|
||||||
#### Graph
|
#### Graph
|
||||||
|
|
||||||
![graph visualization](images/object_detection_desktop_tflite.png){width="400"}
|
![graph visualization](images/object_detection_desktop_tflite.png)
|
||||||
|
|
||||||
To visualize the graph as shown above, copy the text specification of the graph
|
To visualize the graph as shown above, copy the text specification of the graph
|
||||||
below and paste it into
|
below and paste it into
|
||||||
|
|
|
@ -1,8 +1,6 @@
|
||||||
# Object Detection (CPU)
|
# Object Detection (CPU)
|
||||||
|
|
||||||
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
|
This doc focuses on the
|
||||||
general instructions to develop an Android application that uses MediaPipe. This
|
|
||||||
doc focuses on the
|
|
||||||
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_cpu.pbtxt)
|
[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_cpu.pbtxt)
|
||||||
that performs object detection with TensorFlow Lite on CPU.
|
that performs object detection with TensorFlow Lite on CPU.
|
||||||
|
|
||||||
|
@ -11,22 +9,25 @@ This is very similar to the
|
||||||
except that at the beginning and the end of the graph it performs GPU-to-CPU and
|
except that at the beginning and the end of the graph it performs GPU-to-CPU and
|
||||||
CPU-to-GPU image transfer respectively. As a result, the rest of graph, which
|
CPU-to-GPU image transfer respectively. As a result, the rest of graph, which
|
||||||
shares the same configuration as the
|
shares the same configuration as the
|
||||||
[GPU graph](images/mobile/object_detection_android_gpu.png), runs entirely on
|
[GPU graph](images/mobile/object_detection_mobile_gpu.png), runs entirely on
|
||||||
CPU.
|
CPU.
|
||||||
|
|
||||||
![object_detection_android_cpu_gif](images/mobile/object_detection_android_cpu.gif){width="300"}
|
![object_detection_android_cpu_gif](images/mobile/object_detection_android_cpu.gif)
|
||||||
|
|
||||||
## Android
|
## Android
|
||||||
|
|
||||||
The graph is used in the
|
Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
|
||||||
[Object Detection CPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu)
|
general instructions to develop an Android application that uses MediaPipe.
|
||||||
example app. To build the app, run:
|
|
||||||
|
The graph below is used in the
|
||||||
|
[Object Detection CPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu).
|
||||||
|
To build the app, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu
|
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu
|
||||||
```
|
```
|
||||||
|
|
||||||
To further install the app on android device, run:
|
To further install the app on an Android device, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu/objectdetectioncpu.apk
|
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectioncpu/objectdetectioncpu.apk
|
||||||
|
@ -35,13 +36,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
|
||||||
## iOS
|
## iOS
|
||||||
|
|
||||||
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
|
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
|
||||||
instructions to develop an iOS application that uses MediaPipe. The graph below
|
instructions to develop an iOS application that uses MediaPipe.
|
||||||
is used in the
|
|
||||||
[Object Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/objectdetectioncpu).
|
|
||||||
|
|
||||||
To build the iOS app, please see the general
|
The graph below is used in the
|
||||||
|
[Object Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/objectdetectioncpu).
|
||||||
|
To build the app, please see the general
|
||||||
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
|
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
|
||||||
Specifically, run:
|
Specific to this example, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectioncpu:ObjectDetectionCpuApp
|
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectioncpu:ObjectDetectionCpuApp
|
||||||
|
@ -49,11 +50,13 @@ bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectioncpu:
|
||||||
|
|
||||||
## Graph
|
## Graph
|
||||||
|
|
||||||
![object_detection_mobile_cpu_graph](images/mobile/object_detection_mobile_cpu.png){width="400"}
|
![object_detection_mobile_cpu_graph](images/mobile/object_detection_mobile_cpu.png)
|
||||||
|
|
||||||
To visualize the graph as shown above, copy the text specification of the graph
|
To visualize the graph as shown above, copy the text specification of the graph
|
||||||
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
|
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
|
||||||
|
|
||||||
|
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_cpu.pbtxt)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# MediaPipe graph that performs object detection with TensorFlow Lite on CPU.
|
# MediaPipe graph that performs object detection with TensorFlow Lite on CPU.
|
||||||
# Used in the example in
|
# Used in the example in
|
||||||
|
|
|
@ -4,7 +4,7 @@ This doc focuses on the
|
||||||
[below example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt)
|
[below example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt)
|
||||||
that performs object detection with TensorFlow Lite on GPU.
|
that performs object detection with TensorFlow Lite on GPU.
|
||||||
|
|
||||||
![object_detection_android_gpu_gif](images/mobile/object_detection_android_gpu.gif){width="300"}
|
![object_detection_android_gpu_gif](images/mobile/object_detection_android_gpu.gif)
|
||||||
|
|
||||||
## Android
|
## Android
|
||||||
|
|
||||||
|
@ -12,14 +12,14 @@ Please see [Hello World! in MediaPipe on Android](hello_world_android.md) for
|
||||||
general instructions to develop an Android application that uses MediaPipe.
|
general instructions to develop an Android application that uses MediaPipe.
|
||||||
|
|
||||||
The graph below is used in the
|
The graph below is used in the
|
||||||
[Object Detection GPU](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu)
|
[Object Detection GPU Android example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu).
|
||||||
example app. To build the app, run:
|
To build the app, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu
|
bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu
|
||||||
```
|
```
|
||||||
|
|
||||||
To further install the app on android device, run:
|
To further install the app on an Android device, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu.apk
|
adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu.apk
|
||||||
|
@ -28,13 +28,13 @@ adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/a
|
||||||
## iOS
|
## iOS
|
||||||
|
|
||||||
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
|
Please see [Hello World! in MediaPipe on iOS](hello_world_ios.md) for general
|
||||||
instructions to develop an iOS application that uses MediaPipe. The graph below
|
instructions to develop an iOS application that uses MediaPipe.
|
||||||
is used in the
|
|
||||||
[Object Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/objectdetectiongpu)
|
|
||||||
|
|
||||||
To build the iOS app, please see the general
|
The graph below is used in the
|
||||||
|
[Object Detection GPU iOS example app](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/objectdetectiongpu).
|
||||||
|
To build the app, please see the general
|
||||||
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
|
[MediaPipe iOS app building and setup instructions](./mediapipe_ios_setup.md).
|
||||||
Specifically, run:
|
Specific to this example, run:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectiongpu:ObjectDetectionGpuApp
|
bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectiongpu:ObjectDetectionGpuApp
|
||||||
|
@ -42,11 +42,13 @@ bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/objectdetectiongpu:
|
||||||
|
|
||||||
## Graph
|
## Graph
|
||||||
|
|
||||||
![object_detection_mobile_gpu_graph](images/mobile/object_detection_mobile_gpu.png){width="400"}
|
![object_detection_mobile_gpu_graph](images/mobile/object_detection_mobile_gpu.png)
|
||||||
|
|
||||||
To visualize the graph as shown above, copy the text specification of the graph
|
To visualize the graph as shown above, copy the text specification of the graph
|
||||||
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
|
below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev/).
|
||||||
|
|
||||||
|
[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/object_detection/object_detection_mobile_gpu.pbtxt)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# MediaPipe graph that performs object detection with TensorFlow Lite on GPU.
|
# MediaPipe graph that performs object detection with TensorFlow Lite on GPU.
|
||||||
# Used in the example in
|
# Used in the example in
|
||||||
|
|
|
@ -79,4 +79,4 @@ and its associated [subgraph](./framework_concepts.md#subgraph) called
|
||||||
|
|
||||||
* Click on the subgraph block in purple `Hand Detection` and the
|
* Click on the subgraph block in purple `Hand Detection` and the
|
||||||
`hand_detection_gpu.pbtxt` tab will open
|
`hand_detection_gpu.pbtxt` tab will open
|
||||||
![Hand detection subgraph](./images/clicksubgraph_handdetection.png){width="1500"}
|
![Hand detection subgraph](./images/click_subgraph_handdetection.png){width="1500"}
|
||||||
|
|
|
@ -0,0 +1,33 @@
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
|
||||||
|
package="com.google.mediapipe.apps.handdetectiongpu">
|
||||||
|
|
||||||
|
<uses-sdk
|
||||||
|
android:minSdkVersion="21"
|
||||||
|
android:targetSdkVersion="27" />
|
||||||
|
|
||||||
|
<!-- For using the camera -->
|
||||||
|
<uses-permission android:name="android.permission.CAMERA" />
|
||||||
|
<uses-feature android:name="android.hardware.camera" />
|
||||||
|
<uses-feature android:name="android.hardware.camera.autofocus" />
|
||||||
|
<!-- For MediaPipe -->
|
||||||
|
<uses-feature android:glEsVersion="0x00020000" android:required="true" />
|
||||||
|
|
||||||
|
|
||||||
|
<application
|
||||||
|
android:allowBackup="true"
|
||||||
|
android:label="@string/app_name"
|
||||||
|
android:supportsRtl="true"
|
||||||
|
android:theme="@style/AppTheme">
|
||||||
|
<activity
|
||||||
|
android:name=".MainActivity"
|
||||||
|
android:exported="true"
|
||||||
|
android:screenOrientation="portrait">
|
||||||
|
<intent-filter>
|
||||||
|
<action android:name="android.intent.action.MAIN" />
|
||||||
|
<category android:name="android.intent.category.LAUNCHER" />
|
||||||
|
</intent-filter>
|
||||||
|
</activity>
|
||||||
|
</application>
|
||||||
|
|
||||||
|
</manifest>
|
|
@ -0,0 +1,83 @@
|
||||||
|
# Copyright 2019 The MediaPipe Authors.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
licenses(["notice"]) # Apache 2.0
|
||||||
|
|
||||||
|
package(default_visibility = ["//visibility:private"])
|
||||||
|
|
||||||
|
cc_binary(
|
||||||
|
name = "libmediapipe_jni.so",
|
||||||
|
linkshared = 1,
|
||||||
|
linkstatic = 1,
|
||||||
|
deps = [
|
||||||
|
"//mediapipe/graphs/hand_tracking:detection_mobile_calculators",
|
||||||
|
"//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
cc_library(
|
||||||
|
name = "mediapipe_jni_lib",
|
||||||
|
srcs = [":libmediapipe_jni.so"],
|
||||||
|
alwayslink = 1,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Maps the binary graph to an alias (e.g., the app name) for convenience so that the alias can be
|
||||||
|
# easily incorporated into the app via, for example,
|
||||||
|
# MainActivity.BINARY_GRAPH_NAME = "appname.binarypb".
|
||||||
|
genrule(
|
||||||
|
name = "binary_graph",
|
||||||
|
srcs = ["//mediapipe/graphs/hand_tracking:hand_detection_mobile_gpu_binary_graph"],
|
||||||
|
outs = ["handdetectiongpu.binarypb"],
|
||||||
|
cmd = "cp $< $@",
|
||||||
|
)
|
||||||
|
|
||||||
|
android_library(
|
||||||
|
name = "mediapipe_lib",
|
||||||
|
srcs = glob(["*.java"]),
|
||||||
|
assets = [
|
||||||
|
":binary_graph",
|
||||||
|
"//mediapipe/models:palm_detection.tflite",
|
||||||
|
"//mediapipe/models:palm_detection_labelmap.txt",
|
||||||
|
],
|
||||||
|
assets_dir = "",
|
||||||
|
manifest = "AndroidManifest.xml",
|
||||||
|
resource_files = glob(["res/**"]),
|
||||||
|
deps = [
|
||||||
|
":mediapipe_jni_lib",
|
||||||
|
"//mediapipe/java/com/google/mediapipe/components:android_camerax_helper",
|
||||||
|
"//mediapipe/java/com/google/mediapipe/components:android_components",
|
||||||
|
"//mediapipe/java/com/google/mediapipe/framework:android_framework",
|
||||||
|
"//mediapipe/java/com/google/mediapipe/glutil",
|
||||||
|
"//third_party:androidx_appcompat",
|
||||||
|
"//third_party:androidx_constraint_layout",
|
||||||
|
"//third_party:androidx_legacy_support_v4",
|
||||||
|
"//third_party:androidx_material",
|
||||||
|
"//third_party:androidx_recyclerview",
|
||||||
|
"//third_party:opencv",
|
||||||
|
"@androidx_concurrent_futures//jar",
|
||||||
|
"@androidx_lifecycle//jar",
|
||||||
|
"@com_google_code_findbugs//jar",
|
||||||
|
"@com_google_guava_android//jar",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
android_binary(
|
||||||
|
name = "handdetectiongpu",
|
||||||
|
manifest = "AndroidManifest.xml",
|
||||||
|
manifest_values = {"applicationId": "com.google.mediapipe.apps.handdetectiongpu"},
|
||||||
|
multidex = "native",
|
||||||
|
deps = [
|
||||||
|
":mediapipe_lib",
|
||||||
|
],
|
||||||
|
)
|
|
@ -0,0 +1,167 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
package com.google.mediapipe.apps.handdetectiongpu;
|
||||||
|
|
||||||
|
import android.graphics.SurfaceTexture;
|
||||||
|
import android.os.Bundle;
|
||||||
|
import androidx.appcompat.app.AppCompatActivity;
|
||||||
|
import android.util.Size;
|
||||||
|
import android.view.SurfaceHolder;
|
||||||
|
import android.view.SurfaceView;
|
||||||
|
import android.view.View;
|
||||||
|
import android.view.ViewGroup;
|
||||||
|
import com.google.mediapipe.components.CameraHelper;
|
||||||
|
import com.google.mediapipe.components.CameraXPreviewHelper;
|
||||||
|
import com.google.mediapipe.components.ExternalTextureConverter;
|
||||||
|
import com.google.mediapipe.components.FrameProcessor;
|
||||||
|
import com.google.mediapipe.components.PermissionHelper;
|
||||||
|
import com.google.mediapipe.framework.AndroidAssetUtil;
|
||||||
|
import com.google.mediapipe.glutil.EglManager;
|
||||||
|
|
||||||
|
/** Main activity of MediaPipe example apps. */
|
||||||
|
public class MainActivity extends AppCompatActivity {
|
||||||
|
private static final String TAG = "MainActivity";
|
||||||
|
|
||||||
|
private static final String BINARY_GRAPH_NAME = "handdetectiongpu.binarypb";
|
||||||
|
private static final String INPUT_VIDEO_STREAM_NAME = "input_video";
|
||||||
|
private static final String OUTPUT_VIDEO_STREAM_NAME = "output_video";
|
||||||
|
private static final CameraHelper.CameraFacing CAMERA_FACING = CameraHelper.CameraFacing.FRONT;
|
||||||
|
|
||||||
|
// Flips the camera-preview frames vertically before sending them into FrameProcessor to be
|
||||||
|
// processed in a MediaPipe graph, and flips the processed frames back when they are displayed.
|
||||||
|
// This is needed because OpenGL represents images assuming the image origin is at the bottom-left
|
||||||
|
// corner, whereas MediaPipe in general assumes the image origin is at top-left.
|
||||||
|
private static final boolean FLIP_FRAMES_VERTICALLY = true;
|
||||||
|
|
||||||
|
static {
|
||||||
|
// Load all native libraries needed by the app.
|
||||||
|
System.loadLibrary("mediapipe_jni");
|
||||||
|
System.loadLibrary("opencv_java4");
|
||||||
|
}
|
||||||
|
|
||||||
|
// {@link SurfaceTexture} where the camera-preview frames can be accessed.
|
||||||
|
private SurfaceTexture previewFrameTexture;
|
||||||
|
// {@link SurfaceView} that displays the camera-preview frames processed by a MediaPipe graph.
|
||||||
|
private SurfaceView previewDisplayView;
|
||||||
|
|
||||||
|
// Creates and manages an {@link EGLContext}.
|
||||||
|
private EglManager eglManager;
|
||||||
|
// Sends camera-preview frames into a MediaPipe graph for processing, and displays the processed
|
||||||
|
// frames onto a {@link Surface}.
|
||||||
|
private FrameProcessor processor;
|
||||||
|
// Converts the GL_TEXTURE_EXTERNAL_OES texture from Android camera into a regular texture to be
|
||||||
|
// consumed by {@link FrameProcessor} and the underlying MediaPipe graph.
|
||||||
|
private ExternalTextureConverter converter;
|
||||||
|
|
||||||
|
// Handles camera access via the {@link CameraX} Jetpack support library.
|
||||||
|
private CameraXPreviewHelper cameraHelper;
|
||||||
|
|
||||||
|
@Override
|
||||||
|
protected void onCreate(Bundle savedInstanceState) {
|
||||||
|
super.onCreate(savedInstanceState);
|
||||||
|
setContentView(R.layout.activity_main);
|
||||||
|
|
||||||
|
previewDisplayView = new SurfaceView(this);
|
||||||
|
setupPreviewDisplayView();
|
||||||
|
|
||||||
|
// Initilize asset manager so that MediaPipe native libraries can access the app assets, e.g.,
|
||||||
|
// binary graphs.
|
||||||
|
AndroidAssetUtil.initializeNativeAssetManager(this);
|
||||||
|
|
||||||
|
eglManager = new EglManager(null);
|
||||||
|
processor =
|
||||||
|
new FrameProcessor(
|
||||||
|
this,
|
||||||
|
eglManager.getNativeContext(),
|
||||||
|
BINARY_GRAPH_NAME,
|
||||||
|
INPUT_VIDEO_STREAM_NAME,
|
||||||
|
OUTPUT_VIDEO_STREAM_NAME);
|
||||||
|
processor.getVideoSurfaceOutput().setFlipY(FLIP_FRAMES_VERTICALLY);
|
||||||
|
|
||||||
|
PermissionHelper.checkAndRequestCameraPermissions(this);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
protected void onResume() {
|
||||||
|
super.onResume();
|
||||||
|
converter = new ExternalTextureConverter(eglManager.getContext());
|
||||||
|
converter.setFlipY(FLIP_FRAMES_VERTICALLY);
|
||||||
|
converter.setConsumer(processor);
|
||||||
|
if (PermissionHelper.cameraPermissionsGranted(this)) {
|
||||||
|
startCamera();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
protected void onPause() {
|
||||||
|
super.onPause();
|
||||||
|
converter.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void onRequestPermissionsResult(
|
||||||
|
int requestCode, String[] permissions, int[] grantResults) {
|
||||||
|
super.onRequestPermissionsResult(requestCode, permissions, grantResults);
|
||||||
|
PermissionHelper.onRequestPermissionsResult(requestCode, permissions, grantResults);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void setupPreviewDisplayView() {
|
||||||
|
previewDisplayView.setVisibility(View.GONE);
|
||||||
|
ViewGroup viewGroup = findViewById(R.id.preview_display_layout);
|
||||||
|
viewGroup.addView(previewDisplayView);
|
||||||
|
|
||||||
|
previewDisplayView
|
||||||
|
.getHolder()
|
||||||
|
.addCallback(
|
||||||
|
new SurfaceHolder.Callback() {
|
||||||
|
@Override
|
||||||
|
public void surfaceCreated(SurfaceHolder holder) {
|
||||||
|
processor.getVideoSurfaceOutput().setSurface(holder.getSurface());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
|
||||||
|
// (Re-)Compute the ideal size of the camera-preview display (the area that the
|
||||||
|
// camera-preview frames get rendered onto, potentially with scaling and rotation)
|
||||||
|
// based on the size of the SurfaceView that contains the display.
|
||||||
|
Size viewSize = new Size(width, height);
|
||||||
|
Size displaySize = cameraHelper.computeDisplaySizeFromViewSize(viewSize);
|
||||||
|
|
||||||
|
// Connect the converter to the camera-preview frames as its input (via
|
||||||
|
// previewFrameTexture), and configure the output width and height as the computed
|
||||||
|
// display size.
|
||||||
|
converter.setSurfaceTextureAndAttachToGLContext(
|
||||||
|
previewFrameTexture, displaySize.getWidth(), displaySize.getHeight());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void surfaceDestroyed(SurfaceHolder holder) {
|
||||||
|
processor.getVideoSurfaceOutput().setSurface(null);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private void startCamera() {
|
||||||
|
cameraHelper = new CameraXPreviewHelper();
|
||||||
|
cameraHelper.setOnCameraStartedListener(
|
||||||
|
surfaceTexture -> {
|
||||||
|
previewFrameTexture = surfaceTexture;
|
||||||
|
// Make the display view visible to start showing the preview. This triggers the
|
||||||
|
// SurfaceHolder.Callback added to (the holder of) previewDisplayView.
|
||||||
|
previewDisplayView.setVisibility(View.VISIBLE);
|
||||||
|
});
|
||||||
|
cameraHelper.startCamera(this, CAMERA_FACING, /*surfaceTexture=*/ null);
|
||||||
|
}
|
||||||
|
}
|
|
@ -0,0 +1,20 @@
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
|
||||||
|
xmlns:app="http://schemas.android.com/apk/res-auto"
|
||||||
|
xmlns:tools="http://schemas.android.com/tools"
|
||||||
|
android:layout_width="match_parent"
|
||||||
|
android:layout_height="match_parent">
|
||||||
|
|
||||||
|
<FrameLayout
|
||||||
|
android:id="@+id/preview_display_layout"
|
||||||
|
android:layout_width="fill_parent"
|
||||||
|
android:layout_height="fill_parent"
|
||||||
|
android:layout_weight="1">
|
||||||
|
<TextView
|
||||||
|
android:id="@+id/no_camera_access_view"
|
||||||
|
android:layout_height="fill_parent"
|
||||||
|
android:layout_width="fill_parent"
|
||||||
|
android:gravity="center"
|
||||||
|
android:text="@string/no_camera_access" />
|
||||||
|
</FrameLayout>
|
||||||
|
</androidx.constraintlayout.widget.ConstraintLayout>
|
|
@ -0,0 +1,6 @@
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<resources>
|
||||||
|
<color name="colorPrimary">#008577</color>
|
||||||
|
<color name="colorPrimaryDark">#00574B</color>
|
||||||
|
<color name="colorAccent">#D81B60</color>
|
||||||
|
</resources>
|
|
@ -0,0 +1,4 @@
|
||||||
|
<resources>
|
||||||
|
<string name="app_name" translatable="false">Hand Detection GPU</string>
|
||||||
|
<string name="no_camera_access" translatable="false">Please grant camera permissions.</string>
|
||||||
|
</resources>
|
|
@ -0,0 +1,11 @@
|
||||||
|
<resources>
|
||||||
|
|
||||||
|
<!-- Base application theme. -->
|
||||||
|
<style name="AppTheme" parent="Theme.AppCompat.Light.DarkActionBar">
|
||||||
|
<!-- Customize your theme here. -->
|
||||||
|
<item name="colorPrimary">@color/colorPrimary</item>
|
||||||
|
<item name="colorPrimaryDark">@color/colorPrimaryDark</item>
|
||||||
|
<item name="colorAccent">@color/colorAccent</item>
|
||||||
|
</style>
|
||||||
|
|
||||||
|
</resources>
|
|
@ -0,0 +1,33 @@
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
|
||||||
|
package="com.google.mediapipe.apps.handtrackinggpu">
|
||||||
|
|
||||||
|
<uses-sdk
|
||||||
|
android:minSdkVersion="21"
|
||||||
|
android:targetSdkVersion="27" />
|
||||||
|
|
||||||
|
<!-- For using the camera -->
|
||||||
|
<uses-permission android:name="android.permission.CAMERA" />
|
||||||
|
<uses-feature android:name="android.hardware.camera" />
|
||||||
|
<uses-feature android:name="android.hardware.camera.autofocus" />
|
||||||
|
<!-- For MediaPipe -->
|
||||||
|
<uses-feature android:glEsVersion="0x00020000" android:required="true" />
|
||||||
|
|
||||||
|
|
||||||
|
<application
|
||||||
|
android:allowBackup="true"
|
||||||
|
android:label="@string/app_name"
|
||||||
|
android:supportsRtl="true"
|
||||||
|
android:theme="@style/AppTheme">
|
||||||
|
<activity
|
||||||
|
android:name=".MainActivity"
|
||||||
|
android:exported="true"
|
||||||
|
android:screenOrientation="portrait">
|
||||||
|
<intent-filter>
|
||||||
|
<action android:name="android.intent.action.MAIN" />
|
||||||
|
<category android:name="android.intent.category.LAUNCHER" />
|
||||||
|
</intent-filter>
|
||||||
|
</activity>
|
||||||
|
</application>
|
||||||
|
|
||||||
|
</manifest>
|
|
@ -0,0 +1,103 @@
|
||||||
|
# Copyright 2019 The MediaPipe Authors.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
licenses(["notice"]) # Apache 2.0
|
||||||
|
|
||||||
|
package(default_visibility = ["//visibility:private"])
|
||||||
|
|
||||||
|
cc_binary(
|
||||||
|
name = "libmediapipe_jni.so",
|
||||||
|
linkshared = 1,
|
||||||
|
linkstatic = 1,
|
||||||
|
deps = [
|
||||||
|
"//mediapipe/graphs/hand_tracking:mobile_calculators",
|
||||||
|
"//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
cc_library(
|
||||||
|
name = "mediapipe_jni_lib",
|
||||||
|
srcs = [":libmediapipe_jni.so"],
|
||||||
|
alwayslink = 1,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Maps the binary graph to an alias (e.g., the app name) for convenience so that the alias can be
|
||||||
|
# easily incorporated into the app via, for example,
|
||||||
|
# MainActivity.BINARY_GRAPH_NAME = "appname.binarypb".
|
||||||
|
genrule(
|
||||||
|
name = "binary_graph",
|
||||||
|
srcs = ["//mediapipe/graphs/hand_tracking:hand_tracking_mobile_gpu_binary_graph"],
|
||||||
|
outs = ["handtrackinggpu.binarypb"],
|
||||||
|
cmd = "cp $< $@",
|
||||||
|
)
|
||||||
|
|
||||||
|
# To use the 3D model instead of the default 2D model, add "--define 3D=true" to the
|
||||||
|
# bazel build command.
|
||||||
|
config_setting(
|
||||||
|
name = "use_3d_model",
|
||||||
|
define_values = {
|
||||||
|
"3D": "true",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
genrule(
|
||||||
|
name = "model",
|
||||||
|
srcs = select({
|
||||||
|
"//conditions:default": ["//mediapipe/models:hand_landmark.tflite"],
|
||||||
|
":use_3d_model": ["//mediapipe/models:hand_landmark_3d.tflite"],
|
||||||
|
}),
|
||||||
|
outs = ["hand_landmark.tflite"],
|
||||||
|
cmd = "cp $< $@",
|
||||||
|
)
|
||||||
|
|
||||||
|
android_library(
|
||||||
|
name = "mediapipe_lib",
|
||||||
|
srcs = glob(["*.java"]),
|
||||||
|
assets = [
|
||||||
|
":binary_graph",
|
||||||
|
":model",
|
||||||
|
"//mediapipe/models:palm_detection.tflite",
|
||||||
|
"//mediapipe/models:palm_detection_labelmap.txt",
|
||||||
|
],
|
||||||
|
assets_dir = "",
|
||||||
|
manifest = "AndroidManifest.xml",
|
||||||
|
resource_files = glob(["res/**"]),
|
||||||
|
deps = [
|
||||||
|
":mediapipe_jni_lib",
|
||||||
|
"//mediapipe/java/com/google/mediapipe/components:android_camerax_helper",
|
||||||
|
"//mediapipe/java/com/google/mediapipe/components:android_components",
|
||||||
|
"//mediapipe/java/com/google/mediapipe/framework:android_framework",
|
||||||
|
"//mediapipe/java/com/google/mediapipe/glutil",
|
||||||
|
"//third_party:androidx_appcompat",
|
||||||
|
"//third_party:androidx_constraint_layout",
|
||||||
|
"//third_party:androidx_legacy_support_v4",
|
||||||
|
"//third_party:androidx_material",
|
||||||
|
"//third_party:androidx_recyclerview",
|
||||||
|
"//third_party:opencv",
|
||||||
|
"@androidx_concurrent_futures//jar",
|
||||||
|
"@androidx_lifecycle//jar",
|
||||||
|
"@com_google_code_findbugs//jar",
|
||||||
|
"@com_google_guava_android//jar",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
android_binary(
|
||||||
|
name = "handtrackinggpu",
|
||||||
|
manifest = "AndroidManifest.xml",
|
||||||
|
manifest_values = {"applicationId": "com.google.mediapipe.apps.handtrackinggpu"},
|
||||||
|
multidex = "native",
|
||||||
|
deps = [
|
||||||
|
":mediapipe_lib",
|
||||||
|
],
|
||||||
|
)
|
|
@ -0,0 +1,167 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
package com.google.mediapipe.apps.handtrackinggpu;
|
||||||
|
|
||||||
|
import android.graphics.SurfaceTexture;
|
||||||
|
import android.os.Bundle;
|
||||||
|
import androidx.appcompat.app.AppCompatActivity;
|
||||||
|
import android.util.Size;
|
||||||
|
import android.view.SurfaceHolder;
|
||||||
|
import android.view.SurfaceView;
|
||||||
|
import android.view.View;
|
||||||
|
import android.view.ViewGroup;
|
||||||
|
import com.google.mediapipe.components.CameraHelper;
|
||||||
|
import com.google.mediapipe.components.CameraXPreviewHelper;
|
||||||
|
import com.google.mediapipe.components.ExternalTextureConverter;
|
||||||
|
import com.google.mediapipe.components.FrameProcessor;
|
||||||
|
import com.google.mediapipe.components.PermissionHelper;
|
||||||
|
import com.google.mediapipe.framework.AndroidAssetUtil;
|
||||||
|
import com.google.mediapipe.glutil.EglManager;
|
||||||
|
|
||||||
|
/** Main activity of MediaPipe example apps. */
|
||||||
|
public class MainActivity extends AppCompatActivity {
|
||||||
|
private static final String TAG = "MainActivity";
|
||||||
|
|
||||||
|
private static final String BINARY_GRAPH_NAME = "handtrackinggpu.binarypb";
|
||||||
|
private static final String INPUT_VIDEO_STREAM_NAME = "input_video";
|
||||||
|
private static final String OUTPUT_VIDEO_STREAM_NAME = "output_video";
|
||||||
|
private static final CameraHelper.CameraFacing CAMERA_FACING = CameraHelper.CameraFacing.FRONT;
|
||||||
|
|
||||||
|
// Flips the camera-preview frames vertically before sending them into FrameProcessor to be
|
||||||
|
// processed in a MediaPipe graph, and flips the processed frames back when they are displayed.
|
||||||
|
// This is needed because OpenGL represents images assuming the image origin is at the bottom-left
|
||||||
|
// corner, whereas MediaPipe in general assumes the image origin is at top-left.
|
||||||
|
private static final boolean FLIP_FRAMES_VERTICALLY = true;
|
||||||
|
|
||||||
|
static {
|
||||||
|
// Load all native libraries needed by the app.
|
||||||
|
System.loadLibrary("mediapipe_jni");
|
||||||
|
System.loadLibrary("opencv_java4");
|
||||||
|
}
|
||||||
|
|
||||||
|
// {@link SurfaceTexture} where the camera-preview frames can be accessed.
|
||||||
|
private SurfaceTexture previewFrameTexture;
|
||||||
|
// {@link SurfaceView} that displays the camera-preview frames processed by a MediaPipe graph.
|
||||||
|
private SurfaceView previewDisplayView;
|
||||||
|
|
||||||
|
// Creates and manages an {@link EGLContext}.
|
||||||
|
private EglManager eglManager;
|
||||||
|
// Sends camera-preview frames into a MediaPipe graph for processing, and displays the processed
|
||||||
|
// frames onto a {@link Surface}.
|
||||||
|
private FrameProcessor processor;
|
||||||
|
// Converts the GL_TEXTURE_EXTERNAL_OES texture from Android camera into a regular texture to be
|
||||||
|
// consumed by {@link FrameProcessor} and the underlying MediaPipe graph.
|
||||||
|
private ExternalTextureConverter converter;
|
||||||
|
|
||||||
|
// Handles camera access via the {@link CameraX} Jetpack support library.
|
||||||
|
private CameraXPreviewHelper cameraHelper;
|
||||||
|
|
||||||
|
@Override
|
||||||
|
protected void onCreate(Bundle savedInstanceState) {
|
||||||
|
super.onCreate(savedInstanceState);
|
||||||
|
setContentView(R.layout.activity_main);
|
||||||
|
|
||||||
|
previewDisplayView = new SurfaceView(this);
|
||||||
|
setupPreviewDisplayView();
|
||||||
|
|
||||||
|
// Initilize asset manager so that MediaPipe native libraries can access the app assets, e.g.,
|
||||||
|
// binary graphs.
|
||||||
|
AndroidAssetUtil.initializeNativeAssetManager(this);
|
||||||
|
|
||||||
|
eglManager = new EglManager(null);
|
||||||
|
processor =
|
||||||
|
new FrameProcessor(
|
||||||
|
this,
|
||||||
|
eglManager.getNativeContext(),
|
||||||
|
BINARY_GRAPH_NAME,
|
||||||
|
INPUT_VIDEO_STREAM_NAME,
|
||||||
|
OUTPUT_VIDEO_STREAM_NAME);
|
||||||
|
processor.getVideoSurfaceOutput().setFlipY(FLIP_FRAMES_VERTICALLY);
|
||||||
|
|
||||||
|
PermissionHelper.checkAndRequestCameraPermissions(this);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
protected void onResume() {
|
||||||
|
super.onResume();
|
||||||
|
converter = new ExternalTextureConverter(eglManager.getContext());
|
||||||
|
converter.setFlipY(FLIP_FRAMES_VERTICALLY);
|
||||||
|
converter.setConsumer(processor);
|
||||||
|
if (PermissionHelper.cameraPermissionsGranted(this)) {
|
||||||
|
startCamera();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
protected void onPause() {
|
||||||
|
super.onPause();
|
||||||
|
converter.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void onRequestPermissionsResult(
|
||||||
|
int requestCode, String[] permissions, int[] grantResults) {
|
||||||
|
super.onRequestPermissionsResult(requestCode, permissions, grantResults);
|
||||||
|
PermissionHelper.onRequestPermissionsResult(requestCode, permissions, grantResults);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void setupPreviewDisplayView() {
|
||||||
|
previewDisplayView.setVisibility(View.GONE);
|
||||||
|
ViewGroup viewGroup = findViewById(R.id.preview_display_layout);
|
||||||
|
viewGroup.addView(previewDisplayView);
|
||||||
|
|
||||||
|
previewDisplayView
|
||||||
|
.getHolder()
|
||||||
|
.addCallback(
|
||||||
|
new SurfaceHolder.Callback() {
|
||||||
|
@Override
|
||||||
|
public void surfaceCreated(SurfaceHolder holder) {
|
||||||
|
processor.getVideoSurfaceOutput().setSurface(holder.getSurface());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {
|
||||||
|
// (Re-)Compute the ideal size of the camera-preview display (the area that the
|
||||||
|
// camera-preview frames get rendered onto, potentially with scaling and rotation)
|
||||||
|
// based on the size of the SurfaceView that contains the display.
|
||||||
|
Size viewSize = new Size(width, height);
|
||||||
|
Size displaySize = cameraHelper.computeDisplaySizeFromViewSize(viewSize);
|
||||||
|
|
||||||
|
// Connect the converter to the camera-preview frames as its input (via
|
||||||
|
// previewFrameTexture), and configure the output width and height as the computed
|
||||||
|
// display size.
|
||||||
|
converter.setSurfaceTextureAndAttachToGLContext(
|
||||||
|
previewFrameTexture, displaySize.getWidth(), displaySize.getHeight());
|
||||||
|
}
|
||||||
|
|
||||||
|
@Override
|
||||||
|
public void surfaceDestroyed(SurfaceHolder holder) {
|
||||||
|
processor.getVideoSurfaceOutput().setSurface(null);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private void startCamera() {
|
||||||
|
cameraHelper = new CameraXPreviewHelper();
|
||||||
|
cameraHelper.setOnCameraStartedListener(
|
||||||
|
surfaceTexture -> {
|
||||||
|
previewFrameTexture = surfaceTexture;
|
||||||
|
// Make the display view visible to start showing the preview. This triggers the
|
||||||
|
// SurfaceHolder.Callback added to (the holder of) previewDisplayView.
|
||||||
|
previewDisplayView.setVisibility(View.VISIBLE);
|
||||||
|
});
|
||||||
|
cameraHelper.startCamera(this, CAMERA_FACING, /*surfaceTexture=*/ null);
|
||||||
|
}
|
||||||
|
}
|
|
@ -0,0 +1,20 @@
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
|
||||||
|
xmlns:app="http://schemas.android.com/apk/res-auto"
|
||||||
|
xmlns:tools="http://schemas.android.com/tools"
|
||||||
|
android:layout_width="match_parent"
|
||||||
|
android:layout_height="match_parent">
|
||||||
|
|
||||||
|
<FrameLayout
|
||||||
|
android:id="@+id/preview_display_layout"
|
||||||
|
android:layout_width="fill_parent"
|
||||||
|
android:layout_height="fill_parent"
|
||||||
|
android:layout_weight="1">
|
||||||
|
<TextView
|
||||||
|
android:id="@+id/no_camera_access_view"
|
||||||
|
android:layout_height="fill_parent"
|
||||||
|
android:layout_width="fill_parent"
|
||||||
|
android:gravity="center"
|
||||||
|
android:text="@string/no_camera_access" />
|
||||||
|
</FrameLayout>
|
||||||
|
</androidx.constraintlayout.widget.ConstraintLayout>
|
|
@ -0,0 +1,6 @@
|
||||||
|
<?xml version="1.0" encoding="utf-8"?>
|
||||||
|
<resources>
|
||||||
|
<color name="colorPrimary">#008577</color>
|
||||||
|
<color name="colorPrimaryDark">#00574B</color>
|
||||||
|
<color name="colorAccent">#D81B60</color>
|
||||||
|
</resources>
|
|
@ -0,0 +1,4 @@
|
||||||
|
<resources>
|
||||||
|
<string name="app_name" translatable="false">Hand Tracking GPU</string>
|
||||||
|
<string name="no_camera_access" translatable="false">Please grant camera permissions.</string>
|
||||||
|
</resources>
|
|
@ -0,0 +1,11 @@
|
||||||
|
<resources>
|
||||||
|
|
||||||
|
<!-- Base application theme. -->
|
||||||
|
<style name="AppTheme" parent="Theme.AppCompat.Light.DarkActionBar">
|
||||||
|
<!-- Customize your theme here. -->
|
||||||
|
<item name="colorPrimary">@color/colorPrimary</item>
|
||||||
|
<item name="colorPrimaryDark">@color/colorPrimaryDark</item>
|
||||||
|
<item name="colorAccent">@color/colorAccent</item>
|
||||||
|
</style>
|
||||||
|
|
||||||
|
</resources>
|
21
mediapipe/examples/ios/handdetectiongpu/AppDelegate.h
Normal file
|
@ -0,0 +1,21 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#import <UIKit/UIKit.h>
|
||||||
|
|
||||||
|
@interface AppDelegate : UIResponder <UIApplicationDelegate>
|
||||||
|
|
||||||
|
@property(strong, nonatomic) UIWindow *window;
|
||||||
|
|
||||||
|
@end
|
59
mediapipe/examples/ios/handdetectiongpu/AppDelegate.m
Normal file
|
@ -0,0 +1,59 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#import "AppDelegate.h"
|
||||||
|
|
||||||
|
@interface AppDelegate ()
|
||||||
|
|
||||||
|
@end
|
||||||
|
|
||||||
|
@implementation AppDelegate
|
||||||
|
|
||||||
|
- (BOOL)application:(UIApplication *)application
|
||||||
|
didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
|
||||||
|
// Override point for customization after application launch.
|
||||||
|
return YES;
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)applicationWillResignActive:(UIApplication *)application {
|
||||||
|
// Sent when the application is about to move from active to inactive state. This can occur for
|
||||||
|
// certain types of temporary interruptions (such as an incoming phone call or SMS message) or
|
||||||
|
// when the user quits the application and it begins the transition to the background state. Use
|
||||||
|
// this method to pause ongoing tasks, disable timers, and invalidate graphics rendering
|
||||||
|
// callbacks. Games should use this method to pause the game.
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)applicationDidEnterBackground:(UIApplication *)application {
|
||||||
|
// Use this method to release shared resources, save user data, invalidate timers, and store
|
||||||
|
// enough application state information to restore your application to its current state in case
|
||||||
|
// it is terminated later. If your application supports background execution, this method is
|
||||||
|
// called instead of applicationWillTerminate: when the user quits.
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)applicationWillEnterForeground:(UIApplication *)application {
|
||||||
|
// Called as part of the transition from the background to the active state; here you can undo
|
||||||
|
// many of the changes made on entering the background.
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)applicationDidBecomeActive:(UIApplication *)application {
|
||||||
|
// Restart any tasks that were paused (or not yet started) while the application was inactive. If
|
||||||
|
// the application was previously in the background, optionally refresh the user interface.
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)applicationWillTerminate:(UIApplication *)application {
|
||||||
|
// Called when the application is about to terminate. Save data if appropriate. See also
|
||||||
|
// applicationDidEnterBackground:.
|
||||||
|
}
|
||||||
|
|
||||||
|
@end
|
|
@ -0,0 +1,99 @@
|
||||||
|
{
|
||||||
|
"images" : [
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "20x20",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "20x20",
|
||||||
|
"scale" : "3x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "29x29",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "29x29",
|
||||||
|
"scale" : "3x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "40x40",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "40x40",
|
||||||
|
"scale" : "3x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "60x60",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "60x60",
|
||||||
|
"scale" : "3x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "20x20",
|
||||||
|
"scale" : "1x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "20x20",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "29x29",
|
||||||
|
"scale" : "1x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "29x29",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "40x40",
|
||||||
|
"scale" : "1x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "40x40",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "76x76",
|
||||||
|
"scale" : "1x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "76x76",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "83.5x83.5",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ios-marketing",
|
||||||
|
"size" : "1024x1024",
|
||||||
|
"scale" : "1x"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"info" : {
|
||||||
|
"version" : 1,
|
||||||
|
"author" : "xcode"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
@ -0,0 +1,7 @@
|
||||||
|
{
|
||||||
|
"info" : {
|
||||||
|
"version" : 1,
|
||||||
|
"author" : "xcode"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
75
mediapipe/examples/ios/handdetectiongpu/BUILD
Normal file
|
@ -0,0 +1,75 @@
|
||||||
|
# Copyright 2019 The MediaPipe Authors.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
licenses(["notice"]) # Apache 2.0
|
||||||
|
|
||||||
|
MIN_IOS_VERSION = "10.0"
|
||||||
|
|
||||||
|
load(
|
||||||
|
"@build_bazel_rules_apple//apple:ios.bzl",
|
||||||
|
"ios_application",
|
||||||
|
)
|
||||||
|
|
||||||
|
ios_application(
|
||||||
|
name = "HandDetectionGpuApp",
|
||||||
|
bundle_id = "com.google.mediapipe.HandDetectionGpu",
|
||||||
|
families = [
|
||||||
|
"iphone",
|
||||||
|
"ipad",
|
||||||
|
],
|
||||||
|
infoplists = ["Info.plist"],
|
||||||
|
minimum_os_version = MIN_IOS_VERSION,
|
||||||
|
provisioning_profile = "//mediapipe/examples/ios:provisioning_profile",
|
||||||
|
deps = [
|
||||||
|
":HandDetectionGpuAppLibrary",
|
||||||
|
"@ios_opencv//:OpencvFramework",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
objc_library(
|
||||||
|
name = "HandDetectionGpuAppLibrary",
|
||||||
|
srcs = [
|
||||||
|
"AppDelegate.m",
|
||||||
|
"ViewController.mm",
|
||||||
|
"main.m",
|
||||||
|
],
|
||||||
|
hdrs = [
|
||||||
|
"AppDelegate.h",
|
||||||
|
"ViewController.h",
|
||||||
|
],
|
||||||
|
data = [
|
||||||
|
"Base.lproj/LaunchScreen.storyboard",
|
||||||
|
"Base.lproj/Main.storyboard",
|
||||||
|
"//mediapipe/graphs/hand_tracking:hand_detection_mobile_gpu_binary_graph",
|
||||||
|
"//mediapipe/models:palm_detection.tflite",
|
||||||
|
"//mediapipe/models:palm_detection_labelmap.txt",
|
||||||
|
],
|
||||||
|
sdk_frameworks = [
|
||||||
|
"AVFoundation",
|
||||||
|
"CoreGraphics",
|
||||||
|
"CoreMedia",
|
||||||
|
"UIKit",
|
||||||
|
],
|
||||||
|
deps = [
|
||||||
|
"//mediapipe/objc:mediapipe_framework_ios",
|
||||||
|
"//mediapipe/objc:mediapipe_input_sources_ios",
|
||||||
|
"//mediapipe/objc:mediapipe_layer_renderer",
|
||||||
|
] + select({
|
||||||
|
"//mediapipe:ios_i386": [],
|
||||||
|
"//mediapipe:ios_x86_64": [],
|
||||||
|
"//conditions:default": [
|
||||||
|
"//mediapipe/graphs/hand_tracking:detection_mobile_calculators",
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
)
|
|
@ -0,0 +1,25 @@
|
||||||
|
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||||
|
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="13122.16" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" launchScreen="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="01J-lp-oVM">
|
||||||
|
<dependencies>
|
||||||
|
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="13104.12"/>
|
||||||
|
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
|
||||||
|
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
|
||||||
|
</dependencies>
|
||||||
|
<scenes>
|
||||||
|
<!--View Controller-->
|
||||||
|
<scene sceneID="EHf-IW-A2E">
|
||||||
|
<objects>
|
||||||
|
<viewController id="01J-lp-oVM" sceneMemberID="viewController">
|
||||||
|
<view key="view" contentMode="scaleToFill" id="Ze5-6b-2t3">
|
||||||
|
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
|
||||||
|
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
|
||||||
|
<color key="backgroundColor" red="1" green="1" blue="1" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
|
||||||
|
<viewLayoutGuide key="safeArea" id="6Tk-OE-BBY"/>
|
||||||
|
</view>
|
||||||
|
</viewController>
|
||||||
|
<placeholder placeholderIdentifier="IBFirstResponder" id="iYj-Kq-Ea1" userLabel="First Responder" sceneMemberID="firstResponder"/>
|
||||||
|
</objects>
|
||||||
|
<point key="canvasLocation" x="53" y="375"/>
|
||||||
|
</scene>
|
||||||
|
</scenes>
|
||||||
|
</document>
|
|
@ -0,0 +1,41 @@
|
||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
|
||||||
|
<device id="retina4_7" orientation="portrait">
|
||||||
|
<adaptation id="fullscreen"/>
|
||||||
|
</device>
|
||||||
|
<dependencies>
|
||||||
|
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
|
||||||
|
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
|
||||||
|
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
|
||||||
|
</dependencies>
|
||||||
|
<scenes>
|
||||||
|
<!--View Controller-->
|
||||||
|
<scene sceneID="tne-QT-ifu">
|
||||||
|
<objects>
|
||||||
|
<viewController id="BYZ-38-t0r" customClass="ViewController" sceneMemberID="viewController">
|
||||||
|
<view key="view" contentMode="scaleToFill" id="8bC-Xf-vdC">
|
||||||
|
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
|
||||||
|
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
|
||||||
|
<subviews>
|
||||||
|
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
|
||||||
|
<rect key="frame" x="0.0" y="20" width="375" height="647"/>
|
||||||
|
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
|
||||||
|
<color key="backgroundColor" white="0.0" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>
|
||||||
|
<accessibility key="accessibilityConfiguration" label="PreviewDisplayView">
|
||||||
|
<bool key="isElement" value="YES"/>
|
||||||
|
</accessibility>
|
||||||
|
</view>
|
||||||
|
</subviews>
|
||||||
|
<color key="backgroundColor" red="1" green="1" blue="1" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
|
||||||
|
<viewLayoutGuide key="safeArea" id="6Tk-OE-BBY"/>
|
||||||
|
</view>
|
||||||
|
<connections>
|
||||||
|
<outlet property="_liveView" destination="EfB-xq-knP" id="wac-VF-etz"/>
|
||||||
|
</connections>
|
||||||
|
</viewController>
|
||||||
|
<placeholder placeholderIdentifier="IBFirstResponder" id="dkx-z0-nzr" sceneMemberID="firstResponder"/>
|
||||||
|
</objects>
|
||||||
|
<point key="canvasLocation" x="48.799999999999997" y="20.239880059970016"/>
|
||||||
|
</scene>
|
||||||
|
</scenes>
|
||||||
|
</document>
|
42
mediapipe/examples/ios/handdetectiongpu/Info.plist
Normal file
|
@ -0,0 +1,42 @@
|
||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||||
|
<plist version="1.0">
|
||||||
|
<dict>
|
||||||
|
<key>NSCameraUsageDescription</key>
|
||||||
|
<string>This app uses the camera to demonstrate live video processing.</string>
|
||||||
|
<key>CFBundleDevelopmentRegion</key>
|
||||||
|
<string>en</string>
|
||||||
|
<key>CFBundleExecutable</key>
|
||||||
|
<string>$(EXECUTABLE_NAME)</string>
|
||||||
|
<key>CFBundleIdentifier</key>
|
||||||
|
<string>$(PRODUCT_BUNDLE_IDENTIFIER)</string>
|
||||||
|
<key>CFBundleInfoDictionaryVersion</key>
|
||||||
|
<string>6.0</string>
|
||||||
|
<key>CFBundleName</key>
|
||||||
|
<string>$(PRODUCT_NAME)</string>
|
||||||
|
<key>CFBundlePackageType</key>
|
||||||
|
<string>APPL</string>
|
||||||
|
<key>CFBundleShortVersionString</key>
|
||||||
|
<string>1.0</string>
|
||||||
|
<key>CFBundleVersion</key>
|
||||||
|
<string>1</string>
|
||||||
|
<key>LSRequiresIPhoneOS</key>
|
||||||
|
<true/>
|
||||||
|
<key>UILaunchStoryboardName</key>
|
||||||
|
<string>LaunchScreen</string>
|
||||||
|
<key>UIMainStoryboardFile</key>
|
||||||
|
<string>Main</string>
|
||||||
|
<key>UIRequiredDeviceCapabilities</key>
|
||||||
|
<array>
|
||||||
|
<string>armv7</string>
|
||||||
|
</array>
|
||||||
|
<key>UISupportedInterfaceOrientations</key>
|
||||||
|
<array>
|
||||||
|
<string>UIInterfaceOrientationPortrait</string>
|
||||||
|
</array>
|
||||||
|
<key>UISupportedInterfaceOrientations~ipad</key>
|
||||||
|
<array>
|
||||||
|
<string>UIInterfaceOrientationPortrait</string>
|
||||||
|
</array>
|
||||||
|
</dict>
|
||||||
|
</plist>
|
19
mediapipe/examples/ios/handdetectiongpu/ViewController.h
Normal file
|
@ -0,0 +1,19 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#import <UIKit/UIKit.h>
|
||||||
|
|
||||||
|
@interface ViewController : UIViewController
|
||||||
|
|
||||||
|
@end
|
178
mediapipe/examples/ios/handdetectiongpu/ViewController.mm
Normal file
|
@ -0,0 +1,178 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#import "ViewController.h"
|
||||||
|
|
||||||
|
#import "mediapipe/objc/MPPGraph.h"
|
||||||
|
#import "mediapipe/objc/MPPCameraInputSource.h"
|
||||||
|
#import "mediapipe/objc/MPPLayerRenderer.h"
|
||||||
|
|
||||||
|
static NSString* const kGraphName = @"hand_detection_mobile_gpu";
|
||||||
|
|
||||||
|
static const char* kInputStream = "input_video";
|
||||||
|
static const char* kOutputStream = "output_video";
|
||||||
|
static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
|
||||||
|
|
||||||
|
@interface ViewController () <MPPGraphDelegate, MPPInputSourceDelegate>
|
||||||
|
|
||||||
|
// The MediaPipe graph currently in use. Initialized in viewDidLoad, started in viewWillAppear: and
|
||||||
|
// sent video frames on _videoQueue.
|
||||||
|
@property(nonatomic) MPPGraph* mediapipeGraph;
|
||||||
|
|
||||||
|
@end
|
||||||
|
|
||||||
|
@implementation ViewController {
|
||||||
|
/// Handles camera access via AVCaptureSession library.
|
||||||
|
MPPCameraInputSource* _cameraSource;
|
||||||
|
|
||||||
|
/// Inform the user when camera is unavailable.
|
||||||
|
IBOutlet UILabel* _noCameraLabel;
|
||||||
|
/// Display the camera preview frames.
|
||||||
|
IBOutlet UIView* _liveView;
|
||||||
|
/// Render frames in a layer.
|
||||||
|
MPPLayerRenderer* _renderer;
|
||||||
|
|
||||||
|
/// Process camera frames on this queue.
|
||||||
|
dispatch_queue_t _videoQueue;
|
||||||
|
}
|
||||||
|
|
||||||
|
#pragma mark - Cleanup methods
|
||||||
|
|
||||||
|
- (void)dealloc {
|
||||||
|
self.mediapipeGraph.delegate = nil;
|
||||||
|
[self.mediapipeGraph cancel];
|
||||||
|
// Ignore errors since we're cleaning up.
|
||||||
|
[self.mediapipeGraph closeAllInputStreamsWithError:nil];
|
||||||
|
[self.mediapipeGraph waitUntilDoneWithError:nil];
|
||||||
|
}
|
||||||
|
|
||||||
|
#pragma mark - MediaPipe graph methods
|
||||||
|
|
||||||
|
+ (MPPGraph*)loadGraphFromResource:(NSString*)resource {
|
||||||
|
// Load the graph config resource.
|
||||||
|
NSError* configLoadError = nil;
|
||||||
|
NSBundle* bundle = [NSBundle bundleForClass:[self class]];
|
||||||
|
if (!resource || resource.length == 0) {
|
||||||
|
return nil;
|
||||||
|
}
|
||||||
|
NSURL* graphURL = [bundle URLForResource:resource withExtension:@"binarypb"];
|
||||||
|
NSData* data = [NSData dataWithContentsOfURL:graphURL options:0 error:&configLoadError];
|
||||||
|
if (!data) {
|
||||||
|
NSLog(@"Failed to load MediaPipe graph config: %@", configLoadError);
|
||||||
|
return nil;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse the graph config resource into mediapipe::CalculatorGraphConfig proto object.
|
||||||
|
mediapipe::CalculatorGraphConfig config;
|
||||||
|
config.ParseFromArray(data.bytes, data.length);
|
||||||
|
|
||||||
|
// Create MediaPipe graph with mediapipe::CalculatorGraphConfig proto object.
|
||||||
|
MPPGraph* newGraph = [[MPPGraph alloc] initWithGraphConfig:config];
|
||||||
|
[newGraph addFrameOutputStream:kOutputStream outputPacketType:MediaPipePacketPixelBuffer];
|
||||||
|
return newGraph;
|
||||||
|
}
|
||||||
|
|
||||||
|
#pragma mark - UIViewController methods
|
||||||
|
|
||||||
|
- (void)viewDidLoad {
|
||||||
|
[super viewDidLoad];
|
||||||
|
|
||||||
|
_renderer = [[MPPLayerRenderer alloc] init];
|
||||||
|
_renderer.layer.frame = _liveView.layer.bounds;
|
||||||
|
[_liveView.layer addSublayer:_renderer.layer];
|
||||||
|
_renderer.frameScaleMode = MediaPipeFrameScaleFillAndCrop;
|
||||||
|
// When using the front camera, mirror the input for a more natural look.
|
||||||
|
_renderer.mirrored = YES;
|
||||||
|
|
||||||
|
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
|
||||||
|
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
|
||||||
|
_videoQueue = dispatch_queue_create(kVideoQueueLabel, qosAttribute);
|
||||||
|
|
||||||
|
_cameraSource = [[MPPCameraInputSource alloc] init];
|
||||||
|
[_cameraSource setDelegate:self queue:_videoQueue];
|
||||||
|
_cameraSource.sessionPreset = AVCaptureSessionPresetHigh;
|
||||||
|
_cameraSource.cameraPosition = AVCaptureDevicePositionFront;
|
||||||
|
// The frame's native format is rotated with respect to the portrait orientation.
|
||||||
|
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
|
||||||
|
|
||||||
|
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
|
||||||
|
self.mediapipeGraph.delegate = self;
|
||||||
|
// Set maxFramesInFlight to a small value to avoid memory contention for real-time processing.
|
||||||
|
self.mediapipeGraph.maxFramesInFlight = 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
// In this application, there is only one ViewController which has no navigation to other view
|
||||||
|
// controllers, and there is only one View with live display showing the result of running the
|
||||||
|
// MediaPipe graph on the live video feed. If more view controllers are needed later, the graph
|
||||||
|
// setup/teardown and camera start/stop logic should be updated appropriately in response to the
|
||||||
|
// appearance/disappearance of this ViewController, as viewWillAppear: can be invoked multiple times
|
||||||
|
// depending on the application navigation flow in that case.
|
||||||
|
- (void)viewWillAppear:(BOOL)animated {
|
||||||
|
[super viewWillAppear:animated];
|
||||||
|
|
||||||
|
[_cameraSource requestCameraAccessWithCompletionHandler:^void(BOOL granted) {
|
||||||
|
if (granted) {
|
||||||
|
[self startGraphAndCamera];
|
||||||
|
dispatch_async(dispatch_get_main_queue(), ^{
|
||||||
|
_noCameraLabel.hidden = YES;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}];
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)startGraphAndCamera {
|
||||||
|
// Start running self.mediapipeGraph.
|
||||||
|
NSError* error;
|
||||||
|
if (![self.mediapipeGraph startWithError:&error]) {
|
||||||
|
NSLog(@"Failed to start graph: %@", error);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Start fetching frames from the camera.
|
||||||
|
dispatch_async(_videoQueue, ^{
|
||||||
|
[_cameraSource start];
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
#pragma mark - MPPGraphDelegate methods
|
||||||
|
|
||||||
|
// Receives CVPixelBufferRef from the MediaPipe graph. Invoked on a MediaPipe worker thread.
|
||||||
|
- (void)mediapipeGraph:(MPPGraph*)graph
|
||||||
|
didOutputPixelBuffer:(CVPixelBufferRef)pixelBuffer
|
||||||
|
fromStream:(const std::string&)streamName {
|
||||||
|
if (streamName == kOutputStream) {
|
||||||
|
// Display the captured image on the screen.
|
||||||
|
CVPixelBufferRetain(pixelBuffer);
|
||||||
|
dispatch_async(dispatch_get_main_queue(), ^{
|
||||||
|
[_renderer renderPixelBuffer:pixelBuffer];
|
||||||
|
CVPixelBufferRelease(pixelBuffer);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#pragma mark - MPPInputSourceDelegate methods
|
||||||
|
|
||||||
|
// Must be invoked on _videoQueue.
|
||||||
|
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
|
||||||
|
timestamp:(CMTime)timestamp
|
||||||
|
fromSource:(MPPInputSource*)source {
|
||||||
|
if (source != _cameraSource) {
|
||||||
|
NSLog(@"Unknown source: %@", source);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
[self.mediapipeGraph sendPixelBuffer:imageBuffer
|
||||||
|
intoStream:kInputStream
|
||||||
|
packetType:MediaPipePacketPixelBuffer];
|
||||||
|
}
|
||||||
|
|
||||||
|
@end
|
22
mediapipe/examples/ios/handdetectiongpu/main.m
Normal file
|
@ -0,0 +1,22 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#import <UIKit/UIKit.h>
|
||||||
|
#import "AppDelegate.h"
|
||||||
|
|
||||||
|
int main(int argc, char * argv[]) {
|
||||||
|
@autoreleasepool {
|
||||||
|
return UIApplicationMain(argc, argv, nil, NSStringFromClass([AppDelegate class]));
|
||||||
|
}
|
||||||
|
}
|
21
mediapipe/examples/ios/handtrackinggpu/AppDelegate.h
Normal file
|
@ -0,0 +1,21 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#import <UIKit/UIKit.h>
|
||||||
|
|
||||||
|
@interface AppDelegate : UIResponder <UIApplicationDelegate>
|
||||||
|
|
||||||
|
@property(strong, nonatomic) UIWindow *window;
|
||||||
|
|
||||||
|
@end
|
59
mediapipe/examples/ios/handtrackinggpu/AppDelegate.m
Normal file
|
@ -0,0 +1,59 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#import "AppDelegate.h"
|
||||||
|
|
||||||
|
@interface AppDelegate ()
|
||||||
|
|
||||||
|
@end
|
||||||
|
|
||||||
|
@implementation AppDelegate
|
||||||
|
|
||||||
|
- (BOOL)application:(UIApplication *)application
|
||||||
|
didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
|
||||||
|
// Override point for customization after application launch.
|
||||||
|
return YES;
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)applicationWillResignActive:(UIApplication *)application {
|
||||||
|
// Sent when the application is about to move from active to inactive state. This can occur for
|
||||||
|
// certain types of temporary interruptions (such as an incoming phone call or SMS message) or
|
||||||
|
// when the user quits the application and it begins the transition to the background state. Use
|
||||||
|
// this method to pause ongoing tasks, disable timers, and invalidate graphics rendering
|
||||||
|
// callbacks. Games should use this method to pause the game.
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)applicationDidEnterBackground:(UIApplication *)application {
|
||||||
|
// Use this method to release shared resources, save user data, invalidate timers, and store
|
||||||
|
// enough application state information to restore your application to its current state in case
|
||||||
|
// it is terminated later. If your application supports background execution, this method is
|
||||||
|
// called instead of applicationWillTerminate: when the user quits.
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)applicationWillEnterForeground:(UIApplication *)application {
|
||||||
|
// Called as part of the transition from the background to the active state; here you can undo
|
||||||
|
// many of the changes made on entering the background.
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)applicationDidBecomeActive:(UIApplication *)application {
|
||||||
|
// Restart any tasks that were paused (or not yet started) while the application was inactive. If
|
||||||
|
// the application was previously in the background, optionally refresh the user interface.
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)applicationWillTerminate:(UIApplication *)application {
|
||||||
|
// Called when the application is about to terminate. Save data if appropriate. See also
|
||||||
|
// applicationDidEnterBackground:.
|
||||||
|
}
|
||||||
|
|
||||||
|
@end
|
|
@ -0,0 +1,99 @@
|
||||||
|
{
|
||||||
|
"images" : [
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "20x20",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "20x20",
|
||||||
|
"scale" : "3x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "29x29",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "29x29",
|
||||||
|
"scale" : "3x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "40x40",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "40x40",
|
||||||
|
"scale" : "3x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "60x60",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "iphone",
|
||||||
|
"size" : "60x60",
|
||||||
|
"scale" : "3x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "20x20",
|
||||||
|
"scale" : "1x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "20x20",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "29x29",
|
||||||
|
"scale" : "1x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "29x29",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "40x40",
|
||||||
|
"scale" : "1x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "40x40",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "76x76",
|
||||||
|
"scale" : "1x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "76x76",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ipad",
|
||||||
|
"size" : "83.5x83.5",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "ios-marketing",
|
||||||
|
"size" : "1024x1024",
|
||||||
|
"scale" : "1x"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"info" : {
|
||||||
|
"version" : 1,
|
||||||
|
"author" : "xcode"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
@ -0,0 +1,7 @@
|
||||||
|
{
|
||||||
|
"info" : {
|
||||||
|
"version" : 1,
|
||||||
|
"author" : "xcode"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
95
mediapipe/examples/ios/handtrackinggpu/BUILD
Normal file
|
@ -0,0 +1,95 @@
|
||||||
|
# Copyright 2019 The MediaPipe Authors.
|
||||||
|
#
|
||||||
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
# you may not use this file except in compliance with the License.
|
||||||
|
# You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing, software
|
||||||
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
# See the License for the specific language governing permissions and
|
||||||
|
# limitations under the License.
|
||||||
|
|
||||||
|
licenses(["notice"]) # Apache 2.0
|
||||||
|
|
||||||
|
MIN_IOS_VERSION = "10.0"
|
||||||
|
|
||||||
|
load(
|
||||||
|
"@build_bazel_rules_apple//apple:ios.bzl",
|
||||||
|
"ios_application",
|
||||||
|
)
|
||||||
|
|
||||||
|
# To use the 3D model instead of the default 2D model, add "--define 3D=true" to the
|
||||||
|
# bazel build command.
|
||||||
|
config_setting(
|
||||||
|
name = "use_3d_model",
|
||||||
|
define_values = {
|
||||||
|
"3D": "true",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
genrule(
|
||||||
|
name = "model",
|
||||||
|
srcs = select({
|
||||||
|
"//conditions:default": ["//mediapipe/models:hand_landmark.tflite"],
|
||||||
|
":use_3d_model": ["//mediapipe/models:hand_landmark_3d.tflite"],
|
||||||
|
}),
|
||||||
|
outs = ["hand_landmark.tflite"],
|
||||||
|
cmd = "cp $< $@",
|
||||||
|
)
|
||||||
|
|
||||||
|
ios_application(
|
||||||
|
name = "HandTrackingGpuApp",
|
||||||
|
bundle_id = "com.google.mediapipe.HandTrackingGpu",
|
||||||
|
families = [
|
||||||
|
"iphone",
|
||||||
|
"ipad",
|
||||||
|
],
|
||||||
|
infoplists = ["Info.plist"],
|
||||||
|
minimum_os_version = MIN_IOS_VERSION,
|
||||||
|
provisioning_profile = "//mediapipe/examples/ios:provisioning_profile",
|
||||||
|
deps = [
|
||||||
|
":HandTrackingGpuAppLibrary",
|
||||||
|
"@ios_opencv//:OpencvFramework",
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
objc_library(
|
||||||
|
name = "HandTrackingGpuAppLibrary",
|
||||||
|
srcs = [
|
||||||
|
"AppDelegate.m",
|
||||||
|
"ViewController.mm",
|
||||||
|
"main.m",
|
||||||
|
],
|
||||||
|
hdrs = [
|
||||||
|
"AppDelegate.h",
|
||||||
|
"ViewController.h",
|
||||||
|
],
|
||||||
|
data = [
|
||||||
|
"Base.lproj/LaunchScreen.storyboard",
|
||||||
|
"Base.lproj/Main.storyboard",
|
||||||
|
":model",
|
||||||
|
"//mediapipe/graphs/hand_tracking:hand_tracking_mobile_gpu_binary_graph",
|
||||||
|
"//mediapipe/models:palm_detection.tflite",
|
||||||
|
"//mediapipe/models:palm_detection_labelmap.txt",
|
||||||
|
],
|
||||||
|
sdk_frameworks = [
|
||||||
|
"AVFoundation",
|
||||||
|
"CoreGraphics",
|
||||||
|
"CoreMedia",
|
||||||
|
"UIKit",
|
||||||
|
],
|
||||||
|
deps = [
|
||||||
|
"//mediapipe/objc:mediapipe_framework_ios",
|
||||||
|
"//mediapipe/objc:mediapipe_input_sources_ios",
|
||||||
|
"//mediapipe/objc:mediapipe_layer_renderer",
|
||||||
|
] + select({
|
||||||
|
"//mediapipe:ios_i386": [],
|
||||||
|
"//mediapipe:ios_x86_64": [],
|
||||||
|
"//conditions:default": [
|
||||||
|
"//mediapipe/graphs/hand_tracking:mobile_calculators",
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
)
|
|
@ -0,0 +1,25 @@
|
||||||
|
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
|
||||||
|
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="13122.16" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" launchScreen="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="01J-lp-oVM">
|
||||||
|
<dependencies>
|
||||||
|
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="13104.12"/>
|
||||||
|
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
|
||||||
|
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
|
||||||
|
</dependencies>
|
||||||
|
<scenes>
|
||||||
|
<!--View Controller-->
|
||||||
|
<scene sceneID="EHf-IW-A2E">
|
||||||
|
<objects>
|
||||||
|
<viewController id="01J-lp-oVM" sceneMemberID="viewController">
|
||||||
|
<view key="view" contentMode="scaleToFill" id="Ze5-6b-2t3">
|
||||||
|
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
|
||||||
|
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
|
||||||
|
<color key="backgroundColor" red="1" green="1" blue="1" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
|
||||||
|
<viewLayoutGuide key="safeArea" id="6Tk-OE-BBY"/>
|
||||||
|
</view>
|
||||||
|
</viewController>
|
||||||
|
<placeholder placeholderIdentifier="IBFirstResponder" id="iYj-Kq-Ea1" userLabel="First Responder" sceneMemberID="firstResponder"/>
|
||||||
|
</objects>
|
||||||
|
<point key="canvasLocation" x="53" y="375"/>
|
||||||
|
</scene>
|
||||||
|
</scenes>
|
||||||
|
</document>
|
|
@ -0,0 +1,41 @@
|
||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" useSafeAreas="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
|
||||||
|
<device id="retina4_7" orientation="portrait">
|
||||||
|
<adaptation id="fullscreen"/>
|
||||||
|
</device>
|
||||||
|
<dependencies>
|
||||||
|
<plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
|
||||||
|
<capability name="Safe area layout guides" minToolsVersion="9.0"/>
|
||||||
|
<capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
|
||||||
|
</dependencies>
|
||||||
|
<scenes>
|
||||||
|
<!--View Controller-->
|
||||||
|
<scene sceneID="tne-QT-ifu">
|
||||||
|
<objects>
|
||||||
|
<viewController id="BYZ-38-t0r" customClass="ViewController" sceneMemberID="viewController">
|
||||||
|
<view key="view" contentMode="scaleToFill" id="8bC-Xf-vdC">
|
||||||
|
<rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
|
||||||
|
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
|
||||||
|
<subviews>
|
||||||
|
<view contentMode="scaleToFill" fixedFrame="YES" translatesAutoresizingMaskIntoConstraints="NO" id="EfB-xq-knP">
|
||||||
|
<rect key="frame" x="0.0" y="20" width="375" height="647"/>
|
||||||
|
<autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
|
||||||
|
<color key="backgroundColor" white="0.0" alpha="1" colorSpace="custom" customColorSpace="genericGamma22GrayColorSpace"/>
|
||||||
|
<accessibility key="accessibilityConfiguration" label="PreviewDisplayView">
|
||||||
|
<bool key="isElement" value="YES"/>
|
||||||
|
</accessibility>
|
||||||
|
</view>
|
||||||
|
</subviews>
|
||||||
|
<color key="backgroundColor" red="1" green="1" blue="1" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
|
||||||
|
<viewLayoutGuide key="safeArea" id="6Tk-OE-BBY"/>
|
||||||
|
</view>
|
||||||
|
<connections>
|
||||||
|
<outlet property="_liveView" destination="EfB-xq-knP" id="wac-VF-etz"/>
|
||||||
|
</connections>
|
||||||
|
</viewController>
|
||||||
|
<placeholder placeholderIdentifier="IBFirstResponder" id="dkx-z0-nzr" sceneMemberID="firstResponder"/>
|
||||||
|
</objects>
|
||||||
|
<point key="canvasLocation" x="48.799999999999997" y="20.239880059970016"/>
|
||||||
|
</scene>
|
||||||
|
</scenes>
|
||||||
|
</document>
|
42
mediapipe/examples/ios/handtrackinggpu/Info.plist
Normal file
|
@ -0,0 +1,42 @@
|
||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||||
|
<plist version="1.0">
|
||||||
|
<dict>
|
||||||
|
<key>NSCameraUsageDescription</key>
|
||||||
|
<string>This app uses the camera to demonstrate live video processing.</string>
|
||||||
|
<key>CFBundleDevelopmentRegion</key>
|
||||||
|
<string>en</string>
|
||||||
|
<key>CFBundleExecutable</key>
|
||||||
|
<string>$(EXECUTABLE_NAME)</string>
|
||||||
|
<key>CFBundleIdentifier</key>
|
||||||
|
<string>$(PRODUCT_BUNDLE_IDENTIFIER)</string>
|
||||||
|
<key>CFBundleInfoDictionaryVersion</key>
|
||||||
|
<string>6.0</string>
|
||||||
|
<key>CFBundleName</key>
|
||||||
|
<string>$(PRODUCT_NAME)</string>
|
||||||
|
<key>CFBundlePackageType</key>
|
||||||
|
<string>APPL</string>
|
||||||
|
<key>CFBundleShortVersionString</key>
|
||||||
|
<string>1.0</string>
|
||||||
|
<key>CFBundleVersion</key>
|
||||||
|
<string>1</string>
|
||||||
|
<key>LSRequiresIPhoneOS</key>
|
||||||
|
<true/>
|
||||||
|
<key>UILaunchStoryboardName</key>
|
||||||
|
<string>LaunchScreen</string>
|
||||||
|
<key>UIMainStoryboardFile</key>
|
||||||
|
<string>Main</string>
|
||||||
|
<key>UIRequiredDeviceCapabilities</key>
|
||||||
|
<array>
|
||||||
|
<string>armv7</string>
|
||||||
|
</array>
|
||||||
|
<key>UISupportedInterfaceOrientations</key>
|
||||||
|
<array>
|
||||||
|
<string>UIInterfaceOrientationPortrait</string>
|
||||||
|
</array>
|
||||||
|
<key>UISupportedInterfaceOrientations~ipad</key>
|
||||||
|
<array>
|
||||||
|
<string>UIInterfaceOrientationPortrait</string>
|
||||||
|
</array>
|
||||||
|
</dict>
|
||||||
|
</plist>
|
19
mediapipe/examples/ios/handtrackinggpu/ViewController.h
Normal file
|
@ -0,0 +1,19 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#import <UIKit/UIKit.h>
|
||||||
|
|
||||||
|
@interface ViewController : UIViewController
|
||||||
|
|
||||||
|
@end
|
178
mediapipe/examples/ios/handtrackinggpu/ViewController.mm
Normal file
|
@ -0,0 +1,178 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#import "ViewController.h"
|
||||||
|
|
||||||
|
#import "mediapipe/objc/MPPGraph.h"
|
||||||
|
#import "mediapipe/objc/MPPCameraInputSource.h"
|
||||||
|
#import "mediapipe/objc/MPPLayerRenderer.h"
|
||||||
|
|
||||||
|
static NSString* const kGraphName = @"hand_tracking_mobile_gpu";
|
||||||
|
|
||||||
|
static const char* kInputStream = "input_video";
|
||||||
|
static const char* kOutputStream = "output_video";
|
||||||
|
static const char* kVideoQueueLabel = "com.google.mediapipe.example.videoQueue";
|
||||||
|
|
||||||
|
@interface ViewController () <MPPGraphDelegate, MPPInputSourceDelegate>
|
||||||
|
|
||||||
|
// The MediaPipe graph currently in use. Initialized in viewDidLoad, started in viewWillAppear: and
|
||||||
|
// sent video frames on _videoQueue.
|
||||||
|
@property(nonatomic) MPPGraph* mediapipeGraph;
|
||||||
|
|
||||||
|
@end
|
||||||
|
|
||||||
|
@implementation ViewController {
|
||||||
|
/// Handles camera access via AVCaptureSession library.
|
||||||
|
MPPCameraInputSource* _cameraSource;
|
||||||
|
|
||||||
|
/// Inform the user when camera is unavailable.
|
||||||
|
IBOutlet UILabel* _noCameraLabel;
|
||||||
|
/// Display the camera preview frames.
|
||||||
|
IBOutlet UIView* _liveView;
|
||||||
|
/// Render frames in a layer.
|
||||||
|
MPPLayerRenderer* _renderer;
|
||||||
|
|
||||||
|
/// Process camera frames on this queue.
|
||||||
|
dispatch_queue_t _videoQueue;
|
||||||
|
}
|
||||||
|
|
||||||
|
#pragma mark - Cleanup methods
|
||||||
|
|
||||||
|
- (void)dealloc {
|
||||||
|
self.mediapipeGraph.delegate = nil;
|
||||||
|
[self.mediapipeGraph cancel];
|
||||||
|
// Ignore errors since we're cleaning up.
|
||||||
|
[self.mediapipeGraph closeAllInputStreamsWithError:nil];
|
||||||
|
[self.mediapipeGraph waitUntilDoneWithError:nil];
|
||||||
|
}
|
||||||
|
|
||||||
|
#pragma mark - MediaPipe graph methods
|
||||||
|
|
||||||
|
+ (MPPGraph*)loadGraphFromResource:(NSString*)resource {
|
||||||
|
// Load the graph config resource.
|
||||||
|
NSError* configLoadError = nil;
|
||||||
|
NSBundle* bundle = [NSBundle bundleForClass:[self class]];
|
||||||
|
if (!resource || resource.length == 0) {
|
||||||
|
return nil;
|
||||||
|
}
|
||||||
|
NSURL* graphURL = [bundle URLForResource:resource withExtension:@"binarypb"];
|
||||||
|
NSData* data = [NSData dataWithContentsOfURL:graphURL options:0 error:&configLoadError];
|
||||||
|
if (!data) {
|
||||||
|
NSLog(@"Failed to load MediaPipe graph config: %@", configLoadError);
|
||||||
|
return nil;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse the graph config resource into mediapipe::CalculatorGraphConfig proto object.
|
||||||
|
mediapipe::CalculatorGraphConfig config;
|
||||||
|
config.ParseFromArray(data.bytes, data.length);
|
||||||
|
|
||||||
|
// Create MediaPipe graph with mediapipe::CalculatorGraphConfig proto object.
|
||||||
|
MPPGraph* newGraph = [[MPPGraph alloc] initWithGraphConfig:config];
|
||||||
|
[newGraph addFrameOutputStream:kOutputStream outputPacketType:MediaPipePacketPixelBuffer];
|
||||||
|
return newGraph;
|
||||||
|
}
|
||||||
|
|
||||||
|
#pragma mark - UIViewController methods
|
||||||
|
|
||||||
|
- (void)viewDidLoad {
|
||||||
|
[super viewDidLoad];
|
||||||
|
|
||||||
|
_renderer = [[MPPLayerRenderer alloc] init];
|
||||||
|
_renderer.layer.frame = _liveView.layer.bounds;
|
||||||
|
[_liveView.layer addSublayer:_renderer.layer];
|
||||||
|
_renderer.frameScaleMode = MediaPipeFrameScaleFillAndCrop;
|
||||||
|
// When using the front camera, mirror the input for a more natural look.
|
||||||
|
_renderer.mirrored = YES;
|
||||||
|
|
||||||
|
dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(
|
||||||
|
DISPATCH_QUEUE_SERIAL, QOS_CLASS_USER_INTERACTIVE, /*relative_priority=*/0);
|
||||||
|
_videoQueue = dispatch_queue_create(kVideoQueueLabel, qosAttribute);
|
||||||
|
|
||||||
|
_cameraSource = [[MPPCameraInputSource alloc] init];
|
||||||
|
[_cameraSource setDelegate:self queue:_videoQueue];
|
||||||
|
_cameraSource.sessionPreset = AVCaptureSessionPresetHigh;
|
||||||
|
_cameraSource.cameraPosition = AVCaptureDevicePositionFront;
|
||||||
|
// The frame's native format is rotated with respect to the portrait orientation.
|
||||||
|
_cameraSource.orientation = AVCaptureVideoOrientationPortrait;
|
||||||
|
|
||||||
|
self.mediapipeGraph = [[self class] loadGraphFromResource:kGraphName];
|
||||||
|
self.mediapipeGraph.delegate = self;
|
||||||
|
// Set maxFramesInFlight to a small value to avoid memory contention for real-time processing.
|
||||||
|
self.mediapipeGraph.maxFramesInFlight = 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
// In this application, there is only one ViewController which has no navigation to other view
|
||||||
|
// controllers, and there is only one View with live display showing the result of running the
|
||||||
|
// MediaPipe graph on the live video feed. If more view controllers are needed later, the graph
|
||||||
|
// setup/teardown and camera start/stop logic should be updated appropriately in response to the
|
||||||
|
// appearance/disappearance of this ViewController, as viewWillAppear: can be invoked multiple times
|
||||||
|
// depending on the application navigation flow in that case.
|
||||||
|
- (void)viewWillAppear:(BOOL)animated {
|
||||||
|
[super viewWillAppear:animated];
|
||||||
|
|
||||||
|
[_cameraSource requestCameraAccessWithCompletionHandler:^void(BOOL granted) {
|
||||||
|
if (granted) {
|
||||||
|
[self startGraphAndCamera];
|
||||||
|
dispatch_async(dispatch_get_main_queue(), ^{
|
||||||
|
_noCameraLabel.hidden = YES;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}];
|
||||||
|
}
|
||||||
|
|
||||||
|
- (void)startGraphAndCamera {
|
||||||
|
// Start running self.mediapipeGraph.
|
||||||
|
NSError* error;
|
||||||
|
if (![self.mediapipeGraph startWithError:&error]) {
|
||||||
|
NSLog(@"Failed to start graph: %@", error);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Start fetching frames from the camera.
|
||||||
|
dispatch_async(_videoQueue, ^{
|
||||||
|
[_cameraSource start];
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
#pragma mark - MPPGraphDelegate methods
|
||||||
|
|
||||||
|
// Receives CVPixelBufferRef from the MediaPipe graph. Invoked on a MediaPipe worker thread.
|
||||||
|
- (void)mediapipeGraph:(MPPGraph*)graph
|
||||||
|
didOutputPixelBuffer:(CVPixelBufferRef)pixelBuffer
|
||||||
|
fromStream:(const std::string&)streamName {
|
||||||
|
if (streamName == kOutputStream) {
|
||||||
|
// Display the captured image on the screen.
|
||||||
|
CVPixelBufferRetain(pixelBuffer);
|
||||||
|
dispatch_async(dispatch_get_main_queue(), ^{
|
||||||
|
[_renderer renderPixelBuffer:pixelBuffer];
|
||||||
|
CVPixelBufferRelease(pixelBuffer);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#pragma mark - MPPInputSourceDelegate methods
|
||||||
|
|
||||||
|
// Must be invoked on _videoQueue.
|
||||||
|
- (void)processVideoFrame:(CVPixelBufferRef)imageBuffer
|
||||||
|
timestamp:(CMTime)timestamp
|
||||||
|
fromSource:(MPPInputSource*)source {
|
||||||
|
if (source != _cameraSource) {
|
||||||
|
NSLog(@"Unknown source: %@", source);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
[self.mediapipeGraph sendPixelBuffer:imageBuffer
|
||||||
|
intoStream:kInputStream
|
||||||
|
packetType:MediaPipePacketPixelBuffer];
|
||||||
|
}
|
||||||
|
|
||||||
|
@end
|
22
mediapipe/examples/ios/handtrackinggpu/main.m
Normal file
|
@ -0,0 +1,22 @@
|
||||||
|
// Copyright 2019 The MediaPipe Authors.
|
||||||
|
//
|
||||||
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
// you may not use this file except in compliance with the License.
|
||||||
|
// You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing, software
|
||||||
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
// See the License for the specific language governing permissions and
|
||||||
|
// limitations under the License.
|
||||||
|
|
||||||
|
#import <UIKit/UIKit.h>
|
||||||
|
#import "AppDelegate.h"
|
||||||
|
|
||||||
|
int main(int argc, char * argv[]) {
|
||||||
|
@autoreleasepool {
|
||||||
|
return UIApplicationMain(argc, argv, nil, NSStringFromClass([AppDelegate class]));
|
||||||
|
}
|
||||||
|
}
|
BIN
mediapipe/models/hand_landmark.tflite
Normal file
BIN
mediapipe/models/hand_landmark_3d.tflite
Normal file
BIN
mediapipe/models/palm_detection.tflite
Normal file
1
mediapipe/models/palm_detection_labelmap.txt
Normal file
|
@ -0,0 +1 @@
|
||||||
|
Palm
|