diff --git a/.gitignore b/.gitignore index 5bcaac023..aa1bde53e 100644 --- a/.gitignore +++ b/.gitignore @@ -1,8 +1,4 @@ -mediapipe/provisioning_profile.mobileprovision -bazel-bin -bazel-genfiles -bazel-mediapipe-ioss -bazel-out -bazel-testlogs +bazel-* mediapipe/MediaPipe.xcodeproj mediapipe/MediaPipe.tulsiproj/*.tulsiconf-user +mediapipe/provisioning_profile.mobileprovision diff --git a/README.md b/README.md index 2d136d932..5992bd82e 100644 --- a/README.md +++ b/README.md @@ -1,84 +1,141 @@ -![MediaPipe](mediapipe/docs/images/mediapipe_small.png?raw=true "MediaPipe logo") -======================================================================= +--- +layout: default +title: Home +nav_order: 1 +--- -[MediaPipe](http://mediapipe.dev) is the simplest way for researchers and developers to build world-class ML solutions and applications for mobile, edge, cloud and the web. +![MediaPipe](docs/images/mediapipe_small.png) -## ML Solutions in MediaPipe +-------------------------------------------------------------------------------- -* [Face Detection](mediapipe/docs/face_detection_mobile_gpu.md) [(web demo)](https://viz.mediapipe.dev/runner/demos/face_detection/face_detection.html) -* [Face Mesh](mediapipe/docs/face_mesh_mobile_gpu.md) -* [Hand Detection](mediapipe/docs/hand_detection_mobile_gpu.md) -* [Hand Tracking](mediapipe/docs/hand_tracking_mobile_gpu.md) [(web demo)](https://viz.mediapipe.dev/runner/demos/hand_tracking/hand_tracking.html) -* [Multi-hand Tracking](mediapipe/docs/multi_hand_tracking_mobile_gpu.md) -* [Hair Segmentation](mediapipe/docs/hair_segmentation_mobile_gpu.md) [(web demo)](https://viz.mediapipe.dev/runner/demos/hair_segmentation/hair_segmentation.html) -* [Object Detection](mediapipe/docs/object_detection_mobile_gpu.md) -* [Object Detection and Tracking](mediapipe/docs/object_tracking_mobile_gpu.md) -* [Objectron: 3D Object Detection and Tracking](mediapipe/docs/objectron_mobile_gpu.md) -* [AutoFlip: Intelligent Video Reframing](mediapipe/docs/autoflip.md) -* [KNIFT: Template Matching with Neural Image Features](mediapipe/docs/template_matching_mobile_cpu.md) +## Cross-platform ML solutions made simple -![face_detection](mediapipe/docs/images/mobile/face_detection_android_gpu_small.gif) -![face_mesh](mediapipe/docs/images/mobile/face_mesh_android_gpu_small.gif) -![hand_tracking](mediapipe/docs/images/mobile/hand_tracking_android_gpu_small.gif) -![multi-hand_tracking](mediapipe/docs/images/mobile/multi_hand_tracking_3d_android_gpu_small.gif) -![hair_segmentation](mediapipe/docs/images/mobile/hair_segmentation_android_gpu_small.gif) -![object_detection](mediapipe/docs/images/mobile/object_detection_android_gpu_small.gif) -![object_tracking](mediapipe/docs/images/mobile/object_tracking_android_gpu_small.gif) -![objectron_shoes](mediapipe/docs/images/mobile/objectron_shoe_android_gpu_small.gif) -![objectron_chair](mediapipe/docs/images/mobile/objectron_chair_android_gpu_small.gif) -![template_matching](mediapipe/docs/images/mobile/template_matching_android_cpu_small.gif) +[MediaPipe](https://google.github.io/mediapipe/) is the simplest way for researchers +and developers to build world-class ML solutions and applications for mobile, +desktop/cloud, web and IoT devices. -## Installation -Follow these [instructions](mediapipe/docs/install.md). +![accelerated.png](docs/images/accelerated_small.png) | ![cross_platform.png](docs/images/cross_platform_small.png) +:------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------: +***End-to-End acceleration***: *built-in fast ML inference and processing accelerated even on common hardware* | ***Build one, deploy anywhere***: *Unified solution works across Android, iOS, desktop/cloud, web and IoT* +![ready_to_use.png](docs/images/ready_to_use_small.png) | ![open_source.png](docs/images/open_source_small.png) +***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable* + +## ML solutions in MediaPipe + +Face Detection | Face Mesh | Hand | Hair Segmentation +:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :---------------: +[![face_detection](docs/images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](docs/images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![hand](docs/images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hand) | [![hair_segmentation](docs/images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation) + +Object Detection | Box Tracking | Objectron | KNIFT +:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---: +[![object_detection](docs/images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](docs/images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![objectron](docs/images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](docs/images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift) + + + + +[]() | Android | iOS | Desktop | Web | Coral +:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :-: | :---: +[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | ✅ | ✅ +[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | | +[Hand](https://google.github.io/mediapipe/solutions/hand) | ✅ | ✅ | ✅ | ✅ | +[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | ✅ | +[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | ✅ +[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | | +[Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | | +[KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | | +[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | | +[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | | +[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m) | | | ✅ | | + +## MediaPipe on the Web + +MediaPipe on the Web is an effort to run the same ML solutions built for mobile +and desktop also in web browsers. The official API is under construction, but +the core technology has been proven effective. Please see +[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html) +in Google Developers Blog for details. + +You can use the following links to load a demo in the MediaPipe Visualizer, and +over there click the "Runner" icon in the top bar like shown below. The demos +use your webcam video as input, which is processed all locally in real-time and +never leaves your device. + +![visualizer_runner](docs/images/visualizer_runner.png) + +* [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection) +* [MediaPipe Hand](https://viz.mediapipe.dev/demo/hand_tracking) +* [MediaPipe Hand (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection) +* [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation) ## Getting started -See mobile, desktop, web and Google Coral [examples](mediapipe/docs/examples.md). -## Documentation -[MediaPipe Read-the-Docs](https://mediapipe.readthedocs.io/) or [docs.mediapipe.dev](https://docs.mediapipe.dev) +Learn how to [install](https://google.github.io/mediapipe/getting_started/install) +MediaPipe and +[build example applications](https://google.github.io/mediapipe/getting_started/building_examples), +and start exploring our ready-to-use +[solutions](https://google.github.io/mediapipe/solutions/solutions) that you can +further extend and customize. -Check out the [Examples page](https://mediapipe.readthedocs.io/en/latest/examples.html) for tutorials on how to use MediaPipe. [Concepts page](https://mediapipe.readthedocs.io/en/latest/concepts.html) for basic definitions - -## Visualizing MediaPipe graphs -A web-based visualizer is hosted on [viz.mediapipe.dev](https://viz.mediapipe.dev/). Please also see instructions [here](mediapipe/docs/visualizer.md). - -## Google Open Source Code search -Search MediaPipe Github repository using [Google Open Source code search](https://t.co/LSZnbMUUnT?amp=1) - -## Videos -* [YouTube Channel](https://www.youtube.com/channel/UCObqmpuSMx-usADtL_qdMAw) +The source code is hosted in the +[MediaPipe Github repository](https://github.com/google/mediapipe), and you can +run code search using +[Google Open Source Code Search](https://cs.opensource.google/mediapipe/mediapipe). ## Publications -* [MediaPipe KNIFT: Template-based Feature Matching](https://mediapipe.page.link/knift-blog) -* [Alfred Camera: Smart camera features using MediaPipe](https://developers.googleblog.com/2020/03/alfred-camera-smart-camera-features-using-mediapipe.html) -* [MediaPipe Objectron: Real-time 3D Object Detection on Mobile Devices](https://mediapipe.page.link/objectron-aiblog) -* [AutoFlip: An Open Source Framework for Intelligent Video Reframing](https://mediapipe.page.link/autoflip) -* [Google Developer Blog: MediaPipe on the Web](https://mediapipe.page.link/webdevblog) -* [Google Developer Blog: Object Detection and Tracking using MediaPipe](https://mediapipe.page.link/objecttrackingblog) -* [On-Device, Real-Time Hand Tracking with MediaPipe](https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html) -* [MediaPipe: A Framework for Building Perception Pipelines](https://arxiv.org/abs/1906.08172) + +* [MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html) + in Google Developers Blog +* [Alfred Camera: Smart camera features using MediaPipe](https://developers.googleblog.com/2020/03/alfred-camera-smart-camera-features-using-mediapipe.html) + in Google Developers Blog +* [Real-Time 3D Object Detection on Mobile Devices with MediaPipe](https://ai.googleblog.com/2020/03/real-time-3d-object-detection-on-mobile.html) + in Google AI Blog +* [AutoFlip: An Open Source Framework for Intelligent Video Reframing](https://ai.googleblog.com/2020/02/autoflip-open-source-framework-for.html) + in Google AI Blog +* [MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html) + in Google Developers Blog +* [Object Detection and Tracking using MediaPipe](https://developers.googleblog.com/2019/12/object-detection-and-tracking-using-mediapipe.html) + in Google Developers Blog +* [On-Device, Real-Time Hand Tracking with MediaPipe](https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html) + in Google AI Blog +* [MediaPipe: A Framework for Building Perception Pipelines](https://arxiv.org/abs/1906.08172) + +## Videos + +* [YouTube Channel](https://www.youtube.com/channel/UCObqmpuSMx-usADtL_qdMAw) ## Events -* [MediaPipe Seattle Meetup, Google Building Waterside, 13 Feb 2020](https://mediapipe.page.link/seattle2020) -* [AI Nextcon 2020, 12-16 Feb 2020, Seattle](http://aisea20.xnextcon.com/) -* [MediaPipe Madrid Meetup, 16 Dec 2019](https://www.meetup.com/Madrid-AI-Developers-Group/events/266329088/) -* [MediaPipe London Meetup, Google 123 Building, 12 Dec 2019](https://www.meetup.com/London-AI-Tech-Talk/events/266329038) -* [ML Conference, Berlin, 11 Dec 2019](https://mlconference.ai/machine-learning-advanced-development/mediapipe-building-real-time-cross-platform-mobile-web-edge-desktop-video-audio-ml-pipelines/) -* [MediaPipe Berlin Meetup, Google Berlin, 11 Dec 2019](https://www.meetup.com/Berlin-AI-Tech-Talk/events/266328794/) -* [The 3rd Workshop on YouTube-8M Large Scale Video Understanding Workshop](https://research.google.com/youtube8m/workshop2019/index.html) Seoul, Korea ICCV 2019 -* [AI DevWorld 2019](https://aidevworld.com) on Oct 10 in San Jose, California -* [Google Industry Workshop at ICIP 2019](http://2019.ieeeicip.org/?action=page4&id=14#Google) [Presentation](https://docs.google.com/presentation/d/e/2PACX-1vRIBBbO_LO9v2YmvbHHEt1cwyqH6EjDxiILjuT0foXy1E7g6uyh4CesB2DkkEwlRDO9_lWfuKMZx98T/pub?start=false&loop=false&delayms=3000&slide=id.g556cc1a659_0_5) on Sept 24 in Taipei, Taiwan -* [Open sourced at CVPR 2019](https://sites.google.com/corp/view/perception-cv4arvr/mediapipe) on June 17~20 in Long Beach, CA + +* [MediaPipe Seattle Meetup, Google Building Waterside, 13 Feb 2020](https://mediapipe.page.link/seattle2020) +* [AI Nextcon 2020, 12-16 Feb 2020, Seattle](http://aisea20.xnextcon.com/) +* [MediaPipe Madrid Meetup, 16 Dec 2019](https://www.meetup.com/Madrid-AI-Developers-Group/events/266329088/) +* [MediaPipe London Meetup, Google 123 Building, 12 Dec 2019](https://www.meetup.com/London-AI-Tech-Talk/events/266329038) +* [ML Conference, Berlin, 11 Dec 2019](https://mlconference.ai/machine-learning-advanced-development/mediapipe-building-real-time-cross-platform-mobile-web-edge-desktop-video-audio-ml-pipelines/) +* [MediaPipe Berlin Meetup, Google Berlin, 11 Dec 2019](https://www.meetup.com/Berlin-AI-Tech-Talk/events/266328794/) +* [The 3rd Workshop on YouTube-8M Large Scale Video Understanding Workshop, + Seoul, Korea ICCV + 2019](https://research.google.com/youtube8m/workshop2019/index.html) +* [AI DevWorld 2019, 10 Oct 2019, San Jose, CA](https://aidevworld.com) +* [Google Industry Workshop at ICIP 2019, 24 Sept 2019, Taipei, Taiwan](http://2019.ieeeicip.org/?action=page4&id=14#Google) + ([presentation](https://docs.google.com/presentation/d/e/2PACX-1vRIBBbO_LO9v2YmvbHHEt1cwyqH6EjDxiILjuT0foXy1E7g6uyh4CesB2DkkEwlRDO9_lWfuKMZx98T/pub?start=false&loop=false&delayms=3000&slide=id.g556cc1a659_0_5)) +* [Open sourced at CVPR 2019, 17~20 June, Long Beach, CA](https://sites.google.com/corp/view/perception-cv4arvr/mediapipe) ## Community -* [Awesome MediaPipe: curation of code related to MediaPipe](https://mediapipe.org) -* [Slack community for MediaPipe users](https://mediapipe.slack.com) -* [Discuss](https://groups.google.com/forum/#!forum/mediapipe) - General community discussion around MediaPipe + +* [Awesome MediaPipe](https://mediapipe.org) - A curated list of awesome + MediaPipe related frameworks, libraries and software +* [Slack community](https://mediapipe.slack.com) for MediaPipe users +* [Discuss](https://groups.google.com/forum/#!forum/mediapipe) - General + community discussion around MediaPipe ## Alpha Disclaimer -MediaPipe is currently in alpha for v0.7. We are still making breaking API changes and expect to get to stable API by v1.0. + +MediaPipe is currently in alpha at v0.7. We may be still making breaking API +changes and expect to get to stable APIs by v1.0. ## Contributing -We welcome contributions. Please follow these [guidelines](./CONTRIBUTING.md). -We use GitHub issues for tracking requests and bugs. Please post questions to the MediaPipe Stack Overflow with a 'mediapipe' tag. +We welcome contributions. Please follow these +[guidelines](https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md). + +We use GitHub issues for tracking requests and bugs. Please post questions to +the MediaPipe Stack Overflow with a `mediapipe` tag. diff --git a/mediapipe/docs/Makefile b/docs/Makefile similarity index 100% rename from mediapipe/docs/Makefile rename to docs/Makefile diff --git a/docs/_config.yml b/docs/_config.yml new file mode 100644 index 000000000..4da202e75 --- /dev/null +++ b/docs/_config.yml @@ -0,0 +1,29 @@ +# Configuration for GitHub Pages + +remote_theme: pmarsceill/just-the-docs + +# Set a path/url to a logo that will be displayed instead of the title +logo: "images/logo_horizontal_color.png" + +# Enable or disable the site search +search_enabled: true + +# Set the search token separator for hyphenated-word search: +search_tokenizer_separator: /[\s/]+/ + +# Enable or disable heading anchors +heading_anchors: true + +# Aux links for the upper right navigation +aux_links: + "MediaPipe on GitHub": + - "//github.com/google/mediapipe" + +# Footer content appears at the bottom of every page's main content +footer_content: "© 2020 GOOGLE LLC | PRIVACY POLICY | TERMS OF SERVICE" + +# Color scheme currently only supports "dark" or nil (default) +color_scheme: nil + +# Google Analytics Tracking (optional) +ga_tracking: UA-140696581-2 diff --git a/mediapipe/docs/conf.py b/docs/conf.py similarity index 100% rename from mediapipe/docs/conf.py rename to docs/conf.py diff --git a/mediapipe/docs/data/visualizer/sample_trace.binarypb b/docs/data/visualizer/sample_trace.binarypb similarity index 100% rename from mediapipe/docs/data/visualizer/sample_trace.binarypb rename to docs/data/visualizer/sample_trace.binarypb diff --git a/docs/examples.md b/docs/examples.md new file mode 100644 index 000000000..b45a2ee39 --- /dev/null +++ b/docs/examples.md @@ -0,0 +1,96 @@ +--- +nav_exclude: true +--- + +# Examples + +Below are code samples on how to run MediaPipe on both mobile and desktop. We +currently support MediaPipe APIs on mobile for Android only but will add support +for Objective-C shortly. + +## Mobile + +### [Hello World! on Android](./getting_started/hello_world_android.md) + +This should be the first mobile Android example users go through in detail. It +teaches the following: + +* Introduction of a simple MediaPipe graph running on mobile GPUs for + [Sobel edge detection](https://en.wikipedia.org/wiki/Sobel_operator). +* Building a simple baseline Android application that displays "Hello World!". +* Adding camera preview support into the baseline application using the + Android [CameraX] API. +* Incorporating the Sobel edge detection graph to process the live camera + preview and display the processed video in real-time. + +[Sobel edge detection]:https://en.wikipedia.org/wiki/Sobel_operator +[CameraX]:https://developer.android.com/training/camerax + +### [Hello World! on iOS](./getting_started/hello_world_ios.md) + +This is the iOS version of Sobel edge detection example. + +### [Face Detection](./solutions/face_detection.md) + +### [Face Mesh](./solutions/face_mesh.md) + +### [Hand](./solutions/hand.md) + +### [Hair Segmentation](./solutions/hair_segmentation.md) + +### [Object Detection](./solutions/object_detection.md) + +### [Box Tracking](./solutions/box_tracking.md) + +### [Objectron: 3D Object Detection](./solutions/objectron.md) + +### [KNIFT: Template-based Feature Matching](./solutions/knift.md) + +## Desktop + +### [Hello World for C++](./getting_started/hello_world_desktop.md) + +This shows how to run a simple graph using the MediaPipe C++ APIs. + +### [Face Detection](./solutions/face_detection.md) + +### [Face Mesh](./solutions/face_mesh.md) + +### [Hand](./solutions/hand.md) + +### [Hair Segmentation](./solutions/hair_segmentation.md) + +### [Object Detection](./solutions/object_detection.md) + +### [Box Tracking](./solutions/box_tracking.md) + +### [AutoFlip - Semantic-aware Video Cropping](./solutions/autoflip.md) + +### [Preparing Data Sets with MediaSequence](./solutions/media_sequence.md) + +This shows how to use MediaPipe for media processing to prepare video data sets +for training a TensorFlow model. + +### [Feature Extraction and Model Inference for YouTube-8M Challenge](./solutions/youtube_8m.md) + +This shows how to use MediaPipe to prepare training data for the YouTube-8M +Challenge and do the model inference with the baseline model. + +## Google Coral (ML acceleration with Google EdgeTPU) + +### [Face Detection](./solutions/face_detection.md) + +### [Object Detection](./solutions/object_detection.md) + +## Web Browser + +See more details [here](./getting_started/web.md) and +[Google Developer blog post](https://mediapipe.page.link/webdevblog). + +### [Face Detection in Browser](https://viz.mediapipe.dev/demo/face_detection) + +### [Hand Detection in Browser](https://viz.mediapipe.dev/demo/hand_detection) + +### [Hand Tracking in Browser](https://viz.mediapipe.dev/demo/hand_tracking) + +### [Hair Segmentation in Browser](https://viz.mediapipe.dev/demo/hair_segmentation) diff --git a/docs/framework_concepts/calculators.md b/docs/framework_concepts/calculators.md new file mode 100644 index 000000000..66aefb7b1 --- /dev/null +++ b/docs/framework_concepts/calculators.md @@ -0,0 +1,412 @@ +--- +layout: default +title: Calculators +parent: Framework Concepts +nav_order: 1 +--- + +# Calculators +{: .no_toc } + +1. TOC +{:toc} +--- + +Each calculator is a node of a graph. We describe how to create a new +calculator, how to initialize a calculator, how to perform its calculations, +input and output streams, timestamps, and options. Each node in the graph is +implemented as a `Calculator`. The bulk of graph execution happens inside its +calculators. A calculator may receive zero or more input streams and/or side +packets and produces zero or more output streams and/or side packets. + +## CalculatorBase + +A calculator is created by defining a new sub-class of the +[`CalculatorBase`](https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.cc) +class, implementing a number of methods, and registering the new sub-class with +Mediapipe. At a minimum, a new calculator must implement the below four methods + +* `GetContract()` + * Calculator authors can specify the expected types of inputs and outputs + of a calculator in GetContract(). When a graph is initialized, the + framework calls a static method to verify if the packet types of the + connected inputs and outputs match the information in this + specification. +* `Open()` + * After a graph starts, the framework calls `Open()`. The input side + packets are available to the calculator at this point. `Open()` + interprets the node configuration operations (see [Graphs](graphs.md)) + and prepares the calculator's per-graph-run state. This function may + also write packets to calculator outputs. An error during `Open()` can + terminate the graph run. +* `Process()` + * For a calculator with inputs, the framework calls `Process()` repeatedly + whenever at least one input stream has a packet available. The framework + by default guarantees that all inputs have the same timestamp (see + [Synchronization](synchronization.md) for more information). Multiple + `Process()` calls can be invoked simultaneously when parallel execution + is enabled. If an error occurs during `Process()`, the framework calls + `Close()` and the graph run terminates. +* `Close()` + * After all calls to `Process()` finish or when all input streams close, + the framework calls `Close()`. This function is always called if + `Open()` was called and succeeded and even if the graph run terminated + because of an error. No inputs are available via any input streams + during `Close()`, but it still has access to input side packets and + therefore may write outputs. After `Close()` returns, the calculator + should be considered a dead node. The calculator object is destroyed as + soon as the graph finishes running. + +The following are code snippets from +[CalculatorBase.h](https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h). + +```c++ +class CalculatorBase { + public: + ... + + // The subclasses of CalculatorBase must implement GetContract. + // ... + static ::MediaPipe::Status GetContract(CalculatorContract* cc); + + // Open is called before any Process() calls, on a freshly constructed + // calculator. Subclasses may override this method to perform necessary + // setup, and possibly output Packets and/or set output streams' headers. + // ... + virtual ::MediaPipe::Status Open(CalculatorContext* cc) { + return ::MediaPipe::OkStatus(); + } + + // Processes the incoming inputs. May call the methods on cc to access + // inputs and produce outputs. + // ... + virtual ::MediaPipe::Status Process(CalculatorContext* cc) = 0; + + // Is called if Open() was called and succeeded. Is called either + // immediately after processing is complete or after a graph run has ended + // (if an error occurred in the graph). ... + virtual ::MediaPipe::Status Close(CalculatorContext* cc) { + return ::MediaPipe::OkStatus(); + } + + ... +}; +``` + +## Life of a calculator + +During initialization of a MediaPipe graph, the framework calls a +`GetContract()` static method to determine what kinds of packets are expected. + +The framework constructs and destroys the entire calculator for each graph run +(e.g. once per video or once per image). Expensive or large objects that remain +constant across graph runs should be supplied as input side packets so the +calculations are not repeated on subsequent runs. + +After initialization, for each run of the graph, the following sequence occurs: + +* `Open()` +* `Process()` (repeatedly) +* `Close()` + +The framework calls `Open()` to initialize the calculator. `Open()` should +interpret any options and set up the calculator's per-graph-run state. `Open()` +may obtain input side packets and write packets to calculator outputs. If +appropriate, it should call `SetOffset()` to reduce potential packet buffering +of input streams. + +If an error occurs during `Open()` or `Process()` (as indicated by one of them +returning a non-`Ok` status), the graph run is terminated with no further calls +to the calculator's methods, and the calculator is destroyed. + +For a calculator with inputs, the framework calls `Process()` whenever at least +one input has a packet available. The framework guarantees that inputs all have +the same timestamp, that timestamps increase with each call to `Process()` and +that all packets are delivered. As a consequence, some inputs may not have any +packets when `Process()` is called. An input whose packet is missing appears to +produce an empty packet (with no timestamp). + +The framework calls `Close()` after all calls to `Process()`. All inputs will +have been exhausted, but `Close()` has access to input side packets and may +write outputs. After Close returns, the calculator is destroyed. + +Calculators with no inputs are referred to as sources. A source calculator +continues to have `Process()` called as long as it returns an `Ok` status. A +source calculator indicates that it is exhausted by returning a stop status +(i.e. MediaPipe::tool::StatusStop). + +## Identifying inputs and outputs + +The public interface to a calculator consists of a set of input streams and +output streams. In a CalculatorGraphConfiguration, the outputs from some +calculators are connected to the inputs of other calculators using named +streams. Stream names are normally lowercase, while input and output tags are +normally UPPERCASE. In the example below, the output with tag name `VIDEO` is +connected to the input with tag name `VIDEO_IN` using the stream named +`video_stream`. + +```proto +# Graph describing calculator SomeAudioVideoCalculator +node { + calculator: "SomeAudioVideoCalculator" + input_stream: "INPUT:combined_input" + output_stream: "VIDEO:video_stream" +} +node { + calculator: "SomeVideoCalculator" + input_stream: "VIDEO_IN:video_stream" + output_stream: "VIDEO_OUT:processed_video" +} +``` + +Input and output streams can be identified by index number, by tag name, or by a +combination of tag name and index number. You can see some examples of input and +output identifiers in the example below. `SomeAudioVideoCalculator` identifies +its video output by tag and its audio outputs by the combination of tag and +index. The input with tag `VIDEO` is connected to the stream named +`video_stream`. The outputs with tag `AUDIO` and indices `0` and `1` are +connected to the streams named `audio_left` and `audio_right`. +`SomeAudioCalculator` identifies its audio inputs by index only (no tag needed). + +```proto +# Graph describing calculator SomeAudioVideoCalculator +node { + calculator: "SomeAudioVideoCalculator" + input_stream: "combined_input" + output_stream: "VIDEO:video_stream" + output_stream: "AUDIO:0:audio_left" + output_stream: "AUDIO:1:audio_right" +} + +node { + calculator: "SomeAudioCalculator" + input_stream: "audio_left" + input_stream: "audio_right" + output_stream: "audio_energy" +} +``` + +In the calculator implementation, inputs and outputs are also identified by tag +name and index number. In the function below input are output are identified: + +* By index number: The combined input stream is identified simply by index + `0`. +* By tag name: The video output stream is identified by tag name "VIDEO". +* By tag name and index number: The output audio streams are identified by the + combination of the tag name `AUDIO` and the index numbers `0` and `1`. + +```c++ +// c++ Code snippet describing the SomeAudioVideoCalculator GetContract() method +class SomeAudioVideoCalculator : public CalculatorBase { + public: + static ::mediapipe::Status GetContract(CalculatorContract* cc) { + cc->Inputs().Index(0).SetAny(); + // SetAny() is used to specify that whatever the type of the + // stream is, it's acceptable. This does not mean that any + // packet is acceptable. Packets in the stream still have a + // particular type. SetAny() has the same effect as explicitly + // setting the type to be the stream's type. + cc->Outputs().Tag("VIDEO").Set(); + cc->Outputs().Get("AUDIO", 0).Set; + cc->Outputs().Get("AUDIO", 1).Set; + return ::mediapipe::OkStatus(); + } +``` + +## Processing + +`Process()` called on a non-source node must return `::mediapipe::OkStatus()` to +indicate that all went well, or any other status code to signal an error + +If a non-source calculator returns `tool::StatusStop()`, then this signals the +graph is being cancelled early. In this case, all source calculators and graph +input streams will be closed (and remaining Packets will propagate through the +graph). + +A source node in a graph will continue to have `Process()` called on it as long +as it returns `::mediapipe::OkStatus(`). To indicate that there is no more data +to be generated return `tool::StatusStop()`. Any other status indicates an error +has occurred. + +`Close()` returns `::mediapipe::OkStatus()` to indicate success. Any other +status indicates a failure. + +Here is the basic `Process()` function. It uses the `Input()` method (which can +be used only if the calculator has a single input) to request its input data. It +then uses `std::unique_ptr` to allocate the memory needed for the output packet, +and does the calculations. When done it releases the pointer when adding it to +the output stream. + +```c++ +::util::Status MyCalculator::Process() { + const Matrix& input = Input()->Get(); + std::unique_ptr output(new Matrix(input.rows(), input.cols())); + // do your magic here.... + // output->row(n) = ... + Output()->Add(output.release(), InputTimestamp()); + return ::mediapipe::OkStatus(); +} +``` + +## Example calculator + +This section discusses the implementation of `PacketClonerCalculator`, which +does a relatively simple job, and is used in many calculator graphs. +`PacketClonerCalculator` simply produces a copy of its most recent input +packets on demand. + +`PacketClonerCalculator` is useful when the timestamps of arriving data packets +are not aligned perfectly. Suppose we have a room with a microphone, light +sensor and a video camera that is collecting sensory data. Each of the sensors +operates independently and collects data intermittently. Suppose that the output +of each sensor is: + +* microphone = loudness in decibels of sound in the room (Integer) +* light sensor = brightness of room (Integer) +* video camera = RGB image frame of room (ImageFrame) + +Our simple perception pipeline is designed to process sensory data from these 3 +sensors such that at any time when we have image frame data from the camera that +is synchronized with the last collected microphone loudness data and light +sensor brightness data. To do this with MediaPipe, our perception pipeline has 3 +input streams: + +* room_mic_signal - Each packet of data in this input stream is integer data + representing how loud audio is in a room with timestamp. +* room_lightening_sensor - Each packet of data in this input stream is integer + data representing how bright is the room illuminated with timestamp. +* room_video_tick_signal - Each packet of data in this input stream is + imageframe of video data representing video collected from camera in the + room with timestamp. + +Below is the implementation of the `PacketClonerCalculator`. You can see +the `GetContract()`, `Open()`, and `Process()` methods as well as the instance +variable `current_` which holds the most recent input packets. + +```c++ +// This takes packets from N+1 streams, A_1, A_2, ..., A_N, B. +// For every packet that appears in B, outputs the most recent packet from each +// of the A_i on a separate stream. + +#include + +#include "absl/strings/str_cat.h" +#include "mediapipe/framework/calculator_framework.h" + +namespace mediapipe { + +// For every packet received on the last stream, output the latest packet +// obtained on all other streams. Therefore, if the last stream outputs at a +// higher rate than the others, this effectively clones the packets from the +// other streams to match the last. +// +// Example config: +// node { +// calculator: "PacketClonerCalculator" +// input_stream: "first_base_signal" +// input_stream: "second_base_signal" +// input_stream: "tick_signal" +// output_stream: "cloned_first_base_signal" +// output_stream: "cloned_second_base_signal" +// } +// +class PacketClonerCalculator : public CalculatorBase { + public: + static ::mediapipe::Status GetContract(CalculatorContract* cc) { + const int tick_signal_index = cc->Inputs().NumEntries() - 1; + // cc->Inputs().NumEntries() returns the number of input streams + // for the PacketClonerCalculator + for (int i = 0; i < tick_signal_index; ++i) { + cc->Inputs().Index(i).SetAny(); + // cc->Inputs().Index(i) returns the input stream pointer by index + cc->Outputs().Index(i).SetSameAs(&cc->Inputs().Index(i)); + } + cc->Inputs().Index(tick_signal_index).SetAny(); + return ::mediapipe::OkStatus(); + } + + ::mediapipe::Status Open(CalculatorContext* cc) final { + tick_signal_index_ = cc->Inputs().NumEntries() - 1; + current_.resize(tick_signal_index_); + // Pass along the header for each stream if present. + for (int i = 0; i < tick_signal_index_; ++i) { + if (!cc->Inputs().Index(i).Header().IsEmpty()) { + cc->Outputs().Index(i).SetHeader(cc->Inputs().Index(i).Header()); + // Sets the output stream of index i header to be the same as + // the header for the input stream of index i + } + } + return ::mediapipe::OkStatus(); + } + + ::mediapipe::Status Process(CalculatorContext* cc) final { + // Store input signals. + for (int i = 0; i < tick_signal_index_; ++i) { + if (!cc->Inputs().Index(i).Value().IsEmpty()) { + current_[i] = cc->Inputs().Index(i).Value(); + } + } + + // Output if the tick signal is non-empty. + if (!cc->Inputs().Index(tick_signal_index_).Value().IsEmpty()) { + for (int i = 0; i < tick_signal_index_; ++i) { + if (!current_[i].IsEmpty()) { + cc->Outputs().Index(i).AddPacket( + current_[i].At(cc->InputTimestamp())); + // Add a packet to output stream of index i a packet from inputstream i + // with timestamp common to all present inputs + // + } else { + cc->Outputs().Index(i).SetNextTimestampBound( + cc->InputTimestamp().NextAllowedInStream()); + // if current_[i], 1 packet buffer for input stream i is empty, we will set + // next allowed timestamp for input stream i to be current timestamp + 1 + } + } + } + return ::mediapipe::OkStatus(); + } + + private: + std::vector current_; + int tick_signal_index_; +}; + +REGISTER_CALCULATOR(PacketClonerCalculator); +} // namespace mediapipe +``` + +Typically, a calculator has only a .cc file. No .h is required, because +mediapipe uses registration to make calculators known to it. After you have +defined your calculator class, register it with a macro invocation +REGISTER_CALCULATOR(calculator_class_name). + +Below is a trivial MediaPipe graph that has 3 input streams, 1 node +(PacketClonerCalculator) and 3 output streams. + +```proto +input_stream: "room_mic_signal" +input_stream: "room_lighting_sensor" +input_stream: "room_video_tick_signal" + +node { + calculator: "PacketClonerCalculator" + input_stream: "room_mic_signal" + input_stream: "room_lighting_sensor" + input_stream: "room_video_tick_signal" + output_stream: "cloned_room_mic_signal" + output_stream: "cloned_lighting_sensor" + } +``` + +The diagram below shows how the `PacketClonerCalculator` defines its output +packets based on its series of input packets. + +| ![Graph using | +: PacketClonerCalculator](../images/packet_cloner_calculator.png) : +| :--------------------------------------------------------------------------: | +| *Each time it receives a packet on its TICK input stream, the | +: PacketClonerCalculator outputs the most recent packet from each of its input : +: streams. The sequence of output packets is determined by the sequene of : +: input packets and their timestamps. The timestamps are shows along the right : +: side of the diagram.* : diff --git a/mediapipe/docs/concepts.md b/docs/framework_concepts/framework_concepts.md similarity index 62% rename from mediapipe/docs/concepts.md rename to docs/framework_concepts/framework_concepts.md index 37d988e6d..b39adf154 100644 --- a/mediapipe/docs/concepts.md +++ b/docs/framework_concepts/framework_concepts.md @@ -1,24 +1,42 @@ -# MediaPipe Concepts +--- +layout: default +title: Framework Concepts +nav_order: 5 +has_children: true +has_toc: false +--- + +# Framework Concepts +{: .no_toc } + +1. TOC +{:toc} +--- ## The basics ### Packet -The basic data flow unit. A packet consists of a numeric timestamp and a shared pointer to an **immutable** payload. The payload can be of any C++ type, and the payload's type is also referred to as the type of the packet. Packets are value classes and can be copied cheaply. Each copy shares ownership of the payload, with reference-counting semantics. Each copy has its own timestamp. [Details](packets.md). +The basic data flow unit. A packet consists of a numeric timestamp and a shared +pointer to an **immutable** payload. The payload can be of any C++ type, and the +payload's type is also referred to as the type of the packet. Packets are value +classes and can be copied cheaply. Each copy shares ownership of the payload, +with reference-counting semantics. Each copy has its own timestamp. See also +[Packet](packets.md). ### Graph MediaPipe processing takes place inside a graph, which defines packet flow paths between **nodes**. A graph can have any number of inputs and outputs, and data -flow can branch and merge. Generally data flows forward, but -[backward loops](cycles.md) are possible. +flow can branch and merge. Generally data flows forward, but backward loops are +possible. See [Graphs](graphs.md) for details. ### Nodes Nodes produce and/or consume packets, and they are where the bulk of the graph’s work takes place. They are also known as “calculators”, for historical reasons. -Each node’s interface defines a number of input and output **ports**, identified by -a tag and/or an index. +Each node’s interface defines a number of input and output **ports**, identified +by a tag and/or an index. See [Calculators](calculators.md) for details. ### Streams @@ -34,21 +52,23 @@ whereas a stream represents a flow of data that changes over time. ### Packet Ports A port has an associated type; packets transiting through the port must be of -that type. An output stream port can be connected to any number of -input stream ports of the same type; each consumer receives a separate copy of -the output packets, and has its own queue, so it can consume them at its own -pace. Similarly, a side packet output port can be connected to as many side -packet input ports as desired. +that type. An output stream port can be connected to any number of input stream +ports of the same type; each consumer receives a separate copy of the output +packets, and has its own queue, so it can consume them at its own pace. +Similarly, a side packet output port can be connected to as many side packet +input ports as desired. A port can be required, meaning that a connection must be made for the graph to be valid, or optional, meaning it may remain unconnected. -Note: even if a stream connection is required, the stream may not carry a packet for all timestamps. +Note: even if a stream connection is required, the stream may not carry a packet +for all timestamps. ## Input and output Data flow can originate from **source nodes**, which have no input streams and -produce packets spontaneously (e.g. by reading from a file); or from **graph input streams**, which let an application feed packets into a graph. +produce packets spontaneously (e.g. by reading from a file); or from **graph +input streams**, which let an application feed packets into a graph. Similarly, there are **sink nodes** that receive data and write it to various destinations (e.g. a file, a memory buffer, etc.), and an application can also @@ -78,15 +98,15 @@ processed data. ### Input policies -The default input policy is deterministic collation of packets by timestamp. A node receives -all inputs for the same timestamp at the same time, in an invocation of its -Process method; and successive input sets are received in their timestamp order. This can -require delaying the processing of some packets until a packet with the same -timestamp is received on all input streams, or until it can be guaranteed that a -packet with that timestamp will not be arriving on the streams that have not -received it. +The default input policy is deterministic collation of packets by timestamp. A +node receives all inputs for the same timestamp at the same time, in an +invocation of its Process method; and successive input sets are received in +their timestamp order. This can require delaying the processing of some packets +until a packet with the same timestamp is received on all input streams, or +until it can be guaranteed that a packet with that timestamp will not be +arriving on the streams that have not received it. Other policies are also available, implemented using a separate kind of component known as an InputStreamHandler. -See [scheduling](scheduling_sync.md) for more details. +See [Synchronization](synchronization.md) for more details. diff --git a/mediapipe/docs/gpu.md b/docs/framework_concepts/gpu.md similarity index 52% rename from mediapipe/docs/gpu.md rename to docs/framework_concepts/gpu.md index e2d2be983..06355ac44 100644 --- a/mediapipe/docs/gpu.md +++ b/docs/framework_concepts/gpu.md @@ -1,14 +1,18 @@ -## Running on GPUs +--- +layout: default +title: GPU +parent: Framework Concepts +nav_order: 5 +--- -- [Overview](#overview) -- [OpenGL ES Support](#opengl-es-support) -- [Disable OpenGL ES Support](#disable-opengl-es-support) -- [OpenGL ES Setup on Linux Desktop](#opengl-es-setup-on-linux-desktop) -- [TensorFlow CUDA Support and Setup on Linux Desktop](#tensorflow-cuda-support-and-setup-on-linux-desktop) -- [Life of a GPU Calculator](#life-of-a-gpu-calculator) -- [GpuBuffer to ImageFrame Converters](#gpubuffer-to-imageframe-converters) +# GPU +{: .no_toc } -### Overview +1. TOC +{:toc} +--- + +## Overview MediaPipe supports calculator nodes for GPU compute and rendering, and allows combining multiple GPU nodes, as well as mixing them with CPU based calculator nodes. There exist several GPU APIs on mobile platforms (eg, OpenGL ES, Metal and Vulkan). MediaPipe does not attempt to offer a single cross-API GPU abstraction. Individual nodes can be written using different APIs, allowing them to take advantage of platform specific features when needed. @@ -25,7 +29,7 @@ Below are the design principles for GPU support in MediaPipe * Because different platforms may require different techniques for best performance, the API should allow flexibility in the way things are implemented behind the scenes. * A calculator should be allowed maximum flexibility in using the GPU for all or part of its operation, combining it with the CPU if necessary. -### OpenGL ES Support +## OpenGL ES Support MediaPipe supports OpenGL ES up to version 3.2 on Android/Linux and up to ES 3.0 on iOS. In addition, MediaPipe also supports Metal on iOS. @@ -50,172 +54,7 @@ some Android devices. Therefore, our approach is to have one dedicated thread per context. Each thread issues GL commands, building up a serial command queue on its context, which is then executed by the GPU asynchronously. -### Disable OpenGL ES Support - -By default, building MediaPipe (with no special bazel flags) attempts to compile -and link against OpenGL ES (and for iOS also Metal) libraries. - -On platforms where OpenGL ES is not available (see also -[OpenGL ES Setup on Linux Desktop](#opengl-es-setup-on-linux-desktop)), you -should disable OpenGL ES support with: - -``` -$ bazel build --define MEDIAPIPE_DISABLE_GPU=1 -``` - -Note: On Android and iOS, OpenGL ES is required by MediaPipe framework and the -support should never be disabled. - -### OpenGL ES Setup on Linux Desktop - -On Linux desktop with video cards that support OpenGL ES 3.1+, MediaPipe can run -GPU compute and rendering and perform TFLite inference on GPU. - -To check if your Linux desktop GPU can run MediaPipe with OpenGL ES: - -```bash -$ sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev -$ sudo apt-get install mesa-utils -$ glxinfo | grep -i opengl -``` - -For example, it may print: - -```bash -$ glxinfo | grep -i opengl -... -OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 430.50 -OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20 -OpenGL ES profile extensions: -``` - -*Notice the ES 3.20 text above.* - -You need to see ES 3.1 or greater printed in order to perform TFLite inference -on GPU in MediaPipe. With this setup, build with: - -``` -$ bazel build --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 -``` - -If only ES 3.0 or below is supported, you can still build MediaPipe targets that -don't require TFLite inference on GPU with: - -``` -$ bazel build --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 --copt -DMEDIAPIPE_DISABLE_GL_COMPUTE -``` - -Note: MEDIAPIPE_DISABLE_GL_COMPUTE is already defined automatically on all Apple -systems (Apple doesn't support OpenGL ES 3.1+). - -### TensorFlow CUDA Support and Setup on Linux Desktop - -MediaPipe framework doesn't require CUDA for GPU compute and rendering. However, -MediaPipe can work with TensorFlow to perform GPU inference on video cards that -support CUDA. - -To enable TensorFlow GPU inference with MediaPipe, the first step is to follow -the -[TensorFlow GPU documentation](https://www.tensorflow.org/install/gpu#software_requirements) -to install the required NVIDIA software on your Linux desktop. - -After installation, update `$PATH` and `$LD_LIBRARY_PATH` and run `ldconfig` -with: - -``` -$ export PATH=/usr/local/cuda-10.1/bin${PATH:+:${PATH}} -$ export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64,/usr/local/cuda-10.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} -$ sudo ldconfig -``` - -It's recommended to verify the installation of CUPTI, CUDA, CuDNN, and NVCC: - -``` -$ ls /usr/local/cuda/extras/CUPTI -/lib64 -libcupti.so libcupti.so.10.1.208 libnvperf_host.so libnvperf_target.so -libcupti.so.10.1 libcupti_static.a libnvperf_host_static.a - -$ ls /usr/local/cuda-10.1 -LICENSE bin extras lib64 libnvvp nvml samples src tools -README doc include libnsight nsightee_plugins nvvm share targets version.txt - -$ nvcc -V -nvcc: NVIDIA (R) Cuda compiler driver -Copyright (c) 2005-2019 NVIDIA Corporation -Built on Sun_Jul_28_19:07:16_PDT_2019 -Cuda compilation tools, release 10.1, V10.1.243 - -$ ls /usr/lib/x86_64-linux-gnu/ | grep libcudnn.so -libcudnn.so -libcudnn.so.7 -libcudnn.so.7.6.4 -``` - -Setting `$TF_CUDA_PATHS` is the way to declare where the CUDA library is. Note -that the following code snippet also adds `/usr/lib/x86_64-linux-gnu` and -`/usr/include` into `$TF_CUDA_PATHS` for cudablas and libcudnn. - -``` -$ export TF_CUDA_PATHS=/usr/local/cuda-10.1,/usr/lib/x86_64-linux-gnu,/usr/include -``` - -To make MediaPipe get TensorFlow's CUDA settings, find TensorFlow's -[.bazelrc](https://github.com/tensorflow/tensorflow/blob/master/.bazelrc) and -copy the `build:using_cuda` and `build:cuda` section into MediaPipe's .bazelrc -file. For example, as of April 23, 2020, TensorFlow's CUDA setting is the -following: - -``` -# This config refers to building with CUDA available. It does not necessarily -# mean that we build CUDA op kernels. -build:using_cuda --define=using_cuda=true -build:using_cuda --action_env TF_NEED_CUDA=1 -build:using_cuda --crosstool_top=@local_config_cuda//crosstool:toolchain - -# This config refers to building CUDA op kernels with nvcc. -build:cuda --config=using_cuda -build:cuda --define=using_cuda_nvcc=true -``` - -Finally, build MediaPipe with TensorFlow GPU with two more flags `--config=cuda` -and `--spawn_strategy=local`. For example: - -``` -$ bazel build -c opt --config=cuda --spawn_strategy=local \ - --define no_aws_support=true --copt -DMESA_EGL_NO_X11_HEADERS \ - mediapipe/examples/desktop/object_detection:object_detection_tensorflow -``` - -While the binary is running, it prints out the GPU device info: - -``` -I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 -I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1544] Found device 0 with properties: pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5 coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s -I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0 -``` - -You can monitor the GPU usage to verify whether the GPU is used for model -inference. - -``` -$ nvidia-smi --query-gpu=utilization.gpu --format=csv --loop=1 - -0 % -0 % -4 % -5 % -83 % -21 % -22 % -27 % -29 % -100 % -0 % -0% -``` - -### Life of a GPU Calculator +## Life of a GPU Calculator This section presents the basic structure of the Process method of a GPU calculator derived from base class GlSimpleCalculator. The GPU calculator @@ -302,7 +141,7 @@ choices for MediaPipe GPU support: * Data that needs to be shared between all GPU-based calculators is provided as a external input that is implemented as a graph service and is managed by the `GlCalculatorHelper` class. * The combination of calculator-specific helpers and a shared graph service allows us great flexibility in managing the GPU resource: we can have a separate context per calculator, share a single context, share a lock or other synchronization primitives, etc. -- and all of this is managed by the helper and hidden from the individual calculators. -### GpuBuffer to ImageFrame Converters +## GpuBuffer to ImageFrame Converters We provide two calculators called `GpuBufferToImageFrameCalculator` and `ImageFrameToGpuBufferCalculator`. These calculators convert between `ImageFrame` and `GpuBuffer`, allowing the construction of graphs that combine GPU and CPU calculators. They are supported on both iOS and Android @@ -310,6 +149,15 @@ When possible, these calculators use platform-specific functionality to share da The below diagram shows the data flow in a mobile application that captures video from the camera, runs it through a MediaPipe graph, and renders the output on the screen in real time. The dashed line indicates which parts are inside the MediaPipe graph proper. This application runs a Canny edge-detection filter on the CPU using OpenCV, and overlays it on top of the original video using the GPU. -| ![How GPU calculators interact](images/gpu_example_graph.png) | -|:--:| -| *Video frames from the camera are fed into the graph as `GpuBuffer` packets. The input stream is accessed by two calculators in parallel. `GpuBufferToImageFrameCalculator` converts the buffer into an `ImageFrame`, which is then sent through a grayscale converter and a canny filter (both based on OpenCV and running on the CPU), whose output is then converted into a `GpuBuffer` again. A multi-input GPU calculator, GlOverlayCalculator, takes as input both the original `GpuBuffer` and the one coming out of the edge detector, and overlays them using a shader. The output is then sent back to the application using a callback calculator, and the application renders the image to the screen using OpenGL.* | +| ![How GPU calculators interact](../images/gpu_example_graph.png) | +| :--------------------------------------------------------------------------: | +| *Video frames from the camera are fed into the graph as `GpuBuffer` packets. | +: The input stream is accessed by two calculators in parallel. : +: `GpuBufferToImageFrameCalculator` converts the buffer into an `ImageFrame`, : +: which is then sent through a grayscale converter and a canny filter (both : +: based on OpenCV and running on the CPU), whose output is then converted into : +: a `GpuBuffer` again. A multi-input GPU calculator, GlOverlayCalculator, : +: takes as input both the original `GpuBuffer` and the one coming out of the : +: edge detector, and overlays them using a shader. The output is then sent : +: back to the application using a callback calculator, and the application : +: renders the image to the screen using OpenGL.* : diff --git a/docs/framework_concepts/graphs.md b/docs/framework_concepts/graphs.md new file mode 100644 index 000000000..83f95e5bb --- /dev/null +++ b/docs/framework_concepts/graphs.md @@ -0,0 +1,271 @@ +--- +layout: default +title: Graphs +parent: Framework Concepts +nav_order: 2 +--- + +# Graphs +{: .no_toc } + +1. TOC +{:toc} +--- + +## GraphConfig + +A `GraphConfig` is a specification that describes the topology and functionality +of a MediaPipe graph. In the specification, a node in the graph represents an +instance of a particular calculator. All the necessary configurations of the +node, such its type, inputs and outputs must be described in the specification. +Description of the node can also include several optional fields, such as +node-specific options, input policy and executor, discussed in +[Synchronization](synchronization.md). + +`GraphConfig` has several other fields to configure the global graph-level +settings, eg, graph executor configs, number of threads, and maximum queue size +of input streams. Several graph-level settings are useful for tuning the +performance of the graph on different platforms (eg, desktop v.s. mobile). For +instance, on mobile, attaching a heavy model-inference calculator to a separate +executor can improve the performance of a real-time application since this +enables thread locality. + +Below is a trivial `GraphConfig` example where we have series of passthrough +calculators : + +```proto +# This graph named main_pass_throughcals_nosubgraph.pbtxt contains 4 +# passthrough calculators. +input_stream: "in" +node { + calculator: "PassThroughCalculator" + input_stream: "in" + output_stream: "out1" +} +node { + calculator: "PassThroughCalculator" + input_stream: "out1" + output_stream: "out2" +} +node { + calculator: "PassThroughCalculator" + input_stream: "out2" + output_stream: "out3" +} +node { + calculator: "PassThroughCalculator" + input_stream: "out3" + output_stream: "out4" +} +``` + +## Subgraph + +To modularize a `CalculatorGraphConfig` into sub-modules and assist with re-use +of perception solutions, a MediaPipe graph can be defined as a `Subgraph`. The +public interface of a subgraph consists of a set of input and output streams +similar to a calculator's public interface. The subgraph can then be included in +an `CalculatorGraphConfig` as if it were a calculator. When a MediaPipe graph is +loaded from a `CalculatorGraphConfig`, each subgraph node is replaced by the +corresponding graph of calculators. As a result, the semantics and performance +of the subgraph is identical to the corresponding graph of calculators. + +Below is an example of how to create a subgraph named `TwoPassThroughSubgraph`. + +1. Defining the subgraph. + + ```proto + # This subgraph is defined in two_pass_through_subgraph.pbtxt + # and is registered as "TwoPassThroughSubgraph" + + type: "TwoPassThroughSubgraph" + input_stream: "out1" + output_stream: "out3" + + node { + calculator: "PassThroughculator" + input_stream: "out1" + output_stream: "out2" + } + node { + calculator: "PassThroughculator" + input_stream: "out2" + output_stream: "out3" + } + ``` + + The public interface to the subgraph consists of: + + * Graph input streams + * Graph output streams + * Graph input side packets + * Graph output side packets + +2. Register the subgraph using BUILD rule `mediapipe_simple_subgraph`. The + parameter `register_as` defines the component name for the new subgraph. + + ```proto + # Small section of BUILD file for registering the "TwoPassThroughSubgraph" + # subgraph for use by main graph main_pass_throughcals.pbtxt + + mediapipe_simple_subgraph( + name = "twopassthrough_subgraph", + graph = "twopassthrough_subgraph.pbtxt", + register_as = "TwoPassThroughSubgraph", + deps = [ + "//mediapipe/calculators/core:pass_through_calculator", + "//mediapipe/framework:calculator_graph", + ], + ) + ``` + +3. Use the subgraph in the main graph. + + ```proto + # This main graph is defined in main_pass_throughcals.pbtxt + # using subgraph called "TwoPassThroughSubgraph" + + input_stream: "in" + node { + calculator: "PassThroughCalculator" + input_stream: "in" + output_stream: "out1" + } + node { + calculator: "TwoPassThroughSubgraph" + input_stream: "out1" + output_stream: "out3" + } + node { + calculator: "PassThroughCalculator" + input_stream: "out3" + output_stream: "out4" + } + ``` + +## Cycles + + + +By default, MediaPipe requires calculator graphs to be acyclic and treats cycles +in a graph as errors. If a graph is intended to have cycles, the cycles need to +be annotated in the graph config. This page describes how to do that. + +NOTE: The current approach is experimental and subject to change. We welcome +your feedback. + +Please use the `CalculatorGraphTest.Cycle` unit test in +`mediapipe/framework/calculator_graph_test.cc` as sample code. Shown +below is the cyclic graph in the test. The `sum` output of the adder is the sum +of the integers generated by the integer source calculator. + +![a cyclic graph that adds a stream of integers](../images/cyclic_integer_sum_graph.svg "A cyclic graph") + +This simple graph illustrates all the issues in supporting cyclic graphs. + +### Back Edge Annotation + +We require that an edge in each cycle be annotated as a back edge. This allows +MediaPipe’s topological sort to work, after removing all the back edges. + +There are usually multiple ways to select the back edges. Which edges are marked +as back edges affects which nodes are considered as upstream and which nodes are +considered as downstream, which in turn affects the priorities MediaPipe assigns +to the nodes. + +For example, the `CalculatorGraphTest.Cycle` test marks the `old_sum` edge as a +back edge, so the Delay node is considered as a downstream node of the adder +node and is given a higher priority. Alternatively, we could mark the `sum` +input to the delay node as the back edge, in which case the delay node would be +considered as an upstream node of the adder node and is given a lower priority. + +### Initial Packet + +For the adder calculator to be runnable when the first integer from the integer +source arrives, we need an initial packet, with value 0 and with the same +timestamp, on the `old_sum` input stream to the adder. This initial packet +should be output by the delay calculator in the `Open()` method. + +### Delay in a Loop + +Each loop should incur a delay to align the previous `sum` output with the next +integer input. This is also done by the delay node. So the delay node needs to +know the following about the timestamps of the integer source calculator: + +* The timestamp of the first output. + +* The timestamp delta between successive outputs. + +We plan to add an alternative scheduling policy that only cares about packet +ordering and ignores packet timestamps, which will eliminate this inconvenience. + +### Early Termination of a Calculator When One Input Stream is Done + +By default, MediaPipe calls the `Close()` method of a non-source calculator when +all of its input streams are done. In the example graph, we want to stop the +adder node as soon as the integer source is done. This is accomplished by +configuring the adder node with an alternative input stream handler, +`EarlyCloseInputStreamHandler`. + +### Relevant Source Code + +#### Delay Calculator + +Note the code in `Open()` that outputs the initial packet and the code in +`Process()` that adds a (unit) delay to input packets. As noted above, this +delay node assumes that its output stream is used alongside an input stream with +packet timestamps 0, 1, 2, 3, ... + +```c++ +class UnitDelayCalculator : public Calculator { + public: +  static ::util::Status FillExpectations( +      const CalculatorOptions& extendable_options, PacketTypeSet* inputs, +      PacketTypeSet* outputs, PacketTypeSet* input_side_packets) { +    inputs->Index(0)->Set("An integer."); +    outputs->Index(0)->Set("The input delayed by one time unit."); +    return ::mediapipe::OkStatus(); +  } + +  ::util::Status Open() final { +    Output()->Add(new int(0), Timestamp(0)); +    return ::mediapipe::OkStatus(); +  } + +  ::util::Status Process() final { +    const Packet& packet = Input()->Value(); +    Output()->AddPacket(packet.At(packet.Timestamp().NextAllowedInStream())); +    return ::mediapipe::OkStatus(); +  } +}; +``` + +#### Graph Config + +Note the `back_edge` annotation and the alternative `input_stream_handler`. + +```proto +node { +  calculator: 'GlobalCountSourceCalculator' +  input_side_packet: 'global_counter' +  output_stream: 'integers' +} +node { +  calculator: 'IntAdderCalculator' +  input_stream: 'integers' +  input_stream: 'old_sum' +  input_stream_info: { +    tag_index: ':1' # 'old_sum' +    back_edge: true +  } +  output_stream: 'sum' +  input_stream_handler { +    input_stream_handler: 'EarlyCloseInputStreamHandler' +  } +} +node { +  calculator: 'UnitDelayCalculator' +  input_stream: 'sum' +  output_stream: 'old_sum' +} +``` diff --git a/mediapipe/docs/packets.md b/docs/framework_concepts/packets.md similarity index 81% rename from mediapipe/docs/packets.md rename to docs/framework_concepts/packets.md index 2e52ae956..bb0b61d6a 100644 --- a/mediapipe/docs/packets.md +++ b/docs/framework_concepts/packets.md @@ -1,10 +1,21 @@ -### Packets +--- +layout: default +title: Packets +parent: Framework Concepts +nav_order: 3 +--- -- [Creating a packet](#creating-a-packet) +# Packets +{: .no_toc } + +1. TOC +{:toc} +--- Each calculator is a node of of a graph. We describe how to create a new calculator, how to initialize a calculator, how to perform its calculations, input and output streams, timestamps, and options -#### Creating a packet +## Creating a packet + Packets are generally created with `MediaPipe::Adopt()` (from packet.h). ```c++ diff --git a/mediapipe/docs/scheduling_sync.md b/docs/framework_concepts/synchronization.md similarity index 98% rename from mediapipe/docs/scheduling_sync.md rename to docs/framework_concepts/synchronization.md index 7f77a989c..5482aeb76 100644 --- a/mediapipe/docs/scheduling_sync.md +++ b/docs/framework_concepts/synchronization.md @@ -1,4 +1,16 @@ -# Framework Architecture +--- +layout: default +title: Synchronization +parent: Framework Concepts +nav_order: 4 +--- + +# Synchronization +{: .no_toc } + +1. TOC +{:toc} +--- ## Scheduling mechanics diff --git a/mediapipe/docs/android_archive_library.md b/docs/getting_started/android_archive_library.md similarity index 88% rename from mediapipe/docs/android_archive_library.md rename to docs/getting_started/android_archive_library.md index f0fef4113..0b2c6181b 100644 --- a/mediapipe/docs/android_archive_library.md +++ b/docs/getting_started/android_archive_library.md @@ -1,4 +1,16 @@ -## MediaPipe Android Archive Library +--- +layout: default +title: MediaPipe Android Archive +parent: Getting Started +nav_order: 7 +--- + +# MediaPipe Android Archive +{: .no_toc } + +1. TOC +{:toc} +--- ***Experimental Only*** @@ -9,15 +21,15 @@ target to generate a custom AAR file for their own projects. This is necessary in order to include specific resources such as MediaPipe calculators needed for each project. -### Steps to build a MediaPipe AAR +## Steps to build a MediaPipe AAR 1. Create a mediapipe_aar() target. In the MediaPipe directory, create a new mediapipe_aar() target in a BUILD file. You need to figure out what calculators are used in the graph and provide the calculator dependencies to the mediapipe_aar(). For example, to - build an AAR for [face detection gpu](./face_detection_mobile_gpu.md), you - can put the following code into + build an AAR for [MediaPipe Face Detection](../solutions/face_detection.md), + you can put the following code into mediapipe/examples/android/src/java/com/google/mediapipe/apps/aar_example/BUILD. ``` @@ -54,7 +66,7 @@ each project. /absolute/path/to/your/preferred/location ``` -### Steps to use a MediaPipe AAR in Android Studio with Gradle +## Steps to use a MediaPipe AAR in Android Studio with Gradle 1. Start Android Studio and go to your project. @@ -65,7 +77,7 @@ each project. /path/to/your/app/libs/ ``` - ![Screenshot](images/mobile/aar_location.png) + ![Screenshot](../images/mobile/aar_location.png) 3. Make app/src/main/assets and copy assets (graph, model, and etc) into app/src/main/assets. @@ -85,7 +97,7 @@ each project. cp mediapipe/models/face_detection_front_labelmap.txt /path/to/your/app/src/main/assets/ ``` - ![Screenshot](images/mobile/assets_location.png) + ![Screenshot](../images/mobile/assets_location.png) 4. Make app/src/main/jniLibs and copy OpenCV JNI libraries into app/src/main/jniLibs. @@ -100,7 +112,7 @@ each project. cp -R ~/Downloads/OpenCV-android-sdk/sdk/native/libs/arm* /path/to/your/app/src/main/jniLibs/ ``` - ![Screenshot](images/mobile/android_studio_opencv_location.png) + ![Screenshot](../images/mobile/android_studio_opencv_location.png) 5. Modify app/build.gradle to add MediaPipe dependencies and MediaPipe AAR. @@ -127,6 +139,8 @@ each project. ``` 6. Follow our Android app examples to use MediaPipe in Android Studio for your - use case. If you are looking for an example, a face detection - example can be found - [here](https://github.com/jiuqiant/mediapipe_face_detection_aar_example) and a multi-hand tracking example can be found [here](https://github.com/jiuqiant/mediapipe_multi_hands_tracking_aar_example). + use case. If you are looking for an example, a face detection example can be + found + [here](https://github.com/jiuqiant/mediapipe_face_detection_aar_example) and + a multi-hand tracking example can be found + [here](https://github.com/jiuqiant/mediapipe_multi_hands_tracking_aar_example). diff --git a/mediapipe/docs/building_examples.md b/docs/getting_started/building_examples.md similarity index 85% rename from mediapipe/docs/building_examples.md rename to docs/getting_started/building_examples.md index 73c139fea..c69355456 100644 --- a/mediapipe/docs/building_examples.md +++ b/docs/getting_started/building_examples.md @@ -1,8 +1,16 @@ -# Building MediaPipe Examples +--- +layout: default +title: Building MediaPipe Examples +parent: Getting Started +nav_order: 2 +--- -* [Android](#android) -* [iOS](#ios) -* [Desktop](#desktop) +# Building MediaPipe Examples +{: .no_toc } + +1. TOC +{:toc} +--- ## Android @@ -29,8 +37,9 @@ export ANDROID_NDK_HOME= In order to use MediaPipe on earlier Android versions, MediaPipe needs to switch to a lower Android API level. You can achieve this by specifying `api_level = -` in android_ndk_repository() and/or android_sdk_repository() -in the [`WORKSPACE`](https://github.com/google/mediapipe/tree/master/WORKSPACE) file. +$YOUR_INTENDED_API_LEVEL` in android_ndk_repository() and/or +android_sdk_repository() in the +[`WORKSPACE`](https://github.com/google/mediapipe/tree/master/WORKSPACE) file. Please verify all the necessary packages are installed. @@ -42,16 +51,19 @@ Please verify all the necessary packages are installed. ### Option 1: Build with Bazel in Command Line -1. To build an Android example app, for instance, for MediaPipe Hand, run: +1. To build an Android example app, build against the corresponding + `android_binary` build target. For instance, for + [MediaPipe Hand](../solutions/hand.md) the target is `handtrackinggpu` in + the + [BUILD](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/BUILD) + file: Note: To reduce the binary size, consider appending `--linkopt="-s"` to the command below to strip symbols. - ~~~ - ```bash - bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu + ```bash + bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu:handtrackinggpu ``` - ~~~ 1. Install it on a device with: @@ -74,13 +86,13 @@ app: 1. Install and launch Android Studio 3.5. -2. Select `Configure` | `SDK Manager` | `SDK Platforms`. +2. Select `Configure` -> `SDK Manager` -> `SDK Platforms`. * Verify that Android SDK Platform API Level 28 or 29 is installed. * Take note of the Android SDK Location, e.g., `/usr/local/home/Android/Sdk`. -3. Select `Configure` | `SDK Manager` | `SDK Tools`. +3. Select `Configure` -> `SDK Manager` -> `SDK Tools`. * Verify that Android SDK Build-Tools 28 or 29 is installed. * Verify that Android SDK Platform-Tools 28 or 29 is installed. @@ -102,10 +114,10 @@ app: export ANDROID_NDK_HOME=/usr/local/home/Android/Sdk/ndk/ ``` -5. Select `Configure` | `Plugins` install `Bazel`. +5. Select `Configure` -> `Plugins` to install `Bazel`. -6. On Linux, select `File` | `Settings`| `Bazel settings`. On macos, select - `Android Studio` | `Preferences` | `Bazel settings`. Then, modify `Bazel +6. On Linux, select `File` -> `Settings` -> `Bazel settings`. On macos, select + `Android Studio` -> `Preferences` -> `Bazel settings`. Then, modify `Bazel binary location` to be the same as the output of `$ which bazel`. 7. Select `Import Bazel Project`. @@ -132,7 +144,7 @@ app: --host_crosstool_top=@bazel_tools//tools/cpp:toolchain ``` -8. Select `Bazel` | `Sync` | `Sync project with Build files`. +8. Select `Bazel` -> `Sync` -> `Sync project with Build files`. Note: Even after doing step 4, if you still see the error: `"no such package '@androidsdk//': Either the path attribute of android_sdk_repository or the @@ -154,11 +166,11 @@ app: 9. Connect an Android device to the workstation. -10. Select `Run...` | `Edit Configurations...`. +10. Select `Run...` -> `Edit Configurations...`. - * Select `Templates` | `Bazel Command`. + * Select `Templates` -> `Bazel Command`. * Enter Target Expression: - `//mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu` + `//mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu:handtrackinggpu` * Enter Bazel command: `mobile-install`. * Enter Bazel flags: `-c opt --config=android_arm64`. * Press the `[+]` button to add the new configuration. @@ -215,15 +227,14 @@ app: ### Option 1: Build with Bazel in Command Line -1. Modify the `bundle_id` field of the app's `ios_application` target to use - your own identifier. For instance, for - [MediaPipe Hand](./hand_tracking_mobile_gpu.md), the `bundle_id` is in the +1. Modify the `bundle_id` field of the app's `ios_application` build target to + use your own identifier. For instance, for + [MediaPipe Hand](../solutions/hand.md), the `bundle_id` is in the `HandTrackingGpuApp` target in the [BUILD](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handtrackinggpu/BUILD) file. -2. Again using [MediaPipe Hand](./hand_tracking_mobile_gpu.md) for example, - run: +2. Again using [MediaPipe Hand](../solutions/hand.md) for example, run: ```bash bazel build -c opt --config=ios_arm64 mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp @@ -287,7 +298,7 @@ the previous section. ### Option 1: Running on CPU -1. To build, for example, [MediaPipe Hand](./hand_tracking_mobile_gpu.md), run: +1. To build, for example, [MediaPipe Hand](../solutions/hand.md), run: ```bash bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu @@ -306,9 +317,9 @@ the previous section. ### Option 2: Running on GPU Note: This currently works only on Linux, and please first follow -[OpenGL ES Setup on Linux Desktop](./gpu.md#opengl-es-setup-on-linux-desktop). +[OpenGL ES Setup on Linux Desktop](./gpu_support.md#opengl-es-setup-on-linux-desktop). -1. To build, for example, [MediaPipe Hand](./hand_tracking_mobile_gpu.md), run: +1. To build, for example, [MediaPipe Hand](../solutions/hand.md), run: ```bash bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \ diff --git a/mediapipe/docs/how_to_questions.md b/docs/getting_started/faq.md similarity index 92% rename from mediapipe/docs/how_to_questions.md rename to docs/getting_started/faq.md index 321a21b6f..75bf8ad97 100644 --- a/mediapipe/docs/how_to_questions.md +++ b/docs/getting_started/faq.md @@ -1,12 +1,16 @@ -## Questions and Answers +--- +layout: default +title: FAQ +parent: Getting Started +nav_order: 9 +--- -- [How to convert ImageFrames and GpuBuffers](#how-to-convert-imageframes-and-gpubuffers) -- [How to visualize perceived results](#how-to-visualize-perception-results) -- [How to run calculators in parallel](#how-to-run-calculators-in-parallel) -- [Output timestamps when using ImmediateInputStreamHandler](#output-timestamps-when-using-immediateinputstreamhandler) -- [How to change settings at runtime](#how-to-change-settings-at-runtime) -- [How to process real-time input streams](#how-to-process-real-time-input-streams) -- [Can I run MediaPipe on MS Windows?](#can-i-run-mediapipe-on-ms-windows) +# FAQ +{: .no_toc } + +1. TOC +{:toc} +--- ### How to convert ImageFrames and GpuBuffers diff --git a/docs/getting_started/getting_started.md b/docs/getting_started/getting_started.md new file mode 100644 index 000000000..be7150543 --- /dev/null +++ b/docs/getting_started/getting_started.md @@ -0,0 +1,13 @@ +--- +layout: default +title: Getting Started +nav_order: 2 +has_children: true +--- + +# Getting Started +{: .no_toc } + +1. TOC +{:toc} +--- diff --git a/docs/getting_started/gpu_support.md b/docs/getting_started/gpu_support.md new file mode 100644 index 000000000..2aae63a2e --- /dev/null +++ b/docs/getting_started/gpu_support.md @@ -0,0 +1,186 @@ +--- +layout: default +title: GPU Support +parent: Getting Started +nav_order: 6 +--- + +# GPU Support +{: .no_toc } + +1. TOC +{:toc} +--- + +## OpenGL ES Support + +MediaPipe supports OpenGL ES up to version 3.2 on Android/Linux and up to ES 3.0 +on iOS. In addition, MediaPipe also supports Metal on iOS. + +OpenGL ES 3.1 or greater is required (on Android/Linux systems) for running +machine learning inference calculators and graphs. + +## Disable OpenGL ES Support + +By default, building MediaPipe (with no special bazel flags) attempts to compile +and link against OpenGL ES (and for iOS also Metal) libraries. + +On platforms where OpenGL ES is not available (see also +[OpenGL ES Setup on Linux Desktop](#opengl-es-setup-on-linux-desktop)), you +should disable OpenGL ES support with: + +``` +$ bazel build --define MEDIAPIPE_DISABLE_GPU=1 +``` + +Note: On Android and iOS, OpenGL ES is required by MediaPipe framework and the +support should never be disabled. + +## OpenGL ES Setup on Linux Desktop + +On Linux desktop with video cards that support OpenGL ES 3.1+, MediaPipe can run +GPU compute and rendering and perform TFLite inference on GPU. + +To check if your Linux desktop GPU can run MediaPipe with OpenGL ES: + +```bash +$ sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev +$ sudo apt-get install mesa-utils +$ glxinfo | grep -i opengl +``` + +For example, it may print: + +```bash +$ glxinfo | grep -i opengl +... +OpenGL ES profile version string: OpenGL ES 3.2 NVIDIA 430.50 +OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20 +OpenGL ES profile extensions: +``` + +*Notice the ES 3.20 text above.* + +You need to see ES 3.1 or greater printed in order to perform TFLite inference +on GPU in MediaPipe. With this setup, build with: + +``` +$ bazel build --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 +``` + +If only ES 3.0 or below is supported, you can still build MediaPipe targets that +don't require TFLite inference on GPU with: + +``` +$ bazel build --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 --copt -DMEDIAPIPE_DISABLE_GL_COMPUTE +``` + +Note: MEDIAPIPE_DISABLE_GL_COMPUTE is already defined automatically on all Apple +systems (Apple doesn't support OpenGL ES 3.1+). + +## TensorFlow CUDA Support and Setup on Linux Desktop + +MediaPipe framework doesn't require CUDA for GPU compute and rendering. However, +MediaPipe can work with TensorFlow to perform GPU inference on video cards that +support CUDA. + +To enable TensorFlow GPU inference with MediaPipe, the first step is to follow +the +[TensorFlow GPU documentation](https://www.tensorflow.org/install/gpu#software_requirements) +to install the required NVIDIA software on your Linux desktop. + +After installation, update `$PATH` and `$LD_LIBRARY_PATH` and run `ldconfig` +with: + +``` +$ export PATH=/usr/local/cuda-10.1/bin${PATH:+:${PATH}} +$ export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64,/usr/local/cuda-10.1/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} +$ sudo ldconfig +``` + +It's recommended to verify the installation of CUPTI, CUDA, CuDNN, and NVCC: + +``` +$ ls /usr/local/cuda/extras/CUPTI +/lib64 +libcupti.so libcupti.so.10.1.208 libnvperf_host.so libnvperf_target.so +libcupti.so.10.1 libcupti_static.a libnvperf_host_static.a + +$ ls /usr/local/cuda-10.1 +LICENSE bin extras lib64 libnvvp nvml samples src tools +README doc include libnsight nsightee_plugins nvvm share targets version.txt + +$ nvcc -V +nvcc: NVIDIA (R) Cuda compiler driver +Copyright (c) 2005-2019 NVIDIA Corporation +Built on Sun_Jul_28_19:07:16_PDT_2019 +Cuda compilation tools, release 10.1, V10.1.243 + +$ ls /usr/lib/x86_64-linux-gnu/ | grep libcudnn.so +libcudnn.so +libcudnn.so.7 +libcudnn.so.7.6.4 +``` + +Setting `$TF_CUDA_PATHS` is the way to declare where the CUDA library is. Note +that the following code snippet also adds `/usr/lib/x86_64-linux-gnu` and +`/usr/include` into `$TF_CUDA_PATHS` for cudablas and libcudnn. + +``` +$ export TF_CUDA_PATHS=/usr/local/cuda-10.1,/usr/lib/x86_64-linux-gnu,/usr/include +``` + +To make MediaPipe get TensorFlow's CUDA settings, find TensorFlow's +[.bazelrc](https://github.com/tensorflow/tensorflow/blob/master/.bazelrc) and +copy the `build:using_cuda` and `build:cuda` section into MediaPipe's .bazelrc +file. For example, as of April 23, 2020, TensorFlow's CUDA setting is the +following: + +``` +# This config refers to building with CUDA available. It does not necessarily +# mean that we build CUDA op kernels. +build:using_cuda --define=using_cuda=true +build:using_cuda --action_env TF_NEED_CUDA=1 +build:using_cuda --crosstool_top=@local_config_cuda//crosstool:toolchain + +# This config refers to building CUDA op kernels with nvcc. +build:cuda --config=using_cuda +build:cuda --define=using_cuda_nvcc=true +``` + +Finally, build MediaPipe with TensorFlow GPU with two more flags `--config=cuda` +and `--spawn_strategy=local`. For example: + +``` +$ bazel build -c opt --config=cuda --spawn_strategy=local \ + --define no_aws_support=true --copt -DMESA_EGL_NO_X11_HEADERS \ + mediapipe/examples/desktop/object_detection:object_detection_tensorflow +``` + +While the binary is running, it prints out the GPU device info: + +``` +I external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 +I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1544] Found device 0 with properties: pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5 coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s +I external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0 +``` + +You can monitor the GPU usage to verify whether the GPU is used for model +inference. + +``` +$ nvidia-smi --query-gpu=utilization.gpu --format=csv --loop=1 + +0 % +0 % +4 % +5 % +83 % +21 % +22 % +27 % +29 % +100 % +0 % +0% +``` diff --git a/mediapipe/docs/hello_world_android.md b/docs/getting_started/hello_world_android.md similarity index 97% rename from mediapipe/docs/hello_world_android.md rename to docs/getting_started/hello_world_android.md index a83b39a47..2794ea4f8 100644 --- a/mediapipe/docs/hello_world_android.md +++ b/docs/getting_started/hello_world_android.md @@ -1,4 +1,16 @@ -# Hello World! in MediaPipe on Android +--- +layout: default +title: Hello World! on Android +parent: Getting Started +nav_order: 3 +--- + +# Hello World! on Android +{: .no_toc } + +1. TOC +{:toc} +--- ## Introduction @@ -14,7 +26,7 @@ graph on Android. A simple camera app for real-time Sobel edge detection applied to a live video stream on an Android device. -![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif) +![edge_detection_android_gpu_gif](../images/mobile/edge_detection_android_gpu.gif) ## Setup @@ -56,7 +68,7 @@ node: { A visualization of the graph is shown below: -![edge_detection_mobile_gpu](images/mobile/edge_detection_mobile_gpu.png) +![edge_detection_mobile_gpu](../images/mobile/edge_detection_mobile_gpu.png) This graph has a single input stream named `input_video` for all incoming frames that will be provided by your device's camera. @@ -247,7 +259,7 @@ adb install bazel-bin/$APPLICATION_PATH/helloworld.apk Open the application on your device. It should display a screen with the text `Hello World!`. -![bazel_hello_world_android](images/mobile/bazel_hello_world_android.png) +![bazel_hello_world_android](../images/mobile/bazel_hello_world_android.png) ## Using the camera via `CameraX` @@ -364,7 +376,7 @@ Add the following line in the `$APPLICATION_PATH/res/values/strings.xml` file: When the user doesn't grant camera permission, the screen will now look like this: -![missing_camera_permission_android](images/mobile/missing_camera_permission_android.png) +![missing_camera_permission_android](../images/mobile/missing_camera_permission_android.png) Now, we will add the [`SurfaceTexture`] and [`SurfaceView`] objects to `MainActivity`: @@ -740,7 +752,7 @@ And that's it! You should now be able to successfully build and run the application on the device and see Sobel edge detection running on a live camera feed! Congrats! -![edge_detection_android_gpu_gif](images/mobile/edge_detection_android_gpu.gif) +![edge_detection_android_gpu_gif](../images/mobile/edge_detection_android_gpu.gif) If you ran into any issues, please see the full code of the tutorial [here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic). diff --git a/mediapipe/docs/hello_world_desktop.md b/docs/getting_started/hello_world_desktop.md similarity index 93% rename from mediapipe/docs/hello_world_desktop.md rename to docs/getting_started/hello_world_desktop.md index 236b8675e..28a9aea8f 100644 --- a/mediapipe/docs/hello_world_desktop.md +++ b/docs/getting_started/hello_world_desktop.md @@ -1,4 +1,16 @@ -## Hello World for C++ +--- +layout: default +title: Hello World! on Desktop (C++) +parent: Getting Started +nav_order: 5 +--- + +# Hello World! on Desktop (C++) +{: .no_toc } + +1. TOC +{:toc} +--- 1. Ensure you have a working version of MediaPipe. See [installation instructions](./install.md). @@ -52,7 +64,7 @@ You can visualize this graph using [MediaPipe Visualizer](https://viz.mediapipe.dev) by pasting the CalculatorGraphConfig content below into the visualizer. See - [here](./visualizer.md) for help on the visualizer. + [here](../tools/visualizer.md) for help on the visualizer. ```bash input_stream: "in" @@ -72,7 +84,7 @@ This graph consists of 1 graph input stream (`in`) and 1 graph output stream (`out`), and 2 [`PassThroughCalculator`]s connected serially. - ![hello_world graph](./images/hello_world.png) + ![hello_world graph](../images/hello_world.png) 4. Before running the graph, an `OutputStreamPoller` object is connected to the output stream in order to later retrieve the graph output, and a graph run diff --git a/mediapipe/docs/hello_world_ios.md b/docs/getting_started/hello_world_ios.md similarity index 97% rename from mediapipe/docs/hello_world_ios.md rename to docs/getting_started/hello_world_ios.md index ac9f76885..1c6c44961 100644 --- a/mediapipe/docs/hello_world_ios.md +++ b/docs/getting_started/hello_world_ios.md @@ -1,4 +1,16 @@ -# Hello World! in MediaPipe on iOS +--- +layout: default +title: Hello World! on iOS +parent: Getting Started +nav_order: 4 +--- + +# Hello World! on iOS +{: .no_toc } + +1. TOC +{:toc} +--- ## Introduction @@ -14,7 +26,7 @@ graph on iOS. A simple camera app for real-time Sobel edge detection applied to a live video stream on an iOS device. -![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif) +![edge_detection_ios_gpu_gif](../images/mobile/edge_detection_ios_gpu.gif) ## Setup @@ -54,7 +66,7 @@ node: { A visualization of the graph is shown below: -![edge_detection_mobile_gpu](images/mobile/edge_detection_mobile_gpu.png) +![edge_detection_mobile_gpu](../images/mobile/edge_detection_mobile_gpu.png) This graph has a single input stream named `input_video` for all incoming frames that will be provided by your device's camera. @@ -538,7 +550,7 @@ method to receive packets on this output stream and display them on the screen: And that is all! Build and run the app on your iOS device. You should see the results of running the edge detection graph on a live video feed. Congrats! -![edge_detection_ios_gpu_gif](images/mobile/edge_detection_ios_gpu.gif) +![edge_detection_ios_gpu_gif](../images/mobile/edge_detection_ios_gpu.gif) If you ran into any issues, please see the full code of the tutorial [here](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/edgedetectiongpu). diff --git a/mediapipe/docs/help.md b/docs/getting_started/help.md similarity index 89% rename from mediapipe/docs/help.md rename to docs/getting_started/help.md index 6667afb78..a9d2ba7b9 100644 --- a/mediapipe/docs/help.md +++ b/docs/getting_started/help.md @@ -1,17 +1,24 @@ -## Getting Help +--- +layout: default +title: Getting Help +parent: Getting Started +nav_order: 8 +--- -- [Technical questions](#technical-questions) -- [Bugs and feature requests](#bugs-and-feature-requests) +# Getting Help +{: .no_toc } -Below are the various ways to get help: +1. TOC +{:toc} +--- -### Technical questions +## Technical questions For help with technical or algorithmic questions, visit [Stack Overflow](https://stackoverflow.com/questions/tagged/mediapipe) to find answers and support from the MediaPipe community. -### Bugs and feature requests +## Bugs and feature requests To report bugs or make feature requests, [file an issue on GitHub](https://github.com/google/mediapipe/issues). diff --git a/mediapipe/docs/install.md b/docs/getting_started/install.md similarity index 96% rename from mediapipe/docs/install.md rename to docs/getting_started/install.md index f842b60a6..3fa83e95e 100644 --- a/mediapipe/docs/install.md +++ b/docs/getting_started/install.md @@ -1,4 +1,16 @@ -## Installing MediaPipe +--- +layout: default +title: Installation +parent: Getting Started +nav_order: 1 +--- + +# Installation +{: .no_toc } + +1. TOC +{:toc} +--- Note: To interoperate with OpenCV, OpenCV 3.x and above are preferred. OpenCV 2.x currently works but interoperability support may be deprecated in the @@ -11,22 +23,11 @@ Note: To make Mediapipe work with TensorFlow, please set Python 3.7 as the default Python version and install the Python "six" library by running `pip3 install --user six`. -Choose your operating system: +Note: To build and run Android example apps, see these +[instructions](./building_examples.md#android). To build and run iOS example +apps, see these [instructions](./building_examples.md#ios). -- [Installing on Debian and Ubuntu](#installing-on-debian-and-ubuntu) -- [Installing on CentOS](#installing-on-centos) -- [Installing on macOS](#installing-on-macos) -- [Installing on Windows](#installing-on-windows) -- [Installing on Windows Subsystem for Linux (WSL)](#installing-on-windows-subsystem-for-linux-wsl) -- [Installing using Docker](#installing-using-docker) - -To build and run Android example apps, see these -[instuctions](./building_examples.md#android). - -To build and run iOS example apps, see these -[instuctions](./building_examples.md#ios). - -### Installing on Debian and Ubuntu +## Installing on Debian and Ubuntu 1. Checkout MediaPipe repository. @@ -137,7 +138,7 @@ To build and run iOS example apps, see these # Hello World! ``` -### Installing on CentOS +## Installing on CentOS 1. Checkout MediaPipe repository. @@ -220,7 +221,7 @@ To build and run iOS example apps, see these # Hello World! ``` -### Installing on macOS +## Installing on macOS 1. Prework: @@ -352,7 +353,7 @@ To build and run iOS example apps, see these # Hello World! ``` -### Installing on Windows +## Installing on Windows **Disclaimer**: Running MediaPipe on Windows is experimental. @@ -457,7 +458,7 @@ next section. ``` -### Installing on Windows Subsystem for Linux (WSL) +## Installing on Windows Subsystem for Linux (WSL) Note: The pre-built OpenCV packages don't support cameras in WSL. Unless you [compile](https://funvision.blogspot.com/2019/12/opencv-web-camera-and-video-streams-in.html) @@ -580,7 +581,7 @@ cameras. Alternatively, you use a video file as input. # Hello World! ``` -### Installing using Docker +## Installing using Docker This will use a Docker image that will isolate mediapipe's installation from the rest of the system. @@ -630,7 +631,7 @@ This will use a Docker image that will isolate mediapipe's installation from the # Hello World! ``` -4. Build Mediapipe [Android demos](./examples.md). +4. Build a MediaPipe Android example. ```bash $ docker run -it --name mediapipe mediapipe:latest diff --git a/mediapipe/docs/troubleshooting.md b/docs/getting_started/troubleshooting.md similarity index 91% rename from mediapipe/docs/troubleshooting.md rename to docs/getting_started/troubleshooting.md index e407f6198..9d1bedac4 100644 --- a/mediapipe/docs/troubleshooting.md +++ b/docs/getting_started/troubleshooting.md @@ -1,12 +1,16 @@ -# Troubleshooting +--- +layout: default +title: Troubleshooting +parent: Getting Started +nav_order: 10 +--- -- [Native method not found](#native-method-not-found) -- [No registered calculator found](#no-registered-calculator-found) -- [Out Of Memory error](#out-of-memory-error) -- [Graph hangs](#graph-hangs) -- [Calculator is scheduled infrequently](#calculator-is-scheduled-infrequently) -- [Output timing is uneven](#output-timing-is-uneven) -- [CalculatorGraph lags behind inputs](#calculatorgraph-lags-behind-inputs) +# Troubleshooting +{: .no_toc } + +1. TOC +{:toc} +--- ## Native method not found @@ -55,7 +59,7 @@ running MediaPipe graph. This can occur for a number of reasons, such as: For problem (1), it may be necessary to drop some old packets in older to process the more recent packets. For some hints, see: -[How to process realtime input streams](how_to_questions.md). +[`How to process realtime input streams`]. For problem (2), it could be that one input stream is lacking packets for some reason. A device or a calculator may be misconfigured or may produce packets @@ -63,7 +67,7 @@ only sporadically. This can cause downstream calculators to wait for many packets that will never arrive, which in turn causes packets to accumulate on some of their input streams. MediaPipe addresses this sort of problem using "timestamp bounds". For some hints see: -[How to process realtime input streams](how_to_questions.md). +[`How to process realtime input streams`]. The MediaPipe setting [`CalculatorGraphConfig::max_queue_size`] limits the number of packets enqueued on any input stream by throttling inputs to the @@ -122,14 +126,14 @@ each packet as early as possible. Normally the lowest possible latency is the total time required by each calculator along a "critical path" of successive calculators. The latency of the a MediaPipe graph could be worse than the ideal due to delays introduced to display frames a even intervals as described in -[Output timing is uneven](troubleshooting.md?cl=252235797#output-timing-is-uneven). +[Output timing is uneven](#output-timing-is-uneven). If some of the calculators in the graph cannot keep pace with the realtime input streams, then latency will continue to increase, and it becomes necessary to drop some input packets. The recommended technique is to use the MediaPipe calculators designed specifically for this purpose such as [`FlowLimiterCalculator`] as described in -[How to process realtime input streams](how_to_questions.md). +[`How to process realtime input streams`]. [`CalculatorGraphConfig`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto [`CalculatorGraphConfig::max_queue_size`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto @@ -142,3 +146,4 @@ calculators designed specifically for this purpose such as [`Timestamp::Done`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/timestamp.h [`CalculatorBase::Close`]: https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h [`FlowLimiterCalculator`]: https://github.com/google/mediapipe/tree/master/mediapipe/calculators/core/flow_limiter_calculator.cc +[`How to process realtime input streams`]: faq.md#how-to-process-realtime-input-streams diff --git a/docs/images/accelerated.png b/docs/images/accelerated.png new file mode 100644 index 000000000..8c9d241ca Binary files /dev/null and b/docs/images/accelerated.png differ diff --git a/docs/images/accelerated_small.png b/docs/images/accelerated_small.png new file mode 100644 index 000000000..759542dc4 Binary files /dev/null and b/docs/images/accelerated_small.png differ diff --git a/mediapipe/docs/images/add_ipa.png b/docs/images/add_ipa.png similarity index 100% rename from mediapipe/docs/images/add_ipa.png rename to docs/images/add_ipa.png diff --git a/mediapipe/docs/images/app_ipa.png b/docs/images/app_ipa.png similarity index 100% rename from mediapipe/docs/images/app_ipa.png rename to docs/images/app_ipa.png diff --git a/mediapipe/docs/images/app_ipa_added.png b/docs/images/app_ipa_added.png similarity index 100% rename from mediapipe/docs/images/app_ipa_added.png rename to docs/images/app_ipa_added.png diff --git a/mediapipe/docs/images/autoflip_edited_example.gif b/docs/images/autoflip_edited_example.gif similarity index 100% rename from mediapipe/docs/images/autoflip_edited_example.gif rename to docs/images/autoflip_edited_example.gif diff --git a/mediapipe/docs/images/autoflip_graph.png b/docs/images/autoflip_graph.png similarity index 100% rename from mediapipe/docs/images/autoflip_graph.png rename to docs/images/autoflip_graph.png diff --git a/mediapipe/docs/images/autoflip_is_required.gif b/docs/images/autoflip_is_required.gif similarity index 100% rename from mediapipe/docs/images/autoflip_is_required.gif rename to docs/images/autoflip_is_required.gif diff --git a/mediapipe/docs/images/bazel_permission.png b/docs/images/bazel_permission.png similarity index 100% rename from mediapipe/docs/images/bazel_permission.png rename to docs/images/bazel_permission.png diff --git a/mediapipe/docs/images/click_subgraph_handdetection.png b/docs/images/click_subgraph_handdetection.png similarity index 100% rename from mediapipe/docs/images/click_subgraph_handdetection.png rename to docs/images/click_subgraph_handdetection.png diff --git a/mediapipe/docs/images/console_error.png b/docs/images/console_error.png similarity index 100% rename from mediapipe/docs/images/console_error.png rename to docs/images/console_error.png diff --git a/docs/images/cross_platform.png b/docs/images/cross_platform.png new file mode 100644 index 000000000..09dedc96a Binary files /dev/null and b/docs/images/cross_platform.png differ diff --git a/docs/images/cross_platform_small.png b/docs/images/cross_platform_small.png new file mode 100644 index 000000000..7476327b2 Binary files /dev/null and b/docs/images/cross_platform_small.png differ diff --git a/mediapipe/docs/images/cyclic_integer_sum_graph.svg b/docs/images/cyclic_integer_sum_graph.svg similarity index 100% rename from mediapipe/docs/images/cyclic_integer_sum_graph.svg rename to docs/images/cyclic_integer_sum_graph.svg diff --git a/mediapipe/docs/images/device.png b/docs/images/device.png similarity index 100% rename from mediapipe/docs/images/device.png rename to docs/images/device.png diff --git a/mediapipe/docs/images/editor_view.png b/docs/images/editor_view.png similarity index 100% rename from mediapipe/docs/images/editor_view.png rename to docs/images/editor_view.png diff --git a/mediapipe/docs/images/face_detection_desktop.png b/docs/images/face_detection_desktop.png similarity index 100% rename from mediapipe/docs/images/face_detection_desktop.png rename to docs/images/face_detection_desktop.png diff --git a/docs/images/face_mesh_ar_effects.gif b/docs/images/face_mesh_ar_effects.gif new file mode 100644 index 000000000..868a40c4d Binary files /dev/null and b/docs/images/face_mesh_ar_effects.gif differ diff --git a/mediapipe/docs/images/favicon.ico b/docs/images/favicon.ico similarity index 100% rename from mediapipe/docs/images/favicon.ico rename to docs/images/favicon.ico diff --git a/mediapipe/docs/images/faviconv2.ico b/docs/images/faviconv2.ico similarity index 100% rename from mediapipe/docs/images/faviconv2.ico rename to docs/images/faviconv2.ico diff --git a/mediapipe/docs/images/gpu_example_graph.png b/docs/images/gpu_example_graph.png similarity index 100% rename from mediapipe/docs/images/gpu_example_graph.png rename to docs/images/gpu_example_graph.png diff --git a/mediapipe/docs/images/graph_visual.png b/docs/images/graph_visual.png similarity index 100% rename from mediapipe/docs/images/graph_visual.png rename to docs/images/graph_visual.png diff --git a/mediapipe/docs/images/hand_tracking_desktop.png b/docs/images/hand_tracking_desktop.png similarity index 100% rename from mediapipe/docs/images/hand_tracking_desktop.png rename to docs/images/hand_tracking_desktop.png diff --git a/mediapipe/docs/images/hello_world.png b/docs/images/hello_world.png similarity index 100% rename from mediapipe/docs/images/hello_world.png rename to docs/images/hello_world.png diff --git a/mediapipe/docs/images/iconv2.png b/docs/images/iconv2.png similarity index 100% rename from mediapipe/docs/images/iconv2.png rename to docs/images/iconv2.png diff --git a/docs/images/knift_stop_sign.gif b/docs/images/knift_stop_sign.gif new file mode 100644 index 000000000..a84b4aa19 Binary files /dev/null and b/docs/images/knift_stop_sign.gif differ diff --git a/mediapipe/docs/images/logo.png b/docs/images/logo.png similarity index 100% rename from mediapipe/docs/images/logo.png rename to docs/images/logo.png diff --git a/mediapipe/docs/images/logo_horizontal_black.png b/docs/images/logo_horizontal_black.png similarity index 100% rename from mediapipe/docs/images/logo_horizontal_black.png rename to docs/images/logo_horizontal_black.png diff --git a/docs/images/logo_horizontal_color.png b/docs/images/logo_horizontal_color.png new file mode 100644 index 000000000..6779a0d2a Binary files /dev/null and b/docs/images/logo_horizontal_color.png differ diff --git a/mediapipe/docs/images/logo_horizontal_white.png b/docs/images/logo_horizontal_white.png similarity index 100% rename from mediapipe/docs/images/logo_horizontal_white.png rename to docs/images/logo_horizontal_white.png diff --git a/mediapipe/docs/images/logov2.png b/docs/images/logov2.png similarity index 100% rename from mediapipe/docs/images/logov2.png rename to docs/images/logov2.png diff --git a/mediapipe/docs/images/maingraph_visualizer.png b/docs/images/maingraph_visualizer.png similarity index 100% rename from mediapipe/docs/images/maingraph_visualizer.png rename to docs/images/maingraph_visualizer.png diff --git a/mediapipe/docs/images/mediapipe_small.png b/docs/images/mediapipe_small.png similarity index 100% rename from mediapipe/docs/images/mediapipe_small.png rename to docs/images/mediapipe_small.png diff --git a/mediapipe/docs/images/mobile/aar_location.png b/docs/images/mobile/aar_location.png similarity index 100% rename from mediapipe/docs/images/mobile/aar_location.png rename to docs/images/mobile/aar_location.png diff --git a/mediapipe/docs/images/mobile/android_studio_opencv_location.png b/docs/images/mobile/android_studio_opencv_location.png similarity index 100% rename from mediapipe/docs/images/mobile/android_studio_opencv_location.png rename to docs/images/mobile/android_studio_opencv_location.png diff --git a/mediapipe/docs/images/mobile/assets_location.png b/docs/images/mobile/assets_location.png similarity index 100% rename from mediapipe/docs/images/mobile/assets_location.png rename to docs/images/mobile/assets_location.png diff --git a/mediapipe/docs/images/mobile/bazel_hello_world_android.png b/docs/images/mobile/bazel_hello_world_android.png similarity index 100% rename from mediapipe/docs/images/mobile/bazel_hello_world_android.png rename to docs/images/mobile/bazel_hello_world_android.png diff --git a/mediapipe/docs/images/mobile/box_tracking_subgraph.png b/docs/images/mobile/box_tracking_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/box_tracking_subgraph.png rename to docs/images/mobile/box_tracking_subgraph.png diff --git a/mediapipe/docs/images/mobile/edge_detection_android_gpu.gif b/docs/images/mobile/edge_detection_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/edge_detection_android_gpu.gif rename to docs/images/mobile/edge_detection_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/edge_detection_ios_gpu.gif b/docs/images/mobile/edge_detection_ios_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/edge_detection_ios_gpu.gif rename to docs/images/mobile/edge_detection_ios_gpu.gif diff --git a/mediapipe/docs/images/mobile/edge_detection_mobile_gpu.png b/docs/images/mobile/edge_detection_mobile_gpu.png similarity index 100% rename from mediapipe/docs/images/mobile/edge_detection_mobile_gpu.png rename to docs/images/mobile/edge_detection_mobile_gpu.png diff --git a/mediapipe/docs/images/mobile/face_detection_android_gpu.gif b/docs/images/mobile/face_detection_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/face_detection_android_gpu.gif rename to docs/images/mobile/face_detection_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/face_detection_android_gpu_small.gif b/docs/images/mobile/face_detection_android_gpu_small.gif similarity index 100% rename from mediapipe/docs/images/mobile/face_detection_android_gpu_small.gif rename to docs/images/mobile/face_detection_android_gpu_small.gif diff --git a/mediapipe/docs/images/mobile/face_detection_mobile_cpu.png b/docs/images/mobile/face_detection_mobile_cpu.png similarity index 100% rename from mediapipe/docs/images/mobile/face_detection_mobile_cpu.png rename to docs/images/mobile/face_detection_mobile_cpu.png diff --git a/mediapipe/docs/images/mobile/face_detection_mobile_gpu.png b/docs/images/mobile/face_detection_mobile_gpu.png similarity index 100% rename from mediapipe/docs/images/mobile/face_detection_mobile_gpu.png rename to docs/images/mobile/face_detection_mobile_gpu.png diff --git a/mediapipe/docs/images/mobile/face_landmark_front_gpu_subgraph.png b/docs/images/mobile/face_landmark_front_gpu_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/face_landmark_front_gpu_subgraph.png rename to docs/images/mobile/face_landmark_front_gpu_subgraph.png diff --git a/mediapipe/docs/images/mobile/face_mesh_android_gpu.gif b/docs/images/mobile/face_mesh_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/face_mesh_android_gpu.gif rename to docs/images/mobile/face_mesh_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/face_mesh_android_gpu_small.gif b/docs/images/mobile/face_mesh_android_gpu_small.gif similarity index 100% rename from mediapipe/docs/images/mobile/face_mesh_android_gpu_small.gif rename to docs/images/mobile/face_mesh_android_gpu_small.gif diff --git a/mediapipe/docs/images/mobile/face_mesh_mobile.png b/docs/images/mobile/face_mesh_mobile.png similarity index 100% rename from mediapipe/docs/images/mobile/face_mesh_mobile.png rename to docs/images/mobile/face_mesh_mobile.png diff --git a/mediapipe/docs/images/mobile/face_renderer_gpu_subgraph.png b/docs/images/mobile/face_renderer_gpu_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/face_renderer_gpu_subgraph.png rename to docs/images/mobile/face_renderer_gpu_subgraph.png diff --git a/mediapipe/docs/images/mobile/hair_segmentation_android_gpu.gif b/docs/images/mobile/hair_segmentation_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/hair_segmentation_android_gpu.gif rename to docs/images/mobile/hair_segmentation_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/hair_segmentation_android_gpu_small.gif b/docs/images/mobile/hair_segmentation_android_gpu_small.gif similarity index 100% rename from mediapipe/docs/images/mobile/hair_segmentation_android_gpu_small.gif rename to docs/images/mobile/hair_segmentation_android_gpu_small.gif diff --git a/mediapipe/docs/images/mobile/hair_segmentation_mobile_gpu.png b/docs/images/mobile/hair_segmentation_mobile_gpu.png similarity index 100% rename from mediapipe/docs/images/mobile/hair_segmentation_mobile_gpu.png rename to docs/images/mobile/hair_segmentation_mobile_gpu.png diff --git a/mediapipe/docs/images/mobile/hand_crops.png b/docs/images/mobile/hand_crops.png similarity index 100% rename from mediapipe/docs/images/mobile/hand_crops.png rename to docs/images/mobile/hand_crops.png diff --git a/mediapipe/docs/images/mobile/hand_detection_android_gpu.gif b/docs/images/mobile/hand_detection_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/hand_detection_android_gpu.gif rename to docs/images/mobile/hand_detection_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/hand_detection_android_gpu_small.gif b/docs/images/mobile/hand_detection_android_gpu_small.gif similarity index 100% rename from mediapipe/docs/images/mobile/hand_detection_android_gpu_small.gif rename to docs/images/mobile/hand_detection_android_gpu_small.gif diff --git a/mediapipe/docs/images/mobile/hand_detection_gpu_subgraph.png b/docs/images/mobile/hand_detection_gpu_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/hand_detection_gpu_subgraph.png rename to docs/images/mobile/hand_detection_gpu_subgraph.png diff --git a/mediapipe/docs/images/mobile/hand_detection_mobile.png b/docs/images/mobile/hand_detection_mobile.png similarity index 100% rename from mediapipe/docs/images/mobile/hand_detection_mobile.png rename to docs/images/mobile/hand_detection_mobile.png diff --git a/mediapipe/docs/images/mobile/hand_landmark_gpu_subgraph.png b/docs/images/mobile/hand_landmark_gpu_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/hand_landmark_gpu_subgraph.png rename to docs/images/mobile/hand_landmark_gpu_subgraph.png diff --git a/mediapipe/docs/images/mobile/hand_renderer_gpu_subgraph.png b/docs/images/mobile/hand_renderer_gpu_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/hand_renderer_gpu_subgraph.png rename to docs/images/mobile/hand_renderer_gpu_subgraph.png diff --git a/mediapipe/docs/images/mobile/hand_tracking_3d_android_gpu.gif b/docs/images/mobile/hand_tracking_3d_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/hand_tracking_3d_android_gpu.gif rename to docs/images/mobile/hand_tracking_3d_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/hand_tracking_android_gpu.gif b/docs/images/mobile/hand_tracking_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/hand_tracking_android_gpu.gif rename to docs/images/mobile/hand_tracking_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/hand_tracking_android_gpu_small.gif b/docs/images/mobile/hand_tracking_android_gpu_small.gif similarity index 100% rename from mediapipe/docs/images/mobile/hand_tracking_android_gpu_small.gif rename to docs/images/mobile/hand_tracking_android_gpu_small.gif diff --git a/mediapipe/docs/images/mobile/hand_tracking_mobile.png b/docs/images/mobile/hand_tracking_mobile.png similarity index 100% rename from mediapipe/docs/images/mobile/hand_tracking_mobile.png rename to docs/images/mobile/hand_tracking_mobile.png diff --git a/mediapipe/docs/images/mobile/missing_camera_permission_android.png b/docs/images/mobile/missing_camera_permission_android.png similarity index 100% rename from mediapipe/docs/images/mobile/missing_camera_permission_android.png rename to docs/images/mobile/missing_camera_permission_android.png diff --git a/mediapipe/docs/images/mobile/multi_hand_detection_gpu_subgraph.png b/docs/images/mobile/multi_hand_detection_gpu_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/multi_hand_detection_gpu_subgraph.png rename to docs/images/mobile/multi_hand_detection_gpu_subgraph.png diff --git a/mediapipe/docs/images/mobile/multi_hand_landmark_subgraph.png b/docs/images/mobile/multi_hand_landmark_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/multi_hand_landmark_subgraph.png rename to docs/images/mobile/multi_hand_landmark_subgraph.png diff --git a/mediapipe/docs/images/mobile/multi_hand_renderer_gpu_subgraph.png b/docs/images/mobile/multi_hand_renderer_gpu_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/multi_hand_renderer_gpu_subgraph.png rename to docs/images/mobile/multi_hand_renderer_gpu_subgraph.png diff --git a/mediapipe/docs/images/mobile/multi_hand_tracking_3d_android_gpu.gif b/docs/images/mobile/multi_hand_tracking_3d_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/multi_hand_tracking_3d_android_gpu.gif rename to docs/images/mobile/multi_hand_tracking_3d_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/multi_hand_tracking_3d_android_gpu_small.gif b/docs/images/mobile/multi_hand_tracking_3d_android_gpu_small.gif similarity index 100% rename from mediapipe/docs/images/mobile/multi_hand_tracking_3d_android_gpu_small.gif rename to docs/images/mobile/multi_hand_tracking_3d_android_gpu_small.gif diff --git a/mediapipe/docs/images/mobile/multi_hand_tracking_android_gpu.gif b/docs/images/mobile/multi_hand_tracking_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/multi_hand_tracking_android_gpu.gif rename to docs/images/mobile/multi_hand_tracking_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/multi_hand_tracking_mobile.png b/docs/images/mobile/multi_hand_tracking_mobile.png similarity index 100% rename from mediapipe/docs/images/mobile/multi_hand_tracking_mobile.png rename to docs/images/mobile/multi_hand_tracking_mobile.png diff --git a/mediapipe/docs/images/mobile/object_detection_3d_android_gpu.png b/docs/images/mobile/object_detection_3d_android_gpu.png similarity index 100% rename from mediapipe/docs/images/mobile/object_detection_3d_android_gpu.png rename to docs/images/mobile/object_detection_3d_android_gpu.png diff --git a/mediapipe/docs/images/mobile/object_detection_android_cpu.gif b/docs/images/mobile/object_detection_android_cpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/object_detection_android_cpu.gif rename to docs/images/mobile/object_detection_android_cpu.gif diff --git a/mediapipe/docs/images/mobile/object_detection_android_gpu.gif b/docs/images/mobile/object_detection_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/object_detection_android_gpu.gif rename to docs/images/mobile/object_detection_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/object_detection_android_gpu_small.gif b/docs/images/mobile/object_detection_android_gpu_small.gif similarity index 100% rename from mediapipe/docs/images/mobile/object_detection_android_gpu_small.gif rename to docs/images/mobile/object_detection_android_gpu_small.gif diff --git a/mediapipe/docs/images/mobile/object_detection_gpu_subgraph.png b/docs/images/mobile/object_detection_gpu_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/object_detection_gpu_subgraph.png rename to docs/images/mobile/object_detection_gpu_subgraph.png diff --git a/mediapipe/docs/images/mobile/object_detection_mobile_cpu.png b/docs/images/mobile/object_detection_mobile_cpu.png similarity index 100% rename from mediapipe/docs/images/mobile/object_detection_mobile_cpu.png rename to docs/images/mobile/object_detection_mobile_cpu.png diff --git a/mediapipe/docs/images/mobile/object_detection_mobile_gpu.png b/docs/images/mobile/object_detection_mobile_gpu.png similarity index 100% rename from mediapipe/docs/images/mobile/object_detection_mobile_gpu.png rename to docs/images/mobile/object_detection_mobile_gpu.png diff --git a/mediapipe/docs/images/mobile/object_tracking_android_gpu.gif b/docs/images/mobile/object_tracking_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/object_tracking_android_gpu.gif rename to docs/images/mobile/object_tracking_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/object_tracking_android_gpu_detection_only.gif b/docs/images/mobile/object_tracking_android_gpu_detection_only.gif similarity index 100% rename from mediapipe/docs/images/mobile/object_tracking_android_gpu_detection_only.gif rename to docs/images/mobile/object_tracking_android_gpu_detection_only.gif diff --git a/mediapipe/docs/images/mobile/object_tracking_android_gpu_small.gif b/docs/images/mobile/object_tracking_android_gpu_small.gif similarity index 100% rename from mediapipe/docs/images/mobile/object_tracking_android_gpu_small.gif rename to docs/images/mobile/object_tracking_android_gpu_small.gif diff --git a/mediapipe/docs/images/mobile/object_tracking_mobile_gpu.png b/docs/images/mobile/object_tracking_mobile_gpu.png similarity index 100% rename from mediapipe/docs/images/mobile/object_tracking_mobile_gpu.png rename to docs/images/mobile/object_tracking_mobile_gpu.png diff --git a/mediapipe/docs/images/mobile/object_tracking_renderer_gpu_subgraph.png b/docs/images/mobile/object_tracking_renderer_gpu_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/object_tracking_renderer_gpu_subgraph.png rename to docs/images/mobile/object_tracking_renderer_gpu_subgraph.png diff --git a/mediapipe/docs/images/mobile/object_tracking_subgraph.png b/docs/images/mobile/object_tracking_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/object_tracking_subgraph.png rename to docs/images/mobile/object_tracking_subgraph.png diff --git a/mediapipe/docs/images/mobile/objectron_chair_android_gpu.gif b/docs/images/mobile/objectron_chair_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/objectron_chair_android_gpu.gif rename to docs/images/mobile/objectron_chair_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/objectron_chair_android_gpu_small.gif b/docs/images/mobile/objectron_chair_android_gpu_small.gif similarity index 100% rename from mediapipe/docs/images/mobile/objectron_chair_android_gpu_small.gif rename to docs/images/mobile/objectron_chair_android_gpu_small.gif diff --git a/mediapipe/docs/images/mobile/objectron_detection_subgraph.png b/docs/images/mobile/objectron_detection_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/objectron_detection_subgraph.png rename to docs/images/mobile/objectron_detection_subgraph.png diff --git a/mediapipe/docs/images/mobile/objectron_shoe_android_gpu.gif b/docs/images/mobile/objectron_shoe_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/objectron_shoe_android_gpu.gif rename to docs/images/mobile/objectron_shoe_android_gpu.gif diff --git a/mediapipe/docs/images/mobile/objectron_shoe_android_gpu_small.gif b/docs/images/mobile/objectron_shoe_android_gpu_small.gif similarity index 100% rename from mediapipe/docs/images/mobile/objectron_shoe_android_gpu_small.gif rename to docs/images/mobile/objectron_shoe_android_gpu_small.gif diff --git a/mediapipe/docs/images/mobile/objectron_tracking_subgraph.png b/docs/images/mobile/objectron_tracking_subgraph.png similarity index 100% rename from mediapipe/docs/images/mobile/objectron_tracking_subgraph.png rename to docs/images/mobile/objectron_tracking_subgraph.png diff --git a/mediapipe/docs/images/mobile/renderer_gpu.png b/docs/images/mobile/renderer_gpu.png similarity index 100% rename from mediapipe/docs/images/mobile/renderer_gpu.png rename to docs/images/mobile/renderer_gpu.png diff --git a/mediapipe/docs/images/mobile/template_matching_android_cpu.gif b/docs/images/mobile/template_matching_android_cpu.gif similarity index 100% rename from mediapipe/docs/images/mobile/template_matching_android_cpu.gif rename to docs/images/mobile/template_matching_android_cpu.gif diff --git a/mediapipe/docs/images/mobile/template_matching_android_cpu_small.gif b/docs/images/mobile/template_matching_android_cpu_small.gif similarity index 100% rename from mediapipe/docs/images/mobile/template_matching_android_cpu_small.gif rename to docs/images/mobile/template_matching_android_cpu_small.gif diff --git a/mediapipe/docs/images/mobile/template_matching_mobile_graph.png b/docs/images/mobile/template_matching_mobile_graph.png similarity index 100% rename from mediapipe/docs/images/mobile/template_matching_mobile_graph.png rename to docs/images/mobile/template_matching_mobile_graph.png diff --git a/mediapipe/docs/images/mobile/template_matching_mobile_template.jpg b/docs/images/mobile/template_matching_mobile_template.jpg similarity index 100% rename from mediapipe/docs/images/mobile/template_matching_mobile_template.jpg rename to docs/images/mobile/template_matching_mobile_template.jpg diff --git a/mediapipe/docs/images/multi_hand_tracking_android_gpu.gif b/docs/images/multi_hand_tracking_android_gpu.gif similarity index 100% rename from mediapipe/docs/images/multi_hand_tracking_android_gpu.gif rename to docs/images/multi_hand_tracking_android_gpu.gif diff --git a/mediapipe/docs/images/multi_hand_tracking_android_gpu_small.gif b/docs/images/multi_hand_tracking_android_gpu_small.gif similarity index 100% rename from mediapipe/docs/images/multi_hand_tracking_android_gpu_small.gif rename to docs/images/multi_hand_tracking_android_gpu_small.gif diff --git a/mediapipe/docs/images/multi_hand_tracking_desktop.png b/docs/images/multi_hand_tracking_desktop.png similarity index 100% rename from mediapipe/docs/images/multi_hand_tracking_desktop.png rename to docs/images/multi_hand_tracking_desktop.png diff --git a/mediapipe/docs/images/object_detection_desktop_tensorflow.png b/docs/images/object_detection_desktop_tensorflow.png similarity index 100% rename from mediapipe/docs/images/object_detection_desktop_tensorflow.png rename to docs/images/object_detection_desktop_tensorflow.png diff --git a/mediapipe/docs/images/object_detection_desktop_tflite.png b/docs/images/object_detection_desktop_tflite.png similarity index 100% rename from mediapipe/docs/images/object_detection_desktop_tflite.png rename to docs/images/object_detection_desktop_tflite.png diff --git a/docs/images/objectron_data_annotation.gif b/docs/images/objectron_data_annotation.gif new file mode 100644 index 000000000..6466f7735 Binary files /dev/null and b/docs/images/objectron_data_annotation.gif differ diff --git a/docs/images/objectron_example_results.png b/docs/images/objectron_example_results.png new file mode 100644 index 000000000..977da33cc Binary files /dev/null and b/docs/images/objectron_example_results.png differ diff --git a/docs/images/objectron_network_architecture.png b/docs/images/objectron_network_architecture.png new file mode 100644 index 000000000..2f0b6d9b2 Binary files /dev/null and b/docs/images/objectron_network_architecture.png differ diff --git a/docs/images/objectron_sample_network_results.png b/docs/images/objectron_sample_network_results.png new file mode 100644 index 000000000..e3ae90f8a Binary files /dev/null and b/docs/images/objectron_sample_network_results.png differ diff --git a/docs/images/objectron_synthetic_data_generation.gif b/docs/images/objectron_synthetic_data_generation.gif new file mode 100644 index 000000000..77705cca3 Binary files /dev/null and b/docs/images/objectron_synthetic_data_generation.gif differ diff --git a/docs/images/open_source.png b/docs/images/open_source.png new file mode 100644 index 000000000..f337c8748 Binary files /dev/null and b/docs/images/open_source.png differ diff --git a/docs/images/open_source_small.png b/docs/images/open_source_small.png new file mode 100644 index 000000000..c64ca50d3 Binary files /dev/null and b/docs/images/open_source_small.png differ diff --git a/mediapipe/docs/images/packet_cloner_calculator.png b/docs/images/packet_cloner_calculator.png similarity index 100% rename from mediapipe/docs/images/packet_cloner_calculator.png rename to docs/images/packet_cloner_calculator.png diff --git a/docs/images/ready_to_use.png b/docs/images/ready_to_use.png new file mode 100644 index 000000000..fbccbe830 Binary files /dev/null and b/docs/images/ready_to_use.png differ diff --git a/docs/images/ready_to_use_small.png b/docs/images/ready_to_use_small.png new file mode 100644 index 000000000..5091faaf6 Binary files /dev/null and b/docs/images/ready_to_use_small.png differ diff --git a/mediapipe/docs/images/realtime_face_detection.gif b/docs/images/realtime_face_detection.gif similarity index 100% rename from mediapipe/docs/images/realtime_face_detection.gif rename to docs/images/realtime_face_detection.gif diff --git a/mediapipe/docs/images/side_packet.png b/docs/images/side_packet.png similarity index 100% rename from mediapipe/docs/images/side_packet.png rename to docs/images/side_packet.png diff --git a/mediapipe/docs/images/side_packet_code.png b/docs/images/side_packet_code.png similarity index 100% rename from mediapipe/docs/images/side_packet_code.png rename to docs/images/side_packet_code.png diff --git a/mediapipe/docs/images/special_nodes.png b/docs/images/special_nodes.png similarity index 100% rename from mediapipe/docs/images/special_nodes.png rename to docs/images/special_nodes.png diff --git a/mediapipe/docs/images/special_nodes_code.png b/docs/images/special_nodes_code.png similarity index 100% rename from mediapipe/docs/images/special_nodes_code.png rename to docs/images/special_nodes_code.png diff --git a/mediapipe/docs/images/startup_screen.png b/docs/images/startup_screen.png similarity index 100% rename from mediapipe/docs/images/startup_screen.png rename to docs/images/startup_screen.png diff --git a/mediapipe/docs/images/stream_code.png b/docs/images/stream_code.png similarity index 100% rename from mediapipe/docs/images/stream_code.png rename to docs/images/stream_code.png diff --git a/mediapipe/docs/images/stream_ui.png b/docs/images/stream_ui.png similarity index 100% rename from mediapipe/docs/images/stream_ui.png rename to docs/images/stream_ui.png diff --git a/mediapipe/docs/images/upload_2pbtxt.png b/docs/images/upload_2pbtxt.png similarity index 100% rename from mediapipe/docs/images/upload_2pbtxt.png rename to docs/images/upload_2pbtxt.png diff --git a/mediapipe/docs/images/upload_button.png b/docs/images/upload_button.png similarity index 100% rename from mediapipe/docs/images/upload_button.png rename to docs/images/upload_button.png diff --git a/mediapipe/docs/images/upload_graph_button.png b/docs/images/upload_graph_button.png similarity index 100% rename from mediapipe/docs/images/upload_graph_button.png rename to docs/images/upload_graph_button.png diff --git a/mediapipe/docs/images/visualizer/ios_download_container.png b/docs/images/visualizer/ios_download_container.png similarity index 100% rename from mediapipe/docs/images/visualizer/ios_download_container.png rename to docs/images/visualizer/ios_download_container.png diff --git a/mediapipe/docs/images/visualizer/ios_window_devices.png b/docs/images/visualizer/ios_window_devices.png similarity index 100% rename from mediapipe/docs/images/visualizer/ios_window_devices.png rename to docs/images/visualizer/ios_window_devices.png diff --git a/mediapipe/docs/images/visualizer/viz_chart_view.png b/docs/images/visualizer/viz_chart_view.png similarity index 100% rename from mediapipe/docs/images/visualizer/viz_chart_view.png rename to docs/images/visualizer/viz_chart_view.png diff --git a/mediapipe/docs/images/visualizer/viz_click_upload.png b/docs/images/visualizer/viz_click_upload.png similarity index 100% rename from mediapipe/docs/images/visualizer/viz_click_upload.png rename to docs/images/visualizer/viz_click_upload.png diff --git a/mediapipe/docs/images/visualizer/viz_click_upload_trace_file.png b/docs/images/visualizer/viz_click_upload_trace_file.png similarity index 100% rename from mediapipe/docs/images/visualizer/viz_click_upload_trace_file.png rename to docs/images/visualizer/viz_click_upload_trace_file.png diff --git a/docs/images/visualizer_runner.png b/docs/images/visualizer_runner.png new file mode 100644 index 000000000..5224a0949 Binary files /dev/null and b/docs/images/visualizer_runner.png differ diff --git a/mediapipe/docs/images/web_effect.gif b/docs/images/web_effect.gif similarity index 100% rename from mediapipe/docs/images/web_effect.gif rename to docs/images/web_effect.gif diff --git a/mediapipe/docs/images/web_segmentation.gif b/docs/images/web_segmentation.gif similarity index 100% rename from mediapipe/docs/images/web_segmentation.gif rename to docs/images/web_segmentation.gif diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 000000000..95a54cc1a --- /dev/null +++ b/docs/index.md @@ -0,0 +1,141 @@ +--- +layout: default +title: Home +nav_order: 1 +--- + +![MediaPipe](images/mediapipe_small.png) + +-------------------------------------------------------------------------------- + +## Cross-platform ML solutions made simple + +[MediaPipe](https://google.github.io/mediapipe/) is the simplest way for researchers +and developers to build world-class ML solutions and applications for mobile, +desktop/cloud, web and IoT devices. + +![accelerated.png](images/accelerated_small.png) | ![cross_platform.png](images/cross_platform_small.png) +:------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------: +***End-to-End acceleration***: *built-in fast ML inference and processing accelerated even on common hardware* | ***Build one, deploy anywhere***: *Unified solution works across Android, iOS, desktop/cloud, web and IoT* +![ready_to_use.png](images/ready_to_use_small.png) | ![open_source.png](images/open_source_small.png) +***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable* + +## ML solutions in MediaPipe + +Face Detection | Face Mesh | Hand | Hair Segmentation +:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :---------------: +[![face_detection](images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![hand](images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hand) | [![hair_segmentation](images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation) + +Object Detection | Box Tracking | Objectron | KNIFT +:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---: +[![object_detection](images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![objectron](images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift) + + + + +[]() | Android | iOS | Desktop | Web | Coral +:---------------------------------------------------------------------------- | :-----: | :-: | :-----: | :-: | :---: +[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | ✅ | ✅ +[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | | +[Hand](https://google.github.io/mediapipe/solutions/hand) | ✅ | ✅ | ✅ | ✅ | +[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | ✅ | +[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | ✅ +[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | | +[Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | | +[KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | | +[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | | +[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | | +[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m) | | | ✅ | | + +## MediaPipe on the Web + +MediaPipe on the Web is an effort to run the same ML solutions built for mobile +and desktop also in web browsers. The official API is under construction, but +the core technology has been proven effective. Please see +[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html) +in Google Developers Blog for details. + +You can use the following links to load a demo in the MediaPipe Visualizer, and +over there click the "Runner" icon in the top bar like shown below. The demos +use your webcam video as input, which is processed all locally in real-time and +never leaves your device. + +![visualizer_runner](images/visualizer_runner.png) + +* [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection) +* [MediaPipe Hand](https://viz.mediapipe.dev/demo/hand_tracking) +* [MediaPipe Hand (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection) +* [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation) + +## Getting started + +Learn how to [install](https://google.github.io/mediapipe/getting_started/install) +MediaPipe and +[build example applications](https://google.github.io/mediapipe/getting_started/building_examples), +and start exploring our ready-to-use +[solutions](https://google.github.io/mediapipe/solutions/solutions) that you can +further extend and customize. + +The source code is hosted in the +[MediaPipe Github repository](https://github.com/google/mediapipe), and you can +run code search using +[Google Open Source Code Search](https://cs.opensource.google/mediapipe/mediapipe). + +## Publications + +* [MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html) + in Google Developers Blog +* [Alfred Camera: Smart camera features using MediaPipe](https://developers.googleblog.com/2020/03/alfred-camera-smart-camera-features-using-mediapipe.html) + in Google Developers Blog +* [Real-Time 3D Object Detection on Mobile Devices with MediaPipe](https://ai.googleblog.com/2020/03/real-time-3d-object-detection-on-mobile.html) + in Google AI Blog +* [AutoFlip: An Open Source Framework for Intelligent Video Reframing](https://ai.googleblog.com/2020/02/autoflip-open-source-framework-for.html) + in Google AI Blog +* [MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html) + in Google Developers Blog +* [Object Detection and Tracking using MediaPipe](https://developers.googleblog.com/2019/12/object-detection-and-tracking-using-mediapipe.html) + in Google Developers Blog +* [On-Device, Real-Time Hand Tracking with MediaPipe](https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html) + in Google AI Blog +* [MediaPipe: A Framework for Building Perception Pipelines](https://arxiv.org/abs/1906.08172) + +## Videos + +* [YouTube Channel](https://www.youtube.com/channel/UCObqmpuSMx-usADtL_qdMAw) + +## Events + +* [MediaPipe Seattle Meetup, Google Building Waterside, 13 Feb 2020](https://mediapipe.page.link/seattle2020) +* [AI Nextcon 2020, 12-16 Feb 2020, Seattle](http://aisea20.xnextcon.com/) +* [MediaPipe Madrid Meetup, 16 Dec 2019](https://www.meetup.com/Madrid-AI-Developers-Group/events/266329088/) +* [MediaPipe London Meetup, Google 123 Building, 12 Dec 2019](https://www.meetup.com/London-AI-Tech-Talk/events/266329038) +* [ML Conference, Berlin, 11 Dec 2019](https://mlconference.ai/machine-learning-advanced-development/mediapipe-building-real-time-cross-platform-mobile-web-edge-desktop-video-audio-ml-pipelines/) +* [MediaPipe Berlin Meetup, Google Berlin, 11 Dec 2019](https://www.meetup.com/Berlin-AI-Tech-Talk/events/266328794/) +* [The 3rd Workshop on YouTube-8M Large Scale Video Understanding Workshop, + Seoul, Korea ICCV + 2019](https://research.google.com/youtube8m/workshop2019/index.html) +* [AI DevWorld 2019, 10 Oct 2019, San Jose, CA](https://aidevworld.com) +* [Google Industry Workshop at ICIP 2019, 24 Sept 2019, Taipei, Taiwan](http://2019.ieeeicip.org/?action=page4&id=14#Google) + ([presentation](https://docs.google.com/presentation/d/e/2PACX-1vRIBBbO_LO9v2YmvbHHEt1cwyqH6EjDxiILjuT0foXy1E7g6uyh4CesB2DkkEwlRDO9_lWfuKMZx98T/pub?start=false&loop=false&delayms=3000&slide=id.g556cc1a659_0_5)) +* [Open sourced at CVPR 2019, 17~20 June, Long Beach, CA](https://sites.google.com/corp/view/perception-cv4arvr/mediapipe) + +## Community + +* [Awesome MediaPipe](https://mediapipe.org) - A curated list of awesome + MediaPipe related frameworks, libraries and software +* [Slack community](https://mediapipe.slack.com) for MediaPipe users +* [Discuss](https://groups.google.com/forum/#!forum/mediapipe) - General + community discussion around MediaPipe + +## Alpha Disclaimer + +MediaPipe is currently in alpha at v0.7. We may be still making breaking API +changes and expect to get to stable APIs by v1.0. + +## Contributing + +We welcome contributions. Please follow these +[guidelines](https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md). + +We use GitHub issues for tracking requests and bugs. Please post questions to +the MediaPipe Stack Overflow with a `mediapipe` tag. diff --git a/mediapipe/docs/index.rst b/docs/index.rst similarity index 84% rename from mediapipe/docs/index.rst rename to docs/index.rst index 870d02b2b..d112c3cda 100644 --- a/mediapipe/docs/index.rst +++ b/docs/index.rst @@ -41,18 +41,20 @@ User Documentation .. toctree:: :maxdepth: 3 - install - concepts - calculator + getting_started/install Examples - visualizer - measure_performance - how_to_questions - troubleshooting - help - framework_concepts - gpu - scheduling_sync + tools/visualizer + tools/tracing_and_profiling + tools/performance_benchmarking + getting_started/help + getting_started/faq + getting_started/troubleshooting + framework_concepts/framework_concepts + framework_concepts/calculators + framework_concepts/graphs + framework_concepts/packets + framework_concepts/synchronization + framework_concepts/gpu license Indices and tables diff --git a/mediapipe/docs/license.md b/docs/license.md similarity index 99% rename from mediapipe/docs/license.md rename to docs/license.md index 9a98f8910..9787bc748 100644 --- a/mediapipe/docs/license.md +++ b/docs/license.md @@ -1,3 +1,7 @@ +--- +nav_exclude: true +--- + License =============== Copyright 2019 The MediaPipe Authors. All rights reserved. diff --git a/docs/solutions/autoflip.md b/docs/solutions/autoflip.md new file mode 100644 index 000000000..f78b4ae95 --- /dev/null +++ b/docs/solutions/autoflip.md @@ -0,0 +1,356 @@ +--- +layout: default +title: AutoFlip (Saliency-aware Video Cropping) +parent: Solutions +nav_order: 9 +--- + +# AutoFlip: Saliency-aware Video Cropping +{: .no_toc } + +1. TOC +{:toc} +--- + +## Overview + +AutoFlip is an automatic video cropping pipeline built on top of MediaPipe. This +example focuses on demonstrating how to use AutoFlip to convert an input video +to arbitrary aspect ratios. + +For overall context on AutoFlip, please read this +[Google AI Blog](https://ai.googleblog.com/2020/02/autoflip-open-source-framework-for.html). + +![graph is_required](../images/autoflip_edited_example.gif) + +## Building + +Run the following command to build the AutoFlip pipeline: + +Note: AutoFlip currently only works with OpenCV 3. Please verify your OpenCV +version beforehand. + +```bash +bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/autoflip:run_autoflip +``` + +## Running + +```bash +GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/autoflip/run_autoflip \ + --calculator_graph_config_file=mediapipe/examples/desktop/autoflip/autoflip_graph.pbtxt \ + --input_side_packets=input_video_path=/absolute/path/to/the/local/video/file,output_video_path=/absolute/path/to/save/the/output/video/file,aspect_ratio=1:1 +``` + +Use the `aspect_ratio` flag to provide the output aspect ratio. The format +should be `width:height`, where the `width` and `height` are two positive +integers. AutoFlip supports both landscape-to-portrait and portrait-to-landscape +conversions. The pipeline internally compares the target aspect ratio against +the original one, and determines the correct conversion automatically. + +We have put a couple test videos under this +[Google Drive folder](https://drive.google.com/corp/drive/u/0/folders/1KK9LV--Ey0UEVpxssVLhVl7dypgJSQgk). +You could download the videos into your local file system, then modify the +command above accordingly to run AutoFlip against the videos. + +## MediaPipe Graph + +![graph visualization](../images/autoflip_graph.png) + +To visualize the graph as shown above, copy the text specification of the graph +below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev). + +```bash +# Autoflip graph that only renders the final cropped video. For use with +# end user applications. +max_queue_size: -1 + +# VIDEO_PREP: Decodes an input video file into images and a video header. +node { + calculator: "OpenCvVideoDecoderCalculator" + input_side_packet: "INPUT_FILE_PATH:input_video_path" + output_stream: "VIDEO:video_raw" + output_stream: "VIDEO_PRESTREAM:video_header" + output_side_packet: "SAVED_AUDIO_PATH:audio_path" +} + +# VIDEO_PREP: Scale the input video before feature extraction. +node { + calculator: "ScaleImageCalculator" + input_stream: "FRAMES:video_raw" + input_stream: "VIDEO_HEADER:video_header" + output_stream: "FRAMES:video_frames_scaled" + node_options: { + [type.googleapis.com/mediapipe.ScaleImageCalculatorOptions]: { + preserve_aspect_ratio: true + output_format: SRGB + target_width: 480 + algorithm: DEFAULT_WITHOUT_UPSCALE + } + } +} + +# VIDEO_PREP: Create a low frame rate stream for feature extraction. +node { + calculator: "PacketThinnerCalculator" + input_stream: "video_frames_scaled" + output_stream: "video_frames_scaled_downsampled" + node_options: { + [type.googleapis.com/mediapipe.PacketThinnerCalculatorOptions]: { + thinner_type: ASYNC + period: 200000 + } + } +} + +# DETECTION: find borders around the video and major background color. +node { + calculator: "BorderDetectionCalculator" + input_stream: "VIDEO:video_raw" + output_stream: "DETECTED_BORDERS:borders" +} + +# DETECTION: find shot/scene boundaries on the full frame rate stream. +node { + calculator: "ShotBoundaryCalculator" + input_stream: "VIDEO:video_frames_scaled" + output_stream: "IS_SHOT_CHANGE:shot_change" + options { + [type.googleapis.com/mediapipe.autoflip.ShotBoundaryCalculatorOptions] { + min_shot_span: 0.2 + min_motion: 0.3 + window_size: 15 + min_shot_measure: 10 + min_motion_with_shot_measure: 0.05 + } + } +} + +# DETECTION: find faces on the down sampled stream +node { + calculator: "AutoFlipFaceDetectionSubgraph" + input_stream: "VIDEO:video_frames_scaled_downsampled" + output_stream: "DETECTIONS:face_detections" +} +node { + calculator: "FaceToRegionCalculator" + input_stream: "VIDEO:video_frames_scaled_downsampled" + input_stream: "FACES:face_detections" + output_stream: "REGIONS:face_regions" +} + +# DETECTION: find objects on the down sampled stream +node { + calculator: "AutoFlipObjectDetectionSubgraph" + input_stream: "VIDEO:video_frames_scaled_downsampled" + output_stream: "DETECTIONS:object_detections" +} +node { + calculator: "LocalizationToRegionCalculator" + input_stream: "DETECTIONS:object_detections" + output_stream: "REGIONS:object_regions" + options { + [type.googleapis.com/mediapipe.autoflip.LocalizationToRegionCalculatorOptions] { + output_all_signals: true + } + } +} + +# SIGNAL FUSION: Combine detections (with weights) on each frame +node { + calculator: "SignalFusingCalculator" + input_stream: "shot_change" + input_stream: "face_regions" + input_stream: "object_regions" + output_stream: "salient_regions" + options { + [type.googleapis.com/mediapipe.autoflip.SignalFusingCalculatorOptions] { + signal_settings { + type { standard: FACE_CORE_LANDMARKS } + min_score: 0.85 + max_score: 0.9 + is_required: false + } + signal_settings { + type { standard: FACE_ALL_LANDMARKS } + min_score: 0.8 + max_score: 0.85 + is_required: false + } + signal_settings { + type { standard: FACE_FULL } + min_score: 0.8 + max_score: 0.85 + is_required: false + } + signal_settings { + type: { standard: HUMAN } + min_score: 0.75 + max_score: 0.8 + is_required: false + } + signal_settings { + type: { standard: PET } + min_score: 0.7 + max_score: 0.75 + is_required: false + } + signal_settings { + type: { standard: CAR } + min_score: 0.7 + max_score: 0.75 + is_required: false + } + signal_settings { + type: { standard: OBJECT } + min_score: 0.1 + max_score: 0.2 + is_required: false + } + } + } +} + +# CROPPING: make decisions about how to crop each frame. +node { + calculator: "SceneCroppingCalculator" + input_side_packet: "EXTERNAL_ASPECT_RATIO:aspect_ratio" + input_stream: "VIDEO_FRAMES:video_raw" + input_stream: "KEY_FRAMES:video_frames_scaled_downsampled" + input_stream: "DETECTION_FEATURES:salient_regions" + input_stream: "STATIC_FEATURES:borders" + input_stream: "SHOT_BOUNDARIES:shot_change" + output_stream: "CROPPED_FRAMES:cropped_frames" + node_options: { + [type.googleapis.com/mediapipe.autoflip.SceneCroppingCalculatorOptions]: { + max_scene_size: 600 + key_frame_crop_options: { + score_aggregation_type: CONSTANT + } + scene_camera_motion_analyzer_options: { + motion_stabilization_threshold_percent: 0.5 + salient_point_bound: 0.499 + } + padding_parameters: { + blur_cv_size: 200 + overlay_opacity: 0.6 + } + target_size_type: MAXIMIZE_TARGET_DIMENSION + } + } +} + +# ENCODING(required): encode the video stream for the final cropped output. +node { + calculator: "VideoPreStreamCalculator" + # Fetch frame format and dimension from input frames. + input_stream: "FRAME:cropped_frames" + # Copying frame rate and duration from original video. + input_stream: "VIDEO_PRESTREAM:video_header" + output_stream: "output_frames_video_header" +} + +node { + calculator: "OpenCvVideoEncoderCalculator" + input_stream: "VIDEO:cropped_frames" + input_stream: "VIDEO_PRESTREAM:output_frames_video_header" + input_side_packet: "OUTPUT_FILE_PATH:output_video_path" + input_side_packet: "AUDIO_FILE_PATH:audio_path" + node_options: { + [type.googleapis.com/mediapipe.OpenCvVideoEncoderCalculatorOptions]: { + codec: "avc1" + video_format: "mp4" + } + } +} +``` + +## Advanced Parameters + +### Required vs. Best-Effort Saliency Features + +AutoFlip allows users to implement and specify custom features to be used in the +camera trajectory computation. If the user would like to detect and preserve +scenes of lions in a wildlife protection video, for example, they could +implement and add a feature detection calculator for lions into the pipeline. +Refer to `AutoFlipFaceDetectionSubgraph` and `FaceToRegionCalculator`, or +`AutoFlipObjectDetectionSubgraph` and `LocalizationToRegionCalculator` for +examples of how to create new feature detection calculators. + +After adding different feature signals into the graph, use the +`SignalFusingCalculator` node to specify types and weights for different feature +signals. For example, in the graph above, we specified a `face_region` and an +`object_region` input streams, to represent face signals and agnostic object +signals, respectively. + +The larger the weight, the more important the features will be considered when +AutoFlip computes the camera trajectory. Use the `is_required` flag to mark a +feature as a hard constraint, in which case the computed camera trajectory will +try best to cover these feature types in the cropped videos. If for some reason +the required features cannot be all covered (for example, when they are too +spread out in the video), AutoFlip will apply a padding effect to cover as much +salient content as possible. See an illustration below. + +![graph is_required](../images/autoflip_is_required.gif) + +### Stable vs Tracking Camera Motion + +AutoFlip makes a decision on each scene whether to have the cropped viewpoint +follow an object or if the crop should remain stable (centered on detected +objects). The parameter `motion_stabilization_threshold_percent` value is used +to make the decision to track action or keep the camera stable. If, over the +duration of the scene, all detected focus objects remain within this ratio of +the frame (e.g. 0.5 = 50% or 1920 * .5 = 960 pixels on 1080p video) then the +camera is held steady. Otherwise the camera tracks activity within the frame. + +### Snap To Center + +For some scenes the camera viewpoint will remain stable at the center of +activity (see `motion_stabilization_threshold_percent` setting). In this case, +if the determined best stable viewpoint is within +`snap_center_max_distance_percent` of the frame's center the camera will be +shifted to be locked to the center of the frame. This setting is useful for +videos where the camera operator did a good job already centering content or if +titles and logos are expected to appear in the center of the frame. It may be +less useful on raw content where objects are not already well positioned on +screen. + +### Visualization to Facilitate Debugging + +`SceneCroppingCalculator` provides two extra output streams +`KEY_FRAME_CROP_REGION_VIZ_FRAMES` and `SALIENT_POINT_FRAME_VIZ_FRAMES` to +visualize the cropping window as well as salient points detected on each frame. +You could modify the `SceneCroppingCalculator` node like below to enable these +two output streams. + +```bash +node { + calculator: "SceneCroppingCalculator" + input_side_packet: "EXTERNAL_ASPECT_RATIO:aspect_ratio" + input_stream: "VIDEO_FRAMES:video_raw" + input_stream: "KEY_FRAMES:video_frames_scaled_downsampled" + input_stream: "DETECTION_FEATURES:salient_regions" + input_stream: "STATIC_FEATURES:borders" + input_stream: "SHOT_BOUNDARIES:shot_change" + output_stream: "CROPPED_FRAMES:cropped_frames" + output_stream: "KEY_FRAME_CROP_REGION_VIZ_FRAMES:key_frame_crop_viz_frames" + output_stream: "SALIENT_POINT_FRAME_VIZ_FRAMES:salient_point_viz_frames" + node_options: { + [type.googleapis.com/mediapipe.autoflip.SceneCroppingCalculatorOptions]: { + max_scene_size: 600 + key_frame_crop_options: { + score_aggregation_type: CONSTANT + } + scene_camera_motion_analyzer_options: { + motion_stabilization_threshold_percent: 0.5 + salient_point_bound: 0.499 + } + padding_parameters: { + blur_cv_size: 200 + overlay_opacity: 0.6 + } + target_size_type: MAXIMIZE_TARGET_DIMENSION + } + } +} +``` diff --git a/docs/solutions/box_tracking.md b/docs/solutions/box_tracking.md new file mode 100644 index 000000000..84da8565d --- /dev/null +++ b/docs/solutions/box_tracking.md @@ -0,0 +1,146 @@ +--- +layout: default +title: Box Tracking +parent: Solutions +nav_order: 6 +--- + +# MediaPipe Box Tracking +{: .no_toc } + +1. TOC +{:toc} +--- + +## Overview + +MediaPipe Box Tracking has been powering real-time tracking in +[Motion Stills](https://ai.googleblog.com/2016/12/get-moving-with-new-motion-stills.html), +[YouTube's privacy blur](https://youtube-creators.googleblog.com/2016/02/blur-moving-objects-in-your-video-with.html), +and [Google Lens](https://lens.google.com/) for several years, leveraging +classic computer vision approaches. + +The box tracking solution consumes image frames from a video or camera stream, +and starting box positions with timestamps, indicating 2D regions of interest to +track, and computes the tracked box positions for each frame. In this specific +use case, the starting box positions come from object detection, but the +starting position can also be provided manually by the user or another system. +Our solution consists of three main components: a motion analysis component, a +flow packager component, and a box tracking component. Each component is +encapsulated as a MediaPipe calculator, and the box tracking solution as a whole +is represented as a MediaPipe +[subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/subgraphs/box_tracking_gpu.pbtxt). + +Note: To visualize a graph, copy the graph and paste it into +[MediaPipe Visualizer](https://viz.mediapipe.dev/). + +In the +[box tracking subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/subgraphs/box_tracking_gpu.pbtxt), +the MotionAnalysis calculator extracts features (e.g. high-gradient corners) +across the image, tracks those features over time, classifies them into +foreground and background features, and estimates both local motion vectors and +the global motion model. The FlowPackager calculator packs the estimated motion +metadata into an efficient format. The BoxTracker calculator takes this motion +metadata from the FlowPackager calculator and the position of starting boxes, +and tracks the boxes over time. Using solely the motion data (without the need +for the RGB frames) produced by the MotionAnalysis calculator, the BoxTracker +calculator tracks individual objects or regions while discriminating from +others. Please see +[Object Detection and Tracking using MediaPipe](https://developers.googleblog.com/2019/12/object-detection-and-tracking-using-mediapipe.html) +in Google Developers Blog for more details. + +An advantage of our architecture is that by separating motion analysis into a +dedicated MediaPipe calculator and tracking features over the whole image, we +enable great flexibility and constant computation independent of the number of +regions tracked! By not having to rely on the RGB frames during tracking, our +tracking solution provides the flexibility to cache the metadata across a batch +of frame. Caching enables tracking of regions both backwards and forwards in +time; or even sync directly to a specified timestamp for tracking with random +access. + +## Object Detection and Tracking + +MediaPipe Box Tracking can be paired with ML inference, resulting in valuable +and efficient pipelines. For instance, box tracking can be paired with ML-based +object detection to create an object detection and tracking pipeline. With +tracking, this pipeline offers several advantages over running detection per +frame (e.g., [MediaPipe Object Detection](./object_detection.md)): + +* It provides instance based tracking, i.e. the object ID is maintained across + frames. +* Detection does not have to run every frame. This enables running heavier + detection models that are more accurate while keeping the pipeline + lightweight and real-time on mobile devices. +* Object localization is temporally consistent with the help of tracking, + meaning less jitter is observable across frames. + +![object_tracking_android_gpu.gif](../images/mobile/object_tracking_android_gpu.gif) | +:----------------------------------------------------------------------------------: | +*Fig 1. Box tracking paired with ML-based object detection.* | + +The object detection and tracking pipeline can be implemented as a MediaPipe +[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/object_detection_tracking_mobile_gpu.pbtxt), +which internally utilizes an +[object detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/subgraphs/object_detection_gpu.pbtxt), +an +[object tracking subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/subgraphs/object_tracking_gpu.pbtxt), +and a +[renderer subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/subgraphs/renderer_gpu.pbtxt). + +In general, the object detection subgraph (which performs ML model inference +internally) runs only upon request, e.g. at an arbitrary frame rate or triggered +by specific signals. More specifically, in this particular +[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/object_detection_tracking_mobile_gpu.pbtxt) +a PacketResampler calculator temporally subsamples the incoming video frames to +0.5 fps before they are passed into the object detection subgraph. This frame +rate can be configured differently as an option in PacketResampler. + +The object tracking subgraph runs in real-time on every incoming frame to track +the detected objects. It expands the +[box tracking subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/subgraphs/box_tracking_gpu.pbtxt) +with additional functionality: when new detections arrive it uses IoU +(Intersection over Union) to associate the current tracked objects/boxes with +new detections to remove obsolete or duplicated boxes. + +## Example Apps + +Please first see general instructions for +[Android](../getting_started/building_examples.md#android), [iOS](../getting_started/building_examples.md#ios) +and [desktop](../getting_started/building_examples.md#desktop) on how to build MediaPipe +examples. + +Note: To visualize a graph, copy the graph and paste it into +[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how +to visualize its associated subgraphs, please see +[visualizer documentation](../visualizer.md). + +### Mobile + +Note: Object detection is using TensorFlow Lite on GPU while tracking is on CPU. + +* Graph: + [`mediapipe/graphs/tracking/object_detection_tracking_mobile_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/object_detection_tracking_mobile_gpu.pbtxt) +* Android target: + [(or download prebuilt ARM64 APK)](https://drive.google.com/open?id=1UXL9jX4Wpp34TsiVogugV3J3T9_C5UK-) + [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/objecttrackinggpu:objecttrackinggpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objecttrackinggpu/BUILD) +* iOS target: Not available + +### Desktop + +* Running on CPU (both for object detection using TensorFlow Lite and + tracking): + * Graph: + [`mediapipe/graphs/tracking/object_detection_tracking_desktop_live.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/object_detection_tracking_desktop_live.pbtxt) + * Target: + [`mediapipe/examples/desktop/object_tracking:object_tracking_cpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/object_tracking/BUILD) +* Running on GPU: Not available + +## Resources + +* Google Developers Blog: + [Object Detection and Tracking using MediaPipe](https://developers.googleblog.com/2019/12/object-detection-and-tracking-using-mediapipe.html) +* Google AI Blog: + [Get moving with the new Motion Stills](https://ai.googleblog.com/2016/12/get-moving-with-new-motion-stills.html) +* YouTube Creator Blog: [Blur moving objects in your video with the new Custom + blurring tool on + YouTube](https://youtube-creators.googleblog.com/2016/02/blur-moving-objects-in-your-video-with.html) diff --git a/docs/solutions/face_detection.md b/docs/solutions/face_detection.md new file mode 100644 index 000000000..a8e844df4 --- /dev/null +++ b/docs/solutions/face_detection.md @@ -0,0 +1,110 @@ +--- +layout: default +title: Face Detection +parent: Solutions +nav_order: 1 +--- + +# MediaPipe Face Detection +{: .no_toc } + +1. TOC +{:toc} +--- + +## Overview + +MediaPipe Face Detection is an ultrafast face detection solution that comes with +6 landmarks and multi-face support. It is based on +[BlazeFace](https://arxiv.org/abs/1907.05047), a lightweight and well-performing +face detector tailored for mobile GPU inference. The detector's super-realtime +performance enables it to be applied to any live viewfinder experience that +requires an accurate facial region of interest as an input for other +task-specific models, such as 3D facial keypoint or geometry estimation (e.g., +[MediaPipe Face Mesh](./face_mesh.md)), facial features or expression +classification, and face region segmentation. BlazeFace uses a lightweight +feature extraction network inspired by, but distinct from +[MobileNetV1/V2](https://ai.googleblog.com/2018/04/mobilenetv2-next-generation-of-on.html), +a GPU-friendly anchor scheme modified from +[Single Shot MultiBox Detector (SSD)](https://arxiv.org/abs/1512.02325), and an +improved tie resolution strategy alternative to non-maximum suppression. For +more information about BlazeFace, please see the [Resources](#resources) +section. + +![face_detection_android_gpu.gif](../images/mobile/face_detection_android_gpu.gif) + +## Example Apps + +Please first see general instructions for +[Android](../getting_started/building_examples.md#android), [iOS](../getting_started/building_examples.md#ios) +and [desktop](../getting_started/building_examples.md#desktop) on how to build MediaPipe +examples. + +Note: To visualize a graph, copy the graph and paste it into +[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how +to visualize its associated subgraphs, please see +[visualizer documentation](../visualizer.md). + +### Mobile + +#### GPU Pipeline + +* Graph: + [`mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt) +* Android target: + [(or download prebuilt ARM64 APK)](https://drive.google.com/open?id=1DZTCy1gp238kkMnu4fUkwI3IrF77Mhy5) + [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu:facedetectiongpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/BUILD) +* iOS target: + [`mediapipe/examples/ios/facedetectiongpu:FaceDetectionGpuApp`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facedetectiongpu/BUILD) + +#### CPU Pipeline + +This is very similar to the [GPU pipeline](#gpu-pipeline) except that at the +beginning and the end of the pipeline it performs GPU-to-CPU and CPU-to-GPU +image transfer respectively. As a result, the rest of graph, which shares the +same configuration as the GPU pipeline, runs entirely on CPU. + +* Graph: + [`mediapipe/graphs/face_detection/face_detection_mobile_cpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_cpu.pbtxt) +* Android target: + [(or download prebuilt ARM64 APK)](https://drive.google.com/open?id=1npiZY47jbO5m2YaL63o5QoCQs40JC6C7) + [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu:facedetectioncpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectioncpu/BUILD) +* iOS target: + [`mediapipe/examples/ios/facedetectioncpu:FaceDetectionCpuApp`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facedetectioncpu/BUILD) + +### Desktop + +* Running on CPU: + * Graph: + [`mediapipe/graphs/face_detection/face_detection_desktop_live.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_desktop_live.pbtxt) + * Target: + [`mediapipe/examples/desktop/face_detection:face_detection_cpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/face_detection/BUILD) +* Running on GPU + * Graph: + [`mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_detection/face_detection_mobile_gpu.pbtxt) + * Target: + [`mediapipe/examples/desktop/face_detection:face_detection_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/face_detection/BUILD) + +### Web + +Please refer to [these instructions](../index.md#mediapipe-on-the-web). + +### Coral + +Please refer to +[these instructions](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/README.md) +to cross-compile and run MediaPipe examples on the +[Coral Dev Board](https://coral.ai/products/dev-board). + +## Resources + +* Paper: + [BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs](https://arxiv.org/abs/1907.05047) + ([presentation](https://docs.google.com/presentation/d/1YCtASfnYyZtH-41QvnW5iZxELFnf0MF-pPWSLGj8yjQ/present?slide=id.g5bc8aeffdd_1_0)) + ([poster](https://drive.google.com/file/d/1u6aB6wxDY7X2TmeUUKgFydulNtXkb3pu/view)) +* For front-facing/selfie camera: + [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_front.tflite), + [TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite) +* For back-facing camera: + [TFLite model ](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_back.tflite) +* [Model card](https://drive.google.com/file/d/1f39lSzU5Oq-j_OXgS67KfN5wNsoeAZ4V/view) diff --git a/docs/solutions/face_mesh.md b/docs/solutions/face_mesh.md new file mode 100644 index 000000000..17b7b9d16 --- /dev/null +++ b/docs/solutions/face_mesh.md @@ -0,0 +1,160 @@ +--- +layout: default +title: Face Mesh +parent: Solutions +nav_order: 2 +--- + +# MediaPipe Face Mesh +{: .no_toc } + +1. TOC +{:toc} +--- + +## Overview + +MediaPipe Face Mesh is a face geometry solution that estimates 468 3D face +landmarks in real-time even on mobile devices. It employs machine learning (ML) +to infer the 3D surface geometry, requiring only a single camera input without +the need for a dedicated depth sensor. Utilizing lightweight model architectures +together with GPU acceleration throughout the pipeline, the solution delivers +real-time performance critical for live experiences. The core of the solution is +the same as what powers +[YouTube Stories](https://youtube-creators.googleblog.com/2018/11/introducing-more-ways-to-share-your.html)' +creator effects, the +[Augmented Faces API in ARCore](https://developers.google.com/ar/develop/java/augmented-faces/) +and the +[ML Kit Face Contour Detection API](https://firebase.google.com/docs/ml-kit/face-detection-concepts#contours). + +![face_mesh_ar_effects.gif](../images/face_mesh_ar_effects.gif) | +:-------------------------------------------------------------: | +*Fig 1. AR effects utilizing facial surface geometry.* | + +## ML Pipeline + +Our ML pipeline consists of two real-time deep neural network models that work +together: A detector that operates on the full image and computes face locations +and a 3D face landmark model that operates on those locations and predicts the +approximate surface geometry via regression. Having the face accurately cropped +drastically reduces the need for common data augmentations like affine +transformations consisting of rotations, translation and scale changes. Instead +it allows the network to dedicate most of its capacity towards coordinate +prediction accuracy. In addition, in our pipeline the crops can also be +generated based on the face landmarks identified in the previous frame, and only +when the landmark model could no longer identify face presence is the face +detector invoked to relocalize the face. This strategy is similar to that +employed in our [MediaPipe Hand](./hand.md) solution, which uses a palm detector +together with a hand landmark model. + +The pipeline is implemented as a MediaPipe +[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_mesh/face_mesh_mobile.pbtxt) +that uses a +[face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt) +from the +[face landmark module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark), +and renders using a dedicated +[face renderer subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_mesh/subgraphs/face_renderer_gpu.pbtxt). +The +[face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt) +internally uses a +[face_detection_subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front_gpu.pbtxt) +from the +[face detection module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection). + +Note: To visualize a graph, copy the graph and paste it into +[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how +to visualize its associated subgraphs, please see +[visualizer documentation](../visualizer.md). + +## Models + +### Face Detection Model + +The face detector is the same [BlazeFace](https://arxiv.org/abs/1907.05047) +model used in [MediaPipe Face Detection](./face_detection.md). Please refer to +[MediaPipe Face Detection](./face_detection.md) for details. + +### Face Landmark Model + +For 3D face landmarks we employed transfer learning and trained a network with +several objectives: the network simultaneously predicts 3D landmark coordinates +on synthetic rendered data and 2D semantic contours on annotated real-world +data. The resulting network provided us with reasonable 3D landmark predictions +not just on synthetic but also on real-world data. + +The 3D landmark network receives as input a cropped video frame without +additional depth input. The model outputs the positions of the 3D points, as +well as the probability of a face being present and reasonably aligned in the +input. A common alternative approach is to predict a 2D heatmap for each +landmark, but it is not amenable to depth prediction and has high computational +costs for so many points. We further improve the accuracy and robustness of our +model by iteratively bootstrapping and refining predictions. That way we can +grow our dataset to increasingly challenging cases, such as grimaces, oblique +angle and occlusions. + +You can find more information about the face landmark model in this +[paper](https://arxiv.org/abs/1907.06724). + +![face_mesh_android_gpu.gif](../images/mobile/face_mesh_android_gpu.gif) | +:------------------------------------------------------------------------: | +*Fig 2. Output of MediaPipe Face Mesh: the red box indicates the cropped area as input to the landmark model, the red dots represent the 468 landmarks in 3D, and the green lines connecting landmarks illustrate the contours around the eyes, eyebrows, lips and the entire face.* | + +## Example Apps + +Please first see general instructions for +[Android](../getting_started/building_examples.md#android), [iOS](../getting_started/building_examples.md#ios) and +[desktop](../getting_started/building_examples.md#desktop) on how to build MediaPipe examples. + +Note: To visualize a graph, copy the graph and paste it into +[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how +to visualize its associated subgraphs, please see +[visualizer documentation](../visualizer.md). + +### Mobile + +* Graph: + [`mediapipe/graphs/face_mesh/face_mesh_mobile.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_mesh/face_mesh_mobile.pbtxt) +* Android target: + [(or download prebuilt ARM64 APK)](https://drive.google.com/open?id=1pUmd7CXCL_onYMbsZo5p91cH0oNnR4gi) + [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu:facemeshgpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu/BUILD) +* iOS target: + [`mediapipe/examples/ios/facemeshgpu:FaceMeshGpuApp`](http:/mediapipe/examples/ios/facemeshgpu/BUILD) + +Tip: Maximum number of faces to detect/process is set to 1 by default. To change +it, for Android modify `NUM_FACES` in +[MainActivity.java](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facemeshgpu/MainActivity.java), +and for iOS modify `kNumFaces` in +[ViewController.mm](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/facemeshgpu/ViewController.mm). + +### Desktop + +* Running on CPU + * Graph: + [`mediapipe/graphs/face_mesh/face_mesh_desktop_live.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_mesh/face_mesh_desktop_live.pbtxt) + * Target: + [`mediapipe/examples/desktop/face_mesh:face_mesh_cpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/face_mesh/BUILD) +* Running on GPU + * Graph: + [`mediapipe/graphs/face_mesh/face_mesh_desktop_live_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/face_mesh/face_mesh_desktop_live_gpu.pbtxt) + * Target: + [`mediapipe/examples/desktop/face_mesh:face_mesh_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/face_mesh/BUILD) + +Tip: Maximum number of faces to detect/process is set to 1 by default. To change +it, in the graph file modify the option of `ConstantSidePacketCalculator`. + +## Resources + +* Google AI Blog: + [Real-Time AR Self-Expression with Machine Learning](https://ai.googleblog.com/2019/03/real-time-ar-self-expression-with.html) +* TensorFlow Blog: + [Face and hand tracking in the browser with MediaPipe and TensorFlow.js](https://blog.tensorflow.org/2020/03/face-and-hand-tracking-in-browser-with-mediapipe-and-tensorflowjs.html) +* Paper: + [Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs](https://arxiv.org/abs/1907.06724) + ([poster](https://docs.google.com/presentation/d/1-LWwOMO9TzEVdrZ1CS1ndJzciRHfYDJfbSxH_ke_JRg/present?slide=id.g5986dd4b4c_4_212)) +* Face detection model: + [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_detection_front.tflite) +* Face landmark mode: + [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/face_landmark.tflite), + [TF.js model](https://tfhub.dev/mediapipe/facemesh/1) +* [Model card](https://drive.google.com/file/d/1VFC_wIpw4O7xBOiTgUldl79d9LA-LsnA/view) diff --git a/docs/solutions/hair_segmentation.md b/docs/solutions/hair_segmentation.md new file mode 100644 index 000000000..87361040a --- /dev/null +++ b/docs/solutions/hair_segmentation.md @@ -0,0 +1,58 @@ +--- +layout: default +title: Hair Segmentation +parent: Solutions +nav_order: 4 +--- + +# MediaPipe Hair Segmentation +{: .no_toc } + +1. TOC +{:toc} +--- + +![hair_segmentation_android_gpu_gif](../images/mobile/hair_segmentation_android_gpu.gif) + +## Example Apps + +Please first see general instructions for +[Android](../getting_started/building_examples.md#android), [iOS](../getting_started/building_examples.md#ios) +and [desktop](../getting_started/building_examples.md#desktop) on how to build MediaPipe +examples. + +Note: To visualize a graph, copy the graph and paste it into +[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how +to visualize its associated subgraphs, please see +[visualizer documentation](../visualizer.md). + +### Mobile + +* Graph: + [`mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt) +* Android target: + [(or download prebuilt ARM64 APK)](https://drive.google.com/open?id=1mmLtyL8IRfCUbqqu0-E-Hgjr_e6P3XAy) + [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu:hairsegmentationgpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/hairsegmentationgpu/BUILD) +* iOS target: Not available + +### Desktop + +* Running on CPU: Not available +* Running on GPU + * Graph: + [`mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hair_segmentation/hair_segmentation_mobile_gpu.pbtxt) + * Target: + [`mediapipe/examples/desktop/hair_segmentation:hair_segmentation_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/hair_segmentation/BUILD) + +### Web + +Please refer to [these instructions](../index.md#mediapipe-on-the-web). + +## Resources + +* Paper: + [Real-time Hair segmentation and recoloring on Mobile GPUs](https://arxiv.org/abs/1907.06740) + ([presentation](https://drive.google.com/file/d/1C8WYlWdDRNtU1_pYBvkkG5Z5wqYqf0yj/view)) + ([supplementary video](https://drive.google.com/file/d/1LPtM99Ch2ogyXYbDNpEqnUfhFq0TfLuf/view)) +* [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hair_segmentation.tflite) +* [Model card](https://drive.google.com/file/d/1lPwJ8BD_-3UUor4LayQ0xpa_RIC_hoRh/view) diff --git a/docs/solutions/hand.md b/docs/solutions/hand.md new file mode 100644 index 000000000..bef5e220e --- /dev/null +++ b/docs/solutions/hand.md @@ -0,0 +1,231 @@ +--- +layout: default +title: Hand +parent: Solutions +nav_order: 3 +--- + +# MediaPipe Hand +{: .no_toc } + +1. TOC +{:toc} +--- + +## Overview + +The ability to perceive the shape and motion of hands can be a vital component +in improving the user experience across a variety of technological domains and +platforms. For example, it can form the basis for sign language understanding +and hand gesture control, and can also enable the overlay of digital content and +information on top of the physical world in augmented reality. While coming +naturally to people, robust real-time hand perception is a decidedly challenging +computer vision task, as hands often occlude themselves or each other (e.g. +finger/palm occlusions and hand shakes) and lack high contrast patterns. + +MediaPipe Hand is a high-fidelity hand and finger tracking solution. It employs +machine learning (ML) to infer 21 3D landmarks of a hand from just a single +frame. Whereas current state-of-the-art approaches rely primarily on powerful +desktop environments for inference, our method achieves real-time performance on +a mobile phone, and even scales to multiple hands. We hope that providing this +hand perception functionality to the wider research and development community +will result in an emergence of creative use cases, stimulating new applications +and new research avenues. + +![hand_tracking_3d_android_gpu.gif](../images/mobile/hand_tracking_3d_android_gpu.gif) | +:------------------------------------------------------------------------------------: | +*Fig 1. Tracked 3D hand landmarks are represented by dots in different shades, with the brighter ones denoting landmarks closer to the camera.* | + +## ML Pipeline + +MediaPipe Hand utilizes an ML pipeline consisting of multiple models working +together: A palm detection model that operates on the full image and returns an +oriented hand bounding box. A hand landmark model that operates on the cropped +image region defined by the palm detector and returns high-fidelity 3D hand +keypoints. This strategy is similar to that employed in our +[MediaPipe Face Mesh](./face_mesh.md) solution, which uses a face detector +together with a face landmark model. + +Providing the accurately cropped hand image to the hand landmark model +drastically reduces the need for data augmentation (e.g. rotations, translation +and scale) and instead allows the network to dedicate most of its capacity +towards coordinate prediction accuracy. In addition, in our pipeline the crops +can also be generated based on the hand landmarks identified in the previous +frame, and only when the landmark model could no longer identify hand presence +is palm detection invoked to relocalize the hand. + +The pipeline is implemented as a MediaPipe +[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt), +which internally utilizes a +[palm/hand detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/subgraphs/hand_detection_gpu.pbtxt), +a +[hand landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/subgraphs/hand_landmark_gpu.pbtxt) +and a +[renderer subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/subgraphs/renderer_gpu.pbtxt). + +Note: To visualize a graph, copy the graph and paste it into +[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how +to visualize its associated subgraphs, please see +[visualizer documentation](../visualizer.md). + +## Models + +### Palm Detection Model + +To detect initial hand locations, we designed a +[single-shot detector](https://arxiv.org/abs/1512.02325) model optimized for +mobile real-time uses in a manner similar to the face detection model in +[MediaPipe Face Mesh](./face_mesh.md). Detecting hands is a decidedly complex +task: our +[model](https://github.com/google/mediapipe/tree/master/mediapipe/models/palm_detection.tflite) has +to work across a variety of hand sizes with a large scale span (~20x) relative +to the image frame and be able to detect occluded and self-occluded hands. +Whereas faces have high contrast patterns, e.g., in the eye and mouth region, +the lack of such features in hands makes it comparatively difficult to detect +them reliably from their visual features alone. Instead, providing additional +context, like arm, body, or person features, aids accurate hand localization. + +Our method addresses the above challenges using different strategies. First, we +train a palm detector instead of a hand detector, since estimating bounding +boxes of rigid objects like palms and fists is significantly simpler than +detecting hands with articulated fingers. In addition, as palms are smaller +objects, the non-maximum suppression algorithm works well even for two-hand +self-occlusion cases, like handshakes. Moreover, palms can be modelled using +square bounding boxes (anchors in ML terminology) ignoring other aspect ratios, +and therefore reducing the number of anchors by a factor of 3-5. Second, an +encoder-decoder feature extractor is used for bigger scene context awareness +even for small objects (similar to the RetinaNet approach). Lastly, we minimize +the focal loss during training to support a large amount of anchors resulting +from the high scale variance. + +With the above techniques, we achieve an average precision of 95.7% in palm +detection. Using a regular cross entropy loss and no decoder gives a baseline of +just 86.22%. + +### Hand Landmark Model + +After the palm detection over the whole image our subsequent hand landmark +[model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hand_landmark.tflite) +performs precise keypoint localization of 21 3D hand-knuckle coordinates inside +the detected hand regions via regression, that is direct coordinate prediction. +The model learns a consistent internal hand pose representation and is robust +even to partially visible hands and self-occlusions. + +To obtain ground truth data, we have manually annotated ~30K real-world images +with 21 3D coordinates, as shown below (we take Z-value from image depth map, if +it exists per corresponding coordinate). To better cover the possible hand poses +and provide additional supervision on the nature of hand geometry, we also +render a high-quality synthetic hand model over various backgrounds and map it +to the corresponding 3D coordinates. + +| ![hand_crops.png](../images/mobile/hand_crops.png) | +| :-------------------------------------------------------------------------: | +| *Fig 2. Top: Aligned hand crops passed to the tracking network with ground truth annotation. Bottom: Rendered synthetic hand images with ground truth annotation.* | + +## Example Apps + +Please first see general instructions for +[Android](../getting_started/building_examples.md#android), [iOS](../getting_started/building_examples.md#ios) +and [desktop](../getting_started/building_examples.md#desktop) on how to build MediaPipe +examples. + +Note: To visualize a graph, copy the graph and paste it into +[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how +to visualize its associated subgraphs, please see +[visualizer documentation](../visualizer.md). + +### Mobile + +#### Main Example + +* Graph: + [`mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt) +* Android target: + [(or download prebuilt ARM64 APK)](https://drive.google.com/open?id=1uCjS0y0O0dTDItsMh8x2cf4-l3uHW1vE) + [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu:handtrackinggpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handtrackinggpu/BUILD) +* iOS target: + [`mediapipe/examples/ios/handtrackinggpu:HandTrackingGpuApp`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handtrackinggpu/BUILD) + +#### With Multi-hand Support + +* Graph: + [`mediapipe/graphs/hand_tracking/multi_hand_tracking_mobile.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/multi_hand_tracking_mobile.pbtxt) +* Android target: + [(or download prebuilt ARM64 APK)](https://drive.google.com/open?id=1Wk6V9EVaz1ks_MInPqqVGvvJD01SGXDc) + [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/multihandtrackinggpu:multihandtrackinggpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/multihandtrackinggpu/BUILD) +* iOS target: + [`mediapipe/examples/ios/multihandtrackinggpu:MultiHandTrackingGpuApp`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/multihandtrackinggpu/BUILD) + +There are two key differences between this graph and that in the +[main example](#main-example) (which handles only one hand): + +1. There is a `NormalizedRectVectorHasMinSize` calculator, that checks if in + input vector of `NormalizedRect` objects has a minimum size equal to `N`. In + this graph, if the vector contains fewer than `N` objects, + `MultiHandDetection` subgraph runs. Otherwise, the `GateCalculator` doesn't + send any image packets to the `MultiHandDetection` subgraph. This way, the + main graph is efficient in that it avoids running the costly hand detection + step when there are already `N` hands in the frame. +2. The `MergeCalculator` has been replaced by the `AssociationNormRect` + calculator. This `AssociationNormRect` takes as input a vector of + `NormalizedRect` objects from the `MultiHandDetection` subgraph on the + current frame, and a vector of `NormalizedRect` objects from the + `MultiHandLandmark` subgraph from the previous frame, and performs an + association operation between these objects. This calculator ensures that + the output vector doesn't contain overlapping regions based on the specified + `min_similarity_threshold`. + +#### Palm/Hand Detection Only (no landmarks) + +* Graph: + [`mediapipe/graphs/hand_tracking/hand_detection_mobile.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_detection_mobile.pbtxt) +* Android target: + [(or download prebuilt ARM64 APK)](https://drive.google.com/open?id=1qUlTtH7Ydg-wl_H6VVL8vueu2UCTu37E) + [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu:handdetectiongpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/handdetectiongpu/BUILD) +* iOS target: + [`mediapipe/examples/ios/handdetectiongpu:HandDetectionGpuApp`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/ios/handdetectiongpu/BUILD) + +### Desktop + +#### Main Example + +* Running on CPU + * Graph: + [`mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt) + * Target: + [`mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/hand_tracking/BUILD) +* Running on GPU + * Graph: + [`mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt) + * Target: + [`mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/hand_tracking/BUILD) + +#### With Multi-hand Support + +* Running on CPU + * Graph: + [`mediapipe/graphs/hand_tracking/multi_hand_tracking_desktop_live.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/multi_hand_tracking_desktop_live) + * Target: + [`mediapipe/examples/desktop/multi_hand_tracking:multi_hand_tracking_cpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/multi_hand_tracking/BUILD) +* Running on GPU + * Graph: + [`mediapipe/graphs/hand_tracking/multi_hand_tracking_mobile.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/hand_tracking/multi_hand_tracking_mobile.pbtxt) + * Target: + [`mediapipe/examples/desktop/multi_hand_tracking:multi_hand_tracking_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/multi_hand_tracking/BUILD) + +### Web + +Please refer to [these instructions](../index.md#mediapipe-on-the-web). + +## Resources + +* Google AI Blog: [On-Device, Real-Time Hand Tracking with MediaPipe](https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html) +* TensorFlow Blog: [Face and hand tracking in the browser with MediaPipe and + TensorFlow.js](https://blog.tensorflow.org/2020/03/face-and-hand-tracking-in-browser-with-mediapipe-and-tensorflowjs.html) +* Palm detection model: + [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/palm_detection.tflite), + [TF.js model](https://tfhub.dev/mediapipe/handdetector/1) +* Hand landmark model: + [TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/models/hand_landmark.tflite), + [TF.js model](https://tfhub.dev/mediapipe/handskeleton/1) +* [Model card](https://mediapipe.page.link/handmc) diff --git a/docs/solutions/knift.md b/docs/solutions/knift.md new file mode 100644 index 000000000..ec2eec154 --- /dev/null +++ b/docs/solutions/knift.md @@ -0,0 +1,145 @@ +--- +layout: default +title: KNIFT (Template-based Feature Matching) +parent: Solutions +nav_order: 8 +--- + +# MediaPipe KNIFT +{: .no_toc } + +1. TOC +{:toc} +--- + +## Overview + +MediaPipe KNIFT is a template-based feature matching solution using KNIFT +(Keypoint Neural Invariant Feature Transform). + +![knift_stop_sign.gif](../images/knift_stop_sign.gif) | +:-----------------------------------------------------------------------: | +*Fig 1. Matching a real Stop Sign with a Stop Sign template using KNIFT.* | + +In many computer vision applications, a crucial building block is to establish +reliable correspondences between different views of an object or scene, forming +the foundation for approaches like template matching, image retrieval and +structure from motion. Correspondences are usually computed by extracting +distinctive view-invariant features such as +[SIFT](https://en.wikipedia.org/wiki/Scale-invariant_feature_transform) or +[ORB](https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_feature2d/py_orb/py_orb.html#orb-in-opencv) +from images. The ability to reliably establish such correspondences enables +applications like image stitching to create panoramas or template matching for +object recognition in videos. + +KNIFT is a general purpose local feature descriptor similar to SIFT or ORB. +Likewise, KNIFT is also a compact vector representation of local image patches +that is invariant to uniform scaling, orientation, and illumination changes. +However unlike SIFT or ORB, which were engineered with heuristics, KNIFT is an +[embedding](https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture) +learned directly from a large number of corresponding local patches extracted +from nearby video frames. This data driven approach implicitly encodes complex, +real-world spatial transformations and lighting changes in the embedding. As a +result, the KNIFT feature descriptor appears to be more robust, not only to +[affine distortions](https://en.wikipedia.org/wiki/Affine_transformation), but +to some degree of +[perspective distortions](https://en.wikipedia.org/wiki/Perspective_distortion_\(photography\)) +as well. + +For more information, please see +[MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html) +in Google Developers Blog. + +![template_matching_mobile_cpu.gif](../images/mobile/template_matching_android_cpu.gif) | +:-------------------------------------------------------------------------------------: | +*Fig 2. Matching US dollar bills using KNIFT.* | + +## Example Apps + +### Matching US Dollar Bills + +In MediaPipe, we've already provided an +[index file](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_index.pb) +pre-computed from the 3 template images (of US dollar bills) shown below. If +you'd like to use your own template images, see +[Matching Your Own Template Images](#matching-your-own-template-images). + +![template_matching_mobile_template.jpg](../images/mobile/template_matching_mobile_template.jpg) + +Please first see general instructions for +[Android](../getting_started/building_examples.md#android) on how to build MediaPipe examples. + +Note: To visualize a graph, copy the graph and paste it into +[MediaPipe Visualizer](https://viz.mediapipe.dev/). For more information on how +to visualize its associated subgraphs, please see +[visualizer documentation](../visualizer.md). + +* Graph: + [`mediapipe/graphs/template_matching/template_matching_mobile_cpu.pbtxt`](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/template_matching/template_matching_mobile_cpu.pbtxt) +* Android target: + [(or download prebuilt ARM64 APK)](https://drive.google.com/open?id=1tSWRfes9rAM4NrzmJBplguNQQvaeBZSa) + [`mediapipe/examples/android/src/java/com/google/mediapipe/apps/templatematchingcpu:templatematchingcpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/templatematchingcpu/BUILD) + +Note: MediaPipe uses OpenCV 3 by default. However, because of +[issues](https://github.com/opencv/opencv/issues/11488) between NDK 17+ and +OpenCV 3 when using +[knnMatch](https://docs.opencv.org/3.4/db/d39/classcv_1_1DescriptorMatcher.html#a378f35c9b1a5dfa4022839a45cdf0e89), +for this example app please use the following commands to temporarily switch to +OpenCV 4, and switch back to OpenCV 3 afterwards. + +```bash +# Switch to OpenCV 4 +sed -i -e 's:3.4.3/opencv-3.4.3:4.0.1/opencv-4.0.1:g' WORKSPACE +sed -i -e 's:libopencv_java3:libopencv_java4:g' third_party/opencv_android.BUILD + +# Build and install app +bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/templatematchingcpu +adb install -r bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/templatematchingcpu/templatematchingcpu.apk + +# Switch back to OpenCV 3 +sed -i -e 's:4.0.1/opencv-4.0.1:3.4.3/opencv-3.4.3:g' WORKSPACE +sed -i -e 's:libopencv_java4:libopencv_java3:g' third_party/opencv_android.BUILD +``` + +Tip: The example uses the TFLite +[XNNPACK delegate](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/delegates/xnnpack) +by default for faster inference. Users can change the +[option in TfLiteInferenceCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/calculators/tflite/tflite_inference_calculator.proto) +to run regular TFLite inference. + +### Matching Your Own Template Images + +* Step 1: Put all template images in a single directory. + +* Step 2: To build the index file for all templates in the directory, run + + ```bash + bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 \ + mediapipe/examples/desktop/template_matching:template_matching_tflite + ``` + + ```bash + bazel-bin/mediapipe/examples/desktop/template_matching/template_matching_tflite \ + --calculator_graph_config_file=mediapipe/graphs/template_matching/index_building.pbtxt \ + --input_side_packets="file_directory=