cassdlcm/mediapipe
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

update readme

Browse Source
This commit is contained in:
kurshakuz 2020-11-22 16:38:42 +06:00
parent ac2b203dd7
commit c17bc96bae
1 changed files with 117 additions and 136 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

253
README.md
View File

@ -1,171 +1,152 @@
--- # Installation
layout: default
title: Home
nav_order: 1
---
![MediaPipe](docs/images/mediapipe_small.png) ## Installing on Debian and Ubuntu
-------------------------------------------------------------------------------- 1. Checkout MediaPipe repository.
## Live ML anywhere ```bash
$ git clone https://github.com/kurshakuz/mediapipe.git
[MediaPipe](https://google.github.io/mediapipe/) offers cross-platform, customizable # Change directory into MediaPipe root directory
ML solutions for live and streaming media. $ cd mediapipe
```
![accelerated.png](docs/images/accelerated_small.png) | ![cross_platform.png](docs/images/cross_platform_small.png) 2. Install Bazel.
:------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------:
***End-to-End acceleration***: *Built-in fast ML inference and processing accelerated even on common hardware* | ***Build once, deploy anywhere***: *Unified solution works across Android, iOS, desktop/cloud, web and IoT*
![ready_to_use.png](docs/images/ready_to_use_small.png) | ![open_source.png](docs/images/open_source_small.png)
***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable*
## ML solutions in MediaPipe Follow the official
[Bazel documentation](https://docs.bazel.build/versions/master/install-ubuntu.html)
to install Bazel 3.4 or higher.
Face Detection | Face Mesh | Iris | Hands | Pose | Hair Segmentation For Nvidia Jetson and Raspberry Pi devices with ARM Ubuntu, Bazel needs to
:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :---------------: be built from source.
[![face_detection](docs/images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](docs/images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](docs/images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](docs/images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands) | [![pose](docs/images/mobile/pose_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/pose) | [![hair_segmentation](docs/images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation)
Object Detection | Box Tracking | Instant Motion Tracking | Objectron | KNIFT ```bash
:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---: # For Bazel 3.4.0
[![object_detection](docs/images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](docs/images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![instant_motion_tracking](docs/images/mobile/instant_motion_tracking_android_small.gif)](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | [![objectron](docs/images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](docs/images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift) wget https://github.com/bazelbuild/bazel/releases/download/3.4.0/bazel-3.4.0-dist.zip
sudo apt-get install build-essential openjdk-8-jdk python zip unzip
unzip bazel-3.4.0-dist.zip
env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh
sudo cp output/bazel /usr/local/bin/
```
<!-- []() in the first cell is needed to preserve table formatting in GitHub Pages. --> 3. Install OpenCV and FFmpeg.
<!-- Whenever this table is updated, paste a copy to solutions/solutions.md. -->
[]() | Android | iOS | Desktop | Python | Web | Coral Option 1. Use package manager tool to install the pre-compiled OpenCV
:---------------------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---: libraries. FFmpeg will be installed via libopencv-video-dev.
[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | | ✅ | ✅
[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | ✅ | |
[Iris](https://google.github.io/mediapipe/solutions/iris) | ✅ | ✅ | ✅ | | ✅ |
[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | ✅ | ✅ |
[Pose](https://google.github.io/mediapipe/solutions/pose) | ✅ | ✅ | ✅ | ✅ | ✅ |
[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | | ✅ |
[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | | ✅
[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | | |
[Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅ | | | | |
[Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | | |
[KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | | |
[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | | |
[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | | |
[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m) | | | ✅ | | |
See also Note: Debian 9 and Ubuntu 16.04 provide OpenCV 2.4.9. You may want to take
[MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models) option 2 or 3 to install OpenCV 3 or above.
for ML models released in MediaPipe.
## MediaPipe in Python ```bash
$ sudo apt-get install libopencv-core-dev libopencv-highgui-dev \
libopencv-calib3d-dev libopencv-features2d-dev \
libopencv-imgproc-dev libopencv-video-dev
```
MediaPipe Python package is available on Debian 9 and Ubuntu 18.04 install the packages in
[PyPI](https://pypi.org/project/mediapipe/), and can be installed simply by `pip `/usr/lib/x86_64-linux-gnu`. MediaPipe's [`opencv_linux.BUILD`] and
install mediapipe` on Linux and macOS, as described in: [`ffmpeg_linux.BUILD`] are configured for this library path. Ubuntu 20.04
may install the OpenCV and FFmpeg packages in `/usr/local`, Please follow
the option 3 below to modify the [`WORKSPACE`], [`opencv_linux.BUILD`] and
[`ffmpeg_linux.BUILD`] files accordingly.
* [MediaPipe Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh#python) Option 2. Run [`setup_opencv.sh`] to automatically build OpenCV from source
([colab](https://mediapipe.page.link/face_mesh_py_colab)) and modify MediaPipe's OpenCV config.
* [MediaPipe Hands](https://google.github.io/mediapipe/solutions/hands#python)
([colab](https://mediapipe.page.link/hands_py_colab))
* [MediaPipe Pose](https://google.github.io/mediapipe/solutions/pose#python)
([colab](https://mediapipe.page.link/pose_py_colab))
## MediaPipe on the Web Option 3. Follow OpenCV's
[documentation](https://docs.opencv.org/3.4.6/d7/d9f/tutorial_linux_install.html)
to manually build OpenCV from source code.
MediaPipe on the Web is an effort to run the same ML solutions built for mobile 4. For running desktop examples on Linux only (not on OS X) with GPU
and desktop also in web browsers. The official API is under construction, but acceleration.
the core technology has been proven effective. Please see
[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
in Google Developers Blog for details.
You can use the following links to load a demo in the MediaPipe Visualizer, and ```bash
over there click the "Runner" icon in the top bar like shown below. The demos # Requires a GPU with EGL driver support.
use your webcam video as input, which is processed all locally in real-time and # Can use mesa GPU libraries for desktop, (or Nvidia/AMD equivalent).
never leaves your device. sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev
![visualizer_runner](docs/images/visualizer_runner.png) # To compile with GPU support, replace
--define MEDIAPIPE_DISABLE_GPU=1
# with
--copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11
# when building GPU examples.
```
* [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection) 5. Run the [Hello World desktop example](./hello_world_desktop.md).
* [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
* [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
* [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
* [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
* [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking)
* [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
## Getting started ```bash
$ export GLOG_logtostderr=1
Learn how to [install](https://google.github.io/mediapipe/getting_started/install) # if you are running on Linux desktop with CPU only
MediaPipe and $ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
[build example applications](https://google.github.io/mediapipe/getting_started/building_examples), mediapipe/examples/desktop/hello_world:hello_world
and start exploring our ready-to-use
[solutions](https://google.github.io/mediapipe/solutions/solutions) that you can
further extend and customize.
The source code is hosted in the # If you are running on Linux desktop with GPU support enabled (via mesa drivers)
[MediaPipe Github repository](https://github.com/google/mediapipe), and you can $ bazel run --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
run code search using mediapipe/examples/desktop/hello_world:hello_world
[Google Open Source Code Search](https://cs.opensource.google/mediapipe/mediapipe).
## Publications # Should print:
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
# Hello World!
```
* [MediaPipe 3D Face Transform](https://developers.googleblog.com/2020/09/mediapipe-3d-face-transform.html) ## Building required hand tracking module on desktop
in Google Developers Blog
* [Instant Motion Tracking With MediaPipe](https://developers.googleblog.com/2020/08/instant-motion-tracking-with-mediapipe.html)
in Google Developers Blog
* [BlazePose - On-device Real-time Body Pose Tracking](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html)
in Google AI Blog
* [MediaPipe Iris: Real-time Eye Tracking and Depth Estimation](https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-iris-tracking.html)
in Google AI Blog
* [MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html)
in Google Developers Blog
* [Alfred Camera: Smart camera features using MediaPipe](https://developers.googleblog.com/2020/03/alfred-camera-smart-camera-features-using-mediapipe.html)
in Google Developers Blog
* [Real-Time 3D Object Detection on Mobile Devices with MediaPipe](https://ai.googleblog.com/2020/03/real-time-3d-object-detection-on-mobile.html)
in Google AI Blog
* [AutoFlip: An Open Source Framework for Intelligent Video Reframing](https://ai.googleblog.com/2020/02/autoflip-open-source-framework-for.html)
in Google AI Blog
* [MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
in Google Developers Blog
* [Object Detection and Tracking using MediaPipe](https://developers.googleblog.com/2019/12/object-detection-and-tracking-using-mediapipe.html)
in Google Developers Blog
* [On-Device, Real-Time Hand Tracking with MediaPipe](https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html)
in Google AI Blog
* [MediaPipe: A Framework for Building Perception Pipelines](https://arxiv.org/abs/1906.08172)
## Videos ### Option 1: Running on CPU
* [YouTube Channel](https://www.youtube.com/c/MediaPipe) 1. To build, for example, [MediaPipe Hands](../solutions/hands.md), run:
## Events ```bash
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu
```
* [MediaPipe Seattle Meetup, Google Building Waterside, 13 Feb 2020](https://mediapipe.page.link/seattle2020) 2. To run the application:
* [AI Nextcon 2020, 12-16 Feb 2020, Seattle](http://aisea20.xnextcon.com/)
* [MediaPipe Madrid Meetup, 16 Dec 2019](https://www.meetup.com/Madrid-AI-Developers-Group/events/266329088/)
* [MediaPipe London Meetup, Google 123 Building, 12 Dec 2019](https://www.meetup.com/London-AI-Tech-Talk/events/266329038)
* [ML Conference, Berlin, 11 Dec 2019](https://mlconference.ai/machine-learning-advanced-development/mediapipe-building-real-time-cross-platform-mobile-web-edge-desktop-video-audio-ml-pipelines/)
* [MediaPipe Berlin Meetup, Google Berlin, 11 Dec 2019](https://www.meetup.com/Berlin-AI-Tech-Talk/events/266328794/)
* [The 3rd Workshop on YouTube-8M Large Scale Video Understanding Workshop,
Seoul, Korea ICCV
2019](https://research.google.com/youtube8m/workshop2019/index.html)
* [AI DevWorld 2019, 10 Oct 2019, San Jose, CA](https://aidevworld.com)
* [Google Industry Workshop at ICIP 2019, 24 Sept 2019, Taipei, Taiwan](http://2019.ieeeicip.org/?action=page4&id=14#Google)
([presentation](https://docs.google.com/presentation/d/e/2PACX-1vRIBBbO_LO9v2YmvbHHEt1cwyqH6EjDxiILjuT0foXy1E7g6uyh4CesB2DkkEwlRDO9_lWfuKMZx98T/pub?start=false&loop=false&delayms=3000&slide=id.g556cc1a659_0_5))
* [Open sourced at CVPR 2019, 17~20 June, Long Beach, CA](https://sites.google.com/corp/view/perception-cv4arvr/mediapipe)
## Community ```bash
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu \
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt
```
* [Awesome MediaPipe](https://mediapipe.org) - A curated list of awesome This will open up your webcam as long as it is connected and on. Any errors
MediaPipe related frameworks, libraries and software is likely due to your webcam being not accessible.
* [Slack community](https://mediapipe.page.link/joinslack) for MediaPipe users
* [Discuss](https://groups.google.com/forum/#!forum/mediapipe) - General
community discussion around MediaPipe
## Alpha disclaimer ### Option 2: Running on GPU
MediaPipe is currently in alpha at v0.7. We may be still making breaking API Note: This currently works only on Linux, and please first follow
changes and expect to get to stable APIs by v1.0. [OpenGL ES Setup on Linux Desktop](./gpu_support.md#opengl-es-setup-on-linux-desktop).
## Contributing 1. To build, for example, [MediaPipe Hands](../solutions/hands.md), run:
We welcome contributions. Please follow these ```bash
[guidelines](https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md). bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \
mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu
```
We use GitHub issues for tracking requests and bugs. Please post questions to 2. To run the application:
the MediaPipe Stack Overflow with a `mediapipe` tag.
```bash
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu \
--calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt
```
This will open up your webcam as long as it is connected and on. Any errors
is likely due to your webcam being not accessible, or GPU drivers not setup
properly.
## Running the pretrained model for sign language recognition
Insert your data to any folder and pass it as a ```--output_data_path``` variable. The result will appear in the same folder and will be called `result.txt`.
```bash
cd RNN/
python3 predict.py --input_data_path='./test_video/' --output_data_path='./test_video_output/'
```