diff --git a/README.md b/README.md index 300563c47..c8e075922 100644 --- a/README.md +++ b/README.md @@ -1,171 +1,152 @@ ---- -layout: default -title: Home -nav_order: 1 ---- +# Installation -![MediaPipe](docs/images/mediapipe_small.png) +## Installing on Debian and Ubuntu --------------------------------------------------------------------------------- +1. Checkout MediaPipe repository. -## Live ML anywhere + ```bash + $ git clone https://github.com/kurshakuz/mediapipe.git -[MediaPipe](https://google.github.io/mediapipe/) offers cross-platform, customizable -ML solutions for live and streaming media. + # Change directory into MediaPipe root directory + $ cd mediapipe + ``` -![accelerated.png](docs/images/accelerated_small.png) | ![cross_platform.png](docs/images/cross_platform_small.png) -:------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------: -***End-to-End acceleration***: *Built-in fast ML inference and processing accelerated even on common hardware* | ***Build once, deploy anywhere***: *Unified solution works across Android, iOS, desktop/cloud, web and IoT* -![ready_to_use.png](docs/images/ready_to_use_small.png) | ![open_source.png](docs/images/open_source_small.png) -***Ready-to-use solutions***: *Cutting-edge ML solutions demonstrating full power of the framework* | ***Free and open source***: *Framework and solutions both under Apache 2.0, fully extensible and customizable* +2. Install Bazel. -## ML solutions in MediaPipe + Follow the official + [Bazel documentation](https://docs.bazel.build/versions/master/install-ubuntu.html) + to install Bazel 3.4 or higher. -Face Detection | Face Mesh | Iris | Hands | Pose | Hair Segmentation -:----------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------: | :---------------: -[![face_detection](docs/images/mobile/face_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_detection) | [![face_mesh](docs/images/mobile/face_mesh_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/face_mesh) | [![iris](docs/images/mobile/iris_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/iris) | [![hand](docs/images/mobile/hand_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hands) | [![pose](docs/images/mobile/pose_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/pose) | [![hair_segmentation](docs/images/mobile/hair_segmentation_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/hair_segmentation) + For Nvidia Jetson and Raspberry Pi devices with ARM Ubuntu, Bazel needs to + be built from source. -Object Detection | Box Tracking | Instant Motion Tracking | Objectron | KNIFT -:----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------: | :---: -[![object_detection](docs/images/mobile/object_detection_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/object_detection) | [![box_tracking](docs/images/mobile/object_tracking_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/box_tracking) | [![instant_motion_tracking](docs/images/mobile/instant_motion_tracking_android_small.gif)](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | [![objectron](docs/images/mobile/objectron_chair_android_gpu_small.gif)](https://google.github.io/mediapipe/solutions/objectron) | [![knift](docs/images/mobile/template_matching_android_cpu_small.gif)](https://google.github.io/mediapipe/solutions/knift) + ```bash + # For Bazel 3.4.0 + wget https://github.com/bazelbuild/bazel/releases/download/3.4.0/bazel-3.4.0-dist.zip + sudo apt-get install build-essential openjdk-8-jdk python zip unzip + unzip bazel-3.4.0-dist.zip + env EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" bash ./compile.sh + sudo cp output/bazel /usr/local/bin/ + ``` - - +3. Install OpenCV and FFmpeg. -[]() | Android | iOS | Desktop | Python | Web | Coral -:---------------------------------------------------------------------------------------- | :-----: | :-: | :-----: | :----: | :-: | :---: -[Face Detection](https://google.github.io/mediapipe/solutions/face_detection) | ✅ | ✅ | ✅ | | ✅ | ✅ -[Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh) | ✅ | ✅ | ✅ | ✅ | | -[Iris](https://google.github.io/mediapipe/solutions/iris) | ✅ | ✅ | ✅ | | ✅ | -[Hands](https://google.github.io/mediapipe/solutions/hands) | ✅ | ✅ | ✅ | ✅ | ✅ | -[Pose](https://google.github.io/mediapipe/solutions/pose) | ✅ | ✅ | ✅ | ✅ | ✅ | -[Hair Segmentation](https://google.github.io/mediapipe/solutions/hair_segmentation) | ✅ | | ✅ | | ✅ | -[Object Detection](https://google.github.io/mediapipe/solutions/object_detection) | ✅ | ✅ | ✅ | | | ✅ -[Box Tracking](https://google.github.io/mediapipe/solutions/box_tracking) | ✅ | ✅ | ✅ | | | -[Instant Motion Tracking](https://google.github.io/mediapipe/solutions/instant_motion_tracking) | ✅ | | | | | -[Objectron](https://google.github.io/mediapipe/solutions/objectron) | ✅ | | | | | -[KNIFT](https://google.github.io/mediapipe/solutions/knift) | ✅ | | | | | -[AutoFlip](https://google.github.io/mediapipe/solutions/autoflip) | | | ✅ | | | -[MediaSequence](https://google.github.io/mediapipe/solutions/media_sequence) | | | ✅ | | | -[YouTube 8M](https://google.github.io/mediapipe/solutions/youtube_8m) | | | ✅ | | | + Option 1. Use package manager tool to install the pre-compiled OpenCV + libraries. FFmpeg will be installed via libopencv-video-dev. -See also -[MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models) -for ML models released in MediaPipe. + Note: Debian 9 and Ubuntu 16.04 provide OpenCV 2.4.9. You may want to take + option 2 or 3 to install OpenCV 3 or above. -## MediaPipe in Python + ```bash + $ sudo apt-get install libopencv-core-dev libopencv-highgui-dev \ + libopencv-calib3d-dev libopencv-features2d-dev \ + libopencv-imgproc-dev libopencv-video-dev + ``` -MediaPipe Python package is available on -[PyPI](https://pypi.org/project/mediapipe/), and can be installed simply by `pip -install mediapipe` on Linux and macOS, as described in: + Debian 9 and Ubuntu 18.04 install the packages in + `/usr/lib/x86_64-linux-gnu`. MediaPipe's [`opencv_linux.BUILD`] and + [`ffmpeg_linux.BUILD`] are configured for this library path. Ubuntu 20.04 + may install the OpenCV and FFmpeg packages in `/usr/local`, Please follow + the option 3 below to modify the [`WORKSPACE`], [`opencv_linux.BUILD`] and + [`ffmpeg_linux.BUILD`] files accordingly. -* [MediaPipe Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh#python) - ([colab](https://mediapipe.page.link/face_mesh_py_colab)) -* [MediaPipe Hands](https://google.github.io/mediapipe/solutions/hands#python) - ([colab](https://mediapipe.page.link/hands_py_colab)) -* [MediaPipe Pose](https://google.github.io/mediapipe/solutions/pose#python) - ([colab](https://mediapipe.page.link/pose_py_colab)) + Option 2. Run [`setup_opencv.sh`] to automatically build OpenCV from source + and modify MediaPipe's OpenCV config. -## MediaPipe on the Web + Option 3. Follow OpenCV's + [documentation](https://docs.opencv.org/3.4.6/d7/d9f/tutorial_linux_install.html) + to manually build OpenCV from source code. -MediaPipe on the Web is an effort to run the same ML solutions built for mobile -and desktop also in web browsers. The official API is under construction, but -the core technology has been proven effective. Please see -[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html) -in Google Developers Blog for details. +4. For running desktop examples on Linux only (not on OS X) with GPU + acceleration. -You can use the following links to load a demo in the MediaPipe Visualizer, and -over there click the "Runner" icon in the top bar like shown below. The demos -use your webcam video as input, which is processed all locally in real-time and -never leaves your device. + ```bash + # Requires a GPU with EGL driver support. + # Can use mesa GPU libraries for desktop, (or Nvidia/AMD equivalent). + sudo apt-get install mesa-common-dev libegl1-mesa-dev libgles2-mesa-dev -![visualizer_runner](docs/images/visualizer_runner.png) + # To compile with GPU support, replace + --define MEDIAPIPE_DISABLE_GPU=1 + # with + --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 + # when building GPU examples. + ``` -* [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection) -* [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking) -* [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth) -* [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking) -* [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection) -* [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking) -* [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation) +5. Run the [Hello World desktop example](./hello_world_desktop.md). -## Getting started + ```bash + $ export GLOG_logtostderr=1 -Learn how to [install](https://google.github.io/mediapipe/getting_started/install) -MediaPipe and -[build example applications](https://google.github.io/mediapipe/getting_started/building_examples), -and start exploring our ready-to-use -[solutions](https://google.github.io/mediapipe/solutions/solutions) that you can -further extend and customize. + # if you are running on Linux desktop with CPU only + $ bazel run --define MEDIAPIPE_DISABLE_GPU=1 \ + mediapipe/examples/desktop/hello_world:hello_world -The source code is hosted in the -[MediaPipe Github repository](https://github.com/google/mediapipe), and you can -run code search using -[Google Open Source Code Search](https://cs.opensource.google/mediapipe/mediapipe). + # If you are running on Linux desktop with GPU support enabled (via mesa drivers) + $ bazel run --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \ + mediapipe/examples/desktop/hello_world:hello_world -## Publications + # Should print: + # Hello World! + # Hello World! + # Hello World! + # Hello World! + # Hello World! + # Hello World! + # Hello World! + # Hello World! + # Hello World! + # Hello World! + ``` -* [MediaPipe 3D Face Transform](https://developers.googleblog.com/2020/09/mediapipe-3d-face-transform.html) - in Google Developers Blog -* [Instant Motion Tracking With MediaPipe](https://developers.googleblog.com/2020/08/instant-motion-tracking-with-mediapipe.html) - in Google Developers Blog -* [BlazePose - On-device Real-time Body Pose Tracking](https://ai.googleblog.com/2020/08/on-device-real-time-body-pose-tracking.html) - in Google AI Blog -* [MediaPipe Iris: Real-time Eye Tracking and Depth Estimation](https://ai.googleblog.com/2020/08/mediapipe-iris-real-time-iris-tracking.html) - in Google AI Blog -* [MediaPipe KNIFT: Template-based feature matching](https://developers.googleblog.com/2020/04/mediapipe-knift-template-based-feature-matching.html) - in Google Developers Blog -* [Alfred Camera: Smart camera features using MediaPipe](https://developers.googleblog.com/2020/03/alfred-camera-smart-camera-features-using-mediapipe.html) - in Google Developers Blog -* [Real-Time 3D Object Detection on Mobile Devices with MediaPipe](https://ai.googleblog.com/2020/03/real-time-3d-object-detection-on-mobile.html) - in Google AI Blog -* [AutoFlip: An Open Source Framework for Intelligent Video Reframing](https://ai.googleblog.com/2020/02/autoflip-open-source-framework-for.html) - in Google AI Blog -* [MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html) - in Google Developers Blog -* [Object Detection and Tracking using MediaPipe](https://developers.googleblog.com/2019/12/object-detection-and-tracking-using-mediapipe.html) - in Google Developers Blog -* [On-Device, Real-Time Hand Tracking with MediaPipe](https://ai.googleblog.com/2019/08/on-device-real-time-hand-tracking-with.html) - in Google AI Blog -* [MediaPipe: A Framework for Building Perception Pipelines](https://arxiv.org/abs/1906.08172) +## Building required hand tracking module on desktop -## Videos +### Option 1: Running on CPU -* [YouTube Channel](https://www.youtube.com/c/MediaPipe) +1. To build, for example, [MediaPipe Hands](../solutions/hands.md), run: -## Events + ```bash + bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu + ``` -* [MediaPipe Seattle Meetup, Google Building Waterside, 13 Feb 2020](https://mediapipe.page.link/seattle2020) -* [AI Nextcon 2020, 12-16 Feb 2020, Seattle](http://aisea20.xnextcon.com/) -* [MediaPipe Madrid Meetup, 16 Dec 2019](https://www.meetup.com/Madrid-AI-Developers-Group/events/266329088/) -* [MediaPipe London Meetup, Google 123 Building, 12 Dec 2019](https://www.meetup.com/London-AI-Tech-Talk/events/266329038) -* [ML Conference, Berlin, 11 Dec 2019](https://mlconference.ai/machine-learning-advanced-development/mediapipe-building-real-time-cross-platform-mobile-web-edge-desktop-video-audio-ml-pipelines/) -* [MediaPipe Berlin Meetup, Google Berlin, 11 Dec 2019](https://www.meetup.com/Berlin-AI-Tech-Talk/events/266328794/) -* [The 3rd Workshop on YouTube-8M Large Scale Video Understanding Workshop, - Seoul, Korea ICCV - 2019](https://research.google.com/youtube8m/workshop2019/index.html) -* [AI DevWorld 2019, 10 Oct 2019, San Jose, CA](https://aidevworld.com) -* [Google Industry Workshop at ICIP 2019, 24 Sept 2019, Taipei, Taiwan](http://2019.ieeeicip.org/?action=page4&id=14#Google) - ([presentation](https://docs.google.com/presentation/d/e/2PACX-1vRIBBbO_LO9v2YmvbHHEt1cwyqH6EjDxiILjuT0foXy1E7g6uyh4CesB2DkkEwlRDO9_lWfuKMZx98T/pub?start=false&loop=false&delayms=3000&slide=id.g556cc1a659_0_5)) -* [Open sourced at CVPR 2019, 17~20 June, Long Beach, CA](https://sites.google.com/corp/view/perception-cv4arvr/mediapipe) +2. To run the application: -## Community + ```bash + GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu \ + --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt + ``` -* [Awesome MediaPipe](https://mediapipe.org) - A curated list of awesome - MediaPipe related frameworks, libraries and software -* [Slack community](https://mediapipe.page.link/joinslack) for MediaPipe users -* [Discuss](https://groups.google.com/forum/#!forum/mediapipe) - General - community discussion around MediaPipe + This will open up your webcam as long as it is connected and on. Any errors + is likely due to your webcam being not accessible. -## Alpha disclaimer +### Option 2: Running on GPU -MediaPipe is currently in alpha at v0.7. We may be still making breaking API -changes and expect to get to stable APIs by v1.0. +Note: This currently works only on Linux, and please first follow +[OpenGL ES Setup on Linux Desktop](./gpu_support.md#opengl-es-setup-on-linux-desktop). -## Contributing +1. To build, for example, [MediaPipe Hands](../solutions/hands.md), run: -We welcome contributions. Please follow these -[guidelines](https://github.com/google/mediapipe/blob/master/CONTRIBUTING.md). + ```bash + bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 \ + mediapipe/examples/desktop/hand_tracking:hand_tracking_gpu + ``` -We use GitHub issues for tracking requests and bugs. Please post questions to -the MediaPipe Stack Overflow with a `mediapipe` tag. +2. To run the application: + + ```bash + GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_gpu \ + --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_mobile.pbtxt + ``` + + This will open up your webcam as long as it is connected and on. Any errors + is likely due to your webcam being not accessible, or GPU drivers not setup + properly. + +## Running the pretrained model for sign language recognition + +Insert your data to any folder and pass it as a ```--output_data_path``` variable. The result will appear in the same folder and will be called `result.txt`. + +```bash +cd RNN/ + +python3 predict.py --input_data_path='./test_video/' --output_data_path='./test_video_output/' +```