From 494fb5886b30bda34c5fb2b98ac55ee7fe341858 Mon Sep 17 00:00:00 2001 From: Dweep Joshipura Date: Fri, 25 Dec 2020 13:01:01 +0530 Subject: [PATCH 01/12] Fix links in javascript.md the links for control_utils, drawing_utils and camera_utils were incorrect --- docs/getting_started/javascript.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/getting_started/javascript.md b/docs/getting_started/javascript.md index 95a1e2610..f0b9fde12 100644 --- a/docs/getting_started/javascript.md +++ b/docs/getting_started/javascript.md @@ -69,9 +69,9 @@ affecting your work, restrict your request to a `` number. e.g., [F-npm]: https://www.npmjs.com/package/@mediapipe/face_mesh [H-npm]: https://www.npmjs.com/package/@mediapipe/hands [P-npm]: https://www.npmjs.com/package/@mediapipe/pose -[draw-npm]: https://www.npmjs.com/package/@mediapipe/pose -[cam-npm]: https://www.npmjs.com/package/@mediapipe/pose -[ctrl-npm]: https://www.npmjs.com/package/@mediapipe/pose +[draw-npm]: https://www.npmjs.com/package/@mediapipe/drawing_utils +[cam-npm]: https://www.npmjs.com/package/@mediapipe/camera_utils +[ctrl-npm]: https://www.npmjs.com/package/@mediapipe/control_utils [Ho-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/holistic [F-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/face_mesh [H-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/hands From d41f6a88c0b4a2b758eff39846387a2413e152e0 Mon Sep 17 00:00:00 2001 From: Dweep Joshipura Date: Fri, 25 Dec 2020 14:28:32 +0530 Subject: [PATCH 02/12] Create js_utils.md --- docs/getting_started/js_utils.md | 119 +++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 docs/getting_started/js_utils.md diff --git a/docs/getting_started/js_utils.md b/docs/getting_started/js_utils.md new file mode 100644 index 000000000..17bd6f37a --- /dev/null +++ b/docs/getting_started/js_utils.md @@ -0,0 +1,119 @@ +--- +layout: default +title: JavaScript Utilities +parent: MediaPipe in JavaScript +nav_order: 1 +--- + +# JavaScript Utils for MediaPipe +{: .no_toc } + +1. TOC +{:toc} +--- + +## Utilities +MediaPipe offers utilities for enabling users to use the MediaPipe Examples with ease, and conveniently integrate with existing code. +The utilities MediaPipe currently has are ~ +* [Drawing Utilities][draw-npm], +* [Camera Utilities][cam-npm], and +* [Control Panel][ctrl-npm] Utilities. + +### Drawing Utilities +These are for drawing on canvas using output from MediaPipe's solutions. +Currently, this includes three functions ~ +1. *drawLandmarks* +```js +drawLandmarks(canvasContext, landmarks, { + fillColor: fillColor, + color: color, + lineWidth: lineWidth + } +``` +This function is for drawing landmarks. +The value for the `landmarks` parameter can be found using in the `results` array. + +2. *drawConnectors* +```js +drawConnectors(canvasContext, landmarks, LANDMARK_CONSTANTS, { + color: color, + lineWidth: lineWidth +}) +``` +This is a function used for plotting connectors in various MediaPipe Solutions. +The `LANDMARK_CONSTANTS` are specific for every solution, so you could choose these there. + +3. *drawRectangle* +```js +drawRectangle(canvasContext, { + width: width, + height: height, + xCenter: xCenter, + yCenter: yCenter, + rotation: rotation +}, { + color: color +}) +``` +This allows drawing of a rectangle, with a certain width, height and rotation given the coordinates of its location. + +The quickest way to get acclimated is to look at the examples above. Each demo +has a link to a [CodePen][codepen] so that you can edit the code and try it +yourself. We have included a number of utility packages to help you get started: + +* [@mediapipe/drawing_utils][draw-npm] - Utilities to draw landmarks and + connectors. +* [@mediapipe/camera_utils][cam-npm] - Utilities to operate the camera. +* [@mediapipe/control_utils][ctrl-npm] - Utilities to show sliders and FPS + widgets. + +Note: See these demos and more at [MediaPipe on CodePen][codepen] + +All of these solutions are staged in [NPM][npm]. You can install any package +locally with `npm install`. Example: + +``` +npm install @mediapipe/holistic. +``` + +If you would rather not stage these locally, you can rely on a CDN (e.g., +[jsDelivr](https://www.jsdelivr.com/)). This will allow you to add scripts +directly to your HTML: + +``` + + + + +``` + +Note: You can specify version numbers to both NPM and jsdelivr. They are +structured as `..`. To prevent breaking changes from +affecting your work, restrict your request to a `` number. e.g., +`@mediapipe/holistic@0.1`. + +[Ho-pg]: ../solutions/holistic#javascript-solution-api +[F-pg]: ../solutions/face_mesh#javascript-solution-api +[H-pg]: ../solutions/hands#javascript-solution-api +[P-pg]: ../solutions/pose#javascript-solution-api +[Ho-npm]: https://www.npmjs.com/package/@mediapipe/holistic +[F-npm]: https://www.npmjs.com/package/@mediapipe/face_mesh +[H-npm]: https://www.npmjs.com/package/@mediapipe/hands +[P-npm]: https://www.npmjs.com/package/@mediapipe/pose +[draw-npm]: https://www.npmjs.com/package/@mediapipe/pose +[cam-npm]: https://www.npmjs.com/package/@mediapipe/pose +[ctrl-npm]: https://www.npmjs.com/package/@mediapipe/pose +[Ho-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/holistic +[F-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/face_mesh +[H-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/hands +[P-jsd]: https://www.jsdelivr.com/package/npm/@mediapipe/pose +[Ho-pen]: https://code.mediapipe.dev/codepen/holistic +[F-pen]: https://code.mediapipe.dev/codepen/face_mesh +[H-pen]: https://code.mediapipe.dev/codepen/hands +[P-pen]: https://code.mediapipe.dev/codepen/pose +[Ho-demo]: https://mediapipe.dev/demo/holistic +[F-demo]: https://mediapipe.dev/demo/face_mesh +[H-demo]: https://mediapipe.dev/demo/hands +[P-demo]: https://mediapipe.dev/demo/pose +[npm]: https://www.npmjs.com/package/@mediapipe +[codepen]: https://code.mediapipe.dev/codepen From ec58c026d1940be6ffd4c8868762c86c676daef9 Mon Sep 17 00:00:00 2001 From: Dweep Joshipura Date: Fri, 25 Dec 2020 16:44:33 +0530 Subject: [PATCH 03/12] Update js_utils.md --- docs/getting_started/js_utils.md | 98 +++++++++++++++++++++----------- 1 file changed, 65 insertions(+), 33 deletions(-) diff --git a/docs/getting_started/js_utils.md b/docs/getting_started/js_utils.md index 17bd6f37a..84ae9860d 100644 --- a/docs/getting_started/js_utils.md +++ b/docs/getting_started/js_utils.md @@ -56,41 +56,73 @@ drawRectangle(canvasContext, { }) ``` This allows drawing of a rectangle, with a certain width, height and rotation given the coordinates of its location. - -The quickest way to get acclimated is to look at the examples above. Each demo -has a link to a [CodePen][codepen] so that you can edit the code and try it -yourself. We have included a number of utility packages to help you get started: - -* [@mediapipe/drawing_utils][draw-npm] - Utilities to draw landmarks and - connectors. -* [@mediapipe/camera_utils][cam-npm] - Utilities to operate the camera. -* [@mediapipe/control_utils][ctrl-npm] - Utilities to show sliders and FPS - widgets. - -Note: See these demos and more at [MediaPipe on CodePen][codepen] - -All of these solutions are staged in [NPM][npm]. You can install any package -locally with `npm install`. Example: + +### Camera Utilities +This module has a `Camera` object that can be initialized and used easily, like below. +```js +// Our input frames will come from here. +const videoElement = + document.getElementsByClassName('input_video')[0]; +const camera = new Camera(videoElement, { + onFrame: async () => { + // Send it to a demo, for example, hands + await hands.send({image: videoElement}); + }, + // Set the width of camera to be rendered + width: 1280, + // And set the height + height: 720 +}); +// Start the camera. +// Note: It will ask the user for permission, and will error out if this is not given. +camera.start(); +``` +### Control Panel Utilities +To showcase and monitor the model in elegant ways, MediaPipe has this Control Panel module. +This panel is also convenient to use and set up. +```js +// Pass in a
+new ControlPanel(controlsElement, { + // Whether to invert the camera, defaults to false + selfieMode: true, + // Maximum Number of Hands to detect, a placeholder, only when using Hands API + maxNumHands: 2, + // Minimum Confidence score + minDetectionConfidence: 0.5, + // Minimum Tracking score + minTrackingConfidence: 0.5 + }) + .add([ + // a StaticText is simply a label + new StaticText({title: 'MediaPipe Hands'}), + // a Toggle is one with only 2 options (i.e., true or false) + new Toggle({title: 'Selfie Mode', field: 'selfieMode'}), + // A slider can have multiple options. This will have the options 1,2,3 and 4. + // Range determines maximum and minimum allowed value + // Step determines the difference between two options + new Slider( + {title: 'Max Number of Hands', field: 'maxNumHands', range: [1, 4], step: 1}), + new Slider({ + title: 'Min Detection Confidence', + field: 'minDetectionConfidence', + range: [0, 1], + step: 0.01 + }), + new Slider({ + title: 'Min Tracking Confidence', + field: 'minTrackingConfidence', + range: [0, 1], + step: 0.01 + }), + ]) + // This is run when an option is updated. + // the options object will contain parameters for all the above in a array. + .on(options => { + videoElement.classList.toggle('selfie', options.selfieMode); + hands.setOptions(options); + }); ``` -npm install @mediapipe/holistic. -``` - -If you would rather not stage these locally, you can rely on a CDN (e.g., -[jsDelivr](https://www.jsdelivr.com/)). This will allow you to add scripts -directly to your HTML: - -``` - - - - -``` - -Note: You can specify version numbers to both NPM and jsdelivr. They are -structured as `..`. To prevent breaking changes from -affecting your work, restrict your request to a `` number. e.g., -`@mediapipe/holistic@0.1`. [Ho-pg]: ../solutions/holistic#javascript-solution-api [F-pg]: ../solutions/face_mesh#javascript-solution-api From 4a1a5666958a6c1083f228106d46a95c6a32b86b Mon Sep 17 00:00:00 2001 From: Dweep Joshipura Date: Thu, 21 Jan 2021 09:16:17 +0530 Subject: [PATCH 04/12] Add holistic_remote, thanks to @afogel --- docs/getting_started/javascript.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/getting_started/javascript.md b/docs/getting_started/javascript.md index f0b9fde12..249576af9 100644 --- a/docs/getting_started/javascript.md +++ b/docs/getting_started/javascript.md @@ -23,6 +23,8 @@ Solution | NPM Package | Example [Pose][P-pg] | [@mediapipe/pose][P-npm] | [mediapipe.dev/demo/pose][P-demo] [Holistic][Ho-pg] | [@mediapipe/holistic][Ho-npm] | [mediapipe.dev/demo/holistic][Ho-demo] +Note: Also see the [Holistic Remote demo][Ho-Re], which explores more options in the Holistic Demo. + Click on a solution link above for more information, including API and code snippets. @@ -81,6 +83,7 @@ affecting your work, restrict your request to a `` number. e.g., [H-pen]: https://code.mediapipe.dev/codepen/hands [P-pen]: https://code.mediapipe.dev/codepen/pose [Ho-demo]: https://mediapipe.dev/demo/holistic +[Ho-Re]: https://mediapipe.dev/demo/holistic_remote [F-demo]: https://mediapipe.dev/demo/face_mesh [H-demo]: https://mediapipe.dev/demo/hands [P-demo]: https://mediapipe.dev/demo/pose From 24c0280946e3d5e211857190ac95066cf7e93afa Mon Sep 17 00:00:00 2001 From: Dweep Joshipura Date: Thu, 21 Jan 2021 09:32:54 +0530 Subject: [PATCH 05/12] Fixed pg links as per @takahirox #1456 --- docs/getting_started/javascript.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/getting_started/javascript.md b/docs/getting_started/javascript.md index 249576af9..3601d5b4d 100644 --- a/docs/getting_started/javascript.md +++ b/docs/getting_started/javascript.md @@ -63,10 +63,10 @@ structured as `..`. To prevent breaking changes from affecting your work, restrict your request to a `` number. e.g., `@mediapipe/holistic@0.1`. -[Ho-pg]: ../solutions/holistic#javascript-solution-api -[F-pg]: ../solutions/face_mesh#javascript-solution-api -[H-pg]: ../solutions/hands#javascript-solution-api -[P-pg]: ../solutions/pose#javascript-solution-api +[Ho-pg]: ../solutions/holistic.md#javascript-solution-api +[F-pg]: ../solutions/face_mesh.md#javascript-solution-api +[H-pg]: ../solutions/hands.md#javascript-solution-api +[P-pg]: ../solutions/pose.md#javascript-solution-api [Ho-npm]: https://www.npmjs.com/package/@mediapipe/holistic [F-npm]: https://www.npmjs.com/package/@mediapipe/face_mesh [H-npm]: https://www.npmjs.com/package/@mediapipe/hands From 61c71c2d7efe878ff4db1e530052359435f60cdf Mon Sep 17 00:00:00 2001 From: Dweep Joshipura Date: Wed, 17 Feb 2021 10:25:50 +0530 Subject: [PATCH 06/12] Add FPS --- docs/getting_started/js_utils.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/docs/getting_started/js_utils.md b/docs/getting_started/js_utils.md index 84ae9860d..a5f469a61 100644 --- a/docs/getting_started/js_utils.md +++ b/docs/getting_started/js_utils.md @@ -79,8 +79,17 @@ camera.start(); ``` ### Control Panel Utilities To showcase and monitor the model in elegant ways, MediaPipe has this Control Panel module. -This panel is also convenient to use and set up. +This panel is also convenient to use and set up. +One of the most interesting features is the built-in `FPS` object that allows us to monitor the FPS or Frames per Second. ```js +// Initialize the FPS control +const fpsControl = new FPS(); +// In the event handler function onResults() in the [Hands demo][Ho-pg]() +function onResults(results) { + // Tick the FPS, i.e., set Incrementation breaks + fpsControl.tick(); + // then do the required processing on the results object + // Pass in a
new ControlPanel(controlsElement, { // Whether to invert the camera, defaults to false @@ -95,6 +104,8 @@ new ControlPanel(controlsElement, { .add([ // a StaticText is simply a label new StaticText({title: 'MediaPipe Hands'}), + // Add the FPS control to the Control Panel + fpsControl, // a Toggle is one with only 2 options (i.e., true or false) new Toggle({title: 'Selfie Mode', field: 'selfieMode'}), // A slider can have multiple options. This will have the options 1,2,3 and 4. From f3adb04c76d477da39fd3d05f7631da8dbd70fde Mon Sep 17 00:00:00 2001 From: Dweep Joshipura Date: Wed, 17 Feb 2021 10:30:39 +0530 Subject: [PATCH 07/12] Update code --- docs/getting_started/js_utils.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/getting_started/js_utils.md b/docs/getting_started/js_utils.md index a5f469a61..bf8c64926 100644 --- a/docs/getting_started/js_utils.md +++ b/docs/getting_started/js_utils.md @@ -81,10 +81,11 @@ camera.start(); To showcase and monitor the model in elegant ways, MediaPipe has this Control Panel module. This panel is also convenient to use and set up. One of the most interesting features is the built-in `FPS` object that allows us to monitor the FPS or Frames per Second. +> Note that the event handler function mentioned in the code is [this][Ho-pg] ```js // Initialize the FPS control const fpsControl = new FPS(); -// In the event handler function onResults() in the [Hands demo][Ho-pg]() +// In the event handler function onResults() function onResults(results) { // Tick the FPS, i.e., set Incrementation breaks fpsControl.tick(); From 4e09410d145e0536fd61a7095ec3ee044a2f139d Mon Sep 17 00:00:00 2001 From: Dweep Joshipura Date: Wed, 17 Feb 2021 10:32:02 +0530 Subject: [PATCH 08/12] Update js_utils.md --- docs/getting_started/js_utils.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting_started/js_utils.md b/docs/getting_started/js_utils.md index bf8c64926..edb902cb0 100644 --- a/docs/getting_started/js_utils.md +++ b/docs/getting_started/js_utils.md @@ -81,7 +81,7 @@ camera.start(); To showcase and monitor the model in elegant ways, MediaPipe has this Control Panel module. This panel is also convenient to use and set up. One of the most interesting features is the built-in `FPS` object that allows us to monitor the FPS or Frames per Second. -> Note that the event handler function mentioned in the code is [this][Ho-pg] +> Note that the event handler function mentioned in the code is [referred here][H-pg] ```js // Initialize the FPS control const fpsControl = new FPS(); From 718d05c4acb4aa5b8dedcf83a55a820935a2ebee Mon Sep 17 00:00:00 2001 From: Dweep Joshipura Date: Wed, 17 Feb 2021 10:33:29 +0530 Subject: [PATCH 09/12] Update js_utils.md --- docs/getting_started/js_utils.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/getting_started/js_utils.md b/docs/getting_started/js_utils.md index edb902cb0..3a0a6d302 100644 --- a/docs/getting_started/js_utils.md +++ b/docs/getting_started/js_utils.md @@ -136,10 +136,10 @@ new ControlPanel(controlsElement, { ``` -[Ho-pg]: ../solutions/holistic#javascript-solution-api -[F-pg]: ../solutions/face_mesh#javascript-solution-api -[H-pg]: ../solutions/hands#javascript-solution-api -[P-pg]: ../solutions/pose#javascript-solution-api +[Ho-pg]: ../solutions/holistic.md#javascript-solution-api +[F-pg]: ../solutions/face_mesh.md#javascript-solution-api +[H-pg]: ../solutions/hands.md#javascript-solution-api +[P-pg]: ../solutions/pose.md#javascript-solution-api [Ho-npm]: https://www.npmjs.com/package/@mediapipe/holistic [F-npm]: https://www.npmjs.com/package/@mediapipe/face_mesh [H-npm]: https://www.npmjs.com/package/@mediapipe/hands From a49442fa968f74f6603de58db57ef7b689317c7d Mon Sep 17 00:00:00 2001 From: Dweep Joshipura Date: Wed, 17 Feb 2021 10:36:04 +0530 Subject: [PATCH 10/12] Update hands.md --- docs/solutions/hands.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/solutions/hands.md b/docs/solutions/hands.md index 3d07411c2..a6eb70e05 100644 --- a/docs/solutions/hands.md +++ b/docs/solutions/hands.md @@ -317,7 +317,7 @@ Supported configuration options: ``` - +
```javascript