mediapipe/mediapipe/tasks/web/vision
Sebastian Schmidt 61854dc6a3 Create Pose Detector Web API
PiperOrigin-RevId: 526672533
2023-04-24 09:53:05 -07:00
..
core Fixes the typos in tasks internal files. 2023-04-21 09:46:39 -07:00
face_detector FaceDetector Web API 2023-04-04 11:23:00 -07:00
face_landmarker Extract shared types to create and test landmarks 2023-04-19 15:37:42 -07:00
face_stylizer Remove resizing and rotation from face stylizer's postprocessing step. 2023-04-14 19:37:52 -07:00
gesture_recognizer Add HAND_CONNECTIONS to HandLandmarker and GestureRecognizer 2023-04-17 14:31:40 -07:00
hand_landmarker Extract shared types to create and test landmarks 2023-04-19 15:37:42 -07:00
image_classifier Fixes the typos in tasks internal files. 2023-04-21 09:46:39 -07:00
image_embedder Allow users to pass canvas element 2023-03-23 08:46:21 -07:00
image_segmenter This will fix the multiple typos in the new tasks internal files 2023-04-20 10:43:35 -07:00
interactive_segmenter Fix Typo 2023-04-20 15:15:13 -07:00
object_detector Merge pull request #4235 from priankakariatyml:ios-object-detection-containers 2023-04-03 15:12:06 -07:00
pose_landmarker Create Pose Detector Web API 2023-04-24 09:53:05 -07:00
BUILD Create Pose Detector Web API 2023-04-24 09:53:05 -07:00
index.ts Create Pose Detector Web API 2023-04-24 09:53:05 -07:00
README.md Create Pose Detector Web API 2023-04-24 09:53:05 -07:00
types.ts Create Pose Detector Web API 2023-04-24 09:53:05 -07:00

MediaPipe Tasks Vision Package

This package contains the vision tasks for MediaPipe.

Face Detection

The MediaPipe Face Detector task lets you detect the presence and location of faces within images or videos.

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const faceDetector = await FaceDetector.createFromModelPath(vision,
    "https://storage.googleapis.com/mediapipe-tasks/face_detector/face_detection_short_range.tflite"
);
const image = document.getElementById("image") as HTMLImageElement;
const detections = faceDetector.detect(image);

Face Landmark Detection

The MediaPipe Face Landmarker task lets you detect the landmarks of faces in an image. You can use this Task to localize key points of a face and render visual effects over the faces.

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const faceLandmarker = await FaceLandmarker.createFromModelPath(vision,
    "https://storage.googleapis.com/mediapipe-tasks/face_landmarker/face_landmarker.task"
);
const image = document.getElementById("image") as HTMLImageElement;
const landmarks = faceLandmarker.detect(image);

Face Stylizer

The MediaPipe Face Stylizer lets you perform face stylization on images.

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const faceStylizer = await FaceStylizer.createFromModelPath(vision,
    "https://storage.googleapis.com/mediapipe-tasks/face_stylizer/face_stylizer_with_metadata.tflite"
);
const image = document.getElementById("image") as HTMLImageElement;
const stylizedImage = faceStylizer.stylize(image);

Gesture Recognition

The MediaPipe Gesture Recognizer task lets you recognize hand gestures in real time, and provides the recognized hand gesture results along with the landmarks of the detected hands. You can use this task to recognize specific hand gestures from a user, and invoke application features that correspond to those gestures.

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const gestureRecognizer = await GestureRecognizer.createFromModelPath(vision,
    "https://storage.googleapis.com/mediapipe-tasks/gesture_recognizer/gesture_recognizer.task"
);
const image = document.getElementById("image") as HTMLImageElement;
const recognitions = gestureRecognizer.recognize(image);

Hand Landmark Detection

The MediaPipe Hand Landmarker task lets you detect the landmarks of the hands in an image. You can use this Task to localize key points of the hands and render visual effects over the hands.

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const handLandmarker = await HandLandmarker.createFromModelPath(vision,
    "https://storage.googleapis.com/mediapipe-tasks/hand_landmarker/hand_landmarker.task"
);
const image = document.getElementById("image") as HTMLImageElement;
const landmarks = handLandmarker.detect(image);

For more information, refer to the Handlandmark Detection documentation.

Image Classification

The MediaPipe Image Classifier task lets you perform classification on images. You can use this task to identify what an image represents among a set of categories defined at training time.

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const imageClassifier = await ImageClassifier.createFromModelPath(vision,
    "https://storage.googleapis.com/mediapipe-tasks/image_classifier/efficientnet_lite0_uint8.tflite"
);
const image = document.getElementById("image") as HTMLImageElement;
const classifications = imageClassifier.classify(image);

For more information, refer to the Image Classification documentation.

Image Segmentation

The MediaPipe Image Segmenter lets you segment an image into categories.

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const imageSegmenter = await ImageSegmenter.createFromModelPath(vision,
    "https://storage.googleapis.com/mediapipe-tasks/image_segmenter/selfie_segmentation.tflite"
);
const image = document.getElementById("image") as HTMLImageElement;
imageSegmenter.segment(image, (masks, width, height) => {
  ...
});

Interactive Segmentation

The MediaPipe Interactive Segmenter lets you select a region of interest to segment an image by.

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const interactiveSegmenter = await InteractiveSegmenter.createFromModelPath(
    vision,
    "https://storage.googleapis.com/mediapipe-tasks/interactive_segmenter/ptm_512_hdt_ptm_woid.tflite
);
const image = document.getElementById("image") as HTMLImageElement;
interactiveSegmenter.segment(image, { keypoint: { x: 0.1, y: 0.2 } },
    (masks, width, height) => { ... }
);

Object Detection

The MediaPipe Object Detector task lets you detect the presence and location of multiple classes of objects within images or videos.

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const objectDetector = await ObjectDetector.createFromModelPath(vision,
    "https://storage.googleapis.com/mediapipe-tasks/object_detector/efficientdet_lite0_uint8.tflite"
);
const image = document.getElementById("image") as HTMLImageElement;
const detections = objectDetector.detect(image);

For more information, refer to the Object Detector documentation.

Pose Landmark Detection

The MediaPipe Pose Landmarker task lets you detect the landmarks of body poses in an image. You can use this Task to localize key points of a pose and render visual effects over the body.

const vision = await FilesetResolver.forVisionTasks(
    "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const poseLandmarker = await PoseLandmarker.createFromModelPath(vision,
    "model.task"
);
const image = document.getElementById("image") as HTMLImageElement;
const landmarks = poseLandmarker.detect(image);