mediapipe/mediapipe/framework/api2

Experimental new APIs

This directory defines new APIs for MediaPipe:

• Node API, an update to the Calculator API for defining MediaPipe components.
• Builder API, for assembling CalculatorGraphConfigs with C++, as an alternative to using the proto API directly.

The code is working, and the new APIs interoperate fully with the existing framework code. They are considered a work in progress, but are being released now so we can begin adopting them in our calculators.

Developers are welcome to try out these APIs as early adopters, but should expect breaking changes. The placement of this code under the mediapipe::api2 namespace is not final.

Node API

This API can be used to define calculators. It is designed to be more type-safe and less verbose than the original API.

Input/output ports (streams and side packets) can now be declared as typed constants, instead of using plain strings for access.

For example, instead of

constexpr char kSelectTag[] = "SELECT";
if (cc->Inputs().HasTag(kSelectTag)) {
  cc->Inputs().Tag(kSelectTag).Set<int>();
}

you can write

static constexpr Input<int>::Optional kSelect{"SELECT"};

Instead of setting up the contract procedurally in GetContract, add ports to the contract declaratively, as follows:

MEDIAPIPE_NODE_CONTRACT(kInput, kOutput);

To access an input in Process, instead of

int select = cc->Inputs().Tag(kSelectTag).Get<int>();

write

int select = kSelectTag(cc).Get();  // alternative: *kSelectTag(cc)

Sets of multiple ports can be declared with ::Multiple. Note, also, that a tag string must always be provided when declaring a port; use "" for untagged ports. For example:

for (int i = 0; i < cc->Inputs().NumEntries(); ++i) {
  cc->Inputs().Index(i).SetAny();
}

becomes

static constexpr Input<AnyType>::Multiple kIn{""};

For output ports, the payload can be passed directly to the Send method. For example, instead of

cc->Outputs().Index(0).Add(
    new std::pair<Packet, Packet>(cc->Inputs().Index(0).Value(),
                                  cc->Inputs().Index(1).Value()),
    cc->InputTimestamp());

you can write

kPair(cc).Send({kIn(cc)[0].packet(), kIn(cc)[1].packet()});

The input timestamp is propagated to the outputs by default. If your calculator wants to alter timestamps, it must add a TimestampChange entry to its contract declaration. For example:

MEDIAPIPE_NODE_CONTRACT(kMain, kLoop, kPrevLoop,
                        StreamHandler("ImmediateInputStreamHandler"),
                        TimestampChange::Arbitrary());

Several calculators in calculators/core and calculators/tensor have been updated to use this API. Reference them for more examples.

More complete documentation will be provided in the future.

Builder API

Documentation will be provided in the future.