2020-06-06 01:49:27 +02:00
|
|
|
|
---
|
|
|
|
|
layout: default
|
|
|
|
|
title: Graphs
|
|
|
|
|
parent: Framework Concepts
|
|
|
|
|
nav_order: 2
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# Graphs
|
|
|
|
|
{: .no_toc }
|
|
|
|
|
|
|
|
|
|
1. TOC
|
|
|
|
|
{:toc}
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## GraphConfig
|
|
|
|
|
|
|
|
|
|
A `GraphConfig` is a specification that describes the topology and functionality
|
|
|
|
|
of a MediaPipe graph. In the specification, a node in the graph represents an
|
|
|
|
|
instance of a particular calculator. All the necessary configurations of the
|
|
|
|
|
node, such its type, inputs and outputs must be described in the specification.
|
|
|
|
|
Description of the node can also include several optional fields, such as
|
|
|
|
|
node-specific options, input policy and executor, discussed in
|
|
|
|
|
[Synchronization](synchronization.md).
|
|
|
|
|
|
|
|
|
|
`GraphConfig` has several other fields to configure the global graph-level
|
|
|
|
|
settings, eg, graph executor configs, number of threads, and maximum queue size
|
|
|
|
|
of input streams. Several graph-level settings are useful for tuning the
|
|
|
|
|
performance of the graph on different platforms (eg, desktop v.s. mobile). For
|
|
|
|
|
instance, on mobile, attaching a heavy model-inference calculator to a separate
|
|
|
|
|
executor can improve the performance of a real-time application since this
|
|
|
|
|
enables thread locality.
|
|
|
|
|
|
|
|
|
|
Below is a trivial `GraphConfig` example where we have series of passthrough
|
|
|
|
|
calculators :
|
|
|
|
|
|
|
|
|
|
```proto
|
|
|
|
|
# This graph named main_pass_throughcals_nosubgraph.pbtxt contains 4
|
|
|
|
|
# passthrough calculators.
|
|
|
|
|
input_stream: "in"
|
|
|
|
|
node {
|
|
|
|
|
calculator: "PassThroughCalculator"
|
|
|
|
|
input_stream: "in"
|
|
|
|
|
output_stream: "out1"
|
|
|
|
|
}
|
|
|
|
|
node {
|
|
|
|
|
calculator: "PassThroughCalculator"
|
|
|
|
|
input_stream: "out1"
|
|
|
|
|
output_stream: "out2"
|
|
|
|
|
}
|
|
|
|
|
node {
|
|
|
|
|
calculator: "PassThroughCalculator"
|
|
|
|
|
input_stream: "out2"
|
|
|
|
|
output_stream: "out3"
|
|
|
|
|
}
|
|
|
|
|
node {
|
|
|
|
|
calculator: "PassThroughCalculator"
|
|
|
|
|
input_stream: "out3"
|
|
|
|
|
output_stream: "out4"
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Subgraph
|
|
|
|
|
|
|
|
|
|
To modularize a `CalculatorGraphConfig` into sub-modules and assist with re-use
|
|
|
|
|
of perception solutions, a MediaPipe graph can be defined as a `Subgraph`. The
|
|
|
|
|
public interface of a subgraph consists of a set of input and output streams
|
|
|
|
|
similar to a calculator's public interface. The subgraph can then be included in
|
|
|
|
|
an `CalculatorGraphConfig` as if it were a calculator. When a MediaPipe graph is
|
|
|
|
|
loaded from a `CalculatorGraphConfig`, each subgraph node is replaced by the
|
|
|
|
|
corresponding graph of calculators. As a result, the semantics and performance
|
|
|
|
|
of the subgraph is identical to the corresponding graph of calculators.
|
|
|
|
|
|
|
|
|
|
Below is an example of how to create a subgraph named `TwoPassThroughSubgraph`.
|
|
|
|
|
|
|
|
|
|
1. Defining the subgraph.
|
|
|
|
|
|
|
|
|
|
```proto
|
|
|
|
|
# This subgraph is defined in two_pass_through_subgraph.pbtxt
|
|
|
|
|
# and is registered as "TwoPassThroughSubgraph"
|
|
|
|
|
|
|
|
|
|
type: "TwoPassThroughSubgraph"
|
|
|
|
|
input_stream: "out1"
|
|
|
|
|
output_stream: "out3"
|
|
|
|
|
|
|
|
|
|
node {
|
2021-05-05 03:30:15 +02:00
|
|
|
|
calculator: "PassThroughCalculator"
|
2020-06-06 01:49:27 +02:00
|
|
|
|
input_stream: "out1"
|
|
|
|
|
output_stream: "out2"
|
|
|
|
|
}
|
|
|
|
|
node {
|
2021-05-05 03:30:15 +02:00
|
|
|
|
calculator: "PassThroughCalculator"
|
2020-06-06 01:49:27 +02:00
|
|
|
|
input_stream: "out2"
|
|
|
|
|
output_stream: "out3"
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The public interface to the subgraph consists of:
|
|
|
|
|
|
|
|
|
|
* Graph input streams
|
|
|
|
|
* Graph output streams
|
|
|
|
|
* Graph input side packets
|
|
|
|
|
* Graph output side packets
|
|
|
|
|
|
|
|
|
|
2. Register the subgraph using BUILD rule `mediapipe_simple_subgraph`. The
|
|
|
|
|
parameter `register_as` defines the component name for the new subgraph.
|
|
|
|
|
|
|
|
|
|
```proto
|
|
|
|
|
# Small section of BUILD file for registering the "TwoPassThroughSubgraph"
|
|
|
|
|
# subgraph for use by main graph main_pass_throughcals.pbtxt
|
|
|
|
|
|
|
|
|
|
mediapipe_simple_subgraph(
|
|
|
|
|
name = "twopassthrough_subgraph",
|
|
|
|
|
graph = "twopassthrough_subgraph.pbtxt",
|
|
|
|
|
register_as = "TwoPassThroughSubgraph",
|
|
|
|
|
deps = [
|
|
|
|
|
"//mediapipe/calculators/core:pass_through_calculator",
|
|
|
|
|
"//mediapipe/framework:calculator_graph",
|
|
|
|
|
],
|
|
|
|
|
)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
3. Use the subgraph in the main graph.
|
|
|
|
|
|
|
|
|
|
```proto
|
|
|
|
|
# This main graph is defined in main_pass_throughcals.pbtxt
|
|
|
|
|
# using subgraph called "TwoPassThroughSubgraph"
|
|
|
|
|
|
|
|
|
|
input_stream: "in"
|
|
|
|
|
node {
|
|
|
|
|
calculator: "PassThroughCalculator"
|
|
|
|
|
input_stream: "in"
|
|
|
|
|
output_stream: "out1"
|
|
|
|
|
}
|
|
|
|
|
node {
|
|
|
|
|
calculator: "TwoPassThroughSubgraph"
|
|
|
|
|
input_stream: "out1"
|
|
|
|
|
output_stream: "out3"
|
|
|
|
|
}
|
|
|
|
|
node {
|
|
|
|
|
calculator: "PassThroughCalculator"
|
|
|
|
|
input_stream: "out3"
|
|
|
|
|
output_stream: "out4"
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
2022-10-04 07:04:29 +02:00
|
|
|
|
## Graph Options
|
|
|
|
|
|
|
|
|
|
It is possible to specify a "graph options" protobuf for a MediaPipe graph
|
|
|
|
|
similar to the [`Calculator Options`](calculators.md#calculator-options)
|
|
|
|
|
protobuf specified for a MediaPipe calculator. These "graph options" can be
|
|
|
|
|
specified where a graph is invoked, and used to populate calculator options and
|
|
|
|
|
subgraph options within the graph.
|
|
|
|
|
|
|
|
|
|
In a CalculatorGraphConfig, graph options can be specified for a subgraph
|
|
|
|
|
exactly like calculator options, as shown below:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
node {
|
|
|
|
|
calculator: "FlowLimiterCalculator"
|
|
|
|
|
input_stream: "image"
|
|
|
|
|
output_stream: "throttled_image"
|
|
|
|
|
node_options: {
|
|
|
|
|
[type.googleapis.com/mediapipe.FlowLimiterCalculatorOptions] {
|
|
|
|
|
max_in_flight: 1
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
node {
|
|
|
|
|
calculator: "FaceDetectionSubgraph"
|
|
|
|
|
input_stream: "IMAGE:throttled_image"
|
|
|
|
|
node_options: {
|
|
|
|
|
[type.googleapis.com/mediapipe.FaceDetectionOptions] {
|
|
|
|
|
tensor_width: 192
|
|
|
|
|
tensor_height: 192
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
In a CalculatorGraphConfig, graph options can be accepted and used to populate
|
|
|
|
|
calculator options, as shown below:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
graph_options: {
|
|
|
|
|
[type.googleapis.com/mediapipe.FaceDetectionOptions] {}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
node: {
|
|
|
|
|
calculator: "ImageToTensorCalculator"
|
|
|
|
|
input_stream: "IMAGE:multi_backend_image"
|
|
|
|
|
node_options: {
|
|
|
|
|
[type.googleapis.com/mediapipe.ImageToTensorCalculatorOptions] {
|
|
|
|
|
keep_aspect_ratio: true
|
|
|
|
|
border_mode: BORDER_ZERO
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
option_value: "output_tensor_width:options/tensor_width"
|
|
|
|
|
option_value: "output_tensor_height:options/tensor_height"
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
node {
|
|
|
|
|
calculator: "InferenceCalculator"
|
|
|
|
|
node_options: {
|
|
|
|
|
[type.googleapis.com/mediapipe.InferenceCalculatorOptions] {}
|
|
|
|
|
}
|
|
|
|
|
option_value: "delegate:options/delegate"
|
|
|
|
|
option_value: "model_path:options/model_path"
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
In this example, the `FaceDetectionSubgraph` accepts graph option protobuf
|
|
|
|
|
`FaceDetectionOptions`. The `FaceDetectionOptions` is used to define some field
|
|
|
|
|
values in the calculator options `ImageToTensorCalculatorOptions` and some field
|
|
|
|
|
values in the subgraph options `InferenceCalculatorOptions`. The field values
|
|
|
|
|
are defined using the `option_value:` syntax.
|
|
|
|
|
|
|
|
|
|
In the `CalculatorGraphConfig::Node` protobuf, the fields `node_options:` and
|
|
|
|
|
`option_value:` together define the option values for a calculator such as
|
|
|
|
|
`ImageToTensorCalculator`. The `node_options:` field defines a set of literal
|
|
|
|
|
constant values using the text protobuf syntax. Each `option_value:` field
|
|
|
|
|
defines the value for one protobuf field using information from the enclosing
|
|
|
|
|
graph, specifically from field values of the graph options of the enclosing
|
|
|
|
|
graph. In the example above, the `option_value:`
|
|
|
|
|
`"output_tensor_width:options/tensor_width"` defines the field
|
|
|
|
|
`ImageToTensorCalculatorOptions.output_tensor_width` using the value of
|
|
|
|
|
`FaceDetectionOptions.tensor_width`.
|
|
|
|
|
|
|
|
|
|
The syntax of `option_value:` is similar to the syntax of `input_stream:`. The
|
|
|
|
|
syntax is `option_value: "LHS:RHS"`. The LHS identifies a calculator option
|
|
|
|
|
field and the RHS identifies a graph option field. More specifically, the LHS
|
|
|
|
|
and RHS each consists of a series of protobuf field names identifying nested
|
|
|
|
|
protobuf messages and fields separated by '/'. This is known as the "ProtoPath"
|
|
|
|
|
syntax. Nested messages that are referenced in the LHS or RHS must already be
|
|
|
|
|
defined in the enclosing protobuf in order to be traversed using
|
|
|
|
|
`option_value:`.
|
|
|
|
|
|
2020-06-06 01:49:27 +02:00
|
|
|
|
## Cycles
|
|
|
|
|
|
|
|
|
|
<!-- TODO: add discussion of PreviousLoopbackCalculator -->
|
|
|
|
|
|
|
|
|
|
By default, MediaPipe requires calculator graphs to be acyclic and treats cycles
|
|
|
|
|
in a graph as errors. If a graph is intended to have cycles, the cycles need to
|
|
|
|
|
be annotated in the graph config. This page describes how to do that.
|
|
|
|
|
|
|
|
|
|
NOTE: The current approach is experimental and subject to change. We welcome
|
|
|
|
|
your feedback.
|
|
|
|
|
|
|
|
|
|
Please use the `CalculatorGraphTest.Cycle` unit test in
|
|
|
|
|
`mediapipe/framework/calculator_graph_test.cc` as sample code. Shown
|
|
|
|
|
below is the cyclic graph in the test. The `sum` output of the adder is the sum
|
|
|
|
|
of the integers generated by the integer source calculator.
|
|
|
|
|
|
2022-09-06 23:29:51 +02:00
|
|
|
|
![a cyclic graph that adds a stream of integers](https://mediapipe.dev/images/cyclic_integer_sum_graph.svg "A cyclic graph")
|
2020-06-06 01:49:27 +02:00
|
|
|
|
|
|
|
|
|
This simple graph illustrates all the issues in supporting cyclic graphs.
|
|
|
|
|
|
|
|
|
|
### Back Edge Annotation
|
|
|
|
|
|
|
|
|
|
We require that an edge in each cycle be annotated as a back edge. This allows
|
|
|
|
|
MediaPipe’s topological sort to work, after removing all the back edges.
|
|
|
|
|
|
|
|
|
|
There are usually multiple ways to select the back edges. Which edges are marked
|
|
|
|
|
as back edges affects which nodes are considered as upstream and which nodes are
|
|
|
|
|
considered as downstream, which in turn affects the priorities MediaPipe assigns
|
|
|
|
|
to the nodes.
|
|
|
|
|
|
|
|
|
|
For example, the `CalculatorGraphTest.Cycle` test marks the `old_sum` edge as a
|
|
|
|
|
back edge, so the Delay node is considered as a downstream node of the adder
|
|
|
|
|
node and is given a higher priority. Alternatively, we could mark the `sum`
|
|
|
|
|
input to the delay node as the back edge, in which case the delay node would be
|
|
|
|
|
considered as an upstream node of the adder node and is given a lower priority.
|
|
|
|
|
|
|
|
|
|
### Initial Packet
|
|
|
|
|
|
|
|
|
|
For the adder calculator to be runnable when the first integer from the integer
|
|
|
|
|
source arrives, we need an initial packet, with value 0 and with the same
|
|
|
|
|
timestamp, on the `old_sum` input stream to the adder. This initial packet
|
|
|
|
|
should be output by the delay calculator in the `Open()` method.
|
|
|
|
|
|
|
|
|
|
### Delay in a Loop
|
|
|
|
|
|
|
|
|
|
Each loop should incur a delay to align the previous `sum` output with the next
|
|
|
|
|
integer input. This is also done by the delay node. So the delay node needs to
|
|
|
|
|
know the following about the timestamps of the integer source calculator:
|
|
|
|
|
|
|
|
|
|
* The timestamp of the first output.
|
|
|
|
|
|
|
|
|
|
* The timestamp delta between successive outputs.
|
|
|
|
|
|
|
|
|
|
We plan to add an alternative scheduling policy that only cares about packet
|
|
|
|
|
ordering and ignores packet timestamps, which will eliminate this inconvenience.
|
|
|
|
|
|
|
|
|
|
### Early Termination of a Calculator When One Input Stream is Done
|
|
|
|
|
|
|
|
|
|
By default, MediaPipe calls the `Close()` method of a non-source calculator when
|
|
|
|
|
all of its input streams are done. In the example graph, we want to stop the
|
|
|
|
|
adder node as soon as the integer source is done. This is accomplished by
|
|
|
|
|
configuring the adder node with an alternative input stream handler,
|
|
|
|
|
`EarlyCloseInputStreamHandler`.
|
|
|
|
|
|
|
|
|
|
### Relevant Source Code
|
|
|
|
|
|
|
|
|
|
#### Delay Calculator
|
|
|
|
|
|
|
|
|
|
Note the code in `Open()` that outputs the initial packet and the code in
|
|
|
|
|
`Process()` that adds a (unit) delay to input packets. As noted above, this
|
|
|
|
|
delay node assumes that its output stream is used alongside an input stream with
|
|
|
|
|
packet timestamps 0, 1, 2, 3, ...
|
|
|
|
|
|
|
|
|
|
```c++
|
|
|
|
|
class UnitDelayCalculator : public Calculator {
|
|
|
|
|
public:
|
2021-02-27 09:21:16 +01:00
|
|
|
|
static absl::Status FillExpectations(
|
2020-06-06 01:49:27 +02:00
|
|
|
|
const CalculatorOptions& extendable_options, PacketTypeSet* inputs,
|
|
|
|
|
PacketTypeSet* outputs, PacketTypeSet* input_side_packets) {
|
|
|
|
|
inputs->Index(0)->Set<int>("An integer.");
|
|
|
|
|
outputs->Index(0)->Set<int>("The input delayed by one time unit.");
|
2021-02-27 09:21:16 +01:00
|
|
|
|
return absl::OkStatus();
|
2020-06-06 01:49:27 +02:00
|
|
|
|
}
|
|
|
|
|
|
2021-02-27 09:21:16 +01:00
|
|
|
|
absl::Status Open() final {
|
2020-06-06 01:49:27 +02:00
|
|
|
|
Output()->Add(new int(0), Timestamp(0));
|
2021-02-27 09:21:16 +01:00
|
|
|
|
return absl::OkStatus();
|
2020-06-06 01:49:27 +02:00
|
|
|
|
}
|
|
|
|
|
|
2021-02-27 09:21:16 +01:00
|
|
|
|
absl::Status Process() final {
|
2020-06-06 01:49:27 +02:00
|
|
|
|
const Packet& packet = Input()->Value();
|
|
|
|
|
Output()->AddPacket(packet.At(packet.Timestamp().NextAllowedInStream()));
|
2021-02-27 09:21:16 +01:00
|
|
|
|
return absl::OkStatus();
|
2020-06-06 01:49:27 +02:00
|
|
|
|
}
|
|
|
|
|
};
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
#### Graph Config
|
|
|
|
|
|
|
|
|
|
Note the `back_edge` annotation and the alternative `input_stream_handler`.
|
|
|
|
|
|
|
|
|
|
```proto
|
|
|
|
|
node {
|
|
|
|
|
calculator: 'GlobalCountSourceCalculator'
|
|
|
|
|
input_side_packet: 'global_counter'
|
|
|
|
|
output_stream: 'integers'
|
|
|
|
|
}
|
|
|
|
|
node {
|
|
|
|
|
calculator: 'IntAdderCalculator'
|
|
|
|
|
input_stream: 'integers'
|
|
|
|
|
input_stream: 'old_sum'
|
|
|
|
|
input_stream_info: {
|
|
|
|
|
tag_index: ':1' # 'old_sum'
|
|
|
|
|
back_edge: true
|
|
|
|
|
}
|
|
|
|
|
output_stream: 'sum'
|
|
|
|
|
input_stream_handler {
|
|
|
|
|
input_stream_handler: 'EarlyCloseInputStreamHandler'
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
node {
|
|
|
|
|
calculator: 'UnitDelayCalculator'
|
|
|
|
|
input_stream: 'sum'
|
|
|
|
|
output_stream: 'old_sum'
|
|
|
|
|
}
|
|
|
|
|
```
|