8.0 KiB
| layout | title | parent | nav_order |
|---|---|---|---|
| default | FAQ | Getting Started | 9 |
FAQ
{: .no_toc }
- TOC {:toc}
How to convert ImageFrames and GpuBuffers
The Calculators ImageFrameToGpuBufferCalculator and
GpuBufferToImageFrameCalculator convert back and forth between packets of
type ImageFrame and GpuBuffer. ImageFrame refers to image data in
CPU memory in any of a number of bitmap image formats. GpuBuffer refers to
image data in GPU memory. You can find more detail in the Framework Concepts
section
GpuBuffer to ImageFrame Converters.
You can see an example in:
How to visualize perception results
The AnnotationOverlayCalculator allows perception results, such as bounding
boxes, arrows, and ovals, to be superimposed on the video frames aligned with
the recognized objects. The results can be displayed in a diagnostic window when
running on a workstation, or in a texture frame when running on device. You can
see an example use of AnnotationOverlayCalculator in:
How to run calculators in parallel
Within a calculator graph, MediaPipe routinely runs separate calculator nodes in parallel. MediaPipe maintains a pool of threads, and runs each calculator as soon as a thread is available and all of it's inputs are ready. Each calculator instance is only run for one set of inputs at a time, so most calculators need only to be thread-compatible and not thread-safe.
In order to enable one calculator to process multiple inputs in parallel, there are two possible approaches:
- Define multiple calulator nodes and dispatch input packets to all nodes.
- Make the calculator thread-safe and configure its
max_in_flightsetting.
The first approach can be followed using the calculators designed to distribute
packets across other calculators, such as RoundRobinDemuxCalculator. A
single RoundRobinDemuxCalculator can distribute successive packets across
several identically configured ScaleImageCalculator nodes.
The second approach allows up to max_in_flight invocations of the
CalculatorBase::Process method on the same calculator node. The output
packets from CalculatorBase::Process are automatically ordered by timestamp
before they are passed along to downstream calculators.
With either approach, you must be aware that the calculator running in parallel cannot maintain internal state in the same way as a normal sequential calculator.
Output timestamps when using ImmediateInputStreamHandler
The ImmediateInputStreamHandler delivers each packet as soon as it arrives
at an input stream. As a result, it can deliver a packet
with a higher timestamp from one input stream before delivering a packet with a
lower timestamp from a different input stream. If these input timestamps are
both used for packets sent to one output stream, that output stream will
complain that the timestamps are not monotonically increasing. In order to
remedy this, the calculator must take care to output a packet only after
processing is complete for its timestamp. This could be accomplished by waiting
until input packets have been received from all inputstreams for that timestamp,
or by ignoring a packet that arrives with a timestamp that has already been
processed.
How to change settings at runtime
There are two main approaches to changing the settings of a calculator graph while the application is running:
- Restart the calculator graph with modified
CalculatorGraphConfig. - Send new calculator options through packets on graph input-streams.
The first approach has the advantage of leveraging CalculatorGraphConfig
processing tools such as "subgraphs". The second approach has the advantage of
allowing active calculators and packets to remain in-flight while settings
change. Mediapipe contributors are currently investigating alternative approaches
to achieve both of these advantages.
How to process realtime input streams
The mediapipe framework can be used to process data streams either online or offline. For offline processing, packets are pushed into the graph as soon as calculators are ready to process those packets. For online processing, one packet for each frame is pushed into the graph as that frame is recorded.
The MediaPipe framework requires only that successive packets be assigned
monotonically increasing timestamps. By convention, realtime calculators and
graphs use the recording time or the presentation time as the timestamp for each
packet, with each timestamp representing microseconds since
Jan/1/1970:00:00:00. This allows packets from various sources to be processed
in a globally consistent order.
Normally for offline processing, every input packet is processed and processing
continues as long as necessary. For online processing, it is often necessary to
drop input packets in order to keep pace with the arrival of input data frames.
When inputs arrive too frequently, the recommended technique for dropping
packets is to use the MediaPipe calculators designed specifically for this
purpose such as FlowLimiterCalculator and PacketClonerCalculator.
For online processing, it is also necessary to promptly determine when processing
can proceed. MediaPipe supports this by propagating timestamp bounds between
calculators. Timestamp bounds indicate timestamp intervals that will contain no
input packets, and they allow calculators to begin processing for those
timestamps immediately. Calculators designed for realtime processing should
carefully calculate timestamp bounds in order to begin processing as promptly as
possible. For example, the MakePairCalculator uses the SetOffset API to
propagate timestamp bounds from input streams to output streams.
Can I run MediaPipe on MS Windows?
Currently MediaPipe portability supports Debian Linux, Ubuntu Linux, MacOS, Android, and iOS. The core of MediaPipe framework is a C++ library conforming to the C++11 standard, so it is relatively easy to port to additional platforms.