US 11,694,341 B2Granted

Cascaded architecture for disparity and motion prediction with block matching and convolutional neural network (CNN)

AssigneeTexas Instmments Incorporated

InventorsJing Li, Do-Kyoung Kwon, Tarek Aziz Lahlou

Life of this patent

Estimating the term…

Loading patent…

Abstract

A CNN operates on the disparity or motion outputs of a block matching hardware module, such as a DMPAC module, to produce refined disparity or motion streams which improve operations in images having ambiguous regions. As the block matching hardware module provides most of the processing, the CNN can be small and thus able to operate in real time, in contrast to CNNs which are performing all of the processing. In one example, the CNN operation is performed only if the block hardware module output confidence level is below a predetermined amount. The CNN can have a number of different configurations and still be sufficiently small to operate in real time on conventional platforms.

Claims20

Dependent claims

An image processing system comprising:

a block matching hardware module having an input for receiving first and second image streams and an output for providing a displacement stream, wherein the block matching hardware module is configured to provide a confidence stream;
and
a convolutional neural network (CNN) having an input coupled to the block matching hardware module for receiving the displacement stream and for receiving at least one of the first and second image streams, wherein the CNN is configured to selectively provide a refined displacement stream for disparity prediction or for motion prediction;
and
comparator logic configured to:
responsive to a value of the confidence stream being greater than a threshold value, bypass the CNN and output the displacement stream;
responsive to the value of the confidence stream being less than the threshold value, provide and output the refined displacement stream.

4 more elements

A system comprising:

a system on a chip (SoC) including:
a plurality of processors;
a memory controller coupled to the plurality of processors;
onboard memory coupled to the memory controller;
an external memory interface for connecting to external memory;
a high-speed interconnect coupled to the plurality of processors, the memory controller and the external memory interface;
an external communication interface coupled to the high-speed interconnect;
and
a block matching hardware module having an input for receiving first and second image streams and an output for providing a displacement stream for disparity prediction or for motion prediction;
and
an external memory coupled to the external memory interface and storing instructions for execution on a first processor of the plurality of processors to form, when executed on the first processor, a convolutional neural network (CNN) having an input coupled to the block matching hardware module for receiving the displacement stream and for receiving at least one of the first and second image streams,
wherein the instructions are configured to be executable by the first processor for further causing the CNN to selectively provide a refined displacement stream.

9 more elements

Dependencies are inferred from the claim text.