# US 2022/0277164 A1 · Technologies for image signal processing and video processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0277164 A1
- Document ID: US20220277164A1
- Status: Published application
- Kind: A1
- Application number: 17/187,202
- Filed: 2021-02-26
- Published: 2022-09-01
- Earliest claimed: 2021-02-26
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Narendranath MALAYATH
- Web page: https://patentgrep.com/patent/US20220277164A1

## CPC

- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- H04N 19/436: using parallelised computational arrangements
- B60W 30/09: Taking automatic action to avoid collision, e.g. braking and steering
- G05D 1/0088: {characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00)}
- G05D 1/0221: {involving a learning process}
- G05D 1/81: Handing over between on-board automatic and on-board manual control
- G06N 3/08: Learning methods
- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/146: Data rate or code amount at the encoder output
- H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence)
- B60W 2420/403: Image sensing, e.g. optical camera
- B60W 2554/4029: Pedestrians
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems, methods, and computer-readable media are provided for efficient control and data utilization between processing components of a system. An method can include obtaining image data captured by an image sensor; prior to a first computing component performing a first set of operations on the image data and a second computing component performing a second set of operations on the image data, determining one or more common operations included in the first set of operations and the second set of operations, wherein the first set of operations is different than the second set of operations; performing the one or more common operations on the image data; and generating an output of the one or more operations for use by the first computing component to perform the first set of operations and the second computing component to perform the second set of operations.

## Claims

### Claim 1

An apparatus for processing video data, the apparatus comprising:
memory; and
one or more processors coupled to the memory, the one or more processors being
configured to:
obtain image data captured by an image sensor;
prior to a first computing component performing a first set of operations on the
image data and a second computing component performing a second set of operations on the image data, determine one or more common operations included in the first set of operations and the second set of operations, wherein the first set of operations is different than the second set of operations;
perform the one or more common operations on the image data; and
generate an output of the one or more operations for use by the first computing component to perform the first set of operations and the second computing component to perform the second set of operations.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are configured to:
provide the output to at least one of the first computing component, the second computing component, and a storage accessible by the first computing component and the second computing component.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are configured to:
provide, by an image processor, additional image data to at least one of the first computing component and the second computing component; and
receive, from the second computing component, control data indicating one or more adjusted parameters for processing the additional image data prior to providing the additional image data to the second computing component, the one or more adjusted parameters being received by at least one of the image processor and the first computing component.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the one or more adjusted parameters are based on at least one of resource capabilities associated with the second computing component, one or more settings at the second computing component, one or more errors at the second computing component, and one or more characteristics of a different output generated by the second computing component based on the second set of operations, the image data, and the output of the one or more operations.

### Claim 5 (depends on 3)

The apparatus of claim 3, wherein the one or more adjusted parameters are based on a bitrate associated with the second computing component, and wherein the one or more adjusted parameters indicate a filtering level for filtering the additional image data.

### Claim 6 (depends on 3)

The apparatus of claim 3, wherein the one or more adjusted parameters are based on one or more distortions in a different output generated by the second computing component based on the second set of operations, the image data, and the output of the one or more operations, and wherein the one or more adjusted parameters indicate one or more image sharpening parameters for sharpening the additional image data.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the first computing component comprises an image processor and the second computing component comprises at least one of a video encoder, an extended reality component, a frame rate up-converter, and one or more components of an advanced driver-assistance system.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are configured to:
determine one or more types of results from the one or more common operations that the first computing component can use for the first set of operations and the second component can use for the second set of operations; and
generate the output based on the one or more types of results.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein the first set of operations comprises digital image processing operations, wherein the second set of operations comprises video encoding operations.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the first set of operations comprises digital image processing operations, and wherein the second set of operations comprises extended reality operations, the extended reality operations comprising at least one of object detection operations, visual inertial odometry operations, obstacle avoidance operations, and face detection operations.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein the first set of operations comprises digital image processing operations, and wherein the second set of operations comprise advanced driver-assistance system operations, the advanced driver-assistance system operations comprising at least one of speed estimation operations, distance estimation operations, collision avoidance operations, and collision detection operations.

### Claim 12 (depends on 1)

The apparatus of claim 1, wherein the image data comprises a compressed video stream, wherein the second computing component comprises a video decoder, and wherein the second set of operations comprises frame rate up-conversion operations.

### Claim 13 (depends on 1)

The apparatus of claim 1, wherein the one or more common operations comprise at least one of motion estimation, image segmentation, and object detection, and wherein the output comprises at least one of an object map, texture information, and one or more motion vectors.

### Claim 14 (depends on 1)

The apparatus of claim 1, wherein the second computing component is configured to:
obtain the output of the one or more common operations; and
perform the second set of operations using the image data and the output of the one or more common operations.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the second computing component is configured to:
prior to performing the second set of operations, refine the output for use in the second set of operations.

### Claim 16 (depends on 1)

The apparatus of claim 1, wherein the second set of operations comprises encoding the image data, wherein the second computing component is configured to:
generate an encoded video bitstream based on the image data, the output of the one or more common operations, and the second set of operations.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the second computing component is configured to:
transmit the encoded video bitstream.

### Claim 18 (depends on 1)

The apparatus of claim 1, wherein the apparatus is a mobile device.

### Claim 19 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are configured to:
provide the output of the one or more operations to the first computing component, the second computing component, and one or more additional computing components.

### Claim 20

A method for processing video data, the method comprising:
obtaining image data captured by an image sensor;
prior to a first computing component performing a first set of operations on the image data and a second computing component performing a second set of operations on the image data, determining one or more common operations included in the first set of operations and the second set of operations, wherein the first set of operations is different than the second set of operations;
performing the one or more common operations on the image data; and
generating an output of the one or more operations for use by the first computing component to perform the first set of operations and the second computing component to perform the second set of operations.

### Claim 21 (depends on 20)

The method of claim 20, further comprising:
providing the output to at least one of the first computing component, the second computing component, and a storage accessible by the first computing component and the second computing component.

### Claim 22 (depends on 20)

The method of claim 20, further comprising:
providing, by an image processor, additional image data to at least one of the first computing component and the second computing component; and
receiving, from the second computing component, control data indicating one or more adjusted parameters for processing the additional image data prior to providing the additional image data to the second computing component, the one or more adjusted parameters being received by at least one of the image processor and the first computing component.

### Claim 23 (depends on 22)

The method of claim 22, wherein the one or more adjusted parameters are based on at least one of resource capabilities associated with the second computing component, one or more settings at the second computing component, one or more errors at the second computing component, and one or more characteristics of a different output generated by the second computing component based on the second set of operations, the image data, and the output of the one or more operations.

### Claim 24 (depends on 22)

The method of claim 22, wherein the one or more adjusted parameters are based on a bitrate associated with the second computing component, and wherein the one or more adjusted parameters indicate a filtering level for filtering the additional image data.

### Claim 25 (depends on 22)

The method of claim 22, wherein the one or more adjusted parameters are based on one or more distortions in a different output generated by the second computing component based on the second set of operations, the image data, and the output of the one or more operations, and wherein the one or more adjusted parameters indicate one or more image sharpening parameters for sharpening the additional image data.

### Claim 26 (depends on 20)

The method of claim 20, wherein the first computing component comprises an image processor and the second computing component comprises at least one of a video encoder, an extended reality component, a frame rate up-converter, and one or more components of an advanced driver-assistance system.

### Claim 27 (depends on 20)

The method of claim 20, further comprising:
determining one or more types of results from the one or more common operations that the first computing component can use for the first set of operations and the second component can use for the second set of operations; and
generating the output based on the one or more types of results.

### Claim 28 (depends on 20)

The method of claim 20, wherein the first set of operations comprises digital image processing operations, and wherein the second set of operations comprises at least one of video encoding operations and extended reality operations, the extended reality operations comprising at least one of object detection operations, visual inertial odometry operations, obstacle avoidance operations, and face detection operations.

### Claim 29 (depends on 20)

The method of claim 20, wherein the first set of operations comprises digital image processing operations, and wherein the second set of operations comprise advanced driver-assistance system operations, the advanced driver-assistance system operations comprising at least one of speed estimation operations, distance estimation operations, collision avoidance operations, and collision detection operations.

### Claim 30 (depends on 20)

The method of claim 20, wherein the one or more common operations comprise at least one of motion estimation, image segmentation, and object detection, and wherein the output comprises at least one of an object map, texture information, and one or more motion vectors.

### Claim 31 (depends on 20)

The method of claim 20, further comprising:
obtaining, via the second computing component, the output of the one or more common operations; and
performing, via the second computing component, the second set of operations using the image data and the output of the one or more common operations.

### Claim 32 (depends on 31)

The method of claim 31, further comprising:
prior to performing the second set of operations, refining, via the second computing component, the output for use in the second set of operations.

### Claim 33 (depends on 20)

The method of claim 20, wherein the second set of operations comprises encoding the image data, the method further comprising:
generating, via the second computing component, an encoded video bitstream based on the image data, the output of the one or more common operations, and the second set of operations; and
transmitting, via the second computing component, the encoded video bitstream.

### Claim 34 (depends on 20)

The method of claim 20, further comprising:
providing the output of the one or more operations to the first computing component, the second computing component, and one or more additional computing components.

### Claim 35

A non-transitory computer-readable medium having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to:
obtain image data captured by an image sensor;
prior to a first computing component performing a first set of operations on the image data and a second computing component performing a second set of operations on the image data, determine one or more common operations included in the first set of operations and the second set of operations, wherein the first set of operations is different than the second set of operations;
perform the one or more common operations on the image data; and
generate an output of the one or more operations for use by the first computing component to perform the first set of operations and the second computing component to perform the second set of operations.
