# US 2020/0007738 A1 · Exchanging an hdr-combined stream and associated exposures between a camera sensor module and a vision processing system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0007738 A1
- Document ID: US20200007738A1
- Status: Published application
- Kind: A1
- Application number: 16/024,716
- Filed: 2018-06-29
- Published: 2020-01-02
- Earliest claimed: 2018-06-29
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Jeffrey Hao CHU
- Web page: https://patentgrep.com/patent/US20200007738A1

## CPC

- H04N 23/66: Remote control of cameras or camera parts, e.g. by remote control devices
- G06T 5/40: using histogram techniques
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- H04N 9/646: {for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters}
- H04N 23/60: Control of cameras or camera modules
- H04N 23/71: Circuitry for evaluating the brightness variation
- H04N 23/72: Combination of two or more compensation controls
- H04N 23/741: by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
- H04N 23/80: Camera processing pipelines; Components thereof
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 2207/30248: Vehicle exterior or interior
- H04N 23/73: by influencing the exposure time
- H04N 23/88: for colour balance, e.g. white-balance circuits or colour temperature control
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

In an embodiment, a camera sensor module is communicatively coupled to a vision processing system. The camera sensor module captures a plurality of exposures, produces an HDR-combined stream, and sends both the HDR-combined stream and the plurality of exposures to the vision processing system. A first ISP of the vision processing system performs color processing to produce a color-processed HDR-combined stream for neural network processing, presentation to a human, and/or computer vision processing. At least one other ISP of the vision processing system performs color processing to collect exposure-specific statistics on at least one of the plurality of exposures for determining camera calibration data for the camera sensor module.

## Claims

### Claim 1

A method of operating a camera sensor module that is communicatively coupled to a vision processing system, comprising:
capturing a plurality of exposures;
performing high-dynamic range (HDR) combining on the plurality of exposures to produce an HDR-combined stream; and
communicating, over a communications link to the vision processing system, (i) the HDR-combined stream, and (ii) the plurality of exposures.

### Claim 2 (depends on 1)

The method of claim 1, wherein the plurality of exposures includes three or four exposures.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
receiving, from the vision processing system, exposure-specific statistics for at least one of the plurality of exposures; and
calibrating a camera sensor of the camera sensor module based on the exposure-specific statistics.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
receiving, from the vision processing system, camera sensor calibration parameters that are computed at the vision processing system based upon exposure-specific statistics for at least one of the plurality of exposures; and
calibrating a camera sensor of the camera sensor module based on the camera sensor calibration parameters.

### Claim 5 (depends on 4)

The method of claim 4, wherein the camera sensor calibration parameters include automatic exposure (AE) control settings, automatic white-balance (AWB) control settings, or a combination thereof.

### Claim 6 (depends on 1)

The method of claim 1, wherein the camera sensor module and the vision processing system are integrated into a single apparatus.

### Claim 7 (depends on 6)

The method of claim 6, wherein the single apparatus is a phone or a tablet computer.

### Claim 8 (depends on 1)

The method of claim 1, wherein the camera sensor module and the vision processing system are physically separate from each other.

### Claim 9 (depends on 8)

The method of claim 8, wherein the camera sensor module and the vision processing system are part of an image capture system for a vehicle.

### Claim 10 (depends on 1)

The method of claim 1, wherein the communications link comprises a single physical layer interface (PHY).

### Claim 11 (depends on 1)

The method of claim 1, wherein the communications link comprises multiple physical layer interfaces (PHYs).

### Claim 12

A method of operating a vision processing system that is communicatively coupled to a camera sensor module, comprising:
receiving, from the camera sensor module over a communications link, (i) a plurality of exposures captured by the camera sensor module, and (ii) a high-dynamic range (HDR)-combined stream of the plurality of exposures based on HDR-combining performed at the camera sensor module;
performing, by a first image signal processor (ISP) among a plurality of ISPs, color processing on the HDR-combined stream to produce a color-processed HDR-combined stream for neural network processing, presentation to a human, and/or computer vision processing; and
performing, by at least one other ISP among the plurality of ISPs, color processing to collect exposure-specific statistics on at least one of the plurality of exposures for determining camera calibration data for the camera sensor module.

### Claim 13 (depends on 12)

The method of claim 12, wherein the color processing performed by the first ISP on the HDR-combined stream includes Bayer RGB to RGB conversion, bad pixel correction, noise filtering, tone mapping, or any combination thereof.

### Claim 14 (depends on 12)

The method of claim 12, wherein the at least one other ISP includes a single ISP configured to perform the exposure-specific statistics collection on each of the plurality of exposures.

### Claim 15 (depends on 12)

The method of claim 12, wherein the at least one other ISP includes multiple ISPs.

### Claim 16 (depends on 15)

The method of claim 15, wherein each ISP among the multiple ISPs is configured to perform the exposure-specific statistics collection on a different one of the plurality of exposures.

### Claim 17 (depends on 12)

The method of claim 12, further comprising:
dynamically modifying an allocation of the at least one other ISP to the color processing for the exposure-specific statistics collection based on a processing requirement associated with HDR-combined stream processing.

### Claim 18 (depends on 12)

The method of claim 12, wherein the performing of the color processing to collect the exposure-specific statistics skips statistic collection for one or more of the plurality of exposures.

### Claim 19 (depends on 18)

The method of claim 18,
wherein the one or more skipped exposures are skipped based on expiration of the one or more skipped exposures, or
wherein the one or more skipped exposures are skipped based on a processing resource restriction, or
any combination thereof.

### Claim 20 (depends on 12)

The method of claim 12, wherein the plurality of exposures includes three or four exposures.

### Claim 21 (depends on 12)

The method of claim 12, further comprising:
sending, to the camera sensor module, the exposure-specific statistics for the at least one of the plurality of exposures for camera sensor calibration.

### Claim 22 (depends on 12)

The method of claim 12, further comprising:
calculating camera sensor calibration parameters for calibration of a camera sensor at the camera sensor module based upon the exposure-specific statistics for the at least one of the plurality of exposures for camera sensor calibration; and
sending, to the camera sensor module, the camera sensor calibration parameters.

### Claim 23 (depends on 22)

The method of claim 22, wherein the camera sensor calibration parameters include automatic exposure (AE) control settings, automatic white-balance (AWB) control settings, or a combination thereof.

### Claim 24 (depends on 12)

The method of claim 12, wherein the camera sensor module and the vision processing system are integrated into a single apparatus.

### Claim 25 (depends on 24)

The method of claim 24, wherein the single apparatus is a phone or a tablet computer.

### Claim 26 (depends on 12)

The method of claim 12, wherein the camera sensor module and the vision processing system are physically separate from each other.

### Claim 27 (depends on 25)

The method of claim 25, wherein the camera sensor module and the vision processing system are part of an image capture system for a vehicle.

### Claim 28 (depends on 12)

The method of claim 12, wherein the communications link comprises a single physical layer interface (PHY).

### Claim 29 (depends on 12)

The method of claim 12, wherein the communications link comprises multiple physical layer interfaces (PHYs).

### Claim 30

A camera sensor module that is communicatively coupled to a vision processing system, comprising:
a camera sensor configured to capture a plurality of exposures;
an on-sensor processing system configured to perform high-dynamic range (HDR) combining on the plurality of exposures to produce an HDR-combined stream; and
a communications interface configured to communicate, over a communications link to the vision processing system, (i) the HDR-combined stream, and (ii) the plurality of exposures.

### Claim 31 (depends on 30)

The camera sensor module of claim 30,
wherein the communications interface is further configured to receive, from the vision processing system, exposure-specific statistics for at least one of the plurality of exposures; and
wherein the on-sensor processing system is further configured to calibrate a camera sensor of the camera sensor module based on the exposure-specific statistics.

### Claim 32 (depends on 30)

The camera sensor module of claim 30,
wherein the communications interface is further configured to receive, from the vision processing system, camera sensor calibration parameters that are computed at the vision processing system based upon exposure-specific statistics for at least one of the plurality of exposures; and
wherein the on-sensor processing system is further configured to calibrate a camera sensor of the camera sensor module based on the camera sensor calibration parameters.

### Claim 33 (depends on 30)

The camera sensor module of claim 30,
wherein the camera sensor module and the vision processing system are integrated into a single apparatus, or
wherein the camera sensor module and the vision processing system are physically separate from each other.

### Claim 34 (depends on 30)

The camera sensor module of claim 30,
wherein the communications link comprises a single physical layer interface (PHY), or
wherein the communications link comprises multiple PHYs.

### Claim 35

A vision processing system that is communicatively coupled to a camera sensor module, comprising:
a communications interface configured to receive, from the camera sensor module over a communications link, (i) a plurality of exposures captured by the camera sensor module, and (ii) a high-dynamic range (HDR)-combined stream of the plurality of exposures based on HDR-combining performed at the camera sensor module;
a first image signal processor (ISP) among a plurality of ISPs, the first ISP configured to perform color processing on the HDR-combined stream to produce a color-processed HDR-combined stream for neural network processing, presentation to a human, and/or computer vision processing; and
at least one other ISP among the plurality of ISPs, the at least one other ISP configured to perform color processing to collect exposure-specific statistics on at least one of the plurality of exposures for determining camera calibration data for the camera sensor module.

### Claim 36 (depends on 35)

The vision processing system of claim 35, wherein the color processing performed by the first ISP on the HDR-combined stream includes Bayer RGB to RGB conversion, bad pixel correction, noise filtering, tone mapping, or any combination thereof.

### Claim 37 (depends on 35)

The vision processing system of claim 35,
wherein the at least one other ISP includes a single ISP configured to perform the exposure-specific statistics collection on each of the plurality of exposures, or
wherein the at least one other ISP includes multiple ISPs.

### Claim 38 (depends on 35)

The vision processing system of claim 35, wherein an allocation of the at least one other ISP to the color processing for the exposure-specific statistics collection is configured to be dynamically modified based on a processing requirement associated with HDR-combined stream processing.

### Claim 39 (depends on 35)

The vision processing system of claim 35, wherein the at least one other ISP skips statistic collection for one or more of the plurality of exposures.

### Claim 40 (depends on 35)

The vision processing system of claim 35, wherein the communications interface is further configured to send, to the camera sensor module, the exposure-specific statistics for the at least one of the plurality of exposures for camera sensor calibration.

### Claim 41 (depends on 35)

The vision processing system of claim 35, further comprising:
at least one processor configured to calculate camera sensor calibration parameters for calibration of a camera sensor at the camera sensor module based upon the exposure-specific statistics for the at least one of the plurality of exposures for camera sensor calibration,
wherein the communications interface is further configured to send, to the camera sensor module, the camera sensor calibration parameters.

### Claim 42 (depends on 35)

The vision processing system of claim 35,
wherein the camera sensor module and the vision processing system are integrated into a single apparatus, or
wherein the camera sensor module and the vision processing system are physically separate from each other.

### Claim 43 (depends on 35)

The vision processing system of claim 35,
wherein the communications link comprises a single physical layer interface (PHY), or
wherein the communications link comprises multiple PHYs.
