# US 2012/0249833 A1 · Motion robust depth estimation using convolution and wavelet transforms

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0249833 A1
- Document ID: US20120249833A1
- Status: Published application
- Kind: A1
- Application number: 13/077,677
- Filed: 2011-03-31
- Published: 2012-10-04
- Earliest claimed: 2011-03-31
- Assignees: Sony Corporation
- Inventors: Pingshan Li
- Web page: https://patentgrep.com/patent/US20120249833A1

## CPC

- H04N 23/676: {Bracketing for image capture at varying focusing conditions}
- G06T 7/571: from focus
- G06T 2207/20064: Wavelet transform [DWT]
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Apparatus and method for electronically estimating focusing distance between a camera (still and/or video camera) and a subject. Images at different focal positions of a calibration target are collected to arrive at a focus matching model for a given imaging apparatus. In operation, at least two images are captured and convolutions performed which approximate the modeling of blur change as a point spread function. Wavelet transforms are applied to the images after each convolution and images are compared based on the wavelet variance differences to provide a motion robust blur difference determination. Applying the blur differences to the focus matching model provides an estimate of focusing distance, which can be utilized such as for controlling camera focus.

## Claims

### Claim 1

An image capture apparatus, comprising:
an imaging device;
a computer processor coupled to the imaging device;
memory coupled to said computer processor configured for retaining programming executable on said computer processor;
a focus matching model retained in said memory, and programming executable on said computer processor for carrying out steps comprising:
(i) capturing multiple object images;
(ii) performing convolutions to model blur changes as a point spread function between said multiple object images;
(iii) determining blur difference within each convolution in response to performing a wavelet transform, obtaining wavelet variance and comparing differences of wavelet variance for said multiple object images; and
(iv) performing depth estimation in response to said convolutions within said focus matching model.

### Claim 2 (depends on 1)

An apparatus as recited in claim 1, wherein said focus matching model is based on imaging calibration targets obtained at different focal lengths.

### Claim 3 (depends on 1)

An apparatus as recited in claim 1, wherein programming executable on said computer processor is configured for performing said convolutions by at least one size of convolution kernel.

### Claim 4 (depends on 1)

An apparatus as recited in claim 1, wherein programming executable on said computer processor is configured for comparing said differences of wavelet variance in response to an absolute difference of wavelet variance.

### Claim 5 (depends on 1)

An apparatus as recited in claim 1, wherein programming executable on said computer processor is configured for determining said wavelet variance in at least one wavelet subband and at least one wavelet transform level.

### Claim 6 (depends on 1)

An apparatus as recited in claim 1, wherein programming executable on said computer processor is configured for determining said wavelet variance in all wavelet subbands in at least one wavelet transform level.

### Claim 7 (depends on 1)

An apparatus as recited in claim 1, wherein said focus matching model utilizes a polynomial function to reduce mismatching noise.

### Claim 8 (depends on 7)

An apparatus as recited in claim 7, wherein coefficients of the polynomial function are stored in said memory.

### Claim 9 (depends on 1)

An apparatus as recited in claim 1, wherein programming executable on said computer processor is further configured for performing histogram matching of the object images to reduce noise from outliers between focal positions prior to inputting the blur differences into the focus matching model.

### Claim 10 (depends on 9)

An apparatus as recited in claim 9, wherein programming executable on said computer processor is configured for performing said histogram matching in response to steps comprising:
sequentially shifting pixels from a first histogram to a second histogram to equalize the pixels of closest luminance; and
approximating histogram matching utilizing a linear matching function;
wherein noise effects are reduced which have been introduced into said focus matching model in response to undesired physical and environmental variations.

### Claim 11 (depends on 1)

An apparatus as recited in claim 1, wherein the focus matching model is generated in response to a calibration process upon said apparatus in which a series of calibration target images are obtained for registering proper focus, with focus curves obtained for the series of calibration target images; and in which a multi-dimensional model is generated based on matching the focus curves for the series of calibration target images.

### Claim 12 (depends on 1)

An apparatus as recited in claim 1, wherein the imaging device comprises a still image camera, a video image camera, or a combination still and video image camera.

### Claim 13 (depends on 1)

An apparatus as recited in claim 1, further comprising:
a focus control element coupled to the imaging device; and
programming executable on said computer processor for adjusting said focus control element in response to performing depth estimation on object images based on inputting blur differences detected between object images into said focus matching model.

### Claim 14

An image capture apparatus, comprising:
an imaging device;
a computer processor coupled to the imaging device;
memory coupled to said computer processor configured for retaining programming executable on said computer processor;
a focus matching model based on imaging calibration targets at different focal lengths which is retained in said memory, and
programming executable on said computer processor for carrying out steps comprising:
(i) capturing multiple object images;
(ii) performing convolutions by at least one size of convolution kernel to model blur changes as a point spread function between said multiple object images;
(iii) determining blur difference within each convolution in response to performing wavelet transform, obtaining wavelet variance and comparing differences of wavelet variance in at least one wavelet subband and at least one wavelet transform level, for said multiple object images; and
(iv) performing depth estimation in response to said convolutions within said focus matching model.

### Claim 15 (depends on 14)

An apparatus as recited in claim 14, wherein programming executable on said computer processor is configured for comparing said differences of wavelet variance in response to an absolute difference of wavelet variance.

### Claim 16 (depends on 14)

An apparatus as recited in claim 14, wherein programming executable on said computer processor is configured for determining said wavelet variance in at least one wavelet subband and at least one wavelet transform level.

### Claim 17 (depends on 14)

An apparatus as recited in claim 14, wherein programming executable on said computer processor is configured for determining said wavelet variance in all wavelet subbands in at least one wavelet transform level.

### Claim 18 (depends on 14)

An apparatus as recited in claim 14, wherein said focus matching model utilizes a polynomial function, whose coefficients are stored in said memory, to reduce mismatching noise.

### Claim 19 (depends on 14)

An apparatus as recited in claim 14, wherein programming executable on said computer processor is further configured for performing histogram matching of the object images to reduce noise from outliers between focal positions prior to inputting the blur differences into the focus matching model.

### Claim 20

A method of automatic estimation of camera-to-object focal depth, comprising:
generating a multi-dimensional focus matching model in response to detecting blur differences between multiple images of a calibration subject captured at different focal distances;
capturing multiple object images;
determining blur differences between the multiple object images in response to convolutions which model blur changes as a point spread function between said multiple object images,
determining blur difference within each convolution in response to performing wavelet transform, obtaining wavelet variance and comparing differences of wavelet variance for said multiple object images, and
performing depth estimation in response to said convolutions within said focus matching model.
