# US 2023/0274453 A1 · Adaptive face depth image generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274453 A1
- Document ID: US20230274453A1
- Status: Published application
- Kind: A1
- Application number: 17/694,726
- Filed: 2022-03-15
- Published: 2023-08-31
- Earliest claimed: 2022-02-28
- Assignees: Ambarella International LP
- Applicants: Ambarella International LP
- Inventors: Jian Tang; Tao Liu; Jingyang Qiu
- Web page: https://patentgrep.com/patent/US20230274453A1

## CPC

- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/586: from multiple light sources, e.g. photometric stereo
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06V 10/145: Illumination specially adapted for pattern recognition, e.g. using gratings
- G06V 40/166: {using acquisition arrangements}
- G06V 40/45: {Detection of the body part being alive}
- H04N 7/0117: {involving conversion of the spatial resolution of the incoming video signal (for graphics images G09G2340/0407)}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30201: Face
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An apparatus comprising an interface, a light projector and a processor. The interface may be configured to receive pixel data. The light projector may be configured to generate a structured light pattern. The processor may be configured to process the pixel data arranged as video frames comprising the structured light pattern, perform computer vision operations to detect a size of a face area of the video frames, determine a scale ratio in response to the size of the face area, extract the structured light pattern from the video frames, generate a downscaled structured light image and generate a depth map in response to the downscaled structured light image and a downscaled reference image. A downscale operation may be performed in response to the scale ratio to generate the downscaled structured light image. The scale ratio may enable the generation of the downscaled structured light image with sufficient depth pixels.

## Claims

### Claim 1

An apparatus comprising:
an interface configured to receive pixel data;
a structured light projector configured to generate a structured light pattern; and
a processor configured to (i) process said pixel data arranged as video frames comprising said structured light pattern, (ii) perform computer vision operations on said video frames to detect a size of a face area of said video frames, (iii) determine a scale ratio in response to said size of said face area, (iv) extract said structured light pattern from said video frames, (v) generate a downscaled structured light image and (vi) generate a depth map in response to said downscaled structured light image and a downscaled reference image, wherein
(i) a downscale operation is performed in response to said scale ratio and said structured light pattern to generate said downscaled structured light image, and
(ii) said scale ratio is calculated to enable said downscaled structured light image to be generated with a sufficient number of depth pixels.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein said downscaled structured light image is generated with said sufficient number of depth pixels to prevent a block match failure using said downscaled structured light image.

### Claim 3 (depends on 1)

The apparatus according to claim 1, wherein said sufficient number of depth pixels provide an expected range of a depth pixel number for a liveness detection operation.

### Claim 4 (depends on 1)

The apparatus according to claim 1, wherein said downscale operation is performed to reduce a number of calculations performed by said processor to generate said depth map using said downscaled structured light image compared to a second number of calculations performed for generating said depth map from said structured light pattern from said video frames.

### Claim 5 (depends on 1)

The apparatus according to claim 1, wherein said downscaled reference image is generated in response to said downscale operation performed on a reference image.

### Claim 6 (depends on 1)

The apparatus according to claim 1, wherein (i) said downscaled reference image is generated in response to said downscale operation performed on a plurality of reference images, (ii) said plurality of reference images are precalculated at fixed downscale ratios and (iii) said processor is further configured to (a) compare said scale ratio to said fixed downscale ratios and (b) modify said scale ratio to match one of said fixed downscale ratios.

### Claim 7 (depends on 1)

The apparatus according to claim 1, wherein said processor is configured to implement an adaptive generation of face depth maps.

### Claim 8 (depends on 1)

The apparatus according to claim 1, wherein determining said scale ratio enables said depth map to be dynamically created at a range of resolutions that correspond to said size of said face area.

### Claim 9 (depends on 1)

The apparatus according to claim 1, wherein said size of said face area is a variable size in each of said video frames based on a distance of a person from an image sensor that captures said pixel data.

### Claim 10 (depends on 1)

The apparatus according to claim 1, wherein (i) said processor is configured to perform face detection operations on said video frames to determine said size of said face area and (ii) said face detection operations (a) determine whether at least one face is captured by said video frames and (b) record an offset and said size of said face area if said at least of one face is detected.

### Claim 11 (depends on 1)

The apparatus according to claim 1, wherein (i) said scale ratio is calculated in response to said sufficient number of depth pixels, said size of said face area and a tolerance parameter and (ii) said size of said face area comprises a face width and a face height.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein (i) said tolerance parameter is a configurable value set by a user and (ii) said tolerance parameter enables said user to adjust said downscale operation between a detection accuracy and a calculation speed.

### Claim 13 (depends on 11)

The apparatus according to claim 11, wherein said scale ratio a smaller value when said size of said face area is large.

### Claim 14 (depends on 1)

The apparatus according to claim 1, wherein an amount of time added to perform calculations for said downscale operation is less than a time difference to perform depth calculations to generate said depth map in response to said downscaled structured light image and downscaled reference image compared to said depth calculations to generate said depth map in response to said structured light pattern and a full resolution reference image.

### Claim 15 (depends on 1)

The apparatus according to claim 1, wherein said depth map generated in response to said downscaled structured light image and a downscaled reference image has a consistent face depth resolution.

### Claim 16 (depends on 15)

The apparatus according to claim 15, wherein said consistent face depth resolution of said depth map reduces complexity for a post processing operation.

### Claim 17 (depends on 16)

The apparatus according to claim 16, wherein (i) said post processing operation comprises aliveness detection and (ii) said consistent face depth resolution enables said liveness detection to be trained for a face depth with said consistent face depth resolution instead of face depths for multiple resolutions.

### Claim 18 (depends on 1)

The apparatus according to claim 1, wherein said downscaled structured light image has (i) a lower resolution than said structured light pattern and (ii) a same aspect ratio as said structured light pattern after a cropping operation.

### Claim 19 (depends on 1)

The apparatus according to claim 1, wherein (i) said face area is detected by performing said computer vision operations on an RGB subset of said video frames that have been captured when said structured light projector is not generating said structured light pattern and (ii) said downscaled structured light image is generated in response to an IR subset of said video frames that have been captured when said structured light projector is generating said structured light pattern.

### Claim 20 (depends on 19)

The apparatus according to claim 19, wherein said scale ratio is determined for each of said video frames in said RGB subset of said video frames in real time in order to adapt to a change in characteristics of said face area as a person moves in front of a capture device that is generating said pixel data.
