# US 2019/0379819 A1 · Detection of main object for camera auto focus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0379819 A1
- Document ID: US20190379819A1
- Status: Published application
- Kind: A1
- Application number: 16/006,716
- Filed: 2018-06-12
- Published: 2019-12-12
- Earliest claimed: 2018-06-12
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Junji Shimada
- Web page: https://patentgrep.com/patent/US20190379819A1

## CPC

- G06T 7/277: involving stochastic approaches, e.g. using Kalman filters
- G06T 7/248: {involving reference images or patches}
- G06T 7/571: from focus
- G06T 7/74: {involving reference images or patches}
- G06T 7/77: using statistical methods
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/23: {Recognition of whole body movements, e.g. for sport training}
- H04N 23/611: where the recognised objects include parts of the human body
- H04N 23/67: Focus control based on electronic image sensor signals
- H04N 23/675: {comprising setting of focusing regions}
- H04N 23/6811: {based on the image signal}
- H04N 23/80: Camera processing pipelines; Components thereof
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30241: Trajectory
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A camera apparatus and method which selects a main object for camera autofocus control. Captured images are input to a convolution neural network (CNN) which is configured for generating pose information. The pose information is utilized in a process of tracking and determining trajectory similarities between the camera trajectory and the trajectory of each of said multiple objects. A main object of focus is then selected as the main object based on which objects maintains the smallest difference in trajectory between camera and object. The autofocus operation of the camera is based on position and trajectory of this main object.

## Claims

### Claim 1

A camera apparatus, comprising:
(a) an image sensor configured for capturing digital images;
(b) a focusing device coupled to said image sensor for controlling focal length of a digital image being captured;
(c) a processor configured for performing image processing on images captured by said image sensor, and for outputting a signal for controlling focal length set by said focusing device; and
(d) a memory storing programming executable by said processor for estimating depth of focus based on blur differences between images;
(e) said programming when executed performing steps comprising:
(i) inputting an image captured by the camera image sensor into a multiple-branch, multiple-stage convolution neural network (CNN) which is configured for predicting anatomical relationships and generating pose information;
(ii) tracking bounding boxes of multiple objects using a recursive state-space model in combination with a matching algorithm to estimate intersections over union distances (IoU) between the multiple objects;
(iii) determining trajectory similarities between the camera trajectory and the trajectory of each of said multiple objects by obtaining a camera trajectory and trajectories of each of said multiple objects, followed by a dynamic time warping process to estimate trajectory differences across frames;
(iv) selecting a main object of focus as the object from said multiple objects which maintains the smallest difference in trajectory between camera and object; and
(v) performing camera autofocusing based on the position and trajectory of said main object.

### Claim 2 (depends on 1)

The apparatus as recited in claim 1, wherein said instructions when executed by the processor perform steps for selecting a main object of focus to reflect a camera operator's intention since they are tracking that object with the camera.

### Claim 3 (depends on 1)

The apparatus as recited in claim 1, wherein said instructions when executed by the processor are configured for performing said multiple-branch, multiple-stage convolution neural network (CNN) having a first branch configured for predicting confidence maps of body parts for each person object detected within the image.

### Claim 4 (depends on 1)

The apparatus as recited in claim 1, wherein said instructions when executed by the processor are configured for performing said multiple-branch, multiple-stage convolution neural network (CNN) having a second branch for predicting part affinity fields (PAFs) for each person object detected within the image.

### Claim 5 (depends on 1)

The apparatus as recited in claim 1, wherein said instructions when executed by the processor are configured for performing said recursive state-space model as a Kalman filter.

### Claim 6 (depends on 1)

The apparatus as recited in claim 1, wherein said instructions when executed by the processor perform said recursive state-space model based on inputs of horizontal center, vertical center, area, and aspect ratio for a bounding box around each object, as well as derivatives of horizontal center, vertical center and area with respect to time.

### Claim 7 (depends on 1)

The apparatus as recited in claim 1, wherein said camera apparatus is selected from a group of image capture devices consisting of camera systems, camera-enabled cell phones, and other image-capture enabled electronic devices.

### Claim 8

A camera apparatus, comprising:
(a) an image sensor configured for capturing digital images;
(b) a focusing device coupled to said image sensor for controlling focal length of a digital image being captured;
(c) a processor configured for performing image processing on images captured by said image sensor, and for outputting a signal for controlling focal length set by said focusing device; and
(d) a memory storing programming executable by said processor for estimating depth of focus based on blur differences between images;
(e) said programming when executed performing steps comprising:
(i) inputting an image captured by the camera image sensor into a multiple-branch, multiple-stage convolution neural network (CNN), having at least a first branch configured for predicting confidence maps of body parts for each person object detected within the image, and at least a second branch for predicting part affinity fields (PAFs) for each person object detected within the image, with said CNN configured for predicting anatomical relationships and generating pose information;
(ii) tracking bounding boxes of multiple objects using a recursive state-space model in combination with a matching algorithm to estimate intersections over union distances (IoU) between the multiple objects;
(iii) determining trajectory similarities between the camera trajectory and the trajectory of each of said multiple objects by obtaining a camera trajectory and trajectories of each of said multiple objects, followed by a dynamic time warping process to estimate trajectory differences across frames;
(iv) selecting a main object of focus as the object from said multiple objects which maintains the smallest difference in trajectory between camera and object; and
(v) performing camera autofocusing based on the position and trajectory of said main object.

### Claim 9 (depends on 8)

The apparatus as recited in claim 8, wherein said instructions when executed by the processor perform steps for selecting a main object of focus to reflect a camera operator's intention since they are tracking that object with the camera.

### Claim 10 (depends on 8)

The apparatus as recited in claim 8, wherein said instructions when executed by the processor are configured for performing said recursive state-space model as a Kalman filter.

### Claim 11 (depends on 8)

The apparatus as recited in claim 8, wherein said instructions when executed by the processor perform said recursive state-space model based on inputs of horizontal center, vertical center, area, and aspect ratio for a bounding box around each object, as well as derivatives of horizontal center, vertical center and area with respect to time.

### Claim 12 (depends on 8)

The apparatus as recited in claim 8, wherein said camera apparatus is selected from a group of image capture devices consisting of camera systems, camera-enabled cell phones, and other image-capture enabled electronic devices.

### Claim 13

A method for selecting a main object within the field of view of a camera apparatus, comprising:
(a) inputting an image captured by an image sensor of a camera into a multiple-branch, multiple-stage convolution neural network (CNN) which is configured for predicting anatomical relationships and generating pose information;
(b) tracking bounding boxes of multiple objects within an image using a recursive state-space model in combination with a matching algorithm to estimate intersections over union distances (IoU) between the multiple objects;
(c) determining trajectory similarities between a physical trajectory of the camera and the trajectory of each of said multiple objects by obtaining a camera trajectory and trajectories of each of said multiple objects, followed by a dynamic time warping process to estimate trajectory differences across frames;
(d) selecting a main object of focus as the object from said multiple objects which maintains the smallest difference in trajectory between the camera and object; and
(e) performing camera autofocusing based on the position and trajectory of said main object.

### Claim 14 (depends on 13)

The method as recited in claim 13, wherein selecting a main object of focus is performed to reflect a camera operator's intention since they are tracking that object with the camera.

### Claim 15 (depends on 13)

The method as recited in claim 13, further comprising predicting confidence maps of body parts for each object detected by said multiple-branch, multiple-stage convolution neural network (CNN).

### Claim 16 (depends on 13)

The method as recited in claim 13, further comprising predicting part affinity fields (PAFs) of body parts for each object detected by said multiple-branch, multiple-stage convolution neural network (CNN).

### Claim 17 (depends on 13)

The method as recited in claim 13, wherein utilizing said recursive state-space model comprises executing a Kalman filter.

### Claim 18 (depends on 13)

The method as recited in claim 13, wherein said recursive state-space model is performing operations based on inputs of horizontal center, vertical center, area, and aspect ratio for a bounding box around each object, as well as derivatives of horizontal center, vertical center and area with respect to time.

### Claim 19 (depends on 13)

The method as recited in claim 13, wherein said method is configured for being executed on a camera apparatus as selected from a group of image capture devices consisting of camera systems, camera-enabled cell phones, and other image-capture enabled electronic devices.
