# US 2019/0318151 A1 · Image analysis apparatus, method, and program

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318151 A1
- Document ID: US20190318151A1
- Status: Published application
- Kind: A1
- Application number: 16/358,765
- Filed: 2019-03-20
- Published: 2019-10-17
- Earliest claimed: 2018-04-13
- Assignees: OMRON Corporation
- Applicants: OMRON Corporation
- Inventors: Daiki SHICHIJO; Tomoyoshi AIZAWA; Hatsumi AOI
- Web page: https://patentgrep.com/patent/US20190318151A1

## CPC

- G06V 40/165: {using facial parts and geometric relationships}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 7/231: using full search
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness}
- G06V 40/193: {Preprocessing; Feature extraction}
- G06F 18/2135: based on approximation criteria, e.g. principal component analysis
- G06T 2207/30201: Face
- G06V 10/446: {using Haar-like filters, e.g. using integral image techniques}
- G06V 10/449: {Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

In a state where a tracking flag is on, a search controller determines, with respect to a previous frame, whether an amount of change in positional coordinates of a feature point of a face in the current frame is within a predetermined range, whether an amount of change in face orientation is within a predetermined angle range, and whether an amount of change in sight line direction is within a predetermined range. When the conditions are satisfied in all these determinations, the change in the detection result in the current frame with respect to the previous frame is considered as being within an allowable range, and continuously in a subsequent frame, detection processing for a face image is performed in accordance with a face image area saved in a tracking information storage unit.

## Claims

### Claim 1

An image analysis apparatus comprising:
a processor configured with a program to perform operations comprising:
operation as a search unit configured to perform processing to detect an image area comprising an object to be detected, the processing being performed in units of frames from an image that is input in time series and to estimate a state of the object to be detected based on the detected image area;
operation as a reliability detector configured to detect a reliability indicating likelihood of the state of the object to be detected estimated by the search unit; and
operation as a search controller configured to control the processing performed by the search unit based on the reliability detected by the reliability detector, wherein
the processor is configured with the program perform operations such that operation as the search controller comprises:
operation as a first determination unit configured to determine whether a reliability detected by the reliability detector in a first frame of the image satisfies a preset first reliability condition,
operation as a first controller configured to save into a memory a position of the image area detected by the search unit in the first frame and configured to control the search unit such that estimation processing for the state of the object to be detected in a second frame subsequent to the first frame is performed taking the saved position of the image area as a reference, in response to the reliability detected in the first frame being determined to satisfy the reliability condition,
operation as a second determination unit configured to determine whether a change in the state of the object to be detected estimated by the search unit in the second frame from the first frame satisfies a preset determination condition,
operation as a second controller configured to control the search unit such that estimation processing for the state of the object to be detected in a third frame subsequent to the second frame is performed taking the saved position of the image area as a reference, in response to the change in the state of the object to be detected from the first frame being determined to satisfy the determination condition, and
operation as a third controller configured to delete the position of the image area saved in the memory and configured to control the processing performed by the search unit such that the processing performed by the search unit in the third frame subsequent to the second frame is performed from the detection processing for the image area, in response to the change in the state of the object to be detected from the first frame being determined not to satisfy the determination condition.

### Claim 2 (depends on 1)

The image analysis apparatus according to claim 1, wherein the object to be detected comprises a human face, and the processor is configured with the program perform operations such that operation as the search unit is further configured to estimate the state of the object by estimating at least one of: positions of each of a plurality of feature points preset corresponding to a plurality of organs constituting the human face, an orientation of the human face, and a sight line direction of the human face.

### Claim 3 (depends on 2)

The image analysis apparatus according to claim 2, wherein the processor is configured with the program perform operations such that:
operation as the search unit is further configured to perform processing of estimating the positions of the plurality of feature points preset for the plurality of organs constituting the human face in the image area, and
operation as the second determination unit is further configured to perform operations comprising, in response to the determination condition comprising a threshold value defining an allowable amount of interframe change in a position of each of the plurality of feature points estimated by the search unit, determining whether an amount of an interframe change in the position of the feature point between the first frame and the second frame exceeds the threshold value.

### Claim 4 (depends on 2)

The image analysis apparatus according to claim 2, wherein the processor is configured with the program perform operations such that:
operation as the search unit is further configured to perform processing of estimating from the image area the orientation of the human face with respect to a reference direction, and
operation as the second determination unit is further configured to perform operations comprising, in response to the determination condition comprising a threshold value defining an allowable amount of interframe change in the orientation of the human face, determining whether an amount of a change in the orientation of the human face between the first frame and the second frame exceeds the threshold value.

### Claim 5 (depends on 2)

The image analysis apparatus according to claim 2, wherein the processor is configured with the program perform operations such that:
operation as the search unit is further configured to perform processing of estimating from the image area a sight line of the human face, and
operation as the second determination unit is further configured to perform operations comprising, in response to the determination condition comprising a threshold value defining an allowable amount of interframe change in the sight line direction of the human face, determining whether an amount of a change in the sight line direction of the human face between the first frame and the second frame exceeds the threshold value.

### Claim 6

An image analysis method executed by an apparatus that estimates a state of an object to be detected based on an image that is input in time series, the image analysis method comprising:
performing processing of detecting an image area comprising the object to be detected in units of frames from the image input in time series and estimating the state of the object to be detected based on the detected image area;
detecting a reliability that indicates likelihood of the estimated state of the object to be detected;
determining whether a detected reliability in a first frame of the image satisfies a preset first reliability condition;
storing into a memory a position of a detected image area in the first frame and controlling the processing such that estimation for the state of the object to be detected in a second frame subsequent to the first frame is performed taking the saved position of the image area as a reference, in response to the reliability detected in the first frame being determined to satisfy the reliability condition;
determining whether a change in the estimated state of the object to be detected in the second frame from the first frame satisfies a preset determination condition;
controlling the processing such that estimation processing for the state of the object to be detected in a third frame subsequent to the second frame is performed taking the saved position of the image area as a reference, in response to the change in the state of the object to be detected from the first frame being determined to satisfy the determination condition; and
deleting the position of the image area saved in the memory and controlling the processing such that the processing in the third frame subsequent to the second frame is performed from the detection processing for the image area, in response to the change in the state of the object to be detected from the first frame being determined not to satisfy the determination condition.

### Claim 7 (depends on 1)

A non-transitory computer-readable storage medium storing a program, which when read and executed, causes the processor in the image analysis apparatus to perform operations comprising the operations according to claim 1.

### Claim 8 (depends on 2)

A non-transitory computer-readable storage medium storing a program, which when read and executed, causes the processor in the image analysis apparatus to perform operations comprising the operations according to claim 2.

### Claim 9 (depends on 3)

A non-transitory computer-readable storage medium storing a program, which when read and executed, causes the processor in the image analysis apparatus to perform operations comprising the operations according to claim 3.

### Claim 10 (depends on 4)

A non-transitory computer-readable storage medium storing a program, which when read and executed, causes the processor in the image analysis apparatus to perform operations comprising the operations according to claim 4.

### Claim 11 (depends on 5)

A non-transitory computer-readable storage medium storing a program, which when read and executed, causes the processor in the image analysis apparatus to perform operations comprising the operations according to claim 5.