# US 12,216,747 B2 · Apparatus and method with image recognition-based security

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,216,747 B2
- Document ID: US12216747B2
- Status: Granted
- Kind: B2
- Application number: 17/675,015
- Filed: 2022-02-18
- Published: 2023-02-16
- Granted: 2025-02-04
- Earliest claimed: 2021-08-13
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Joohyeon Kim; Donghoon Lee; Changyong Son; Do-Ha Hwang
- Web page: https://patentgrep.com/patent/US12216747B2

## CPC

- G06F 21/32: using biometric data, e.g. fingerprints, iris scans or voiceprints
- G06T 7/194: involving foreground-background segmentation
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/172: {Classification, e.g. identification}
- G06T 2207/30201: Face
- G06T 2207/30232: Surveillance
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus and a method with image recognition-based security are disclosed. The method includes, for an unlocked terminal, tracking a face detected in a previous frame, detecting a background region change between the previous frame and a current frame based on a region of the tracked face, when the background region change is not detected, determining whether a state maintenance time fails to meet a preset time, in response to the state maintenance time failing to meet the preset time, determining an operation mode to be a first operation mode for determining whether recognition succeeds for the current frame, performing the first operation mode, including performing face detection with respect to the current frame, and maintaining the unlocked state of the terminal for the current frame when the face is detected as a result of the performing of the face detection, representing that the recognition succeeded for the current frame.

## Claims

### Claim 1

A processor-implemented method, comprising:
determining between a locked state and an unlocked state of a terminal; and
when a result of the determining between the locked state and the unlocked state of the terminal is that the terminal is in the unlocked state:
tracking, in a current frame, a face detected in a previous frame;
detecting for a background region change between the previous frame and the current frame based on a background region, of the current frame, distinguished from a region of the tracked face in the current frame;
when a result of the detecting for the background region change is that the background region change is not detected, determining whether a state maintenance time of the unlocked state fails to meet an unlocked state maintenance preset time, where the state maintenance time of the unlocked state is a time for which the unlocked state of the terminal has been maintained for one or more previous frames; and
in response to the state maintenance time of the unlocked state failing to meet the unlocked state maintenance preset time,
determining an operation mode to be a first operation mode for determining whether recognition succeeds for the current frame,
performing the first operation mode, including performing face detection with respect to the current frame, and
maintaining the unlocked state of the terminal for the current frame when the face is detected as a result of the performing of the face detection, representing that the recognition succeeded for the current frame.

### Claim 2 (depends on 1)

The method of claim 1, further comprising, in response to the state maintenance time of the unlocked state meeting the unlocked state maintenance preset time:
determining the operation mode to be a second operation mode for the determining of whether the recognition succeeds for the current frame;
performing the second operation mode, including sequentially performing the face detection, facial recognition, and unauthorized object detection with respect to the current frame; and
initializing the state maintenance time of the unlocked state.

### Claim 3 (depends on 1)

The method of claim 1, further comprising, when the result of the detecting for the background region change is that the background region change is detected:
determining the operation mode to be a third operation mode for the determining of whether the recognition succeeds for the current frame;
performing the third operation mode, including sequentially performing unauthorized object detection, the face detection, and face recognition with respect to the current frame; and
initializing the state maintenance time of the unlocked state.

### Claim 4 (depends on 1)

The method of claim 1, further comprising, when the result of the determining between the locked state and the unlocked state is that is that the terminal is in the locked state, dependent on an unauthorized object having been detected in the previous frame:
tracking, in the current frame, for the detected unauthorized object;
determining whether a tracking reliability of a result of the tracking for the detected unauthorized object meets a threshold value;
in response to the tracking reliability meeting the threshold value, determining whether a state maintenance time of the locked state fails to meet a locked state maintenance preset time; and
in response to the state maintenance time of the locked state failing to meet the locked state maintenance preset time,
determining the operation mode to be a fourth operation mode for the determining whether the recognition succeeds for the current frame, and
maintaining, for the fourth operation mode, the locked state of the terminal with respect to the current frame.

### Claim 5 (depends on 4)

The method of claim 4, further comprising, in response to the tracking reliability failing to meet the threshold value:
determining the operation mode to be a third operation mode for the determining of whether the recognition succeeds for the current frame;
performing the third operation mode, including sequentially performing unauthorized object detection, the face detection, and face recognition with respect to the current frame; and
initializing the state maintenance time of the locked state.

### Claim 6 (depends on 4)

The method of claim 4, further comprising, in response to the state maintenance time of the locked state meeting the locked state maintenance preset time:
determining the operation mode to be a third operation mode for the determining of whether the recognition succeeds for the current frame;
performing the third operation mode, including sequentially performing unauthorized object detection, the face detection, and face recognition with respect to the current frame; and
initializing the state maintenance time of the locked state.

### Claim 7 (depends on 1)

The method of claim 1, further comprising, when the result of the determining between the locked state and the unlocked state is that is that the terminal is in the locked state, dependent on a foreground object having not been detected in the previous frame:
performing the detecting for the background region change between the previous frame and the current frame;
when a result of the performing of the detecting for the background region change is that the background region change is not detected, determining whether a state maintenance time of the locked state fails to meet a locked state maintenance preset time; and
in response to the state maintenance time of the locked state failing to meet the locked state maintenance preset time,
determining the operation mode to be a fourth operation mode for the determining whether the recognition succeeds for the current frame, and
maintaining, for the fourth operation model, the locked state of the terminal with respect to the current frame.

### Claim 8 (depends on 7)

The method of claim 7, further comprising, when the result of the performing of the detecting for the background region change is that the background region change is detected:
determining the operation mode to be a second operation mode for the determining of whether the recognition succeeds for the current frame;
performing the second operation mode, including sequentially performing the face detection, face recognition, and unauthorized object detection with respect to the current frame; and
initializing the state maintenance time.

### Claim 9 (depends on 7)

The method of claim 7, further comprising, when the result of the performing of the detecting for the background region change is that the background region change is not detected, and the state maintenance time of the locked state meets the locked state maintenance preset time:
determining the operation mode to be a second operation mode for the determining of whether the recognition succeeds for the current frame; and
performing the second operation mode, including sequentially performing the face detection, face recognition, and unauthorized object detection with respect to the current frame.

### Claim 10 (depends on 1)

The method of claim 1, further comprising, when the result of the determining between the locked state and the unlocked state is that the terminal is in the locked state, dependent on a face recognition for the previous frame having failed or a plurality of faces having been detected in the previous frame:
determining the operation mode to be a second operation mode for the determining of whether the recognition succeeds for the current frame; and
performing the second operation mode, including sequentially performing the face detection, face recognition, and unauthorized object detection with respect to the current frame.

### Claim 11 (depends on 1)

The method of claim 1, wherein the detecting for the background region change comprises:
determining a threshold value for the background region change based on the region of the tracked face;
calculating a difference between a size of a background region in the previous frame and a size of a background region in the current frame; and
detecting the background region change in response to the calculated difference meeting the threshold value.

### Claim 12

An apparatus, comprising:
one or more processors configured to:
determine between a locked state and an unlocked state of a terminal; and
when a result of the determination between the locked state and the unlocked state of the terminal is that the terminal is in the unlocked state:
track, in a current frame, a face detected in a previous frame;
detect for a background region change between the previous frame and the current frame based on a background region, of the current frame, distinguished from a region of the tracked face in the current frame;
when a result of the detection for the background region change is that the background region change is not detected, determine whether a state maintenance time of the unlocked state fails to meet an unlocked state maintenance preset time, where the state maintenance time of the unlocked state is a time for which the unlocked state of the terminal has been maintained for one or more previous frames; and
in response to the state maintenance time of the unlocked state failing to meet the unlocked state maintenance preset time,
determine an operation mode to be a first operation mode for determining whether recognition succeeds for the current frame,
perform the first operation mode, including performance of face detection with respect to the current frame, and
maintain the unlocked state of the terminal for the current frame when the face is detected as a result of the performance of the face detection, representing that the recognition succeeded for the current frame.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the processor is configured to:
in response to the state maintenance time of the unlocked state meeting the unlocked state maintenance preset time:
determine the operation mode to be a second operation mode for the determining of whether the recognition succeeds for the current frame;
perform the second operation mode, including a sequential performance of the face detection, facial recognition, and unauthorized object detection with respect to the current frame; and
initialize the state maintenance time of the unlocked state.

### Claim 14 (depends on 12)

The apparatus of claim 12, wherein the processor is configured to:
when the result of the detection for the background region change is that the background region change is detected:
determine the operation mode to be a third operation mode for the determining of whether the recognition succeeds for the current frame;
perform the third operation mode, including a sequential performance of unauthorized object detection, the face detection, and face recognition with respect to the current frame; and
initialize the state maintenance time of the unlocked state.

### Claim 15 (depends on 12)

The apparatus of claim 12, wherein the processor is configured to:
when the result of the determination between the locked state and the unlocked state is that is that the terminal is in the locked state, dependent on an unauthorized object having been detected in the previous frame:
track, in the current frame, for the detected unauthorized object;
determine whether a tracking reliability of a result of the tracking for the detected unauthorized object meets a threshold value;
in response to the tracking reliability meeting the threshold value, determine whether a state maintenance time of the locked state fails to meet a locked state maintenance preset time; and
in response to the state maintenance time of the locked state failing to meet the locked state maintenance preset time,
determine the operation mode to be a fourth operation mode for the determining whether the recognition succeeds for the current frame, and
maintain, for the fourth operation mode, the locked state of the terminal with respect to the current frame.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the processor is configured to:
in response to the tracking reliability failing to meet the threshold value:
determine the operation mode to be a third operation mode for the determining of whether the recognition succeeds for the current frame;
perform the third operation mode, including a sequential performance of unauthorized object detection, the face detection, and face recognition with respect to the current frame; and
initialize the state maintenance time of the locked state.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein the processor is configured to:
in response to the state maintenance time of the locked state meeting the locked state maintenance preset time:
determine the operation mode to be a third operation mode for the determining of whether the recognition succeeds for the current frame;
perform the third operation mode, including a sequential performance of unauthorized object detection, the face detection, and face recognition with respect to the current frame; and
initialize the state maintenance time of the locked state.

### Claim 18 (depends on 12)

The apparatus of claim 12, wherein the processor is configured to:
when the result of the determination between the locked state and the unlocked state is that is that the terminal is in the locked state, dependent on a foreground object having not been detected in the previous frame:
perform the detection for the background region change between the previous frame and the current frame;
when a result of the performance of the detection for the background region change is that the background region change is not detected, determine whether a state maintenance time of the locked state fails to meet a locked state maintenance preset time; and
in response to the state maintenance time of the locked state failing to meet the locked state maintenance preset time,
determine the operation mode to be a fourth operation mode for the determining whether the recognition succeeds for the current frame, and
maintain, for the fourth operation model, the locked state of the terminal with respect to the current frame.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the processor is configured to:
when the result of the performance of the detection for the background region change is that the background region change is detected:
determine the operation mode to be a second operation mode for the determining of whether the recognition succeeds for the current frame;
perform the second operation mode, including a sequential performance of the face detection, face recognition, and unauthorized object detection with respect to the current frame; and
initialize the state maintenance time.

### Claim 20 (depends on 18)

The apparatus of claim 18, wherein the processor is configured to:
when the result of the performance of the detection for the background region change is that the background region change is not detected, and the state maintenance time of the locked state meets the locked state maintenance preset time:
determine the operation mode to be a second operation mode for the determining of whether the recognition succeeds for the current frame; and
perform the second operation mode, including a sequential performance of the face detection, face recognition, and unauthorized object detection with respect to the current frame.

### Claim 21 (depends on 12)

The apparatus of claim 12, wherein the processor is configured to:
when the result of the determination between the locked state and the unlocked state is that the terminal is in the locked state, dependent on a face recognition for the previous frame having failed or a plurality of faces having been detected in the previous frame:
determine the operation mode to be a second operation mode for the determining of whether the recognition succeeds for the current frame;
perform the second operation mode, including a sequential performance of the face detection, face recognition, and unauthorized object detection with respect to the current frame; and
selectively permit a user to access the terminal dependent on results of the performed second operation mode.

### Claim 22

An apparatus, comprising:
one or more processors configured to:
perform a first operation mode which includes a face detection with respect to a current frame;
perform another operation mode which includes the face detection and at least one of facial recognition and unauthorized object detection with respect to the current frame, where the first operation mode is performed without performing the facial recognition and without performing the unauthorized object detection;
determine between a locked state and an unlocked state of a terminal for the current frame;
when a result of the determination between the locked state and the unlocked state of the terminal is that the terminal is in the unlocked state, perform one operation mode selected between the first operation mode and the other operation mode for determining whether recognition of a user succeeds for the current frame; and
when a result of the determination between the locked state and the unlocked state of the terminal is that the terminal is in the locked state dependent on a previous frame, determine one or more reasons why the terminal is in the locked state dependent on the previous frame and selectively maintain the terminal in the locked state dependent on the determined one or more reasons, for the determining of whether the recognition succeeds for the current frame.