# US 2023/0267744 A1 · Non-transitory computer readable recording medium, setting method, detection method, setting apparatus, and detection appratus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267744 A1
- Document ID: US20230267744A1
- Status: Published application
- Kind: A1
- Application number: 17/978,980
- Filed: 2022-11-02
- Published: 2023-08-24
- Earliest claimed: 2022-02-18
- Assignees: Fujitsu Limited
- Applicants: Fujitsu Limited
- Inventors: Sho IWASAKI; Daisuke UCHIDA; Genta Suzuki
- Web page: https://patentgrep.com/patent/US20230267744A1

## CPC

- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06T 7/50: Depth or shape recovery
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20044: Skeletonization; Medial axis transform
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A non-transitory computer-readable recording medium has stored therein a setting program that causes a computer to execute a process, the process including acquiring a video from a camera, identifying a depth indicating a distance from the camera to each of constituent elements of the video acquired from the camera, generating a three-dimensional in-store model, generating skeleton information on a person who moves inside the store from the video acquired from the camera, setting a range and a direction of an aisle in the store in the generated three-dimensional in-store model based on a change in the generated skeleton information and setting a detection line in the storage based on the range and the direction of the aisle in the store, the detection line for detecting that the person has extended a hand to a product.

## Claims

### Claim 1

A non-transitory computer-readable recording medium having stored therein a setting program that causes a computer to execute a process, the process comprising:
acquiring a video from a camera that is set in a store that has a storage in which a product is stored;
identifying a depth indicating a distance from the camera to each of constituent elements of the video acquired from the camera, by inputting the acquired video into a machine learning model;
generating a three-dimensional in-store model that is configured with the identified depth indicating the distance from the camera to each of the constituent elements of the video;
generating skeleton information on a person who moves inside the store from the video acquired from the camera;
setting a range and a direction of an aisle in the store in the generated three-dimensional in-store model based on a change in the generated skeleton information; and
setting a detection line in the storage based on the range and the direction of the aisle in the store, the detection line for detecting that the person has extended a hand to a product.

### Claim 2 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein the setting the range and the direction of the aisle in the store includes
identifying a change of a position of a foot included in the skeleton information on the person; and
setting, as the range of the aisle, a polygon that includes a change of the identified position of the foot.

### Claim 3 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein the setting the range and the direction of the aisle in the store includes
identifying a change of a position of a foot included in the skeleton information on the person; and
setting the direction based on a displacement directional vector that is based on the change of the position of the foot.

### Claim 4

A non-transitory computer-readable recording medium having stored therein a detection program that causes a computer to execute a process, the process comprising:
identifying a detection line for detecting that a person has extended a hand to a product, the detection line being included in setting information, the setting information being set in a storage, based on range and a direction of an aisle in a store, the range and the direction being included in a three-dimensional in-store model that is configured with a depth indicating a distance from a camera that is set in the store to each of constituent elements of a video from the camera;
identifying a part corresponding to a hand of the person based on skeleton information on the person generated from a video in which an inside of the store is captured; and
detecting that the person has extended the hand to a product that is stored in the storage, based on a positional relationship between the identified part corresponding to the hand of the person and the detection line included in the setting information.

### Claim 5

A setting method comprising:
acquiring a video from a camera that is set in a store that has a storage in which a product is stored;
identifying a depth indicating a distance from the camera to each of constituent elements of the video acquired from the camera, by inputting the acquired video into a machine learning model;
generating a three-dimensional in-store model that is configured with the identified depth indicating the distance from the camera to each of the constituent elements of the video;
generating skeleton information on a person who moves inside the store from the video acquired from the camera;
setting a range and a direction of an aisle in the store in the generated three-dimensional in-store model based on a change in the generated skeleton information; and
setting a detection line in the storage based on the range and the direction of the aisle in the store, the detection line for detecting that the person has extended a hand to a product, by using a processor.

### Claim 6 (depends on 5)

The setting method according to claim 5, wherein the setting the range and the direction of the aisle in the store includes
identifying a change of a position of a foot included in the skeleton information on the person; and
setting, as the range of the aisle, a polygon that includes a change of the identified position of the foot.

### Claim 7 (depends on 5)

The setting method according to claim 5, wherein setting the range and the direction of the aisle in the store includes
identifying a change of a position of a foot included in the skeleton information on the person; and
setting the direction based on a displacement directional vector that is based on the change of the position of the foot.

### Claim 8

A setting apparatus comprising:
a memory; and
a processor coupled to the memory and configured to:
acquire a video from a camera that is set in a store that has a storage in which a product is stored;
identify a depth indicating a distance from the camera to each of constituent elements of the video acquired from the camera, by inputting the acquired video into a machine learning model;
generate a three-dimensional in-store model that is configured with the identified depth indicating the distance from the camera to each of the constituent elements of the video;
generate skeleton information on a person who moves inside the store from the video acquired from the camera;
set a range and a direction of an aisle in the store in the generated three-dimensional in-store model based on a change in the generated skeleton information; and
set a detection line in the storage based on the range and the direction of the aisle in the store, the detection line for detecting that the person has extended a hand to a product.

### Claim 9 (depends on 8)

The setting apparatus according to claim 8, wherein the processor is further configured to
identify a change of a position of a foot included in the skeleton information on the person; and
set, as the range of the aisle, a polygon that includes a change of the identified position of the foot.

### Claim 10 (depends on 8)

A setting apparatus according to claim 8, wherein the processor is further configured to
identify a change of a position of a foot included in the skeleton information on the person; and
set the direction based on a displacement directional vector that is based on the change of the position of the foot.
