# US 2024/0202934 A1 · Information processing program, information processing method, and information processing device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202934 A1
- Document ID: US20240202934A1
- Status: Published application
- Kind: A1
- Application number: 18/452,200
- Filed: 2023-08-18
- Published: 2024-06-20
- Earliest claimed: 2022-12-14
- Assignees: Fujitsu Limited
- Applicants: Fujitsu Limited
- Inventors: Shun KOHATA
- Web page: https://patentgrep.com/patent/US20240202934A1

## CPC

- G06Q 30/015: Providing customer assistance, e.g. assisting a customer within a business location or via helpdesk
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06N 3/045: Combinations of networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06N 3/091: Active learning
- G06Q 30/0204: Market segmentation
- G06T 2207/30196: Human being; Person
- G06V 10/82: using neural networks
- G06V 2201/07: Target detection
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A non-transitory computer-readable recording medium has stored therein an information processing program that causes a computer to execute a process including acquiring a video analyzing the acquired video identifying, based on a result of the analyzing, a first area containing a first object included in the acquired video, a second area containing a second object included in the acquired video, and a relationship that identifies interaction between the first object and the second object based on the specified relationship, selecting a model that is relevant to any one of the first object and the second object from a plurality of models and outputting the selected model.

## Claims

### Claim 1

A non-transitory computer-readable recording medium having stored therein an information processing program that causes a computer to execute a process comprising:
acquiring a video;
analyzing the acquired video;
identifying, based on a result of the analyzing, a first area containing a first object included in the acquired video, a second area containing a second object included in the acquired video, and a relationship that identifies interaction between the first object and the second object;
based on the specified relationship, selecting a model that is relevant to any one of the first object and the second object from a plurality of models; and
outputting the selected model.

### Claim 2 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein the process further includes:
tracing, based on the result of the analyzing, the first object and the second object;
based on the specified relationship, selecting a machine learning model to be applied to the second object from the machine learning models;
inputting an image of the tracked second object to the selected machine learning model;
specifying relevant information on the second object based on the output from the selected machine learning model; and
causing a display device that is relevant to the tracked second object to output the specified relevant information relevant to the second object.

### Claim 3 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein
the first object is a person,
the second object is a product, and
the process further includes:
referring to a storage in which the specified relationship and a plurality of serving-customer models that define content of serving a customer are associated;
specifying a serving-customer model corresponding to the specified relationship from the serving-customer models;
based on the specified serving-customer model, specifying content of serving a customer that is for the person represented by the first object and that is relevant to the object represented by the second object; and
transmitting the specified content of serving a customer to a terminal device that a sales staff uses.

### Claim 4 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein
the first object is a person,
the second object is a product, and
the process further includes:
referring to a storage in which the specified relationship and a plurality of trained machine learning models that output product information are associated;
specifying a machine learning model corresponding to the specified relationship from the trained machine learning models;
inputting an image of the product represented by the specified second object to the specified machine learning model;
specifying product information outputted from the specified machine learning model; and
causing a terminal device that the person represented by the first object uses to display the specified product information.

### Claim 5 (depends on 4)

The non-transitory computer-readable recording medium according to claim 4, wherein the process further includes:
tracing, based on the result of the analyzing, a position of a person in a store;
based on the tracked position of the person in the store, specifying a terminal device that the person uses; and
causing the specified terminal device to display the specified product information.

### Claim 6 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein the process further includes:
inputting the video to a given model; and
specifying, based on the output from the given model, the first area, the second area, and the relationship and
the given model is a model for HOID (Human Object Interaction Detection) on which machine learning is executed in order to identify first area information that presents a first class presenting a person who purchases a product and an area in which the person appears, second area information that presents a second class presenting an object containing a product and an area in which the object appears, and interaction between the first class and the second class.

### Claim 7

An information processing method comprising:
acquiring a video;
analyzing the acquired video;
identifying, based on a result of the analyzing, a first area containing a first object included in the acquired video, second area containing a second object included in the acquired video, and a relationship that identifies interaction between the first object and the second object;
based on the specified relationship, selecting a model that is relevant to any one of the first object and the second object from a plurality of models by using a processor; and
outputting the selected model.

### Claim 8 (depends on 7)

The information processing method according to claim 7, further including:
tracing, based on the result of the analyzing, the first object and the second object;
based on the specified relationship, selecting a machine learning model to be applied to the second object from the machine learning models;
inputting an image of the tracked second object to the selected machine learning model;
specifying relevant information on the second object based on the output from the selected machine learning model; and
causing a display device that is relevant to the tracked second object to output the specified relevant information relevant to the second object.

### Claim 9 (depends on 7)

The information processing method according to claim 7, wherein
the first object is a person,
the second object is a product, and
the method further includes:
referring to a storage in which the specified relationship and a plurality of serving-customer models that define content of serving a customer are associated;
specifying a serving-customer model corresponding to the specified relationship from the serving-customer models;
based on the specified serving-customer model, specifying content of serving a customer that is for the person represented by the first object and that is relevant to the object represented by the second object; and
transmitting the specified content of serving a customer to a terminal device that a sales staff uses.

### Claim 10 (depends on 7)

The information processing method according to claim 7, wherein
the first object is a person,
the second object is a product, and
the method further includes:
referring to a storage in which the specified relationship and a plurality of trained machine learning models that output product information are associated;
specifying a machine learning model corresponding to the specified relationship from the trained machine learning models;
inputting an image of the product represented by the specified second object to the specified machine learning model;
specifying product information outputted from the specified machine learning model; and
causing a terminal device that the person represented by the first object uses to display the specified product information.

### Claim 11 (depends on 10)

The information processing method according to claim 10, including:
tracing, based on the result of the analyzing, a position of a person in a store;
based on the tracked position of the person in the store, specifying a terminal device that the person uses; and
causing the specified terminal device to display the specified product information.

### Claim 12 (depends on 7)

The information processing method according to claim 7, including:
inputting the video to a given model; and
specifying, based on the output from the given model, the first area, the second area, and the relationship, and
the given model is a model for HOID (Human Object Interaction Detection) on which machine learning is executed in order to identify first area information that presents a first class presenting a person who purchases a product and an area in which the person appears, second area information that presents a second class presenting an object containing a product and an area in which the object appears, and interaction between the first class and the second class.

### Claim 13

An information processing device comprising:
a memory; and
a processor coupled to the memory and configured to:
acquire a video;
analyze the acquired video;
identify, based on analyzing the acquired video, a first area containing a first object included in the acquired video, a second area containing a second object included in the acquired video, and a relationship that identifies interaction between the first object and the second object;
based on the specified relationship, select a model that is relevant to any one of the first object and the second object from a plurality of models; and
output the selected model.

### Claim 14 (depends on 13)

The information processing device according to claim 13, wherein the processor is further configured to:
trace, based on analyzing the acquired video, the first object and the second object;
based on the specified relationship, select a machine learning model to be applied to the second object from the machine learning models;
input an image of the tracked second object to the selected machine learning model;
specify relevant information on the second object based on the output from the selected machine learning model; and
cause a display device that is relevant to the tracked second object to output the specified relevant information relevant to the second object.

### Claim 15 (depends on 13)

The information processing device according to claim 13, wherein
the first object is a person,
the second object is a product, and
the processor is further configured to:
refer to a storage in which the specified relationship and a plurality of serving-customer models that define content of serving a customer are associated;
specify a serving-customer model corresponding to the specified relationship from the serving-customer models;
based on the specified serving-customer model, specify content of serving a customer that is for the person represented by the first object and that is relevant to the object represented by the second object; and
transmit the specified content of serving a customer to a terminal device that a sales staff uses.

### Claim 16 (depends on 13)

The information processing device according to claim 13, wherein
the first object is a person,
the second object is a product, and
the processor is further configured to:
refer to a storage in which the specified relationship and a plurality of trained machine learning models that output product information are associated;
specify a machine learning model corresponding to the specified relationship from the trained machine learning models;
input an image of the product represented by the specified second object to the specified machine learning model;
specify product information outputted from the specified machine learning model; and
cause a terminal device that the person represented by the first object uses to display the specified product information.

### Claim 17 (depends on 16)

The information processing device according to claim 16, wherein the processor is further configured to:
trace, based on the result of the analyzing, a position of a person in a store;
based on the tracked position of the person in the store, specify a terminal device that the person uses; and
cause the specified terminal device to display the specified product information.

### Claim 18 (depends on 13)

The information processing device according to claim 13, wherein the processor is further configured to:
input the video to a given model; and
specify, based on the output from the given model, the first area, the second area, and the relationship, and
the given model is a model for HOID (Human Object Interaction Detection) on which machine learning is executed in order to identify first area information that presents a first class presenting a person who purchases a product and an area in which the person appears, second area information that presents a second class presenting an object containing a product and an area in which the object appears, and interaction between the first class and the second class.
