# US 2024/0203164 A1 · Computer-readable recording medium storing information processing program, information processing method, and information processing device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0203164 A1
- Document ID: US20240203164A1
- Status: Published application
- Kind: A1
- Application number: 18/472,098
- Filed: 2023-09-21
- Published: 2024-06-20
- Earliest claimed: 2022-12-19
- Assignees: Fujitsu Limited
- Applicants: Fujitsu Limited
- Inventors: Takuma YAMAMOTO
- Web page: https://patentgrep.com/patent/US20240203164A1

## CPC

- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- G06V 10/22: by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/103: {Static body considered as a whole, e.g. static pedestrian or occupant recognition}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A non-transitory computer-readable recording medium stores an information processing program for causing a computer to execute processing including: acquiring a video; specifying a first region that includes an object included in the video, a second region that includes a person included in the video, and a relation that identifies an interaction between the object and the person, by analyzing the acquired video; determining whether or not the person included in the second region performs abnormal behavior, based on the specified first region and the specified relation; and notifying an alert related to appearance of the person who performs the abnormal behavior in a case of determining that the person performs the abnormal behavior.

## Claims

### Claim 1

A non-transitory computer-readable recording medium storing an information processing program for causing a computer to execute processing comprising:
acquiring a video;
specifying a first region that includes an object included in the video, a second region that includes a person included in the video, and a relation that identifies an interaction between the object and the person, by analyzing the acquired video;
determining whether or not the person included in the second region performs abnormal behavior, based on the specified first region and the specified relation; and
notifying an alert related to appearance of the person who performs the abnormal behavior in a case of determining that the person performs the abnormal behavior.

### Claim 2 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein
the determining whether or not the person performs abnormal behavior includes
identifying a type of the object included in the first region, by analyzing the video, and
determining whether or not the person included in the second region performs the abnormal behavior, by comparing a combination of the identified type of the object and the specified relation with a preset rule.

### Claim 3 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein
the specifying the first region, the second region, and the relation includes
specifying the first region, the second region, and a type of the relation, by analyzing the acquired video, and
the determining whether or not the person performs abnormal behavior includes
specifying a first person that indicates the different relation with respect to the predetermined object, from among a plurality of the persons included in the video, based on the object included in the specified first region and the specified type of the relation, and
determining that the first person performs abnormal behavior.

### Claim 4 (depends on 3)

The non-transitory computer-readable recording medium according to claim 3, wherein
the determining that the first person performs abnormal behavior includes
determining that the first person performs the abnormal behavior, in a case where only the first person causes a first relation that identifies an interaction between the object and the person, with respect to the predetermined object and the person other than the first person indicates a second relation different from the first relation with respect to the predetermined object.

### Claim 5 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein
the specifying the first region, the second region, and the relation includes
generating a scene graph that specifies the first region, the second region, and the relation, for each of the person included in the video, by inputting the acquired video into a machine learning model, and
the determining whether or not the person performs abnormal behavior includes
specifying a first person that indicates the predetermined relation with respect to the predetermined object, from among a plurality of the persons included in the video, by analyzing the scene graph and
determining that the first person performs the abnormal behavior.

### Claim 6 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein
the specifying the first region, the second region, and the relation includes
extracting a first feature amount that corresponds to the first region or the second region from the video,
detecting the object and the person included in the video from the extracted first feature amount,
generating a second feature amount obtained by combining the first feature amounts included in the object or the person in at least one pair of a plurality of the detected objects, a plurality of the persons, or the object and the person, or any combination of the plurality of detected objects, the plurality of persons, and the object and the person,
generating a first map that indicates the relation that identifies at least one of the interactions of the plurality of objects, or the plurality of persons, or the object and the person, or any combination of the plurality of objects, the plurality of persons, or the object and the person, based on the first feature amount and the second feature amount,
extracting a fourth feature amount, based on a third feature amount obtained by converting the first feature amount and the first map, and
specifying the relation from the fourth feature amount.

### Claim 7 (depends on 1)

The non-transitory computer-readable recording medium according to claim 1, wherein
the specifying the first region, the second region, and the relation includes
specifying the first region, the second region, and the relation, by inputting the acquired video into a machine learning model, and
the machine learning model is a model for human object interaction detection (HOID) trained to identify information regarding a first class that indicates the object and the first region that indicates a region where the object appears, information regarding a second class that indicates the person and the second region that indicates a region where the person appears, and an interaction between the first class and the second class.

### Claim 8

An information processing method comprising:
acquiring a video;
specifying a first region that includes an object included in the video, a second region that includes a person included in the video, and a relation that identifies an interaction between the object and the person, by analyzing the acquired video;
determining whether or not the person included in the second region performs abnormal behavior, based on the specified first region and the specified relation; and
notifying an alert related to appearance of the person who performs the abnormal behavior in a case of determining that the person performs the abnormal behavior.

### Claim 9 (depends on 8)

The information processing method according to claim 8, wherein
the determining whether or not the person performs abnormal behavior includes
identifying a type of the object included in the first region, by analyzing the video, and
determining whether or not the person included in the second region performs the abnormal behavior, by comparing a combination of the identified type of the object and the specified relation with a preset rule.

### Claim 10 (depends on 8)

The information processing method according to claim 8, wherein
the specifying the first region, the second region, and the relation includes
specifying the first region, the second region, and a type of the relation, by analyzing the acquired video, and
the determining whether or not the person performs abnormal behavior includes
specifying a first person that indicates the different relation with respect to the predetermined object, from among a plurality of the persons included in the video, based on the object included in the specified first region and the specified type of the relation, and
determining that the first person performs abnormal behavior.

### Claim 11 (depends on 10)

The information processing method according to claim 10, wherein
the determining that the first person performs abnormal behavior includes
determining that the first person performs the abnormal behavior, in a case where only the first person causes a first relation that identifies an interaction between the object and the person, with respect to the predetermined object and the person other than the first person indicates a second relation different from the first relation with respect to the predetermined object.

### Claim 12 (depends on 8)

The information processing method according to claim 8, wherein
the specifying the first region, the second region, and the relation includes
generating a scene graph that specifies the first region, the second region, and the relation, for each of the person included in the video, by inputting the acquired video into a machine learning model, and
the determining whether or not the person performs abnormal behavior includes
specifying a first person that indicates the predetermined relation with respect to the predetermined object, from among a plurality of the persons included in the video, by analyzing the scene graph and
determining that the first person performs the abnormal behavior.

### Claim 13 (depends on 8)

The information processing method according to claim 8, wherein
the specifying the first region, the second region, and the relation includes
extracting a first feature amount that corresponds to the first region or the second region from the video,
detecting the object and the person included in the video from the extracted first feature amount,
generating a second feature amount obtained by combining the first feature amounts included in the object or the person in at least one pair of a plurality of the detected objects, a plurality of the persons, or the object and the person, or any combination of the plurality of detected objects, the plurality of persons, and the object and the person,
generating a first map that indicates the relation that identifies at least one of the interactions of the plurality of objects, or the plurality of persons, or the object and the person, or any combination of the plurality of objects, the plurality of persons, or the object and the person, based on the first feature amount and the second feature amount,
extracting a fourth feature amount, based on a third feature amount obtained by converting the first feature amount and the first map, and
specifying the relation from the fourth feature amount.

### Claim 14 (depends on 8)

The information processing method according to claim 8, wherein
the specifying the first region, the second region, and the relation includes
specifying the first region, the second region, and the relation, by inputting the acquired video into a machine learning model, and
the machine learning model is a model for human object interaction detection (HOID) trained to identify information regarding a first class that indicates the object and the first region that indicates a region where the object appears, information regarding a second class that indicates the person and the second region that indicates a region where the person appears, and an interaction between the first class and the second class.

### Claim 15

An information processing device comprising:
a memory; and
a processor coupled to the memory and configured to perform a processing of:
acquiring a video;
specifying a first region that includes an object included in the video, a second region that includes a person included in the video, and a relation that identifies an interaction between the object and the person, by analyzing the acquired video;
determining whether or not the person included in the second region performs abnormal behavior, based on the specified first region and the specified relation; and
notifying an alert related to appearance of the person who performs the abnormal behavior in a case of determining that the person performs the abnormal behavior.

### Claim 16 (depends on 15)

The information processing device according to claim 15, wherein
the determining whether or not the person performs abnormal behavior includes
identifying a type of the object included in the first region, by analyzing the video, and
determining whether or not the person included in the second region performs the abnormal behavior, by comparing a combination of the identified type of the object and the specified relation with a preset rule.

### Claim 17 (depends on 15)

The information processing device according to claim 15, wherein
the specifying the first region, the second region, and the relation includes
specifying the first region, the second region, and a type of the relation, by analyzing the acquired video, and
the determining whether or not the person performs abnormal behavior includes
specifying a first person that indicates the different relation with respect to the predetermined object, from among a plurality of the persons included in the video, based on the object included in the specified first region and the specified type of the relation, and
determining that the first person performs abnormal behavior.

### Claim 18 (depends on 17)

The information processing device according to claim 17, wherein
the determining that the first person performs abnormal behavior includes
determining that the first person performs the abnormal behavior, in a case where only the first person causes a first relation that identifies an interaction between the object and the person, with respect to the predetermined object and the person other than the first person indicates a second relation different from the first relation with respect to the predetermined object.

### Claim 19 (depends on 15)

The information processing device according to claim 15, wherein
the specifying the first region, the second region, and the relation includes
generating a scene graph that specifies the first region, the second region, and the relation, for each of the person included in the video, by inputting the acquired video into a machine learning model, and
the determining whether or not the person performs abnormal behavior includes
specifying a first person that indicates the predetermined relation with respect to the predetermined object, from among a plurality of the persons included in the video, by analyzing the scene graph and
determining that the first person performs the abnormal behavior.

### Claim 20 (depends on 15)

The information processing device according to claim 15, wherein
the specifying the first region, the second region, and the relation includes
extracting a first feature amount that corresponds to the first region or the second region from the video,
detecting the object and the person included in the video from the extracted first feature amount,
generating a second feature amount obtained by combining the first feature amounts included in the object or the person in at least one pair of a plurality of the detected objects, a plurality of the persons, or the object and the person, or any combination of the plurality of detected objects, the plurality of persons, and the object and the person,
generating a first map that indicates the relation that identifies at least one of the interactions of the plurality of objects, or the plurality of persons, or the object and the person, or any combination of the plurality of objects, the plurality of persons, or the object and the person, based on the first feature amount and the second feature amount,
extracting a fourth feature amount, based on a third feature amount obtained by converting the first feature amount and the first map, and
specifying the relation from the fourth feature amount.
