# US 2019/0171864 A1 · Inter-object relation recognition apparatus, learned model, recognition method and non-transitory computer readable medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0171864 A1
- Document ID: US20190171864A1
- Status: Published application
- Kind: A1
- Application number: 16/209,116
- Filed: 2018-12-04
- Published: 2019-06-06
- Earliest claimed: 2017-12-06
- Assignees: The University of Tokyo; Toyota Jidosha Kabushiki Kaisha
- Inventors: Hideki Nakayama; Kento Masui; Shintaro Yoshizawa; Akiyoshi Ochiai
- Web page: https://patentgrep.com/patent/US20190171864A1

## CPC

- G06N 3/008: based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
- G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/768: {using context analysis, e.g. recognition aided by known co-occurring patterns}
- G06V 10/82: using neural networks
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/64: Three-dimensional [3D] objects
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An Inter-object relation recognition apparatus includes: a first learning device receiving an image, and outputting a first feature amount of the image; a second learning device receiving the first feature amount, outputting a second feature amount, and having a plurality of storage units holding internal states for predetermined steps; a triplet unit having a plurality of triplet-units, the triplet-units receiving the second feature amount, recognizing first to third elements, and constituted of first to third recognition units outputting probability information. The triplet unit selects at least one combination of the first to third elements, based on the probability information of the first to third elements output from the first to third recognition units of each of the triplet units, from the combinations of the first to third elements output from each of the triplet units, and recognizes and outputs the combination as a relation between objects.

## Claims

### Claim 1

An inter-object relation recognition apparatus configured to perform learning based on learning data associated with an image, and a plurality of relations between objects included in the image, and to recognize and output the relations between objects included in the image using a result of the learning, the apparatus comprising:
a first learning device configured to receive the image and output a first feature amount indicating a feature of the image;
a second learning device configured to receive the first feature amount output from the first learning device and output a second feature amount of a lower dimension number than the first feature amount, and the second learning device including a plurality of storage units holding internal states for predetermined steps; and
a triplet unit including a plurality of triplet-units, the triplet-units connected to each storage unit of the second learning device, receiving the second feature amount output from each of the storage units, constituted of first, second, and third recognition units, and respectively outputting combinations of the first, second, and third elements, the first, second, and third recognition units respectively recognizing the first, second, and third elements based on the received second feature amount, outputting probability information of the first, second, and third elements,
wherein the triplet unit selects at least one combination of the first, second, and third elements, based on the probability information of the first, second, and third elements output from the first, second, and third recognition units of each of the triplet-units, from the combinations of the first, second, and third elements output from each of the triplet-units, and recognizes and outputs the selected combination of the first, second, and third elements as the relation between objects included in the image.

### Claim 2 (depends on 1)

The inter-object relation recognition apparatus according to claim 1, wherein
the second learning device is a recurrent neural network, and
the second learning device and the triplet unit perform the learning by optimizing parameters of a predetermined function, based on learning data associating the image and the relations between objects and hold the optimized parameter as the result of the learning.

### Claim 3 (depends on 1)

The inter-object relation recognition apparatus according to claim 1, wherein
the first and third recognition units respectively output the probability information of the first and third elements to the second recognition unit, and
the second recognition unit recognizes the second element, based on the first and third elements output from the first and third recognition units, and the second feature amount output from the corresponding storage unit of the second learning device, and outputs the probability information of the second element.

### Claim 4 (depends on 1)

The inter-object relation recognition apparatus according to claim 1,
wherein the storage unit of the second learning device is an LSTM (Long Short-Term Memory).

### Claim 5 (depends on 1)

The inter-object relation recognition apparatus according to claim 1,
wherein the first learning device is configured as a convolution type neural network.

### Claim 6

A learned model for making a computer function comprising:
a first learning device configured to receive an image, output a first feature amount indicating a feature of the image;
a second learning device configured to receive the first feature amount output from the first learning device, output a second feature amount of a lower dimension number than the first feature amount, and have a plurality of storage units holding internal states for predetermined steps; and
a triplet unit including a plurality of triplet-units, the triplet-units connected to each storage unit of the second learning device, receiving the second feature amount output from each of the storage units, constituted of first, second, and third recognition units, and outputting combinations of the first, second, and third elements, the first, second, and third recognition units respectively recognizing the first, second, and third elements based on the received second feature amount, outputting probability information of the first, second, and third elements, wherein,
the triplet unit is configured to select at least one combination of the first, second, and third elements, based on the probability information of the first, second, and third elements output from the first, second, and third recognition units of each of the triplet-units, from the combinations of the first, second, and third elements output from each of the triplet-units, and recognize the selected combination of the first, second, and third elements as the relation between objects included in the image,
a weighting coefficient of the first learning device, the second learning device, and the triplet unit is learned, based on learning data associating the image and the relations between objects included in the image, and
when an image of a recognition target is input, it is intended that the first learning device, the second learning device, and the triplet unit perform calculations based on the learned weighting coefficients, and each relation between objects included in the image of the recognition target is recognized.

### Claim 7

A recognition method of an inter-object relation recognition apparatus comprising:
a first learning device configured to receive an image, output a first feature amount indicating a feature of the image;
a second learning device configured to receive the first feature amount output from the first learning device, output a second feature amount of a lower dimension number than the first feature amount, and have a plurality of storage units holding internal states for predetermined steps; and
a triplet unit including a plurality of triplet-units, the triplet-units connected to each storage unit of the second learning device, receiving the second feature amount output from each of the storage units, constituted of first, second, and third recognition units, and respectively outputting combinations of the first, second, and third elements, the first, second, and third recognition units respectively recognizing the first, second, and third elements based on the received second feature amount, outputting probability information of the first, second, and third elements,
wherein the triplet unit selects at least one combination of the first, second, and third elements, based on the probability information of the first, second, and third elements output from the first, second, and third recognition units of each of the triplet-units, from the combinations of the first, second, and third elements output from each of the triplet-units, and recognizes and outputs the selected combination of the first, second, and third elements as the relation between objects included in the image.

### Claim 8

A non-transitory computer readable medium storing a program of an inter-object relation recognition apparatus comprising:
a first learning device configured to receive an image, output a first feature amount indicating a feature of the image;
a second learning device configured to receive the first feature amount output from the first learning device, output a second feature amount of a lower dimension number than the first feature amount, and have a plurality of storage units holding internal states for predetermined steps; and
a triplet unit including a plurality of triplet-units, the triplet-units connected to each storage unit of the second learning device, receiving the second feature amount output from each of the storage units, constituted of first, second, and third recognition units, and respectively outputting combinations of the first, second, and third elements, the first, second, and third recognition units respectively recognizing the first, second, and third elements based on the received second feature amount, outputting probability information of the first, second, and third elements,
wherein the triplet unit makes a computer to execute a process of selecting at least one combination of the first, second, and third elements, based on the probability information of the first, second, and third elements output from the first, second, and third recognition units of each of the triplet-units, from the combinations of the first, second, and third elements output from each of the triplet-units, and recognizing and outputting the selected combination of the first, second, and third elements as the relations between objects included in the image.
