Inter-object relation recognition apparatus, learned model, recognition method and non-transitory computer readable medium

A learned model for making a computer function comprising:

• a first learning device configured to receive an image, output a first feature amount indicating a feature of the image;
• a second learning device configured to receive the first feature amount output from the first learning device, output a second feature amount of a lower dimension number than the first feature amount, and have a plurality of storage units holding internal states for predetermined steps;
• and
• a triplet unit including a plurality of triplet-units, the triplet-units connected to each storage unit of the second learning device, receiving the second feature amount output from each of the storage units, constituted of first, second, and third recognition units, and outputting combinations of the first, second, and third elements, the first, second, and third recognition units respectively recognizing the first, second, and third elements based on the received second feature amount, outputting probability information of the first, second, and third elements, wherein,
• the triplet unit is configured to select at least one combination of the first, second, and third elements, based on the probability information of the first, second, and third elements output from the first, second, and third recognition units of each of the triplet-units, from the combinations of the first, second, and third elements output from each of the triplet-units, and recognize the selected combination of the first, second, and third elements as the relation between objects included in the image,
• a weighting coefficient of the first learning device, the second learning device, and the triplet unit is learned, based on learning data associating the image and the relations between objects included in the image, and
• when an image of a recognition target is input, it is intended that the first learning device, the second learning device, and the triplet unit perform calculations based on the learned weighting coefficients, and each relation between objects included in the image of the recognition target is recognized.

A recognition method of an inter-object relation recognition apparatus comprising:

• a first learning device configured to receive an image, output a first feature amount indicating a feature of the image;
• a second learning device configured to receive the first feature amount output from the first learning device, output a second feature amount of a lower dimension number than the first feature amount, and have a plurality of storage units holding internal states for predetermined steps;
• and
• a triplet unit including a plurality of triplet-units, the triplet-units connected to each storage unit of the second learning device, receiving the second feature amount output from each of the storage units, constituted of first, second, and third recognition units, and respectively outputting combinations of the first, second, and third elements, the first, second, and third recognition units respectively recognizing the first, second, and third elements based on the received second feature amount, outputting probability information of the first, second, and third elements,
• wherein the triplet unit selects at least one combination of the first, second, and third elements, based on the probability information of the first, second, and third elements output from the first, second, and third recognition units of each of the triplet-units, from the combinations of the first, second, and third elements output from each of the triplet-units, and recognizes and outputs the selected combination of the first, second, and third elements as the relation between objects included in the image.

A non-transitory computer readable medium storing a program of an inter-object relation recognition apparatus comprising:

• a first learning device configured to receive an image, output a first feature amount indicating a feature of the image;
• a second learning device configured to receive the first feature amount output from the first learning device, output a second feature amount of a lower dimension number than the first feature amount, and have a plurality of storage units holding internal states for predetermined steps;
• and
• a triplet unit including a plurality of triplet-units, the triplet-units connected to each storage unit of the second learning device, receiving the second feature amount output from each of the storage units, constituted of first, second, and third recognition units, and respectively outputting combinations of the first, second, and third elements, the first, second, and third recognition units respectively recognizing the first, second, and third elements based on the received second feature amount, outputting probability information of the first, second, and third elements,
• wherein the triplet unit makes a computer to execute a process of selecting at least one combination of the first, second, and third elements, based on the probability information of the first, second, and third elements output from the first, second, and third recognition units of each of the triplet-units, from the combinations of the first, second, and third elements output from each of the triplet-units, and recognizing and outputting the selected combination of the first, second, and third elements as the relations between objects included in the image.