Interaction detection model for identifying human-object interactions in image content

#This disclosure relates generally to machine-learning systems that transform an image by generating metadata or other explanatory data providing descriptions of the image's content. More specifically, but not by way of limitation, this disclosure relates to enhancing image content for searches and other applications by using interaction detection models to identify human-object interactions in the image content.

#Image-analysis applications are often used to derive information about content depicted in one or more images. For instance, objection localization algorithms determine the location of objects in images. Image captioning applications can use these algorithms to describe objects in pictures by combining identified objects with a relationship based on the relative location of the objects. Additionally, neural networks can be used to determine the interaction between people and objects using machine learning to associate person-object pairs with certain interactions.

#Existing image-analysis techniques present disadvantages. For example, current algorithms make assumptions about the interaction based on the person and object identified. If a person is identified next to a bicycle, these algorithms may assume the person is riding a bicycle because that is the most common relationship between the two objects. Other algorithms will attempt not only to identify interactions based on the person and object identified, but will compare the current image to other images of the same action. Therefore, if a person is next to a bicycle, riding a bicycle and washing a bicycle may both be possible interactions between the two objects. The application would then evaluate the image for similarities to other images involving riding and washing.

#Features, embodiments, and advantages of the present disclosure are better understood when the following Detailed Description is read with reference to the accompanying drawings.

#FIG. 1 depicts an example of an operating environment for enhancing image content for use by an image search system or other applications by identifying human-object interactions in the image content, according to certain embodiments of the present disclosure.

#FIG. 2 depicts an example of a process for generating human-interaction metadata that describes or otherwise indicates human-object interactions detected in image content, according to certain embodiments of the present disclosure.

#Certain embodiments involve image-analysis algorithms that use interaction detection models to identify human-object interactions in the image content. For example, an interaction detection model receives an input image and uses various subnets, in combination, to generate metadata or other information describing the human-object interactions. A first subnet (e.g., a pose estimation subnet, such as a convolutional pose machine) identifies which human joints (e.g., elbows, feet, etc.) are depicted in the image, and second subnet (e.g., an object contact subnet) identifies which contact points (e.g., parts of an object) are depicted in the image.

#The following non-limiting example is provided to introduce certain embodiments. In this example, an image-analysis engine accesses an input image that depicts a person riding a bicycle. The image-analysis engine applies an interaction detection network to the input image and thereby generates human-object interaction metadata that can be used to augment or otherwise transform the input image. For instance, the interaction detection network includes a pose estimation subnet for identifying joint locations of humans in an image (e.g., the rider's hands, elbows, feet, etc.) and an object contact subnet for identifying potential contact points in the image (e.g., the bicycle's seat, handles, wheels, etc.). The pose estimation subnet outputs a joint-location heat map indicating detected joints in the input image, and the object contact subnet outputs a contact-point heat map indicating potential contact points on the bicycle in the input image.

#Continuing with this example, these outputted heat maps are provided to an interaction-detection subnet, which is trained to identify an interaction based on one or more joint-object contact pairs in the input image. The interaction-detection subnet determines which parts of the bicycle rider and part of the bicycle are likely to be in contact in input image. For instance, the interaction-detection subnet determines, based on the location of a person's hands, hips, and feet, that a person's hands and feet are in contact with the bicycle's handles, seat, and pedals, respectively. Based on this joint-object contact information, the engine can specifically determine that the person is riding the bicycle rather than interacting with the bicycle in some other manner (e.g., washing the bicycle, carrying the bicycle, or simply walking past the bicycle).