# US 10,121,076 B2 · Recognizing entity interactions in visual media > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,121,076 B2 - Document ID: US10121076B2 - Status: Granted - Kind: B2 - Application number: 15/144,279 - Filed: 2016-05-02 - Published: 2016-08-25 - Granted: 2018-11-06 - Earliest claimed: 2013-03-15 - Assignees: SRI International - Applicants: SRI International - Inventors: Ishani Chakraborty; Hui Cheng; Omar Javed - Web page: https://patentgrep.com/patent/US10121076B2 ## CPC - G06V 20/30: in albums, collections or shared content, e.g. social network photos or video - G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) - G06V 40/16: Human faces, e.g. facial parts, sketches or expressions - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An entity interaction recognition system algorithmically recognizes a variety of different types of entity interactions that may be captured in two-dimensional images. In some embodiments, the system estimates the three-dimensional spatial configuration or arrangement of entities depicted in the image. In some embodiments, the system applies a proxemics-based analysis to determine an interaction type. In some embodiments, the system infers, from a characteristic of an entity detected in an image, an area or entity of interest in the image. ## Claims ### Claim 1 A method for inferring an area of interest in a two-dimensional image depicting at least one person, the method comprising, with a computing system, algorithmically: locating the at least one person in the image; determining, from the image, a spatial configuration of at least a portion of the at least one person located in the image; estimating a three-dimensional position of the person from the determined spatial configuration; analyzing the three-dimensional position of the person using a proxemics analysis; determining a type of human interaction likely depicted in the image based on the proxemics analysis; and inferring an area of interest in the image based on the determined type of human interaction, the area of interest at least partially spaced from the at least one person, and the area of interest having a size that is greater than zero and less than the size of the entire image. ### Claim 2 (depends on 1) The method of claim 1, wherein the method comprises further inferring the area of interest based on a location of the person in the image and a spatial configuration determined for a plurality of characteristics of the person. ### Claim 3 (depends on 2) The method of claim 2, wherein the plurality of characteristics comprises a face pose and a hand location of the person, and the method further comprises estimating the face pose, estimating the hand location based on the estimated face pose, and inferring the area of interest based on the estimated face pose and the estimated hand location. ### Claim 4 (depends on 2) The method of claim 2, wherein the spatial configuration comprises an estimated face pose of the person, and the method comprises computing a direction of the person's gaze from the estimated face pose, and inferring the area of interest based on the computed direction of the person's gaze. ### Claim 5 (depends on 4) The method of claim 4, further comprising estimating a location of the person's hand and inferring the area of interest based on the location of the person's hand. ### Claim 6 (depends on 1) The method of claim 1, wherein the area of interest comprises a plurality of entities of possible interest, and the method comprises filtering the plurality of entities of possible interest to a smaller number of entities of possible interest based on a determined spatial configuration of at least one entity depicted in the image. ### Claim 7 (depends on 1) The method of claim 1, comprising detecting an attribute of the person depicted in the image, computing a size of the attribute detected in the image, comparing the size of the detected attribute to a threshold size, and inferring the area of interest in the image based on the comparison of the size of the detected attribute to the threshold size. ### Claim 8 (depends on 7) The method of claim 7, wherein the attribute of the person comprises the person's face, and the method comprises computing the size of the detected face, comparing the size of the detected face to a threshold face size and inferring the area of interest in the image based on the comparison of the size of the detected face to the threshold face size. ### Claim 9 (depends on 1) The method of claim 1, wherein the inferred area of interest comprises a plurality of entities of possible interest and the method comprises using one or more characteristics of one or more of the entities of possible interest to filter the plurality of entities of possible of interest to a smaller number of entities of possible interest. ### Claim 10 (depends on 1) The method of claim 1, further comprising a surface, and the method comprises determining a three-dimensional arrangement of the surface and another entity depicted in the image based on a characteristic of the surface. ### Claim 11 (depends on 10) The method of claim 10, comprising associating a surface affordance with the surface based on the characteristic of the surface and the three-dimensional arrangement of the surface and a person depicted in the image. ### Claim 12 A non-transitory computer readable medium for storing computer instructions that, when executed by one or more processors causes the one or more processors to perform a method for inferring an area of interest in a two-dimensional image depicting at least one person, the method comprising, algorithmically: locating the at least one person in the image; determining, from the image, a spatial configuration of at least a portion of the at least one person located in the image; estimating a three-dimensional position of the person from the determined spatial configuration; analyzing the three-dimensional position of the person using a proxemics analysis; determining a type of human interaction likely depicted in the image based on the proxemics analysis; and inferring an area of interest in the image based on the determined type of human interaction, the area of interest at least partially spaced from the at least one person, and the area of interest having a size that is greater than zero and less than the size of the entire image. ### Claim 13 A computing system comprising one or more of an image/video tagger, an information retrieval system, and an intelligent assistant and one or more processors to perform a method for inferring an area of interest in a two-dimensional image depicting at least one person, the method comprising, algorithmically: locating the at least one person in the image; determining, from the image, a spatial configuration of at least a portion of the at least one person located in the image; estimating a three-dimensional position of the person from the determined spatial configuration; analyzing the three-dimensional position of the person using a proxemics analysis; determining a type of human interaction likely depicted in the image based on the proxemics analysis; and inferring an area of interest in the image based on the determined type of human interaction, the area of interest at least partially spaced from the at least one person, and the area of interest having a size that is greater than zero and less than the size of the entire image. ### Claim 14 A method for inferring an area of interest in a recorded two-dimensional image using a characteristic of a gaze of a person depicted in the recorded two-dimensional image, the method comprising, with a computing system, algorithmically: detecting the person in the recorded two-dimensional image; estimating a three-dimensional spatial configuration of at least a portion of the person in the image; inferring a characteristic of the person's gaze in the image based on the estimated spatial configuration; determining a proxemics class associated with the recorded two-dimensional image based on the determined three-dimensional spatial configuration; and inferring an area of interest in the image based on at least one of the inferred characteristic of the person's gaze or the proxemics class. ### Claim 15 (depends on 14) The method of claim 14, wherein the estimated spatial configuration comprises an estimated head pose of the detected person and the method comprises inferring a characteristic of the person's gaze in the image based on the estimated head pose. ### Claim 16 (depends on 14) The method of claim 14, comprising classifying the recorded two-dimensional image as depicting a type of human interaction based on the inferred characteristic of the person's gaze. ### Claim 17 (depends on 16) The method of claim 16, wherein the recorded image comprises a plurality of images and the method comprises tracking the person's gaze by repeating the detecting, estimating, and inferring for the plurality of images over a time period, and reclassifying the recorded image as depicting a different type of human interaction based on the tracking of the person's gaze over the time period. ### Claim 18 (depends on 14) The method of claim 14, comprising inferring a direction of the person's gaze, and inferring an area of interest in the recorded two-dimensional image based on the direction of the person's gaze, wherein the area of interest at least partially spaced from the person, and the area of interest has a size that is greater than zero and less than the size of the entire recorded two-dimensional image. ### Claim 19 (depends on 18) The method of claim 18, wherein the inferring an area of interest comprises inferring a three-dimensional area of interest from the recorded two-dimensional image. ### Claim 20 A non-transitory computer readable medium for storing computer instructions that, when executed by one or more processors, causes the one or more processors to perform a method for inferring an area of interest in a recorded two-dimensional image using a characteristic of a gaze of a person depicted in the recorded two-dimensional image, the method comprising, algorithmically: detecting the person in the recorded two-dimensional image; estimating a three-dimensional spatial configuration of at least a portion of the person in the image; inferring a characteristic of the person's gaze in the image based on the estimated spatial configuration; determining a proxemics class associated with the recorded two-dimensional image based on the determined three-dimensional spatial configuration; and inferring an area of interest in the image based on at least one of the inferred characteristic of the person's gaze or the proxemics class. ### Claim 21 A computing system comprising one or more of an image/video tagger, an information retrieval system, and an intelligent assistant and one or more processors to perform a method for inferring an area of interest in a recorded two-dimensional image using a characteristic of a gaze of a person depicted in the recorded two-dimensional image, the method comprising, algorithmically: detecting the person in the recorded two-dimensional image; estimating a three-dimensional spatial configuration of at least a portion of the person in the image; inferring a characteristic of the person's gaze in the image based on the estimated spatial configuration; determining a proxemics class associated with the recorded two-dimensional image based on the determined three-dimensional spatial configuration; and inferring an area of interest in the image based on at least one of the inferred characteristic of the person's gaze or the proxemics class.