# US 11,907,269 B2 · Detecting non-obvious relationships between entities from visual data sources > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,269 B2 - Document ID: US11907269B2 - Status: Granted - Kind: B2 - Application number: 17/107,991 - Filed: 2020-12-01 - Published: 2022-06-02 - Granted: 2024-02-20 - Earliest claimed: 2020-12-01 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Srinivasan S. Muthuswamy; Mukesh Kumar; Subhendu Das - Web page: https://patentgrep.com/patent/US11907269B2 ## CPC - G06F 16/288: {Entity relationship models} - G06F 16/24578: {using ranking} - G06F 16/5854: {using shape and object relationship} - G06F 16/587: using geographical or spatial information, e.g. location - G06F 16/9536: Search customisation based on social or collaborative filtering - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract In an approach for detecting non-obvious relationships between entities from visual data sources, a processor calculates a co-occurrence frequency score for an entity pair from visual data. A processor calculates a distance proximity score for the entity pair from the visual data. A processor determines an event type in the visual data. A processor determines a timeline relationship in the visual data. A processor calculates a relationship score based on the co-occurrence frequency score, the distance proximity score, the event type, and the timeline relationship. A processor detects a relationship between the entity pair based on the relationship score. ## Claims ### Claim 1 A computer-implemented method incorporated into a risk analysis application, the computer-implemented method comprising: calculating, by one or more processors, a co-occurrence frequency score for an entity pair from the visual data; calculating, by the one or more processors, a distance proximity score for the entity pair from the visual data; determining, by the one or more processors, an event type in the visual data based on event metadata indicating an environment setting including a nature of the event, information of the number of attendees and a background scene; determining, by the one or more processors, a timeline relationship in the visual data by determining an initial point of a co-occurrence of the entity pair, a frequency of the co-occurrence over time, a time gap in which the co-occurrence does not occur, and the most recent point of the co-occurrence; calculating, by the one or more processors, a relationship score based on the co-occurrence frequency score, the distance proximity score, the event type, and the timeline relationship; and adjusting, by the one or more processors, the relationship score, in a risk analysis application, based on changing weighting values of parameters comprising the co-occurrence frequency score, the distance proximity score, the event type and the timeline relationship, after a relationship risk analysis; and detecting, by the one or more processors, a relationship between the entity pair based on the relationship score. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the visual data includes images from a social media. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein calculating the co-occurrence frequency score comprises counting co-occurrence of the entity pair in each image from the visual data. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, wherein calculating the distance proximity score comprises summing each distance proximity score in each image, each distance proximity score calculated as one divided by the number of entities in between the entity pair. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the event type is a public event. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein calculating the relationship score comprises summing the co-occurrence frequency score and the distance proximity score in consideration of the event type and the timeline relationship, wherein a small setting event has a higher score factor compared to a public event, wherein the timeline relationship is related to the relationship score. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein a higher relationship score indicates a closer relationship between the entity pair. ### Claim 8 A computer program product incorporated into a risk analysis application, the computer program product comprising: one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions comprising: program instructions to calculate a co-occurrence frequency score for an entity pair from the visual data; program instructions to calculate a distance proximity score for the entity pair from the visual data; program instructions to determine an event type in the visual data based on event metadata indicating an environment setting including a nature of the event, information of the number of attendees and a background scene; program instructions to determine a timeline relationship in the visual data by determining an initial point of a co-occurrence of the entity pair, a frequency of the co-occurrence over time, a time gap in which the co-occurrence does not occur, and the most recent point of the co-occurrence; program instructions to calculate a relationship score based on the co-occurrence frequency score, the distance proximity score, the event type, and the timeline relationship; program instructions to adjust the relationship score, in a risk analysis application, based on changing weighting values of parameters comprising the co-occurrence frequency score, the distance proximity score, the event type and the timeline relationship, after a relationship risk analysis; and program instructions to detect a relationship between the entity pair based on the relationship score. ### Claim 9 (depends on 8) The computer program product of claim 8, wherein the visual data includes images from a social media. ### Claim 10 (depends on 9) The computer program product of claim 9, wherein program instructions to calculate the co-occurrence frequency score comprise program instructions to count co-occurrence of the entity pair in each image from the visual data. ### Claim 11 (depends on 9) The computer program product of claim 9, wherein program instructions to calculate the distance proximity score comprise program instructions to sum each distance proximity score in each image, each distance proximity score calculated as one divided by the number of entities in between the entity pair. ### Claim 12 (depends on 8) The computer program product of claim 8, wherein the event type is a public event. ### Claim 13 (depends on 8) The computer program product of claim 8, wherein program instructions to calculate the relationship score comprise program instructions to sum the co-occurrence frequency score and the distance proximity score in consideration of the event type and the timeline relationship, wherein a small setting event has a higher score factor compared to a public event, wherein the timeline relationship is related to the relationship score. ### Claim 14 (depends on 8) The computer program product of claim 8, wherein a higher relationship score indicates a closer relationship between the entity pair. ### Claim 15 A computer system incorporated into a risk analysis application, the computer system comprising: one or more computer processors, one or more computer readable storage media, and program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising: program instructions to calculate a co-occurrence frequency score for an entity pair from the visual data; program instructions to calculate a distance proximity score for the entity pair from the visual data; program instructions to determine an event type in the visual data based on event metadata indicating an environment setting including a nature of the event, information of the number of attendees and a background scene; program instructions to determine a timeline relationship in the visual data by determining an initial point of a co-occurrence of the entity pair, a frequency of the co-occurrence over time, a time gap in which the co-occurrence does not occur, and the most recent point of the co-occurrence; program instructions to calculate a relationship score based on the co-occurrence frequency score, the distance proximity score, the event type, and the timeline relationship; program instructions to adjust the relationship score, in a risk analysis application, based on changing weighting values of parameters comprising the co-occurrence frequency score, the distance proximity score, the event type and the timeline relationship, after a relationship risk analysis; and program instructions to detect a relationship between the entity pair based on the relationship score. ### Claim 16 (depends on 15) The computer system of claim 15, wherein the visual data includes images from a social media. ### Claim 17 (depends on 16) The computer system of claim 16, wherein program instructions to calculate the co-occurrence frequency score comprise program instructions to count co-occurrence of the entity pair in each image from the visual data. ### Claim 18 (depends on 16) The computer system of claim 16, wherein program instructions to calculate the distance proximity score comprise program instructions to sum each distance proximity score in each image, each distance proximity score calculated as one divided by the number of entities in between the entity pair. ### Claim 19 (depends on 15) The computer system of claim 15, wherein program instructions to calculate the relationship score comprise program instructions to sum the co-occurrence frequency score and the distance proximity score in consideration of the event type and the timeline relationship, wherein a small setting event has a higher score factor compared to a public event, wherein the timeline relationship is related to the relationship score. ### Claim 20 (depends on 15) The computer system of claim 15, wherein a higher relationship score indicates a closer relationship between the entity pair.