# US 11,308,334 B2 · Method and apparatus for integration of detected object identifiers and semantic scene graph networks for captured visual scene behavior estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,308,334 B2 - Document ID: US11308334B2 - Status: Granted - Kind: B2 - Application number: 16/659,503 - Filed: 2019-10-21 - Published: 2020-02-13 - Granted: 2022-04-19 - Earliest claimed: 2017-11-28 - Assignees: Visual Semantics, Inc. - Applicants: Visual Semantics, Inc. - Inventors: Gareth Ian Block - Web page: https://patentgrep.com/patent/US11308334B2 ## CPC - G06N 3/08: Learning methods - G06V 10/82: using neural networks - G06F 18/24: Classification techniques - G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06V 10/764: using classification, e.g. of video objects - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A computer system executing code instructions of a visual scene feature extractor and classifier system comprising a processor executing object recognition on a captured image of a visual scene of interest including a plurality of objects in the visual scene of interest to identify a first object from the plurality of objects with a set of feature attributes wherein the feature attributes include an image view of the first object where the processor executing code instructions for the visual scene feature extractor and classifier to generate a first feature vector value for a first feature attribute of the first object. The processor to generate a first feature space including a neighborhood of labeled, previously-observed first feature vector values around the first feature vector value and to generate a second feature vector value for a second feature attribute of the first object, wherein the second feature vector value is within a second feature space including a neighborhood of labeled, previously-observed second feature vector values and the processor to generate association edges between the first and second feature vector values and the first object and between labeled, previously-observed first and second feature vector values and candidate objects corresponding to previous observations to establish a multiple feature graph for the first object for determining weighted likelihood that the first object may be identified as at least one previously observed candidate object. ## Claims ### Claim 1 A computer system executing code of visual scene feature extractor and feature classifier system comprising: a processor executing object recognition on a captured image of a visual scene of interest to identify a first object and a second object in the visual scene of interest and a set of feature attributes captured for the first and second object; the processor executing code instructions for the visual scene feature extractor and classifier to assign a first object candidate node for the first object and a second object candidate node for the second object; the processor associating feature vector values for feature attributes of the first object with the first candidate object node and of the second object with the second object candidate node to relate to labeled, previously-observed feature vector values for assessment of relationship to the first object or the second object; the processor to determine an association edge between the first object candidate node and the second object candidate node, wherein the association edge represents a quantified association likelihood between the first object candidate node and the second object candidate node; the processor to determine a likelihood for accuracy of conclusions of predicted semantic behavior of the first object based in part on the feature vector values and boosted by the association edge providing contextualization between the first object candidate node and the second object candidate node; and the processor generating an indication of the predicted semantic behavior. ### Claim 2 (depends on 1) The computer system of claim 1 further comprising: the processor to generate labels for association edges of previously-observed instances of candidate objects having the plurality of labeled, previously-observed feature vector values to generate likelihoods for the identity of at least the first object or the second object. ### Claim 3 (depends on 1) The computer system of claim 1, wherein the feature attributes include an image view of the first object and an image view of the second object. ### Claim 4 (depends on 1) The computer system of claim 1 wherein the feature attributes include a plurality of image views of the first object or the second object having feature vector values for the first object or the second object in a multi-feature graph to assess the first object via the first object candidate node. ### Claim 5 (depends on 1) The computer system of claim 1 further comprising: the processor to determine a location attribute candidate node for the visual scene of interest and a location association edge between the location attribute candidate node and the first object candidate node or the second object candidate node including a likelihood of accuracy for that location association edge boosting conclusions of predicted semantic behavior of the first object via contextual information about the location of the first object or second object. ### Claim 6 (depends on 1) The computer system of claim 1 further comprising: the processor to determine an attribute node for the visual scene of interest and an attribute association edge between the attribute node and the first or second object candidate node boosting conclusions of predicted semantic behavior of the first object, wherein the attribute node may relate to an environment attribute of the visual scene of interest. ### Claim 7 (depends on 1) The computer system of claim 1 further comprising: the processor executing code to generate conclusions of predicted semantic behavior of the first object veracity of the identity of the first object and the identity of the second object and relationships between the first object and the second object indicated by at least one association edge. ### Claim 8 (depends on 1) The computer system of claim 1 further comprising: the processor executing object recognition to selectively identify object candidate nodes for the first object and the second object in the visual scene of interest based on a predefined, minimum threshold of similarity. ### Claim 9 (depends on 1) The computer system of claim 1 wherein the first object is a first person in the visual scene of interest, the second object is a second person in the visual scene of interest, and the association edge between the first object candidate node and the second object candidate node includes an affinity label predicting a relationship between the first person and the second person. ### Claim 10 A computer system executing code of visual scene feature extractor and feature classifier system comprising: a processor executing object recognition on a captured image of a visual scene of interest to identify a first object and a second object in the visual scene of interest and a set of feature attributes captured for the first and second object; the processor executing code instructions for the visual scene feature extractor and classifier to assign a first object candidate node for the first object and a second object candidate node for the second object and associating feature vector values for feature attributes of the first object with the first candidate object node and of the second object with the second object candidate node, the feature vector values relating to currently observable or previously-observed features of the first object or the second object; the processor to determine an association edge between the first object candidate node and the second object candidate node, where the association edge represents a quantified association likelihood between the first object candidate node and the second object candidate node to boost an ontological conclusion about the first object such as an identity of the first object; and the processor generating an indication of the likelihood of the ontological conclusion about the first object. ### Claim 11 (depends on 10) The computer system of claim 10 further comprising: the visual scene feature extractor to predict a semantic behavior of the first object and the second object by determining a weighted likelihood value for the association edge with a labeled affinity between the object candidate nodes for the first object and the second object describing a proposed relationship conclusion. ### Claim 12 (depends on 10) The computer system of claim 10, wherein the feature attributes include an image view of the first object and an image view of the second object. ### Claim 13 (depends on 10) The computer system of claim 10 further comprising: the processor to generate labels for association edges of previously-observed instances of candidate objects having the plurality of labeled, previously-observed feature vector values to further the indication of the likelihood of the ontological conclusion about the first object. ### Claim 14 (depends on 10) The computer system of claim 10 further comprising: the processor to determine a location attribute candidate node for the visual scene of interest and a location association edge between the location attribute candidate node and the first object candidate node or the second object candidate node including a likelihood of accuracy for that location association edge boosting conclusions of predicted semantic behavior of the first object via contextual information about the location of the first object or second object. ### Claim 15 (depends on 10) The computer system of claim 10 further comprising: the processor to determine an attribute node for the visual scene of interest and an attribute association edge between the attribute node and the first or second object candidate node boosting conclusions of predicted semantic behavior of the first object, wherein the attribute node may relate to an environment attribute of the visual scene of interest. ### Claim 16 (depends on 10) The computer system of claim 10, wherein the first object is a first person in the visual scene of interest, the second object is a second person in the visual scene of interest, and the association edge between the first object candidate node and the second object candidate node includes an affinity label predicting a relationship between the first person and the second person. ### Claim 17 A computer system executing code of visual scene feature extractor and feature classifier system comprising: a processor executing object recognition on a captured image of a visual scene of interest to identify a first object and a second object in the visual scene of interest and a first set of feature attributes captured for the first and a second set of feature attributes captured for the second object; the processor executing code instructions for the visual scene feature extractor and classifier to assign a first object candidate node for the first object and a second object candidate node for the second object and associating feature vector values for the first feature attributes of the first object with the first candidate object node and for the second feature attributes of the second object with the second object candidate node, the feature vector values relating to currently observable or previously-observed features of the first object or the second object; the processor to determine an association edge between the first object candidate node and the second object candidate node, where the association edge represents a quantified association likelihood between the first object candidate node and the second object candidate node to boost an ontological conclusion about the first object such as an identity of the first object, where the association edge between the first object candidate node and the second object candidate node includes an affinity label predicting a relationship between the first object and the second object; and the processor generating an indication of the likelihood of the ontological conclusion about the first object. ### Claim 18 (depends on 17) The computer system of claim 17 further comprising: the visual scene feature extractor to predict a semantic behavior of the first object and the second object by determining a weighted likelihood value for the association edge with a labeled affinity between the object candidate nodes for the first object and the second object describing a proposed relationship conclusion. ### Claim 19 (depends on 17) The computer system of claim 17 wherein the first object is a first person in the visual scene of interest, the second object is a second person in the visual scene of interest, and the association edge between the first object candidate node and the second object candidate node includes an affinity label predicting a relationship between the first person and the second person. ### Claim 20 (depends on 17) The computer system of claim 17, wherein the feature attributes include an image view of the first object and an image view of the second object.