# US 10,452,923 B2 · Method and apparatus for integration of detected object identifiers and semantic scene graph networks for captured visual scene behavior estimation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,452,923 B2 - Document ID: US10452923B2 - Status: Granted - Kind: B2 - Application number: 15/823,916 - Filed: 2017-11-28 - Published: 2019-05-30 - Granted: 2019-10-22 - Earliest claimed: 2017-11-28 - Assignees: Visual Semantics, Inc. - Applicants: Visual Semantics, Inc. - Inventors: Gareth Ian Block - Web page: https://patentgrep.com/patent/US10452923B2 ## CPC - G06N 3/08: Learning methods - G06V 10/82: using neural networks - G06F 18/24: Classification techniques - G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/09: Supervised learning - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06V 10/764: using classification, e.g. of video objects - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06F 18/2411: based on the proximity to a decision surface, e.g. support vector machines - G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound - G06N 20/10: using kernel methods, e.g. support vector machines [SVM] - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A computer system executing code instructions of a visual scene feature extractor and classifier system comprising a processor executing object recognition on a captured image of a visual scene of interest including a plurality of objects in the visual scene of interest to identify a first object from the plurality of objects with a set of feature attributes wherein the feature attributes include an image view of the first object where the processor executing code instructions for the visual scene feature extractor and classifier to generate a first feature vector value for a first feature attribute of the first object. The processor to generate a first feature space including a neighborhood of labeled, previously-observed first feature vector values around the first feature vector value and to generate a second feature vector value for a second feature attribute of the first object, wherein the second feature vector value is within a second feature space including a neighborhood of labeled, previously-observed second feature vector values and the processor to generate association edges between the first and second feature vector values and the first object and between labeled, previously-observed first and second feature vector values and candidate objects corresponding to previous observations to establish a multiple feature graph for the first object for determining weighted likelihood that the first object may be identified as at least one previously observed candidate object. ## Claims ### Claim 1 A computer system executing code instructions of a visual scene feature extractor and classifier system comprising: a bus interface for receiving a captured image of a visual scene of interest including a plurality of objects in the visual scene of interest; a processor executing object recognition to identify a first object from the plurality of objects with a set of feature attributes wherein the feature attributes include an image view of the first object; the processor executing code instructions for the visual scene feature extractor and classifier to generate a first feature vector value for a first feature attribute of the first object; the processor to generate a first feature space including a neighborhood of labeled, previously-observed first feature vector values around the first feature vector value; the processor to generate a second feature vector value for a second feature attribute of the first object, wherein the second feature vector value is within a second feature space including a neighborhood of labeled, previously-observed second feature vector values; the processor to generate association edges between the first and second feature vector values and the first object; and the processor to generate association edges between labeled, previously-observed first and second feature vector values and candidate objects corresponding to previous observations to establish a multiple feature graph for the first object for determining weighted likelihood that the first object may be identified as at least one previously observed candidate object. ### Claim 2 (depends on 1) The computer system operating the system of claim 1 further comprising: the processor to determine nearness of the first feature vector value to the plurality of labeled, prior-observed first feature vector values within the first feature space to assign a weighted likelihood value to a label for the first object as being the previously observed candidate object. ### Claim 3 (depends on 2) The computer system operating the system of claim 2 further comprising: the processor to determine nearness of the second feature vector value to the plurality of labeled, prior-observed second feature vector values within the second feature space to assign the weighted likelihood value to the label for the first object as being the previously observed candidate object. ### Claim 4 (depends on 1) The computer system operating the system of claim 1 further comprising: the processor to determine the number of the plurality of labeled, prior-observed first feature vector values labeled as the previously observed candidate object within the neighborhood of the first feature vector value to assign the weighted likelihood value to the label for the first object as being the previously observed candidate object. ### Claim 5 (depends on 2) The computer system operating the system of claim 2 further comprising: the processor to determine nearness of the first feature vector value to the plurality of labeled, prior-observed first feature vector values within the first feature space to assign a second weighted likelihood value to a label for the first object as being a second previously observed candidate object. ### Claim 6 (depends on 1) The computer system of claim 1 wherein the first feature attribute of the first feature space is a frontal image of the first object. ### Claim 7 (depends on 1) The computer system of claim 1 wherein the first feature attribute of the first feature space is a profile image of first object. ### Claim 8 (depends on 1) The computer system of claim 1 wherein the second feature attribute of the second feature space is a sound signature of the first object. ### Claim 9 A computer system executing code of visual scene feature extractor and feature classifier system comprising: a processor executing object recognition on a captured image of a visual scene of interest to identify a first object and a second object in the visual scene of interest and a set of feature attributes captured for the first and second object, wherein the feature attributes include an image view of the first object and an image view of the second object; the processor executing code instructions for the visual scene feature extractor and classifier to generate a semantic scene graph including a object candidate node for the first object and a plurality of object candidate nodes for the second object and a location attribute node for the visual scene of interest; the processor associating a multi-feature graph for a plurality of object candidate nodes, wherein the multi-feature graph includes feature vector values for feature attributes of the first object or the second object in feature spaces each including a neighborhood of labeled, previously-observed feature vector values for assessment of nearness to an observed feature vector value of the first object or the second object; the processor to generate an association edge between the first object candidate node and a second object candidate node and to generate an affinity label for the association edge; the processor to generate a likelihood for accuracy of that affinity label for the association edge based on probabilistic graph analysis of other object candidate nodes, the attribute location node, and association edges in the semantic scene graph including at least one multi-feature graph for at least the first object or the second object; and the processor transmitting a graphical output indication of the likelihood for accuracy for the affinity label for the association edge. ### Claim 10 (depends on 9) The computer system operating the system of claim 9 further comprising: the processor to generate labels for association edges of previously-observed instances of candidate objects for the plurality of labeled, previously-observed feature vector values in the multi-feature graph and to assess the semantic scene graph with probabilistic graph analysis to generate likelihoods for the identity of at least the first object or the second object. ### Claim 11 (depends on 9) The computer system operating the system of claim 9 wherein the feature attributes include a plurality of image views of the first object or the second object having feature vector values for the first object or the second object in the multi-feature graph. ### Claim 12 (depends on 9) The computer system operating the system of claim 9 further comprising: the processor to generate a location association edge between an object node labeled for a candidate object identity and a location attribute node including a likelihood of accuracy for that location association edge derived from contextual information accessed about the candidate object location. ### Claim 13 (depends on 9) The computer system operating the system of claim 9 further comprising: the processor to assess a semantic scene graph with probabilistic graph analysis to determine likelihood values for a plurality of object candidate nodes, attribute nodes, and association edges between object candidate nodes or attribute nodes throughout the semantic scene graph. ### Claim 14 (depends on 9) The computer system of claim 9 further comprising: the processor executing object recognition to identify a first object and a second object in the visual scene of interest and a set of feature attributes captured for the first and second object, wherein the feature attributes include an image view of the first object and an image view of the second object. ### Claim 15 (depends on 9) The computer system of claim 9 further comprising: the processor executing object recognition to selectively identify object candidate nodes for the first object and the second object in the visual scene of interest based on a predefined, minimum threshold of similarity. ### Claim 16 A computer system executing code of visual scene feature extractor and feature classifier system comprising: a network interface for receiving a captured image of a visual scene of interest including a plurality of objects in the visual scene of interest and a position identifier of a location of the visual scene of interest; a processor executing object recognition to identify a first object and a second object of the plurality of objects within the visual scene of interest for analysis in a semantic scene graph and a visual scene feature extractor to classify the first object by association with a first set of feature attributes in a multi-feature graph wherein the feature attributes include an image view of the first object at the location; the processor executing code to access available contextual data for a plurality of object candidate nodes corresponding to at least the identified first object and second object within the visual scene of interest and generate an association edge with labeled affinity between the object candidate nodes for the first object and the second object; the processor to generate a weighted likelihood for object candidate nodes as being the first object from analysis of nearness of previously-observed feature vector nodes to currently observed vector nodes within feature spaces for the first set of feature attributes, frequency of previously-observed feature vector nodes within neighborhoods of the feature spaces, and association edges within the semantic scene graph relating to the first object. ### Claim 17 (depends on 16) The computer system of claim 16 wherein the first object is a first person and the second object is a second person. ### Claim 18 (depends on 16) The computer system of claim 16 further comprising: the visual scene feature extractor to predict a semantic behavior of the first object and second object by determining a weighted likelihood value for the association edge with labeled affinity between the object candidate nodes for the first object and the second object. ### Claim 19 (depends on 16) The computer system of claim 16 wherein the first set of feature attributes includes at least one of a sound signature, a frontal view image, a left profile image, or a right profile image of the first or second object. ### Claim 20 (depends on 16) The computer system of claim 16 further comprising: the processor executing code to access available contextual data for a plurality of object candidate nodes corresponding to at least the identified first object and second object within the visual scene of interest and generate an association edge for locations of candidate objects between the object candidate nodes for the first object or the second object and a location attribute node in the semantic scene graph.