# US 2021/0174155 A1 · Automated semantic inference of visual features and scenes > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0174155 A1 - Document ID: US20210174155A1 - Status: Published application - Kind: A1 - Application number: 16/947,096 - Filed: 2020-07-17 - Published: 2021-06-10 - Earliest claimed: 2017-12-28 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Ned M. Smith; Katalin Klara Bartfai-Walcott; Eve M. Schooler; Shao-Wen Yang - Web page: https://patentgrep.com/patent/US20210174155A1 ## CPC - G06F 21/44: Program or device authentication - G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues} - G06F 9/5044: {considering hardware capabilities} - G06F 9/5066: {Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451)} - G06F 9/5072: {Grid computing} - G06F 16/535: Filtering based on additional data, e.g. user or group profiles - G06F 16/538: Presentation of query results - G06F 16/54: Browsing; Visualisation therefor - G06F 16/951: Indexing; Web crawling techniques - G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation - G06F 18/211: Selection of the most significant subset of features - G06F 18/213: Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods - G06F 18/2163: {Partitioning the feature space} - G06F 18/22: Matching criteria, e.g. proximity measures - G06F 18/24: Classification techniques - G06F 18/24143: {Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]} - G06F 21/45: Structures or tools for the administration of authentication - G06F 21/53: by executing in a restricted environment, e.g. sandbox or secure virtual machine - G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification} - G06F 21/64: Protecting data integrity, e.g. using checksums, certificates or signatures - G06K 15/1886: {Storage of the print image data or of parts thereof} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/063: using electronic means - G06N 3/08: Learning methods - G06N 5/022: Knowledge engineering; Knowledge acquisition - G06T 7/11: Region-based segmentation - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06V 10/20: Image preprocessing - G06V 10/40: Extraction of image or video features - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G06V 10/75: Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries - G06V 10/771: Feature selection, e.g. selecting representative features from a multi-dimensional feature space - G06V 10/82: using neural networks - G06V 10/95: {structured as a network, e.g. client-server architectures} - G06V 10/96: Management of image or video recognition tasks - G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60) - G06V 30/19173: {Classification techniques} - G06V 30/274: {Syntactic or semantic context, e.g. balancing} - G06V 40/161: {Detection; Localisation; Normalisation} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - H04L 9/0643: {Hash functions, e.g. MD5, SHA, HMAC or f9 MAC} - H04L 9/3239: {involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD} - H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks - H04L 67/51: Discovery or management thereof, e.g. service location protocol [SLP] or web services - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence) - H04W 4/70: Services for machine-to-machine communication [M2M] or machine type communication [MTC] - G06F 18/24323: {Tree-organised classifiers} - G06F 2209/503: Resource availability - G06F 2209/506: Constraint - G06F 2221/2117: User registration - G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) - G06T 7/223: using block-matching - G06T 2207/10016: Video; Image sequence - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20024: Filtering details - G06T 2207/20052: Discrete cosine transform [DCT] - G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT] - G06T 2207/20064: Wavelet transform [DWT] - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/20221: Image fusion; Image merging - G06T 2207/30242: Counting objects in image - G06V 30/194: References adjustable by an adaptive method, e.g. learning - G06V 2201/10: Recognition assisted with metadata - H04L 9/50: {using hash chains, e.g. blockchains or hash trees} - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) - H04N 19/12: Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264 - H04N 19/124: Quantisation - H04N 19/167: Position within a video image, e.g. region of interest [ROI] - H04N 19/172: the region being a picture, frame or field - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence) - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - H04N 19/48: using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data (motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513) - H04N 19/513: Processing of motion vectors - H04N 19/625: using discrete cosine transform [DCT] - H04N 19/63: using sub-band based transform, e.g. wavelets - H04W 12/02: Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII] - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06K: GRAPHICAL DATA READING (image or video recognition or understanding G06V); PRESENTATION OF DATA; RECORD CARRIERS; HANDLING RECORD CARRIERS - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract In one embodiment, an apparatus comprises a memory and a processor. The memory stores visual data captured by one or more sensors. The processor detects one or more first objects in the visual data based on a machine learning model and one or more first reference templates. The processor further determines, based on an object ontology, that the visual data is expected to contain a second object, wherein the object ontology indicates that the second object is related to the one or more first objects. The processor further detects the second object in the visual data based on the machine learning model and a second reference template. The processor further determines, based on an inference rule, that the visual data is expected to contain a third object. The processor further detects the third object in the visual data based on the machine learning model and a third reference template. ## Claims ### Claim at position 1 1-25. (canceled) ### Claim 26 A processing device to perform scene recognition, comprising: interface circuitry to receive visual data captured by one or more sensors; and processing circuitry to: detect a first set of objects in the visual data based on one or more first machine learning models, wherein the one or more first machine learning models are trained to detect the first set of objects; determine, based on detecting the first set of objects, that the visual data is expected to contain a second object, wherein the second object is related to at least a subset of the first set of objects; detect the second object in the visual data based on a second machine learning model, wherein the second machine learning model is trained to detect the second object; and recognize a scene captured in the visual data, wherein the scene is recognized based on detecting the first set of objects and the second object in the visual data. ### Claim 27 (depends on 26) The processing device of claim 26, wherein the processing circuitry to determine, based on detecting the first set of objects, that the visual data is expected to contain the second object is further to: determine that the visual data is expected to contain the second object based on an object ontology, wherein the object ontology indicates that the second object is related to at least the subset of the first set of objects. ### Claim 28 (depends on 27) The processing device of claim 27, wherein the object ontology indicates hierarchical relationships among a plurality of objects at a plurality of levels of abstraction. ### Claim 29 (depends on 28) The processing device of claim 28, wherein the processing circuitry to determine that the visual data is expected to contain the second object based on the object ontology is further to: determine, based on the hierarchical relationships among the plurality of objects, that the second object is a parent of at least the subset of the first set of objects. ### Claim 30 (depends on 28) The processing device of claim 28, wherein the processing circuitry to determine that the visual data is expected to contain the second object based on the object ontology is further to: determine, based on the hierarchical relationships among the plurality of objects, that the second object is a child of at least the subset of the first set of objects. ### Claim 31 (depends on 26) The processing device of claim 26, wherein the processing circuitry to recognize the scene captured in the visual data is further to: evaluate a scene inference rule against the visual data, wherein the scene inference rule indicates a set of criteria for recognizing the scene, and wherein the scene inference rule is evaluated based on the first set of objects and the second object detected in the visual data; determine that the visual data satisfies the set of criteria for recognizing the scene; and infer, based on determining that the visual data satisfies the set of criteria for recognizing the scene, that the scene is captured in the visual data. ### Claim 32 (depends on 31) The processing device of claim 31, wherein the set of criteria indicates expected content within the scene. ### Claim 33 (depends on 26) The processing device of claim 26, wherein the processing circuitry is further to: send, via the interface circuitry, a request to obtain the second machine learning model from a repository over a network, wherein the request is sent based at least in part on determining that the visual data is expected to contain the second object; and receive, via the interface circuitry, the second machine learning model from the repository over the network. ### Claim 34 (depends on 26) The processing device of claim 26, wherein the processing circuitry is further to: determine, based on recognizing the scene captured in the visual data, that the visual data is expected to contain a third object, wherein the third object is related to the scene; and detect the third object in the visual data based on a third machine learning model, wherein the third machine learning model is trained to detect the third object. ### Claim 35 (depends on 26) The processing device of claim 26, wherein the one or more sensors comprise a camera. ### Claim 36 At least one non-transitory machine accessible storage medium having instructions stored thereon, wherein the instructions, when executed on processing circuitry, cause the processing circuitry to: receive, via interface circuitry, visual data captured by one or more sensors; detect a first set of objects in the visual data based on one or more first machine learning models, wherein the one or more first machine learning models are trained to detect the first set of objects; determine, based on detecting the first set of objects, that the visual data is expected to contain a second object, wherein the second object is related to at least a subset of the first set of objects; detect the second object in the visual data based on a second machine learning model, wherein the second machine learning model is trained to detect the second object; and recognize a scene captured in the visual data, wherein the scene is recognized based on detecting the first set of objects and the second object in the visual data. ### Claim 37 (depends on 36) The storage medium of claim 36, wherein the instructions that cause the processing circuitry to determine, based on detecting the first set of objects, that the visual data is expected to contain the second object further cause the processing circuitry to: determine that the visual data is expected to contain the second object based on an object ontology, wherein the object ontology indicates that the second object is related to at least the subset of the first set of objects. ### Claim 38 (depends on 37) The storage medium of claim 37, wherein the object ontology indicates hierarchical relationships among a plurality of objects at a plurality of levels of abstraction. ### Claim 39 (depends on 38) The storage medium of claim 38, wherein the instructions that cause the processing circuitry to determine that the visual data is expected to contain the second object based on the object ontology further cause the processing circuitry to: determine, based on the hierarchical relationships among the plurality of objects, that the second object is a parent of at least the subset of the first set of objects. ### Claim 40 (depends on 38) The storage medium of claim 38, wherein the instructions that cause the processing circuitry to determine that the visual data is expected to contain the second object based on the object ontology further cause the processing circuitry to: determine, based on the hierarchical relationships among the plurality of objects, that the second object is a child of at least the subset of the first set of objects. ### Claim 41 (depends on 36) The storage medium of claim 36, wherein the instructions that cause the processing circuitry to recognize the scene captured in the visual data further cause the processing circuitry to: evaluate a scene inference rule against the visual data, wherein the scene inference rule indicates a set of criteria for recognizing the scene, and wherein the scene inference rule is evaluated based on the first set of objects and the second object detected in the visual data; determine that the visual data satisfies the set of criteria for recognizing the scene; and infer, based on determining that the visual data satisfies the set of criteria for recognizing the scene, that the scene is captured in the visual data. ### Claim 42 (depends on 41) The storage medium of claim 41, wherein the set of criteria indicates expected content within the scene. ### Claim 43 (depends on 36) The storage medium of claim 36, wherein the instructions further cause the processing circuitry to: send, via the interface circuitry, a request to obtain the second machine learning model from a repository over a network, wherein the request is sent based at least in part on determining that the visual data is expected to contain the second object; and receive, via the interface circuitry, the second machine learning model from the repository over the network. ### Claim 44 (depends on 36) The storage medium of claim 36, wherein the instructions further cause the processing circuitry to: determine, based on recognizing the scene captured in the visual data, that the visual data is expected to contain a third object, wherein the third object is related to the scene; and detect the third object in the visual data based on a third machine learning model, wherein the third machine learning model is trained to detect the third object. ### Claim 45 A method of performing scene recognition, comprising: receiving, via interface circuitry, visual data captured by one or more sensors; detecting a first set of objects in the visual data based on one or more first machine learning models, wherein the one or more first machine learning models are trained to detect the first set of objects; determining, based on detecting the first set of objects, that the visual data is expected to contain a second object, wherein the second object is related to at least a subset of the first set of objects; detecting the second object in the visual data based on a second machine learning model, wherein the second machine learning model is trained to detect the second object; and recognizing a scene captured in the visual data, wherein the scene is recognized based on detecting the first set of objects and the second object in the visual data. ### Claim 46 (depends on 45) The method of claim 45, wherein determining, based on detecting the first set of objects, that the visual data is expected to contain the second object comprises: determining that the visual data is expected to contain the second object based on an object ontology, wherein the object ontology indicates that the second object is related to at least the subset of the first set of objects. ### Claim 47 (depends on 45) The method of claim 45, wherein recognizing the scene captured in the visual data comprises: evaluating a scene inference rule against the visual data, wherein the scene inference rule indicates a set of criteria for recognizing the scene, and wherein the scene inference rule is evaluated based on the first set of objects and the second object detected in the visual data; determining that the visual data satisfies the set of criteria for recognizing the scene; and inferring, based on determining that the visual data satisfies the set of criteria for recognizing the scene, that the scene is captured in the visual data. ### Claim 48 (depends on 45) The method of claim 45, further comprising: determining, based on recognizing the scene captured in the visual data, that the visual data is expected to contain a third object, wherein the third object is related to the scene; and detecting the third object in the visual data based on a third machine learning model, wherein the third machine learning model is trained to detect the third object. ### Claim 49 A system for performing scene recognition, comprising: a camera to capture visual data of an environment; and processing circuitry to: detect a first set of objects in the visual data based on one or more first machine learning models, wherein the one or more first machine learning models are trained to detect the first set of objects; determine, based on detecting the first set of objects, that the visual data is expected to contain a second object, wherein the second object is related to at least a subset of the first set of objects; detect the second object in the visual data based on a second machine learning model, wherein the second machine learning model is trained to detect the second object; and recognize a scene captured in the visual data, wherein the scene is recognized based on detecting the first set of objects and the second object in the visual data. ### Claim 50 (depends on 49) The system of claim 49, wherein the processing circuitry comprises: a central processing unit; and an artificial intelligence accelerator.