# US 12,002,149 B2 · Machine learning based image attribute determination > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,149 B2 - Document ID: US12002149B2 - Status: Granted - Kind: B2 - Application number: 17/540,173 - Filed: 2021-12-01 - Published: 2022-03-24 - Granted: 2024-06-04 - Earliest claimed: 2016-03-25 - Assignees: Outward, Inc. - Applicants: Outward, Inc. - Inventors: Clarence Chui; Manu Parmar - Web page: https://patentgrep.com/patent/US12002149B2 ## CPC - G06F 16/78: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually - G06T 15/205: {Image-based rendering} - G06F 16/58: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 7/32: using correlation-based methods - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Techniques for machine learning based image attribute determination are disclosed. In some embodiments, one or more unknown attributes of a received image comprising a prescribed environment are determined using a machine learning based framework. The machine learning based framework is at least in part trained on image data sets comprising a model environment that substantially simulates the prescribed environment. ## Claims ### Claim 1 A method, comprising: receiving an image for which one or more attributes are unknown, wherein the image comprises a prescribed environment; and determining one or more unknown attributes for the received image using a machine learning based framework, wherein the machine learning based framework is at least in part trained on image data sets comprising a model environment that substantially simulates the prescribed environment. ### Claim 2 (depends on 1) The method of claim 1, wherein the received image comprises a rendering and the prescribed environment comprises the model environment. ### Claim 3 (depends on 1) The method of claim 1, wherein the received image comprises a photograph and the prescribed environment comprises an actual physical environment. ### Claim 4 (depends on 1) The method of claim 1, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel location in three-dimensional space. ### Claim 5 (depends on 1) The method of claim 1, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel xyz coordinates. ### Claim 6 (depends on 1) The method of claim 1, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel depth. ### Claim 7 (depends on 1) The method of claim 1, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel surface normal vectors. ### Claim 8 (depends on 1) The method of claim 1, wherein the prescribed environment is constrained and known. ### Claim 9 (depends on 1) The method of claim 1, wherein known information about the prescribed environment comprises known structure and geometry of the prescribed environment. ### Claim 10 (depends on 1) The method of claim 1, wherein the prescribed environment comprises an apparatus or rig for photographing objects or items. ### Claim 11 (depends on 1) The method of claim 1, wherein known information about the prescribed environment comprises known camera information. ### Claim 12 (depends on 1) The method of claim 1, wherein known information about the prescribed environment comprises known lighting information. ### Claim 13 (depends on 1) The method of claim 1, wherein the training image data sets comprising the model environment are rendered from available three-dimensional models of corresponding assets. ### Claim 14 (depends on 1) The method of claim 1, wherein the training image data sets comprising the model environment are labeled or otherwise associated with relevant metadata. ### Claim 15 (depends on 1) The method of claim 1, wherein various attributes learned by the machine learning based framework from the training image data sets are derived or inferred from labels or metadata associated with the training image data sets. ### Claim 16 (depends on 1) The method of claim 1, wherein the machine learning based framework comprises one or more neural networks. ### Claim 17 (depends on 1) The method of claim 1, wherein the machine learning based framework comprises one or more convolutional neural networks. ### Claim 18 (depends on 1) The method of claim 1, wherein the received image comprises a still image or a frame of a video sequence. ### Claim 19 A system, comprising: a processor configured to: receive an image for which one or more attributes are unknown, wherein the image comprises a prescribed environment; and use a machine learning based framework to determine one or more unknown attributes for the received image, wherein the machine learning based framework is at least in part trained on image data sets comprising a model environment that substantially simulates the prescribed environment; and a memory coupled to the processor and configured to provide the processor with instructions. ### Claim 20 (depends on 19) The system of claim 19, wherein the received image comprises a rendering and the prescribed environment comprises the model environment. ### Claim 21 (depends on 19) The system of claim 19, wherein the received image comprises a photograph and the prescribed environment comprises an actual physical environment. ### Claim 22 (depends on 19) The system of claim 19, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel location in three-dimensional space. ### Claim 23 (depends on 19) The system of claim 19, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel xyz coordinates. ### Claim 24 (depends on 19) The system of claim 19, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel depth. ### Claim 25 (depends on 19) The system of claim 19, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel surface normal vectors. ### Claim 26 (depends on 19) The system of claim 19, wherein the prescribed environment is constrained and known. ### Claim 27 (depends on 19) The system of claim 19, wherein known information about the prescribed environment comprises known structure and geometry of the prescribed environment. ### Claim 28 (depends on 19) The system of claim 19, wherein the prescribed environment comprises an apparatus or rig for photographing objects or items. ### Claim 29 (depends on 19) The system of claim 19, wherein known information about the prescribed environment comprises known camera information. ### Claim 30 (depends on 19) The system of claim 19, wherein known information about the prescribed environment comprises known lighting information. ### Claim 31 (depends on 19) The system of claim 19, wherein the training image data sets comprising the model environment are rendered from available three-dimensional models of corresponding assets. ### Claim 32 (depends on 19) The system of claim 19, wherein the training image data sets comprising the model environment are labeled or otherwise associated with relevant metadata. ### Claim 33 (depends on 19) The system of claim 19, wherein various attributes learned by the machine learning based framework from the training image data sets are derived or inferred from labels or metadata associated with the training image data sets. ### Claim 34 (depends on 19) The system of claim 19, wherein the machine learning based framework comprises one or more neural networks. ### Claim 35 (depends on 19) The system of claim 19, wherein the machine learning based framework comprises one or more convolutional neural networks. ### Claim 36 (depends on 19) The system of claim 19, wherein the received image comprises a still image or a frame of a video sequence. ### Claim 37 A computer program product, the computer program product being embodied in a non-transitory computer readable medium and comprising computer instructions which when executed cause a computer to: receive an image for which one or more attributes are unknown, wherein the image comprises a prescribed environment; and determine one or more unknown attributes for the received image using a machine learning based framework, wherein the machine learning based framework is at least in part trained on image data sets comprising a model environment that substantially simulates the prescribed environment. ### Claim 38 (depends on 37) The computer program product of claim 37, wherein the received image comprises a rendering and the prescribed environment comprises the model environment. ### Claim 39 (depends on 37) The computer program product of claim 37, wherein the received image comprises a photograph and the prescribed environment comprises an actual physical environment. ### Claim 40 (depends on 37) The computer program product of claim 37, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel location in three-dimensional space. ### Claim 41 (depends on 37) The computer program product of claim 37, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel xyz coordinates. ### Claim 42 (depends on 37) The computer program product of claim 37, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel depth. ### Claim 43 (depends on 37) The computer program product of claim 37, wherein the one or more unknown attributes of the received image determined by the machine learning based framework comprise pixel surface normal vectors. ### Claim 44 (depends on 37) The computer program product of claim 37, wherein the prescribed environment is constrained and known. ### Claim 45 (depends on 37) The computer program product of claim 37, wherein known information about the prescribed environment comprises known structure and geometry of the prescribed environment. ### Claim 46 (depends on 37) The computer program product of claim 37, wherein the prescribed environment comprises an apparatus or rig for photographing objects or items. ### Claim 47 (depends on 37) The computer program product of claim 37, wherein known information about the prescribed environment comprises known camera information. ### Claim 48 (depends on 37) The computer program product of claim 37, wherein known information about the prescribed environment comprises known lighting information. ### Claim 49 (depends on 37) The computer program product of claim 37, wherein the training image data sets comprising the model environment are rendered from available three-dimensional models of corresponding assets. ### Claim 50 (depends on 37) The computer program product of claim 37, wherein the training image data sets comprising the model environment are labeled or otherwise associated with relevant metadata. ### Claim 51 (depends on 37) The computer program product of claim 37, wherein various attributes learned by the machine learning based framework from the training image data sets are derived or inferred from labels or metadata associated with the training image data sets. ### Claim 52 (depends on 37) The computer program product of claim 37, wherein the machine learning based framework comprises one or more neural networks. ### Claim 53 (depends on 37) The computer program product of claim 37, wherein the machine learning based framework comprises one or more convolutional neural networks. ### Claim 54 (depends on 37) The computer program product of claim 37, wherein the received image comprises a still image or a frame of a video sequence.