# US 2019/0378204 A1 · Generating and providing augmented reality representations of recommended products based on style similarity in relation to real-world surroundings > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0378204 A1 - Document ID: US20190378204A1 - Status: Published application - Kind: A1 - Application number: 16/004,787 - Filed: 2018-06-11 - Published: 2019-12-12 - Earliest claimed: 2018-06-11 - Assignees: Adobe Inc. - Applicants: Adobe Inc. - Inventors: Kumar Ayush; Gaurush Hiranandani - Web page: https://patentgrep.com/patent/US20190378204A1 ## CPC - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06Q 30/0631: {Recommending goods or services} - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The present disclosure relates to systems, methods, and non-transitory computer readable media for generating augmented reality representations of recommended products based on style similarity with real-world surroundings. For example, the disclosed systems can identify a real-world object within a camera feed and can utilize a 2D-3D alignment algorithm to identify a three-dimensional model that matches the real-world object. In addition, the disclosed systems can utilize a style similarity algorithm to generate style similarity scores for products in relation to the identified three-dimensional model. The disclosed systems can also utilize a color compatibility algorithm to generate color compatibility scores for products, and the systems can determine overall scores for products based on a combination of style similarity scores and color compatibility scores. The disclosed systems can further generate AR representations of recommended products based on the overall scores. ## Claims ### Claim 1 In a digital medium environment for providing context-aware product recommendations, a computer-implemented method for generating a three-dimensional digital representation of a product based on a similarity to real-world surroundings of a user, comprising: determining, based on a camera feed received from a user client device, a viewpoint that depicts a real-world environment associated with the user client device; performing a step for generating an augmented reality representation of a recommended product based on three-dimensional models that match a style of real-world objects depicted within the viewpoint; and providing the augmented reality representation for display on the user client device. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein determining the viewpoint comprises: selecting a time instant during an application session associated with the user client device where the user client device satisfies a stillness threshold; and capturing a frame of the camera feed at the time instant. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, further comprising determining that the user client device satisfies the stillness threshold by accessing accelerometer information associated with the user client device. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein providing the augmented reality representation for display on the user client device comprises providing an augmented reality scene in which a real-world object has been replaced by the augmented reality representation of a recommended product. ### Claim 5 A non-transitory computer readable medium for generating a three-dimensional digital representation of a product based on a similarity to real-world surroundings of a user, the non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause a user client device to: analyze a real-world environment depicted within a viewpoint of a camera feed received from the user client device to identify one or more real-world objects; identify, from a model database, a matching three-dimensional model that matches an identified real-world object of the one or more real-world objects; generate, utilizing a style similarity algorithm for the matching three-dimensional model, one or more recommended products based on style similarity in relation to the matching three-dimensional model that matches the identified real-world object; and generate an augmented reality representation of the one or more recommended products to overlay in place of a real-world object of the one or more real-world objects. ### Claim 6 (depends on 5) The non-transitory computer readable medium of claim 5, further comprising instructions that, when executed by the at least one processor, cause the user client device to generate a final product recommendation by: determining overall scores for the one or more recommended products based on weighted linear combinations of style similarity and color compatibility; and selecting a recommended product of the one or more recommended products based on the overall scores. ### Claim 7 (depends on 5) The non-transitory computer readable medium of claim 5, further comprising instructions that, when executed by the at least one processor, cause the user client device to determine, in relation to the real-world environment, a location and an orientation of the real-world object. ### Claim 8 (depends on 7) The non-transitory computer readable medium of claim 7, further comprising instructions that, when executed by the at least one processor, cause the user client device to provide the augmented reality representation of the one or more recommended products by positioning, within the real-world environment of the camera feed, the augmented reality representation of the one or more recommended products to match the location and the orientation of the real-world object. ### Claim 9 (depends on 7) The non-transitory computer readable medium of claim 7, wherein the instructions cause the user client device to generate the augmented reality representation of the one or more recommended products by utilizing a color compatibility algorithm to, based on identifying colors within the viewpoint, determine one or more colors for the augmented reality representation of the one or more recommended products. ### Claim 10 (depends on 5) The non-transitory computer readable medium of claim 5, further comprising instructions that, when executed by the at least one processor, cause the user client device to determine the viewpoint by: selecting a time instant during an application session associated with the user client device where the user client device satisfies a stillness threshold; and capturing a frame of the camera feed at the time instant. ### Claim 11 (depends on 5) The non-transitory computer readable medium of claim 5, wherein the instructions cause the user client device to analyze the viewpoint to identify the real-world object by utilizing a neural network to generate proposed regions of the viewpoint with corresponding probabilities of containing real-world objects. ### Claim 12 (depends on 5) The non-transitory computer readable medium of claim 5, wherein the instructions cause the user client device to generate the one or more recommended products by utilizing the style similarity algorithm to determine similarity scores for a plurality of products within a product database, wherein the similarity scores are based on comparing geometric features of the three-dimensional model with geometric features of the plurality of products. ### Claim 13 (depends on 5) The non-transitory computer readable medium of claim 5, wherein the instructions cause the user client device to identify the matching three-dimensional model by implementing a 2D-3D alignment algorithm to determine, for each of a plurality of three-dimensional models with the model database, an alignment relative to the real-world object. ### Claim 14 A system for generating a three-dimensional digital representation of a product based on a similarity to real-world surroundings of a user, the system comprising: at least one processor; and a non-transitory computer readable medium comprising: a model database; and instructions that, when executed by the at least one processor, cause the system to: analyze a real-world environment depicted within a viewpoint of a camera feed received from a user client device to identify one or more real-world objects by utilizing a neural network to determine proposed regions within the viewpoint that each satisfy a probability of containing a real-world object; identify, from a model database, a matching three-dimensional model that matches an identified real-world object of the one or more real-world objects; generate, utilizing a style similarity algorithm for the matching three-dimensional model, one or more recommended products based on style similarity in relation to the matching three-dimensional model that matches the identified real-world object; in response to receiving an indication from the user client device to replace a real-world object of the one or more real-world objects depicted within the viewpoint: generate an augmented reality representation of the one or more recommended products to overlay in place of the real-world object. ### Claim 15 (depends on 14) The system of claim 14, further comprising instructions that, when executed by the at least one processor, cause the system to generate a final product recommendation by: determining overall scores for the one or more recommended products based on weighted linear combinations of style similarity and color compatibility; and selecting a recommended product of the one or more recommended products based on the overall scores. ### Claim 16 (depends on 14) The system of claim 14, further comprising instructions that, when executed by the at least one processor, cause the system to: generate, utilizing a region proposal network, object bounds that satisfy a probability of containing real-world objects within the real-world environment; and determine, utilizing a fast region-based convolutional neural network that shares convolutional features with the region proposal network, object classes for the one or more real-world objects depicted within the object bounds within the real-world environment. ### Claim 17 (depends on 16) The system of claim 16, further comprising instructions that, when executed by the at least one processor, cause the system to provide the augmented reality representation of the one or more recommended products by positioning, within the real-world environment of the camera feed, the augmented reality representation of the one or more recommended products to match the location and the orientation of the real-world object. ### Claim 18 (depends on 17) The system of claim 17, wherein the instructions cause the system to generate the augmented reality representation of the one or more recommended products by utilizing a color compatibility algorithm to, based on identifying colors within the viewpoint, determine one or more colors for the augmented reality representation of the one or more recommended products. ### Claim 19 (depends on 14) The system of claim 14, further comprising instructions that, when executed by the at least one processor, cause the system to determine the viewpoint by: selecting a time instant during an application session associated with the user client device where the user client device satisfies a stillness threshold; and capturing a frame of the camera feed at the time instant. ### Claim 20 (depends on 14) The system of claim 14, wherein the indication to replace the real-world object comprises a user input to elect to replace the real-world object within the viewpoint.