# US 10,121,190 B2 · System and method of sharing an augmented environment with a companion > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,121,190 B2 - Document ID: US10121190B2 - Status: Granted - Kind: B2 - Application number: 15/664,705 - Filed: 2017-07-31 - Published: 2018-06-28 - Granted: 2018-11-06 - Earliest claimed: 2016-12-22 - Assignees: Capital One Services, LLC - Applicants: Capital One Services, LLC - Inventors: Karen Nickerson; Justin Wishne; Drew Jacobs; Justin Smith; Marco S. Giampaolo; Hannes Jouhikainen - Web page: https://patentgrep.com/patent/US10121190B2 ## CPC - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay} - G06F 3/147: using display panels - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06Q 30/0601: Electronic shopping [e-shopping] - G06T 11/00: Two-dimensional [2D] image generation - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06F 3/16: Sound input; Sound output (speech processing G10L) - G09G 2370/02: Networking aspects - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract A system includes one or more memory devices storing instructions, and one or more processors configured to execute the instructions to perform steps of a method for providing an interactive virtual environment. The system may receive environmental data, and augment the environmental data by adding virtual environmental data. The system may then provide this virtual environment to two or more devices. The system can then receive user input, and modify the virtual environment in response to those inputs. ## Claims ### Claim 1 A system for providing an interactive virtual environment, comprising: one or more processors; and a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to: receive, from a first computing device, environmental data comprising audio and visual information representing a physical environment; augment the environmental data by adding virtual environmental data comprising data associated with a merchant; transmit, to the first computing device, the virtual environmental data to create an augmented environment in combination with the physical environment; transmit, to at least a second computing device, the environmental data and the virtual environmental data to create a virtual environment; receive user input data associated with one or more predefined user inputs from the second computing device, the user input data comprising an instruction to modify a product displayed in the virtual environment; and modify, based on the user input data, the virtual environmental data to include enhanced content representing a modification of the product, wherein the enhanced content is unrepresented in the physical environment; transmit, to the first computing device, the modified virtual environmental data to update the augmented environment to display the enhanced content representing the modification of the product; and transmit, to the second computing device, the modified virtual environmental data to update the virtual environment to display the enhanced content representing the modification of the product. ### Claim 2 (depends on 1) The system of claim 1, wherein the virtual environmental data includes one or more of: product price information, product inventory information, product manufacturing information, product marketing information or description, product ratings or reviews, social information, related products, available discounts, and aesthetic visual effects. ### Claim 3 (depends on 1) The system of claim 1, wherein the first and second computing devices each comprise one or more of: a mobile phone, a virtual reality headset, an augmented reality headset, a wearable device, internet of things (IoT) devices, a tablet computer, and a personal computer. ### Claim 4 (depends on 1) The system of claim 1, wherein the step of augmenting the environmental data further comprises identifying one or more products in the visual information. ### Claim 5 (depends on 4) The system of claim 4, wherein the step of augmenting the environmental data further comprises displaying information from the merchant that is specific to one of the one or more products identified in the visual information. ### Claim 6 (depends on 1) The system of claim 1, wherein the user input data associated with one or more predefined user inputs from the second computing device comprises a selection of a product displayed in the virtual environment. ### Claim 7 (depends on 1) The system of claim 1, wherein the modification of a product displayed in the virtual environment comprises one or more of: a color change, a size change, and a style change. ### Claim 8 (depends on 1) The system of claim 1, wherein the one or more processors are further configured to execute the instructions to: receive first user input data associated with one or more predefined user inputs from the first computing device; and modify the enhanced content displayed in the virtual environment on the second computing device based on the first user input data. ### Claim 9 (depends on 8) The system of claim 8, wherein the modifications to the enhanced content include one or more of: highlighting one or more products identified in the visual information, removing one or more products identified in the visual information, moving one or more products identified in the visual information to a different location in the virtual environment. ### Claim 10 A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause a computer system to: receive, from a first computing device, environmental data comprising audio and visual information representing a physical environment; augment the environmental data by adding virtual environmental data comprising data associated with a merchant; transmit, to the first computing device, the virtual environmental data to create an augmented environment in combination with the physical environment; transmit, to at least a second computing device, the environmental data and the virtual environmental data to create a virtual environment; receive user input data associated with one or more predefined user inputs from the second computing device, the user input data comprising an instruction to modify a product displayed in the virtual environment; and modify, based on the user input data, the virtual environmental data to include enhanced content representing a modification of the product, wherein the enhanced content is unrepresented in the physical environment; transmit, to the first computing device, the modified virtual environmental data to update the augmented environment to display the enhanced content representing the modification of the product; and transmit, to the second computing device, the modified virtual environmental data to update the virtual environment to display the enhanced content representing the modification of the product. ### Claim 11 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the virtual environmental data includes one or more of: product price information, product inventory information, product manufacturing information, product marketing information or description, product ratings or reviews, social information, related products, available discounts, and aesthetic visual effects. ### Claim 12 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the step of augmenting the environmental data further comprises identifying one or more products in the visual information. ### Claim 13 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the user input data associated with one or more predefined user inputs from the second computing device comprises a selection of a product displayed in the virtual environment. ### Claim 14 (depends on 10) The non-transitory computer-readable medium of claim 10, wherein the one or more processors are further configured to execute the instructions to: receive first user input data associated with one or more predefined user inputs from the first computing device; and modify the enhanced content displayed in the virtual environment on the second computing device based on the first user input data. ### Claim 15 (depends on 14) The non-transitory computer-readable medium of claim 14, wherein the modifications to the enhanced content include one or more of: highlighting one or more products identified in the visual information, removing one or more products identified in the visual information, moving one or more products identified in the visual information to a different location in the virtual environment. ### Claim 16 A method of providing an interactive virtual environment, comprising: receiving, by a processor and from a first computing device, environmental data comprising audio and visual information representing a physical environment; augmenting, by the processor, the environmental data by adding virtual environmental data comprising data associated with a merchant; transmitting, by the processor to the first computing device, the virtual environmental data to create an augmented environment in combination with the physical environment; transmitting, by the processor to at least a second computing device, the environmental data and the virtual environmental data to create a virtual environment; receiving, by the processor, user input data associated with one or more predefined user inputs from the second computing device, the user input data comprising an instruction to modify a product displayed in the virtual environment; and modifying, by the processor and based on the user input data, the virtual environmental data to include enhanced content representing a modification of the product, wherein the enhanced content is unrepresented in the physical environment; transmitting, by the processor to the first computing device, the modified virtual environmental data to update the augmented environment to display the enhanced content representing the modification of the product; and transmitting, by the processor to the second computing device, the modified virtual environmental data to update the virtual environment to display the enhanced content representing the modification of the product. ### Claim 17 (depends on 16) The method of claim 16, wherein the virtual environmental data includes one or more of: product price information, product inventory information, product manufacturing information, product marketing information or description, product ratings or reviews, social information, related products, available discounts, and aesthetic visual effects. ### Claim 18 (depends on 16) The method of claim 16, wherein the step of augmenting the environmental data further comprises identifying one or more products in the visual information.