# US 11,443,490 B2 · Snapping, virtual inking, and accessibility in augmented reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,490 B2 - Document ID: US11443490B2 - Status: Granted - Kind: B2 - Application number: 17/181,406 - Filed: 2021-02-22 - Published: 2021-07-01 - Granted: 2022-09-13 - Earliest claimed: 2019-01-09 - Assignees: VMware, Inc. - Applicants: VMware, Inc. - Inventors: Shubham Verma; Prabhakara Rao Narnakaje Venugopala; Kannan Balasubramanian - Web page: https://patentgrep.com/patent/US11443490B2 ## CPC - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06F 3/0486: Drag-and-drop - G06T 11/00: Two-dimensional [2D] image generation - G06T 11/40: Filling planar surfaces by adding surface attributes, e.g. adding colours or textures - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures - G06T 2219/2004: Aligning objects, relative positioning of parts - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Embodiments of the present disclosure relate to providing an augmented reality experience. Embodiments include receiving, from an augmented reality or virtual reality (AR/VR) device, first input that identifies application content and a location of an artifact in an AR/VR environment. Embodiments include identifying a region in the environment based on the location, the region corresponding to the artifact. Embodiments include receiving the application content from an application. Embodiments include composing a first scene, the application content being overlaid onto the region. Embodiments include transporting the first scene to the AR/VR device for display. Embodiments include receiving, from the client device, second input in the region in the environment. Embodiments include composing a second scene wherein the application content is modified based on the second input. Embodiments include transporting the second scene to the AR/VR device for display. ## Claims ### Claim 1 A method for providing an augmented reality experience, comprising: receiving, by a computing device from an augmented reality or virtual reality (AR/VR) device, first input that identifies application content and a location of an artifact in an AR/VR environment; determining, by the computing device, that the artifact in the AR/VR environment comprises a touch input device based on determining that the touch input device is visible, at the location, in one or more images received from the AR/VR device; receiving, by the computing device, the application content from an application; composing, by the computing device, a first scene wherein the application content is overlaid onto the touch input device; transporting, by the computing device, the first scene to the AR/VR device, wherein the AR/VR device displays the first scene; receiving, by the computing device from the touch input device, touch input provided by a user; composing, by the computing device, a second scene wherein the application content overlaid onto the touch input device is modified based on the touch input, wherein the second scene comprises a visual representation of the touch input, and wherein composing, by the computing device, the second scene comprises: providing the touch input to the application by redirecting coordinates of input determined by a driver of the touch input device to the application; and receiving modified application content from the application, wherein the modified application content comprises the visual representation of the touch input; and transporting, by the computing device, the second scene to the AR/VR device, wherein the AR/VR device displays the second scene. ### Claim 2 (depends on 1) The method of claim 1, wherein composing, by the computing device, the first scene comprises augmenting the touch input device in the first scene with the application content based on default augmentation configuration values. ### Claim 3 (depends on 1) The method of claim 1, wherein composing, by the computing device, the first scene comprises scaling the application content based on dimensions of the touch input device. ### Claim 4 (depends on 1) The method of claim 1, wherein the application content comprises a partial area of an application screen in which touch input is to be entered. ### Claim 5 A system for providing an augmented reality experience, comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor and the at least one memory configured to: receive, by a computing device from an augmented reality or virtual reality (AR/VR) device, first input that identifies application content and a location of an artifact in an AR/VR environment; determine, by the computing device, that the artifact in the AR/VR environment comprises a touch input device based on determining that the touch input device is visible, at the location, in one or more images received from the AR/VR device; receive, by the computing device, the application content from an application; compose, by the computing device, a first scene wherein the application content is overlaid onto the touch input device; transport, by the computing device, the first scene to the AR/VR device, wherein the AR/VR device displays the first scene; receive, by the computing device from the touch input device, touch input provided by a user; compose, by the computing device, a second scene wherein the application content overlaid onto the touch input device is modified based on the touch input wherein the second scene comprises a visual representation of the touch input, and wherein composing, by the computing device, the second scene comprises: providing the touch input to the application by redirecting coordinates of input determined by a driver of the touch input device to the application; and receiving modified application content from the application, wherein the modified application content comprises the visual representation of the touch input; and transport, by the computing device, the second scene to the AR/VR device, wherein the AR/VR device displays the second scene. ### Claim 6 (depends on 5) The system of claim 5, wherein composing, by the computing device, the first scene comprises augmenting the touch input device in the first scene with the application content based on default augmentation configuration values. ### Claim 7 (depends on 5) The system of claim 5, wherein composing, by the computing device, the first scene comprises scaling the application content based on dimensions of the touch input device. ### Claim 8 (depends on 5) The system of claim 5, wherein the application content comprises a partial area of an application screen in which touch input is to be entered. ### Claim 9 A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receive, by a computing device from an augmented reality or virtual reality (AR/VR) device, first input that identifies application content and a location of an artifact in an AR/VR environment; determine, by the computing device, that the artifact in the AR/VR environment comprises a touch input device based on determining that the touch input device is visible, at the location, in one or more images received from the AR/VR device; receive, by the computing device, the application content from an application; compose, by the computing device, a first scene wherein the application content is overlaid onto the touch input device; transport, by the computing device, the first scene to the AR/VR device, wherein the AR/VR device displays the first scene; receive, by the computing device from the touch input device, touch input provided by a user; compose, by the computing device, a second scene wherein the application content overlaid onto the touch input device is modified based on the touch input, wherein the second scene comprises a visual representation of the touch input, and wherein composing, by the computing device, the second scene comprises: providing the touch input to the application by redirecting coordinates of input determined by a driver of the touch input device to the application; and receiving modified application content from the application, wherein the modified application content comprises the visual representation of the touch input; and transport, by the computing device, the second scene to the AR/VR device, wherein the AR/VR device displays the second scene. ### Claim 10 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein composing, by the computing device, the first scene comprises augmenting the touch input device in the first scene with the application content based on default augmentation configuration values. ### Claim 11 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein composing, by the computing device, the first scene comprises scaling the application content based on dimensions of the touch input device. ### Claim 12 (depends on 9) The non-transitory computer-readable medium of claim 9, wherein the application content comprises a partial area of an application screen in which touch input is to be entered.