# US 2022/0026984 A1 · Porting physical object into virtual reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0026984 A1
- Document ID: US20220026984A1
- Status: Published application
- Kind: A1
- Application number: 17/494,496
- Filed: 2021-10-05
- Published: 2022-01-27
- Earliest claimed: 2019-11-01
- Applicants: Facebook Technologies, LLC
- Inventors: Jan Herling; Adrian Brian Ratter
- Web page: https://patentgrep.com/patent/US20220026984A1

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/02: Input arrangements using manually operated switches, e.g. using keyboards or dials
- G06F 3/0481: based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 20/20: in augmented reality scenes
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed herein are related to a system and a method for porting a physical object in a physical space into a virtual reality. In one approach, the method includes detecting an input device in a physical space relative to a user of the input device. In one approach, the method includes presenting, by a display device to the user, a virtual model of the detected input device in a virtual space at a location and an orientation. The location and the orientation of the virtual model in the virtual space may correspond to a location and an orientation of the input device in the physical space relative to the user. In one approach, the method includes visually providing relative to the virtual model in the virtual space, through the display device, spatial feedback on the user's interaction with the input device in the physical space.

## Claims

### Claim 1

A method comprising:
detecting a first contour or a first layout of an input device in a physical space;
identifying, from a plurality of candidate models having various levels of similarity to the first contour or the first layout of the input device and corresponding to a same device type as the input device, a candidate model having a second contour or a second layout in a virtual space, that is closest to the first contour or the first layout of the input device in the physical space, as a virtual model of the input device; and
presenting, by a display device, the virtual model of the input device in the virtual space.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining a first location and a first orientation of the virtual model in the virtual space according to a second location and a second orientation of the input device in the physical space; and
rendering an image of the virtual space including the virtual model of the input device, according to the determined first location and the first orientation of the virtual model in the virtual space.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
detecting a movement of a user in the physical space;
determining a change in the first location and the first orientation of the virtual model according to the detected movement of the user; and
rendering another image of the virtual space including the virtual model of the input device, according to the change in the first location and the first orientation of the virtual model.

### Claim 4 (depends on 1)

The method of claim 1, wherein detecting the first contour or the first layout of the input device in the physical space includes:
capturing an image of the input device, and
detecting the first contour or the first layout of the input device in the captured image.

### Claim 5 (depends on 4)

The method of claim 4, wherein detecting the first contour or the first layout of the input device in the captured image includes:
determining an arrangement of keys on the input device.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
detecting a product identification of the input device; and
determining the candidate model corresponding to the detected product identification.

### Claim 7 (depends on 1)

The method of claim 1, wherein the input device comprises a keyboard.

### Claim 8

A system comprising:
a display device;
a processor; and
a non-transitory computer readable medium storing instructions when executed by the processor cause the processor to:
detect a first contour or a first layout of an input device in a physical space,
identify, from a plurality of candidate models having various levels of similarity to the first contour or the first layout of the input device and corresponding to a same device type as the input device, a candidate model having a second contour or a second layout in a virtual space, that is closest to the first contour or the first layout of the input device in the physical space, as a virtual model of the input device, and
present the virtual model of the input device in the virtual space.

### Claim 9 (depends on 8)

The system of claim 8, wherein the non-transitory computer readable medium stores instructions when executed by the processor cause the processor to:
determine a first location and a first orientation of the virtual model in the virtual space according to a second location and a second orientation of the input device in the physical space; and
render an image of the virtual space including the virtual model of the input device, according to the determined first location and the first orientation of the virtual model in the virtual space.

### Claim 10 (depends on 9)

The system of claim 9, wherein the non-transitory computer readable medium stores instructions when executed by the processor cause the processor to:
detect a movement of a user in the physical space;
determine a change in the first location and the first orientation of the virtual model according to the detected movement of the user; and
render another image of the virtual space including the virtual model of the input device, according to the change in the first location and the first orientation of the virtual model.

### Claim 11 (depends on 8)

The system of claim 8, wherein the processor is configured to detect the input device in the physical space by:
capturing an image of the input device, and
detecting the first contour or the first layout of the input device in the captured image.

### Claim 12 (depends on 11)

The system of claim 11, wherein the processor is configured to detect the first contour or the first layout of the input device in the captured image by:
determining an arrangement of keys on the input device.

### Claim 13 (depends on 8)

The system of claim 8, wherein the non-transitory computer readable medium stores instructions when executed by the processor cause the processor to:
detect a product identification of the input device; and
determine the candidate model corresponding to the detected product identification.

### Claim 14 (depends on 8)

The system of claim 8, wherein the input device comprises a keyboard.

### Claim 15

A head mounted display comprising:
an imaging sensor configured to capture an image of an input device in a physical space;
a processor configured to:
detect from the captured image, a first contour or a first layout of the input device in the physical space, and
identify, from a plurality of candidate models having various levels of similarity to the first contour or the first layout of the input device and corresponding to a same device type as the input device, a candidate model having a second contour or a second layout in a virtual space, that is closest to the first contour or the first layout of the input device in the physical space, as a virtual model of the input device; and
a display device configured to:
present the virtual model of the input device in the virtual space.

### Claim 16 (depends on 15)

The head mounted display of claim 15, wherein:
the processor is configured to determine a first location and a first orientation of the virtual model in the virtual space according to a second location and a second orientation of the input device in the physical space; and
the display device is configured to render an image of the virtual space including the virtual model of the input device, according to the determined first location and the first orientation of the virtual model in the virtual space.

### Claim 17 (depends on 16)

The head mounted display of claim 16, wherein:
the processor is configured to detect a movement of a user in the physical space and determine a change in the first location and the first orientation of the virtual model according to the detected movement of the user, and
the display device is configured to render another image of the virtual space including the virtual model of the input device, according to the change in the first location and the first orientation of the virtual model.

### Claim 18 (depends on 15)

The head mounted display of claim 15, wherein the processor is configured to detect the first contour or the first layout of the input device in the captured image by:
determining an arrangement of keys on the input device.

### Claim 19 (depends on 15)

The head mounted display of claim 15, wherein the processor is configured to:
detect a product identification of the input device; and
determine the candidate model corresponding to the detected product identification.

### Claim 20 (depends on 15)

The head mounted display of claim 15, wherein the input device is a keyboard.
