# US 2019/0377474 A1 · Systems and methods for a mixed reality user interface

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0377474 A1
- Document ID: US20190377474A1
- Status: Published application
- Kind: A1
- Application number: 16/204,765
- Filed: 2018-11-29
- Published: 2019-12-12
- Earliest claimed: 2018-06-06
- Assignees: FactualVR, Inc.
- Applicants: FactualVR, Inc.
- Inventors: EDUARDO NEETER
- Web page: https://patentgrep.com/patent/US20190377474A1

## CPC

- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G02B 27/017: {Head mounted}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for a mixed reality user interface are provided. The systems and methods relate to equipment and an interface that a user can employ to interact with a real or a virtual environment. The equipment includes a head device, a torso device, a waist device, and a hands device(s). The interface includes user interface elements, referred to as “widgets”, that are attachable and removable from a grid of user interface space locations, referred to as “magnets”. The user is able to use the widgets to interact with the virtual environment.

## Claims

### Claim 1

A system for generating user interface (“UI”) elements in a virtual environment, comprising:
a head device worn by a user, the head device displaying the virtual environment for the user;
at least one wearable device worn by the user; and
a processor in communication with the head device and the at least one wearable device, the processor generating a persistent virtual toolkit including at least one UI element corresponding to the at least one wearable device worn by the user, and causing the head device to display the persistent virtual toolkit in the virtual environment, the persistent virtual toolkit movable in the virtual environment by the user to a desired location within the virtual environment, the virtual toolkit persisting at the desired location while the user moves within the virtual environment.

### Claim 2 (depends on 1)

The system of claim 1, wherein the virtual toolkit further comprises:
a first UI area is associated with the head device, the first UI area including a first magnet grid and the first magnet grid comprises a first group of magnets,
wherein the magnets provide positional information relating to a location where the at least one UI element can be attached.

### Claim 3 (depends on 2)

The system of claim 2, wherein the virtual toolkit further comprises:
a second UI area associated with the at least one wearable device worn by the user, the second UI area including a second magnet grid and the second magnet grid comprises a second group of magnets.

### Claim 4 (depends on 3)

The system of claim 3, wherein the first magnet grid is defined using a first set of parameters and attributes, and the second magnet grid is defined using a second set of parameters and attributes.

### Claim 5 (depends on 1)

The system of claim 1, wherein the user interacts with the at least one UI element using a hand gesture, a voice command, an eye command, a controller, or a sensor.

### Claim 6 (depends on 1)

The system of claim 1, wherein the at least one UI element comprises at least one of a video, a floorplan, a tool, a sensor, evidence, or a defined area where a user can perform a hand gesture.

### Claim 7 (depends on 1)

The system of claim 1, wherein the at least one UI element comprises a menu of buttons.

### Claim 8 (depends on 1)

The system of claim 1, further comprising:
a second wearable device in communication with the processor, wherein the processor;
generates a second UI element corresponding to the second wearable device, the second UI element forming part of the virtual toolkit; and
causes the head device to display the second UI element in the virtual environment.

### Claim 9 (depends on 1)

The system of claim 1, wherein the at least one wearable device comprises a device worn on a user's torso, waist, arm, wrist, or hand.

### Claim 10 (depends on 1)

The system of claim 1, wherein the head device comprises one of a head-mounted display, virtual reality glasses, or smart glasses.

### Claim 11 (depends on 1)

The system of claim 1, wherein the head device or the at least one wearable device comprises at least one of a live feed camera, a night vision camera, a thermal camera, an infrared camera, or a 3D scanning camera.

### Claim 12 (depends on 1)

The system of claim 1, wherein the head device comprises at least one of a microphone, a speaker, a heat sensor, a chemical sensor, a humidity sensor, a pressure sensor, or a depth sensor.

### Claim 13 (depends on 1)

The system of claim 1, wherein the virtual environment comprises one of a virtual reality environment, an augmented reality environment, or a mixed reality environment.

### Claim 14 (depends on 1)

The system of claim 1, wherein the virtual toolkit further comprises a tracker for indicating a type of device, and the processor causes the head device to display the tracker in the virtual environment.

### Claim 15 (depends on 1)

The system of claim 1, wherein the virtual toolkit further comprises a navigational tool capable of indicating a physical path travelled by a user, the path including at least one of an actual path, a possible path, or a hypothetical path; and the processor causes the head device to display the path as a series of sequential steps in the virtual environment.

### Claim 16 (depends on 1)

The system of claim 1, wherein the virtual toolkit further comprises a mini map tool showing a 2D or a 3D virtual representation or a virtual map of an environment where an incident is occurring; and the processor causes the head device to display the mini map tool in the virtual environment.

### Claim 17 (depends on 16)

The system of claim 16, wherein the mini map tool displays one or more trackers.

### Claim 18 (depends on 1)

The system of claim 1, wherein the virtual toolkit further comprises a video feed tool streaming a live view or a recording from a camera associated with another user, an object, or a surveillance camera; and the processor causes the head device to display the video feed tool in the virtual environment.

### Claim 19 (depends on 1)

The system of claim 1, wherein the virtual toolkit further comprises at least one of a rear-view mirror tool, a voice communication tool, a biometric tool, or a dashboard tool.

### Claim 20 (depends on 1)

The system of claim 1, wherein the desired location is set relative to the user or set relative to the virtual environment.

### Claim 21

A method for generating user interface (“UI”) elements in a virtual environment, comprising the steps of:
displaying the virtual environment in a head device worn by a user;
generating a persistent virtual toolkit by a processor in communication with the head device and at least one wearable device worn by the user, the persistent virtual toolkit including at least one UI element corresponding to the at least one wearable device;
displaying the persistent virtual toolkit in the virtual environment;
allowing the user to move the persistent virtual toolkit to a desired location within the virtual environment; and
maintaining the persistent virtual toolkit at the desired location while the user moves within the virtual environment.

### Claim 22 (depends on 21)

The method of claim 21, wherein the step of generating the virtual toolkit further comprises generating a first UI area is associated with the head device, the first UI area including a first magnet grid and the first magnet grid comprises a first group of magnets, wherein the magnets provide positional information relating to a location where the at least one UI element can be attached.

### Claim 23 (depends on 22)

The method of claim 22, wherein the step of generating the virtual toolkit further comprises generating a second UI area associated with the at least one wearable device worn by the user, the second UI area including a second magnet grid and the second magnet grid comprises a second group of magnets.

### Claim 24 (depends on 23)

The method of claim 23, wherein the first magnet grid is defined using a first set of parameters and attributes, and the second magnet grid is defined using a second set of parameters and attributes.

### Claim 25 (depends on 21)

The method of claim 21, further comprising the step of generating a second UI element corresponding to a second wearable device, the second UI element forming part of the virtual toolkit, and displaying the second UI element in the virtual environment.

### Claim 26 (depends on 21)

The method of claim 21, further comprising displaying in the head device at least one of a live feed camera, a night vision camera, a thermal camera, an infrared camera, or a 3D scanning camera.

### Claim 27 (depends on 21)

The method of claim 21, further comprising generating a tracker for indicating a type of device, and displaying the tracker in the virtual environment.

### Claim 28 (depends on 21)

The method of claim 21, further comprising generating a navigational tool capable of indicating a physical path travelled by a user, the path including at least one of an actual path, a possible path, or a hypothetical path; and displaying the path in the head device as a series of sequential steps in the virtual environment.

### Claim 29 (depends on 21)

The method of claim 21, wherein the desired location is set relative to the user or set relative to the virtual environment.

### Claim 30 (depends on 21)

The method of claim 21, further comprising generating a mini map tool showing a 2D or a 3D virtual representation or a virtual map of an environment where an incident is occurring; and displaying the mini map tool in the virtual environment in the head device.

### Claim 31 (depends on 30)

The method of claim 30, wherein the mini map tool displays one or more trackers.

### Claim 32 (depends on 21)

The method of claim 21, further comprising generating a live video feed tool streaming a live view from a camera associated with another user; and displaying the live video feed tool in the virtual environment in the head device.

### Claim 33 (depends on 21)

The method of claim 21, wherein the step of generating the virtual toolkit further comprises generating at least one of a rear-view mirror tool, a voice communication tool, a biometric tool, or a dashboard tool.
