# US 10,771,350 B2 · Method and apparatus for changeable configuration of objects using a mixed reality approach with augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,771,350 B2
- Document ID: US10771350B2
- Status: Granted
- Kind: B2
- Application number: 15/715,509
- Filed: 2017-09-26
- Published: 2019-03-28
- Granted: 2020-09-08
- Earliest claimed: 2017-09-26
- Assignees: Siemens Aktiengesellschaft
- Applicants: Siemens Aktiengesellschaft
- Inventors: Mareike Kritzler
- Web page: https://patentgrep.com/patent/US10771350B2

## CPC

- H04L 41/22: comprising specially adapted graphical user interfaces [GUI]
- G02B 27/017: {Head mounted}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 20/20: in augmented reality scenes
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system for enabling an augmented reality application includes in a mobile device of a user, displaying a real-world object and allowing the user to interact with the real-world object by drawing a virtual outline of the object and build up the current configuration or its virtual twin. A computer processor retrieves a list of possible component objects of the real-world object. The user interacts with a visual depiction of the scene and move candidate component objects to the virtual outline of the real-world object. The configuration of the real-world object is saved allowing the AR application to retrieve the verified configuration and produce augmented overlays overtop the visual depiction of the real-world object. The configuration is verified by the user using spatial relationships of the component objects associated with the real-world object. The verified object configuration may be tracked in the AR application without needing a recognition procedure.

## Claims

### Claim 1

A system for enabling an augmented reality (AR) application comprising:
a computer processor;
a mobile device in communication with the computer processor;
an image sensing device in communication with the mobile device;
a memory in communication with the computer processor, the memory storing instructions executable on a computer, which when executed by the computer processor cause the computer processor to:
receive an input from a user indicative of a spatial property of a real-world object, wherein the user provides the input by dragging a reconfigurable component object representative of the real-world object to a spatial location in an environment sensed by the user;
track the real-world object with the AR application based on the user input; and
display a visual overlay in the user's field of view sensed by the image sensing device containing the real-world object,
wherein the real-world object is a computer rack having a first configuration and a second configuration, the first configuration of the computer rack defining a first placement of a first plurality of computer components in the computer rack in a first specific order, and the second configuration of the computer rack defining a second placement of a second plurality of computer components in the computer rack in a second specific order that is different than the first specific order.

### Claim 2 (depends on 1)

The system of claim 1, wherein the mobile device is a device wearable by the user.

### Claim 3 (depends on 1)

The system of claim 1, wherein the mobile device is a tablet computer.

### Claim 4 (depends on 1)

The system of claim 1, wherein the input received from the user comprises the user tracing a virtual outline of the real-world object.

### Claim 5 (depends on 1)

The system of claim 1, wherein the input received from the user comprises the user selecting a type of real-world object from list containing a plurality of types of real-world objects.

### Claim 6 (depends on 1)

The system of claim 1, wherein the real-world object is a computer rack and a first configuration of the computer rack is a placement of a plurality of computer components in the computer rack in a first specific order.

### Claim 7 (depends on 1)

The system of claim 1, wherein the real-world object is an inventoried component within a room, and a first configuration is a spatial position of the inventoried component within the room.

### Claim 8 (depends on 7)

The system of claim 7, wherein displaying a visual overlay comprises:
displaying in the mobile device, stored information relating to the inventoried object.

### Claim 9

A method for displaying information in an augmented reality (AR) application comprising:
in a computer processor, receiving an input from a user relating to a spatial coordinate within a field of view of the user, wherein the input received from the user is provided by the user dragging a reconfigurable component object representative of a real-world object to the spatial coordinate in the field of view the user;
in the computer processor, associating the user input with a first configuration of the real-world object; and
in the AR application, superimposing in a viewable mobile device, augmented reality data over a depiction of the real-world object
wherein the real-world object is a computer rack having the first configuration and a second configuration, the first configuration of the computer rack defining a first placement of a first plurality of computer components in the computer rack in a first specific order, and the second configuration of the computer rack defining a second placement of a second plurality of computer components in the computer rack in a second specific order that is different than the first specific order.

### Claim 10 (depends on 9)

The method of claim 9, wherein the mobile device is a device wearable by the user.

### Claim 11 (depends on 9)

The method of claim 9, wherein the mobile device is a tablet computer.

### Claim 12 (depends on 9)

The method of claim 9, wherein the input received from the user comprises the user tracing a virtual outline of the real-world object or user environment.

### Claim 13 (depends on 9)

The method of claim 9, wherein the input received from the user comprises the user selecting a type of real-world object from list containing a plurality of types of real-world objects.

### Claim 14 (depends on 9)

The method of claim 9, wherein displaying a visual overlay comprises:
displaying in the mobile device, an animated maintenance procedure on one computer component of the first configuration or the second configuration of the computer rack.

### Claim 15 (depends on 9)

The method of claim 9, wherein the real-world object is an inventoried component within a room, and the first configuration is a spatial position of the inventoried component within the room.

### Claim 16 (depends on 15)

The method of claim 15, wherein displaying a visual overlay comprises: displaying in the mobile device, stored information relating to the inventoried object.