# US 10,866,093 B2 · Method and system for retrieving data in response to user input

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,866,093 B2
- Document ID: US10866093B2
- Status: Granted
- Kind: B2
- Application number: 14/707,266
- Filed: 2015-05-08
- Published: 2015-09-03
- Granted: 2020-12-15
- Earliest claimed: 2013-07-12
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Rony Abovitz; Brian T. Schowengerdt; Matthew D. Watson
- Web page: https://patentgrep.com/patent/US10866093B2

## CPC

- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A63F 13/00: Video games, i.e. games using an electronically generated display having two or more dimensions
- A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence)
- A63F 13/428: involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
- G01B 11/303: {using photoelectric detection means}
- G02B 6/10: of the optical waveguide type (G02B6/02, G02B6/24 take precedence; devices or arrangements for the control of light by electric, magnetic, electro-magnetic or acoustic means G02F1/00; transferring the modulation of modulated light G02F2/00; optical logic elements G02F3/00; optical analogue/digital converters G02F7/00)
- G02B 6/34: utilising prism or grating {(G02B6/293 takes precedence)}
- G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)}
- G02B 27/017: {Head mounted}
- G02B 27/0172: {characterised by optical features}
- G02B 27/42: Diffraction optics {, i.e. systems including a diffractive element being designed for providing a diffractive effect}(G02B27/60 takes precedence)
- G02B 27/4205: {having a diffractive optical element [DOE] contributing to image formation, e.g. whereby modulation transfer function MTF or optical aberrations are relevant}
- G02B 27/4227: {in image scanning systems}
- G06F 3/005: {Input arrangements through a video camera}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/0485: Scrolling or panning
- G06F 3/0487: using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 16/7837: {using objects detected or recognised in the video content}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06Q 30/0643: {graphically representing goods, e.g. 3D product representation}
- G06T 7/60: Analysis of geometric attributes
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 10/758: {Involving statistics of pixels or of feature values, e.g. histogram matching}
- G06V 20/10: Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59)
- G06V 20/20: in augmented reality scenes
- G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44)
- G06V 20/653: {by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G16H 40/00: ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H 40/20: for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
- G16H 40/67: for remote operation
- H04B 10/25891: {Transmission components (H04B10/40 takes precedence)}
- A61B 2034/101: {Computer-aided simulation of surgical operations}
- G02B 2027/0105: {Holograms with particular structures}
- G02B 2027/0127: {comprising devices increasing the depth of field}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G02B 2027/0185: {Displaying image at variable distance}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06T 2200/04: involving 3D image data
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/30196: Human being; Person
- G06T 2210/41: Medical
- G06T 2219/024: Multi-user, collaborative environment
- G06V 40/113: {Recognition of static hand signs}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- H04B: TRANSMISSION

## Abstract

A method of retrieving data comprises receiving a user input in the form of a command, determining that the user input is a gesture, sending a request to a cloud server based on the determined gesture, receiving data from a knowledge base connected to the cloud server based on the request, and displaying virtual content at the user device based on the received data.

## Claims

### Claim 1

A method of retrieving and presenting data, the method comprising:
receiving a visual user input in the form of a gesture performed by a user;
based at least partly on interpreting the gesture, sending a request to a cloud server in response to the interpreted gesture;
receiving, from the cloud server, the data from a knowledge base connected to the cloud server, the data retrieved from the knowledge base by the cloud server in response to the request, the data including a navigation menu;
receiving, from the cloud server, a set of map points associated with a position of a hand of the user;
displaying a virtual user interface comprising the navigation menu using the set of map points, wherein the virtual user interface is displayed in real time at a position that corresponds to a current position of the hand as described by the set of map points;
detecting a further gesture, the further gesture triggering an expansion of the plurality of first level menu items by spreading apart fingers of a hand, wherein the expansion of the plurality of first level menu items renders a plurality of second level menu items that appears on the finger of the user;
recognizing, based on the detected further gesture, a second active command to create a second virtual user interface comprising the plurality of second level menu items such that the second virtual user interface, when viewed by the user, appears to be stationary at a second position of the hand; and
rendering, in real-time and based on the detected further gesture, the second virtual user interface at the determined map points associated with the second position of the hand such that the plurality of first level menu items of the virtual user interface and the plurality of second level menu items of the second virtual user interface are simultaneously viewable on the finger of the user, the second level menu items being derived from the first level menu items as lower level menu items.

### Claim 2 (depends on 1)

The method of claim 1, further comprising determining if the user input is valid before sending the request to the cloud server.

### Claim 3 (depends on 1)

The method of claim 1, further comprising selecting the knowledge base from a plurality of knowledge bases connected to the cloud server in response to the request sent to the cloud server.

### Claim 4 (depends on 1)

The method of claim 1, wherein different predetermined gestures are respectively mapped to different corresponding commands, and wherein interpreting the meaning of the gesture comprises matching the gesture to one of the predetermined gestures, thereby determining the command corresponding to the one predetermined gesture.

### Claim 5 (depends on 1)

The method of claim 1, wherein the gesture is a finger/hand gesture.

### Claim 6 (depends on 5)

The method of claim 5, wherein the finger/hand gesture is one of an opening or holding up of the palm of the hand.

### Claim 7 (depends on 1)

The method of claim 1, wherein additional user input is received via the virtual user interface.

### Claim 8 (depends on 1)

The method of claim 1, wherein a camera receives the visual user input from the user.

### Claim 9 (depends on 8)

The method of claim 8, wherein the camera is an FOV camera.

### Claim 10 (depends on 1)

The method of claim 1, wherein an item, icon, or field of the navigation menu is displayed on each of a plurality of fingers of the user, and
wherein an item, icon, or field of the navigation menu is not displayed on a thumb of the user.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
receiving another visual user input from the user in the form of another gesture comprising touching one of the plurality of fingers with the thumb;
interpreting the meaning of the other gesture;
sending another request to the cloud server; and
selecting the item, icon, or field of the navigation menu that is displayed on the touched finger in response to the request.

### Claim 12 (depends on 11)

The method of claim 11, wherein the item, icon, or field of the navigation menu comprises one of an instance application, function, menu, tool, model, game, and virtual room or virtual space.

### Claim 13 (depends on 10)

The method of claim 10, further comprising:
receiving another visual user input from the user in the form of another gesture comprising making a circle motion in a palm of the hand of the user with a finger of the other hand of the user;
interpreting the meaning of the other gesture;
sending another request to the cloud server; and
scrolling the displayed items, icons, or fields of the navigation menu successively between the plurality of fingers.

### Claim 14 (depends on 13)

The method of claim 13, wherein if the circle motion is clockwise, the displayed items, icons, or fields of the navigation are successively scrolled in a first direction between the plurality of fingers, and if the circle motion is counterclockwise, the displayed items, icons, or fields of the navigation are successively scrolled in a second opposite direction between the plurality of fingers.

### Claim 15 (depends on 1)

The method of claim 1, wherein the cloud server comprises clusters of computer systems located different geographically dispersed locations.

### Claim 16 (depends on 1)

The method of claim 1, wherein the cloud server comprises geometry mapping data, semantic mapping data, and user personal information.