# US 9,519,640 B2 · Intelligent translations in personal see through display

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,519,640 B2
- Document ID: US9519640B2
- Status: Granted
- Kind: B2
- Application number: 13/464,945
- Filed: 2012-05-04
- Published: 2013-11-07
- Granted: 2016-12-13
- Earliest claimed: 2012-05-04
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Kathryn Stone Perez; John Clavin; Kevin A. Geisner; Stephen G. Latta; Brian J. Mount; Arthur C. Tomlin; Adam G. Poulos
- Inventors: Kathryn Stone Perez; John Clavin; Kevin A. Geisner; Stephen G. Latta; Brian J. Mount; Arthur C. Tomlin; Adam G. Poulos
- Web page: https://patentgrep.com/patent/US9519640B2

## CPC

- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G02B 27/017: {Head mounted}
- G06F 1/163: {Wearable computers, e.g. on a belt}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A see-through, near-eye, mixed reality display apparatus for providing translations of real world data for a user. A wearer's location and orientation with the apparatus is determined and input data for translation is selected using sensors of the apparatus. Input data can be audio or visual in nature, and selected by reference to the gaze of a wearer. The input data is translated for the user relative to user profile information bearing on accuracy of a translation and determining from the input data whether a linguistic translation, knowledge addition translation or context translation is useful.

## Claims

### Claim 1

A method for presenting a translation of a real world expression to a wearer of a see-through head mounted display apparatus, comprising:
determining a three dimensional location of one or more objects in a field of view of the wearer through the see-through head mounted display apparatus, the determining of the three dimensional location of the object is performed using one or more sensors;
receiving a selection of data for translation in the field of view of the wearer by reference to a gaze of the wearer at the item of data;
analyzing the data for translation to provide input data, the input data having one or more of a linguistic translation input, supplemental information input and contextual input;
analyzing the input data to determine whether each of a linguistic translation, an informational translation, and a context translation should be provided based on the input data and a wearer profile, wherein the linguistic translation is a translation from one language to another, the information translation provides additional information regarding the selected item and the contextual translation is a translation based at least in part on a context of another user providing the input data, wherein the linguistic, informational and context translations are different types of translations;
determining wearer knowledge of subject matter in the analyzed input data from the wearer profile;
determining a complexity of the translation based on the determined wearer knowledge;
determining one of a plurality of output formats for the translation based on the input data;
translating the input data into a translated form for the wearer, the translation based on the determined translation type and the determined translated complexity, the translated form based on the determined one of the plurality of output formats; and
rendering the translation in at least one of the plurality of output formats determined, the output format selected from at least an audio translation format and a visual translation format rendered in the see-through head mounted display.

### Claim 2 (depends on 1)

The method of claim 1 wherein said step of analyzing the input data to determine whether each of a linguistic translation, an informational translation, and a context translation should be provided determines a context translation should be provided, the method further comprising evaluating the input data relative to the wearer information to determine an accuracy of the context translation, the accuracy so determined being presented to the user.

### Claim 3 (depends on 1)

The method of claim 1 wherein the step of translating comprises converting the input data from a first language to a second language.

### Claim 4 (depends on 1)

The method of claim 1 wherein the step of translating comprises providing supplemental knowledge for the input data on a subject matter identified in the input data.

### Claim 5 (depends on 4)

The method of claim 4 wherein the supplemental knowledge includes a selectable link for the wearer to provide additional supplemental knowledge on the subject matter.

### Claim 6 (depends on 1)

The method of claim 1 wherein the step of translating comprises evaluating input data in the form of a person and a context, determining a location of the apparatus, and providing contextual information based on at least the location and the input data.

### Claim 7 (depends on 1)

The method of claim 1 further comprising receiving information of an event from user-specific scheduling or profile data, and using information from the event to provide the input data for the translation.

### Claim 8 (depends on 1)

The method of claim 1 wherein the sensors include audio sensors and the step of receiving a selection of data includes receiving audio data and analyzing the data including isolating audio data based on the gaze to provide the input data.

### Claim 9 (depends on 1)

The method of claim 1 further including a step of providing input data comprising training data from the wearer in written or audio form; performing the receiving, analyzing, translating and rendering steps; and receiving from the wearer a corrected translation of the training data.

### Claim 10

A method for presenting a translation of information at a location to a wearer of a see-through head mounted display apparatus, comprising:
receiving from the wearer a selection of input data for translation at the location of the wearer;
determining three dimensional locations of objects within a field of view of the wearer at the location;
receiving a selection of input data based on an object within the field of view for translation in the field of view of the wearer;
retrieving wearer profile information bearing on an accuracy of a translation;
analyzing the input data to select a type of translation to perform, the types of translations comprising each of a linguistic translation from a first language to a second language, an informational translation which provides additional information regarding a particular subject or scene in the input data and a context translation which provides information on an emotional state of a person in the field of view should be provided based on the input data, wherein the linguistic, informational and context translations are different types of translations;
determining wearer proficiency in a language of subject matter in the analyzed input data from the wearer profile information;
determining a complexity of the translation based on the determined wearer knowledge;
determining one of a plurality of output formats for the translation;
performing one or more of the linguistic translation, the informational translation or the context translation on the input data based on determining a translation type and the determined translation complexity, the translation based on wearer feedback on translation accuracy of previous translations; and
rendering the translation in at least one of the plurality of output formats, the plurality of output formats comprising an audio output format and a visual output format in the see-through head mounted display.

### Claim 11 (depends on 10)

The method of claim 10 wherein an intelligent translation includes determining subject matter in the input data, retrieving supplementation information on the subject matter; determining wearer knowledge of the subject matter from the wearer profile information; determining a complexity of supplemental knowledge information and outputting the information.

### Claim 12 (depends on 10)

The method of claim 10 wherein a context translation includes determining bodily characteristics of a person in the field of view of the wearer, determining a context of the person in the field of view of the wearer; and comparing the bodily characteristics and context to known characteristics and contexts to provide an emotional translation of the person in the field of view of the wearer.

### Claim 13 (depends on 10)

The method of claim 10 further including a step of rendering the translation output to the wearer and receiving from the wearer a corrected translation of the input data.