# US 10,769,858 B2 · Systems and methods for sign language recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,769,858 B2 - Document ID: US10769858B2 - Status: Granted - Kind: B2 - Application number: 16/801,684 - Filed: 2020-02-26 - Published: 2020-06-18 - Granted: 2020-09-08 - Earliest claimed: 2016-09-13 - Assignees: Magic Leap, Inc. - Applicants: Magic Leap, Inc. - Inventors: Eric C. Browy; Michael Janusz Woods; Andrew Rabinovich - Web page: https://patentgrep.com/patent/US10769858B2 ## CPC - G02B 27/017: {Head mounted} - G02B 27/0172: {characterised by optical features} - G06F 1/163: {Wearable computers, e.g. on a belt} - G06F 1/1686: {the I/O peripheral being an integrated camera} - G06F 1/1688: {the I/O peripheral being integrated loudspeakers} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/20: in augmented reality scenes - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G02B 27/0179: {Display position adjusting means not related to the information to be displayed} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G02B 2027/014: {comprising information/image processing systems} - G02B 2027/0141: {characterised by the informative content of the display} - G02B 2027/0178: {Eyeglass type (eyeglass details G02C)} - G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye} - G06F 3/16: Sound input; Sound output (speech processing G10L) - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 2207/20084: Artificial neural networks [ANN] - G06V 30/10: Character recognition - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A sensory eyewear system for a mixed reality device can facilitate user's interactions with the other people or with the environment. As one example, the sensory eyewear system can recognize and interpret a sign language, and present the translated information to a user of the mixed reality device. The wearable system can also recognize text in the user's environment, modify the text (e.g., by changing the content or display characteristics of the text), and render the modified text to occlude the original text. ## Claims ### Claim 1 A computing system comprising: a hardware computer processor; a non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the computing system to perform operations comprising: communicate with a plurality of wearable AR devices via one or more networks; receive image information captured by an outward facing imaging system of a first wearable AR device of the plurality of wearable AR devices, wherein the image information comprises images of one or more hands of a first wearer of the first wearable AR device; automatically detect sign language in the received image information, based at least on analysis of the images of the one or more hands of the first wearer; convert the detected sign language into text; and transmit, through the one or more networks, the converted text to a second wearable AR device in the plurality of wearable AR devices, wherein the second wearable AR device is configured to display the converted text to a second wearer of the second wearable AR device. ### Claim 2 (depends on 1) The system of claim 1, wherein the software instructions are further configured to cause the computing system to: transmit, through the one or more networks, a world map of the first wearer of the first wearable AR device. ### Claim 3 (depends on 2) The system of claim 2, wherein the world map of the first wearer comprises an avatar of the first wearer of the first wearable AR device. ### Claim 4 (depends on 1) The system of claim 1, wherein the software instructions are further configured to cause the computing system to identify a target language. ### Claim 5 (depends on 1) The system of claim 1, wherein the software instructions are further configured to cause the computing system to: identify a target language understood by the second wearer of the second AR device based on at least one of: speech as captured by the second AR device, the location of the second AR device, or an input from the wearer of the second AR device. ### Claim 6 (depends on 5) The system of claim 5, wherein the converted text is in the target language. ### Claim 7 (depends on 1) The system of claim 1, wherein said automatically detecting sign language comprises detecting a series of gestures in the received image information. ### Claim 8 (depends on 7) The system of claim 7, wherein said converting the detected sign language into text comprises accessing a sign language dictionary to translate a detected gesture into text. ### Claim 9 (depends on 1) The system of claim 1, wherein said converting the detected sign language into text comprises use of a machine learning algorithm. ### Claim 10 (depends on 1) The system of claim 1, wherein the software instructions are further configured to cause the computing system to: determine a source of the detected sign language, the source comprising a person, and convert the detected sign language based on the determined source. ### Claim 11 (depends on 10) The system of claim 10, wherein said determining the source of the detected sign language is based on at least the size of the one or more hands in the image information. ### Claim 12 (depends on 1) The system of claim 1, wherein the software instructions are further configured to cause the computing system to: transmit an audio stream to the second wearable AR device based on the converted text. ### Claim 13 A method of facilitating communication between wearable AR devices, the method comprising: communicating with a plurality of wearable AR devices via one or more networks; receiving image information captured by an outward facing imaging system of a first wearable AR device of the plurality of wearable AR devices, wherein the image information comprises images of one or more hands of a first wearer of the first wearable AR device; automatically detecting sign language in the received image information, based at least on analysis of the images of the one or more hands of the first wearer; converting the detected sign language into text; and transmitting, through one or more networks, the converted text to a second wearable AR device in the plurality of wearable AR devices, wherein the second wearable AR device is configured to display the converted text to a second wearer of the second wearable AR device. ### Claim 14 (depends on 13) The method of claim 13, comprising transmitting, through the one or more networks, a world map of the first wearer of the first wearable AR device. ### Claim 15 (depends on 14) The method of claim 14, wherein the world map of the first wearer comprises an avatar of the first wearer of the first wearable AR device. ### Claim 16 (depends on 13) The method of claim 13, comprising identifying a target language. ### Claim 17 (depends on 13) The method of claim 13, comprising identifying a target language understood by the second wearer of the second AR device based on at least one of: speech as captured by the second AR device, the location of the second AR device, or an input from the wearer of the second AR device. ### Claim 18 (depends on 17) The method of claim 17, wherein the converted text is in the target language. ### Claim 19 (depends on 13) The method of claim 13, wherein said automatically detecting sign language comprises detecting a series of gestures in the received image information. ### Claim 20 (depends on 13) The method of claim 13, wherein said converting the detected sign language into text comprises accessing a sign language dictionary to translate a detected gesture into text. ### Claim 21 (depends on 13) The method of claim 13, wherein said converting the detected sign language into text comprises use of a machine learning algorithm. ### Claim 22 (depends on 13) The method of claim 13 comprising: determining a source of the detected sign language, the source comprising a person, and converting the detected sign language based on the determined source. ### Claim 23 (depends on 13) The method of claim 13 comprising determining a source of the detected sign language is the wearer of the first wearable AR device based on the size of the one or more hands in the image information. ### Claim 24 (depends on 13) The method of claim 13 comprising transmitting an audio stream to the second wearable AR device based on the converted text.