# US 12,008,696 B2 · Translation method and AR device > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,696 B2 - Document ID: US12008696B2 - Status: Granted - Kind: B2 - Application number: 17/707,690 - Filed: 2022-03-29 - Published: 2023-03-16 - Granted: 2024-06-11 - Earliest claimed: 2021-09-14 - Assignees: Beijing Xiaomi Mobile Software Co., Ltd. - Applicants: Beijing Xiaomi Mobile Software Co., Ltd. - Inventors: Zhixiong Yu; Zifei Dou; Xiang Li - Web page: https://patentgrep.com/patent/US12008696B2 ## CPC - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 11/60: Creating or editing images; Combining images with text - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06V 30/1456: {based on user interactions} - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06T 2219/2016: Rotation, translation, scaling - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A translation method is performed by an augmented reality (AR) device. The method includes obtaining a text to be translated from an environment image in response to a translation trigger instruction; obtaining a translation result of the text to be translated; and displaying the translation result based on a preset display mode in an AR space. The AR space is a virtual space constructed by the AR device based on a real environment. ## Claims ### Claim 1 A translation method, performed by an augmented reality (AR) device, comprising: obtaining a text to be translated from an environment image in response to a trigger instruction; obtaining a translation result of the text to be translated; and displaying the translation result based on a preset display mode in an AR space, wherein the AR space is a virtual space constructed by the AR device based on a real environment; wherein obtaining the text to be translated from the environment image in response to the trigger instruction comprises: recognizing a user gesture in the environment image; determining a target text image pointed by the user gesture; generating the trigger instruction based on the target text image; and determining the target text image as an image of the text to be translated based on the trigger instruction; wherein determining the target text image pointed by the user gesture comprises: recognizing a direction of the user gesture in the environment image and at least one first text image in the environment image; determining a text image of the at least one first text image along the recognized direction as the target text image; or collecting a new environment image based on the user gesture; recognizing a direction of the user gesture in the new environment image and at least one second text image in the new environment image; determining a text image of the at least one second text image along the recognized direction as the target text image. ### Claim 2 (depends on 1) The method of claim 1, wherein obtaining the text to be translated from the environment image in response to the trigger instruction further comprises: obtaining the text to be translated by performing image recognition on the image of the text to be translated. ### Claim 3 (depends on 2) The method of claim 2, wherein obtaining the image of the text to be translated from the environment image in response to the trigger instruction comprises: obtaining the image of the text to be translated by inputting the environment image into a text image recognition model in response to a voice trigger instruction sent by a user. ### Claim 4 (depends on 2) The method of claim 2, wherein obtaining the image of the text to be translated from the environment image in response to the trigger instruction comprises: generating the trigger instruction in response to a text image being recognized from the environment image; and determining the text image as the image of the text to be translated based on the trigger instruction. ### Claim 5 (depends on 2) The method of claim 2, further comprising: displaying the image of the text to be translated in the AR space. ### Claim 6 (depends on 5) The method of claim 5, wherein displaying the image of the text to be translated in the AR space comprises: obtaining an interface image from the environment image, wherein the interface image is a virtual image corresponding to an electronic display screen interface of a real display device in the real environment; and displaying the interface image in the AR space, wherein the interface image comprises the image of the text to be translated, and covers the electronic display screen interface of the real display device in the real environment. ### Claim 7 (depends on 5) The method of claim 5, wherein the image of the text to be translated is displayed in the AR space with a style being at least one of: a gradually increased display size, a gradually increased transparency of a text background, highlighted marking, underlined marking, and displaying a preset color. ### Claim 8 (depends on 5) The method of claim 5, further comprising: obtaining a target operation gesture for the image of the text to be translated displayed in the AR space; and selecting a target operation mode for the target operation gesture from a mapping relationship table between operation gestures and operation modes, wherein the target operation mode is configured to operate the image of the text to be translated displayed in the AR space. ### Claim 9 (depends on 8) The method of claim 8, wherein the gesture with a palm facing up and fingers open corresponds to an operation of zooming out the image of the text to be translated, the first gesture corresponds to an operation of zooming in the image of the text to be translated, and the OK gesture corresponds to an operation of gradually increasing a text background transparency of the image of the text to be translated. ### Claim 10 (depends on 1) The method of claim 1, wherein obtaining the translation result of the text to be translated comprises at least one of: obtaining a translation result sent by an interaction device bound to the AR device; obtaining a translation result sent by a cloud server, wherein the translation result is generated by translating a text to be translated sent by the AR device or the interaction device; and obtaining the text to be translated sent by the interaction device, and generating the translation result by translating the text to be translated. ### Claim 11 (depends on 1) The method of claim 1, wherein displaying the translation result in the AR space based on the preset display mode comprises: adjusting a display definition and/or a display font size of the translation result in the AR space. ### Claim 12 (depends on 11) The method of claim 11, wherein adjusting the display definition and/or the display font size of the translation result in the AR device comprises: gradually reducing the display definition of the translation result in the AR space and gradually increasing the display font size of the translation result. ### Claim 13 An augmented reality (AR) device, comprising: a frame, a camera arranged on the frame, configured to collect an environment image, a microphone arranged on the frame, configured to receive a trigger instruction sent by a user, and a memory and a processor arranged on the frame, the memory being configured to store instructions executable by a processor, the processor being configured to: obtain a text to be translated from the environment image in response to a trigger instruction; obtain a translation result of the text to be translated and control display of the translation result based on a preset display mode in an AR space, wherein the AR space is a virtual space constructed by the AR device based on a real environment; wherein the processor is further configured to: recognize a user gesture in the environment image; determine a target text image pointed by the user gesture; generate the trigger instruction based on the target text image; and determine the target text image as an image of the text to be translated based on the trigger instruction; wherein the processor is further configured to perform: recognizing a direction of the user gesture in the environment image and at least one first text image in the environment image; determining a text image of the at least one first text image along the recognized direction as the target text image; or collecting a new environment image based on the user gesture; recognizing a direction of the user gesture in the new environment image and at least one second text image in the new environment image; determining a text image of the at least one second text image along the recognized direction as the target text image. ### Claim 14 (depends on 13) The AR device of claim 13, wherein the processor is further configured to: obtain the text to be translated by performing image recognition on the image of the text to be translated. ### Claim 15 (depends on 14) The AR device of claim 14, wherein the camera is configured to receive a voice trigger instruction, and the processor is further configured to: obtain the image of the text to be translated by inputting the environment image into a text image recognition model in response to the voice trigger instruction. ### Claim 16 (depends on 14) The AR device of claim 14, wherein the processor is further configured to: generate the trigger instruction in response to a text image being recognized from the environment image; and determine the text image as the image of the text to be translated based on the trigger instruction. ### Claim 17 A non-transitory computer readable storage medium having a computer program stored thereon, wherein when the computer program is performed by a processor, a translation method performed by an augmented reality (AR) device, the method comprising: obtaining a text to be translated from an environment image in response to a trigger instruction; obtaining a translation result of the text to be translated; and displaying the translation result based on a preset display mode in an AR space, wherein the AR space is a virtual space constructed by the AR device based on a real environment; wherein obtaining the text to be translated from the environment image in response to the trigger instruction comprises: recognizing a user gesture in the environment image; determining a target text image pointed by the user gesture; generating the trigger instruction based on the target text image; and determining the target text image as an image of the text to be translated based on the trigger instruction; wherein determining the target text image pointed by the user gesture comprises: recognizing a direction of the user gesture in the environment image and at least one first text image in the environment image; determining a text image of the at least one first text image along the recognized direction as the target text image; or collecting a new environment image based on the user gesture; recognizing a direction of the user gesture in the new environment image and at least one second text image in the new environment image; determining a text image of the at least one second text image along the recognized direction as the target text image.