# US 12,002,173 B2 · Image processing device, image processing method, and program > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,002,173 B2 - Document ID: US12002173B2 - Status: Granted - Kind: B2 - Application number: 18/133,252 - Filed: 2023-04-11 - Published: 2023-08-03 - Granted: 2024-06-04 - Earliest claimed: 2010-02-02 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Yasuhiro Suto; Masaki Fukuchi; Kenichirou Ooi; Jingjing Guo; Kouichi Matsuda - Web page: https://patentgrep.com/patent/US12002173B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 27/0172: {characterised by optical features} - G06T 11/60: Creating or editing images; Combining images with text - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 19/003: {Navigation within 3D models or images} - G06V 20/20: in augmented reality scenes - G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69) - G09G 5/00: Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators (image data processing or generation, in general G06T) - G02B 2027/014: {comprising information/image processing systems} - G06T 2210/61: Scene description - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) ## Abstract There is provided an image processing device including: a data storage unit storing feature data indicating a feature of appearance of one or more physical objects; an environment map building unit for building an environment map based on an input image obtained by imaging a real space and the feature data, the environment map representing a position of a physical object present in the real space; a control unit for acquiring procedure data for a set of procedures of operation to be performed in the real space, the procedure data defining a correspondence between a direction for each procedure and position information designating a position at which the direction is to be displayed; and a superimposing unit for generating an output image by superimposing the direction for each procedure at a position in the input image determined based on the environment map and the position information, using the procedure data. ## Claims ### Claim 1 An image processing device comprising: one or more processors configured to: acquire procedure data for a set of procedures of operation to be performed in a real space, wherein at least one procedure of the set of procedures is related to an object; control to initiate a display of first procedure information based on a first procedure of the acquired procedure data; acquire a first progress condition; and control to initiate a display of second procedure information based on a second procedure of the acquired procedure data, wherein the first procedure information is related to a first object in the real space, and wherein the second procedure information is related to a second object in the real space. ### Claim 2 (depends on 1) The image processing device according to claim 1, wherein the first progress condition is detected based on whether or not a predetermined time has elapsed. ### Claim 3 (depends on 1) The image processing device according to claim 1, wherein the first progress condition is detected based on whether or not a predetermined time has elapsed from detecting a user operation. ### Claim 4 (depends on 1) The image processing device according to claim 1, wherein the first procedure information is displayed as a text box. ### Claim 5 (depends on 1) The image processing device according to claim 1, wherein the first progress condition is detected based on a result of recognizing a state of a hand of a user. ### Claim 6 (depends on 1) The image processing device according to claim 1, wherein the one or more processors is further configured to control a display of the first procedure information at a position associated with the first object in the real space. ### Claim 7 (depends on 1) The image processing device according to claim 1, wherein the procedure data defines a correspondence between a direction for each procedure and position information designating a position at which the direction is to be displayed or a condition for progressing displaying of each direction. ### Claim 8 (depends on 7) The image processing device according to claim 7, wherein the condition for progressing displaying of each direction includes a condition according to a position or a posture of the object. ### Claim 9 (depends on 7) The image processing device according to claim 7, wherein the displaying of the direction for each procedure included in the set of procedures is controlled according to the procedure data. ### Claim 10 (depends on 1) The image processing device according to claim 1, wherein the first progress condition comprises at least one of a state of the object, passage of a given time, an external event, or a combination thereof. ### Claim 11 (depends on 1) The image processing device according to claim 1, wherein the first procedure information is displayed prior to the second procedure information being displayed. ### Claim 12 (depends on 1) The image processing device according to claim 1, wherein the one or more processors is further configured to store feature data that indicates features of appearance of objects, and wherein the first object and the second object are identified based on correspondence between the stored feature data. ### Claim 13 (depends on 1) The image processing device according to claim 1, wherein the set of procedures comprises more than two procedures. ### Claim 14 (depends on 13) The image processing device according to claim 13, wherein the image processing device is further caused to: detect a second progress condition associated with the second procedure when the second procedure information is displayed; and control, when the second progress condition is detected, a display of third procedure image data based on a third procedure of the acquired procedure data, wherein the first procedure information is displayed prior to the second procedure information being displayed, and the second procedure information is displayed prior to the third procedure image data being displayed, and wherein the third procedure image data is related to a third object in the real space. ### Claim 15 (depends on 1) The image processing device according to claim 1, wherein the set of procedures comprises text based information. ### Claim 16 (depends on 1) The image processing device according to claim 1, wherein the one or more processors is further configured to acquire the first progress condition when the first procedure information is displayed. ### Claim 17 An image processing method comprising: acquiring procedure data for a set of procedures of operation to be performed in a real space, where at least one procedure of the set of procedures is related to an object; controlling a display of first procedure information based on a first procedure of the acquired procedure data; acquiring a first progress condition; and controlling a display of second procedure information based on a second procedure of the acquired procedure data, wherein the first procedure information is related to a first object in the real space, and wherein the second procedure information is related to a second object in the real space. ### Claim 18 A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer causes the computer to execute a method, the method comprising: acquiring procedure data for a set of procedures of operation to be performed in a real space, wherein at least one procedure of the set of procedures is related to an object; controlling a display of first procedure information based on a first procedure of the acquired procedure data; acquiring a first progress condition; and controlling a display of second procedure information based on a second procedure of the acquired procedure data, wherein the first procedure information is related to a first object in the real space, and wherein the second procedure information is related to a second object in the real space.