# US 10,373,382 B2 · Information processing apparatus, information processing method, and storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,373,382 B2 - Document ID: US10373382B2 - Status: Granted - Kind: B2 - Application number: 15/195,118 - Filed: 2016-06-28 - Published: 2017-01-05 - Granted: 2019-08-06 - Earliest claimed: 2015-06-30 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Kazuki Takemoto - Web page: https://patentgrep.com/patent/US10373382B2 ## CPC - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G02B 27/017: {Head mounted} - G06T 17/05: Geographic models - G06T 2200/04: involving 3D image data - G06T 2210/21: Collision detection, intersection - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS ## Abstract An apparatus includes an image acquisition unit configured to acquire a captured image of a real space, a position and orientation acquisition unit configured to acquire a position and orientation of each of a plurality of virtual objects, an interference determination unit configured to determine whether interference is present between the virtual objects based on the position and orientation of each of the plurality of virtual objects and model data of the plurality of virtual objects, an interference information recording unit configured to record an outline of an interference part where the virtual objects interfere with each other as interference information, based on a result of the interference determination unit, a generation unit configured to generate a composite image including the captured image, an image of the plurality of virtual objects, and an image representing the outline of the interference part. ## Claims ### Claim 1 An information processing apparatus comprising: one or more processors; and at least one memory device that stores a program executed by the one or more processors in order for the image processing apparatus to function as: a storage unit configured to store model data of a plurality of virtual objects; a position and orientation acquisition unit configured to acquire a position and orientation of each of the plurality of virtual objects; an interference determination unit configured to determine whether interference is present between the virtual objects based on the position and orientation of each of the plurality of virtual objects and the model data of the plurality of virtual objects; an interference area recording unit configured to, in response to being determined that the interference is present between the virtual objects, determine a set of lines of intersections between interfered virtual objects as a contour of an interference area on a surface of an interfered virtual object and record the contour of the interference area; a repeating unit configured to record contours of the interference area by repeating execution of the interference area recording unit at a predetermined timing; a generation unit configured to generate a composite image including a captured image, an image of the plurality of virtual objects, and an image representing the recorded contours of the interference area, after an end of the interference; and an output unit configured to output the generated composite image. ### Claim 2 (depends on 1) The information processing apparatus according to claim 1, further comprising a position and orientation measurement unit configured to measure a position and orientation of an image capturing unit producing the captured image, wherein the generation unit is configured to generate the image of the virtual objects and the image representing the outline of the interference part, based on the position and orientation of the image capturing unit. ### Claim 3 (depends on 1) The information processing apparatus according to claim 1, wherein the interference part is a line of intersection of the model data when the plurality of virtual objects interfere with each other. ### Claim 4 (depends on 1) The information processing apparatus according to claim 1, further comprising an interference information image generation unit configured to generate an image representing the outline of the interference part. ### Claim 5 (depends on 4) The information processing apparatus according to claim 4, wherein the interference information image generation unit generates an image representing a history of interference in images captured before the captured image, by referring to the interference information recording unit. ### Claim 6 (depends on 5) The information processing apparatus according to claim 5, wherein the interference information image generation unit includes a deletion unit configured to delete past data starting from oldest past data, in a case where a number of pieces of data included in the history of the interference is equal to or more than a threshold. ### Claim 7 (depends on 5) The information processing apparatus according to claim 5, wherein the interference information image generation unit incorporates data into the history of the interference, a distance between a position of the virtual objects in the captured image and a position of the virtual objects in an image captured before the captured image is greater than a threshold. ### Claim 8 (depends on 7) The information processing apparatus according to claim 7, wherein the position is a barycentric position determined by calculating an average of all three-dimensional vertex positions of the model data of the virtual objects. ### Claim 9 (depends on 1) The information processing apparatus according to claim 1, further comprising a derivation unit configured to derive reliability of the position and orientation acquired by the position and orientation acquisition unit, wherein the interference information recording unit does not record the interference information, in a case where the reliability is equal to or less than a threshold. ### Claim 10 (depends on 1) The information processing apparatus according to claim 1, wherein the interference determination unit further determines whether interference is present between any of the virtual objects and a predetermined part of a human body. ### Claim 11 (depends on 1) The information processing apparatus according to claim 1, wherein the interference information recording unit records, as the interference information, at least one of a frame number of an image, a time, identification information of the interfering virtual objects, a shape of the interference part, a three-dimensional position of the interference part, at the time of occurrence of the interference. ### Claim 12 (depends on 1) The information processing apparatus according to claim 1, wherein the output unit outputs the composite image to a head mounted display. ### Claim 13 A system comprising a head mounted display and an information processing apparatus, wherein the head mounted display comprises an image capturing unit configured to capture an image of a real space, and wherein the information processing apparatus comprises, one or more processors; and at least one memory that stores a program executed by the one or more processors in order for the image processing apparatus to function as: a storage unit configured to store model data of a plurality of virtual objects; a position and orientation acquisition unit configured to acquire a position and orientation of each of the plurality of virtual objects; an interference determination unit configured to determine whether interference is present between the virtual objects based on the position and orientation of each of the plurality of virtual objects and the model data of the plurality of virtual objects; an interference area recording unit configured to, in response to being determined that the interference is present between the virtual objects, determine a set of lines of intersections between interfered virtual objects as a contour of an interference area on a surface of an interfered virtual object and record the contour of the interference area; a repeating unit to record contours of the interference area by repeating execution of the interference area recording unit at a predetermined timing; a generation unit configured to generate a composite image including the captured image captured by the image capturing unit, an image of the plurality of virtual objects, and an image representing the contours of the recorded interference area, after an end of the interference; and an output unit configured to output the generated composite image to the head mounted display. ### Claim 14 An information processing method comprising: acquiring a captured image of a real space; acquiring a position and orientation of each of a plurality of virtual objects; determining whether interference is present between the virtual objects based on the position and orientation of each of the plurality of virtual objects and model data of the plurality of virtual objects; recording a contour of an interference area, in response to being determined that the interference is present between the virtual objects, determine a set of lines of intersections between interfered virtual objects as a contour of an interference area on a surface of an interfered virtual object at a predetermined timing; recording contours of the interference area by repeating the recording a contour of an interference area at a predetermined timing; generating a composite image including the captured image, an image of the plurality of virtual objects, and an image representing the recorded contours of the interference area, after an end of the interference; and outputting the generated composite image. ### Claim 15 A non-transitory computer-readable storage medium storing a program for causing a computer to perform a method for controlling an information processing apparatus, the information processing method comprising: acquiring a captured image of a real space; acquiring a position and orientation of each of a plurality of virtual objects; determining whether interference is present between the virtual objects based on the position and orientation of each of the plurality of virtual objects and model data of the plurality of virtual objects; recording a contour of an interference area, in response to being determined that the interference is present between the virtual objects, determine a set of lines of intersections between interfered virtual objects as a contour of an interference area on a surface of an interfered virtual object; recording contours of the interference area by repeating the recording a contour of an interference area at a predetermined timing; generating a composite image including the captured image, an image of the plurality of virtual objects, and an image representing the recorded contours of the interference area, after an end of the interference; and outputting the generated composite image.