# US 2020/0401805 A1 · Image processing apparatus and method of controlling the same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2020/0401805 A1 - Document ID: US20200401805A1 - Status: Published application - Kind: A1 - Application number: 16/906,355 - Filed: 2020-06-19 - Published: 2020-12-24 - Earliest claimed: 2019-06-24 - Assignees: Canon Kabushiki Kaisha - Applicants: Canon Kabushiki Kaisha - Inventors: Takuya Kotoyori - Web page: https://patentgrep.com/patent/US20200401805A1 ## CPC - G06T 7/73: using feature-based methods - G06T 7/50: Depth or shape recovery - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 7/90: Determination of colour characteristics - G06V 10/56: relating to colour - G06V 20/20: in augmented reality scenes - G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)} - G06T 2207/10016: Video; Image sequence - G06T 2207/10024: Color image - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30244: Camera pose - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract In order to make it possible for the user to perceive the possibility of a collision with an object in the real world, an image processing apparatus comprises: a location estimation unit configured to, based on a video obtained by an image capturing unit for capturing a physical space, estimating a self-location of the image capturing unit in the physical space; a recognition unit configured to recognize a physical object existing within a certain distance from the self-location based on the video; an area decision unit configured to decide a predetermined area in the physical space in relation to the video; and a determination unit configured to determine whether or not a warning is given in accordance with whether or not a physical object recognized by the recognition unit is included in the predetermined area. ## Claims ### Claim 1 An image processing apparatus comprising: a location estimation unit configured to, based on a video obtained by an image capturing unit for capturing a physical space, estimating a self-location of the image capturing unit in the physical space; a recognition unit configured to recognize a physical object existing within a certain distance from the self-location based on the video; an area decision unit configured to decide a predetermined area in the physical space in relation to the video; and a determination unit configured to determine whether or not a warning is given in accordance with whether or not a physical object recognized by the recognition unit is included in the predetermined area. ### Claim 2 (depends on 1) The image processing apparatus according to claim 1, wherein the area decision unit decides the predetermined area based on a predetermined image feature included in the video. ### Claim 3 (depends on 2) The image processing apparatus according to claim 2, wherein the predetermined image feature is a given index image, and the area decision unit decides an area including a physical object to which the given index image is attached as the predetermined area. ### Claim 4 (depends on 3) The image processing apparatus according to claim 3, further comprising detecting means for accepting registration of the given index image and detecting the given index image included in the video. ### Claim 5 (depends on 2) The image processing apparatus according to claim 2, wherein the predetermined image feature is a given color, and the area decision unit decides an area including a physical object having the given color as the predetermined area. ### Claim 6 (depends on 5) The image processing apparatus according to claim 5, further comprising a color detection unit configured to accept registration of the given color and detecting the given color included in the video. ### Claim 7 (depends on 1) The image processing apparatus according to claim 1, further comprising an acquiring unit configured to acquire a depth of each point in the video, wherein the area decision unit decides a predetermined area in the physical space based on a movement feature of a depth area of each point in the video acquired by the acquiring unit. ### Claim 8 (depends on 7) The image processing apparatus according to claim 7, wherein the area decision unit divides the video into a plurality of partial areas based on the movement feature of the depth area of each point in the video acquired by the acquiring unit, and decides the predetermined area based on one or more partial areas excluding the widest partial area among the plurality of partial areas. ### Claim 9 (depends on 7) The image processing apparatus according to claim 7, wherein the area decision unit excludes, from the predetermined area, a range in which a physical object which continues to approach the image capturing unit for a predetermined period of time appears in the video, based on the movement feature of the depth area of each point in the video acquired by the acquiring unit. ### Claim 10 (depends on 1) The image processing apparatus according to claim 1, wherein the area decision unit decides a predetermined area in the physical space based on a history of the self-location estimated by the location estimation unit based on the video. ### Claim 11 (depends on 10) The image processing apparatus according to claim 10, wherein the area decision unit decides a vertically downward direction in the physical space based on the history, and makes an area in which a cylinder of a predetermined radius from a latest self-location estimated by the location estimation unit extends in the vertically downward direction be a self-area. ### Claim 12 (depends on 1) The image processing apparatus according to claim 1, wherein when the physical object recognized by the recognition unit is not included in the predetermined area, the determination unit determines to give a warning. ### Claim 13 (depends on 1) The image processing apparatus according to claim 1, wherein the warning includes at least one of a warning display on the display unit and a warning sound playback in a sound reproduction unit. ### Claim 14 (depends on 1) The image processing apparatus according to claim 1, wherein the image capturing unit and the image processing apparatus are configured as part of a video see-through type head-mounted display. ### Claim 15 (depends on 14) The image processing apparatus according to claim 14, wherein the predetermined area indicates an area in which at least a part of a body of a user wearing head-mounted display is estimated to exist in the video. ### Claim 16 (depends on 14) The image processing apparatus according to claim 14, wherein the predetermined area is an area in which a physical object moving in synchronism with a movement of a user wearing a head-mounted display is estimated to exist in the video. ### Claim 17 (depends on 16) The image processing apparatus according to claim 16, wherein the physical object that moves in synchronism with the movement of the user is an object that the user holds in his hand and moves. ### Claim 18 A method of controlling an image processing apparatus for processing a video obtained by an image capturing unit for capturing a physical space, the method comprising: based on the video, estimating a self-location of the image capturing unit in the physical space; recognizing a physical object existing within a certain distance from the self-location based on the video; deciding a predetermined area in the physical space in relation to the video; and determining whether or not a warning is given in accordance with whether or not the recognized physical object is included in the predetermined area. ### Claim 19 A non-transitory computer-readable recording medium storing a program that causes a computer to function as an image processing apparatus comprising: a location estimation unit configured to, based on a video obtained by an image capturing unit for capturing a physical space, estimating a self-location of the image capturing unit in the physical space; a recognition unit configured to recognize a physical object existing within a certain distance from the self-location based on the video; an area decision unit configured to decide a predetermined area in the physical space in relation to the video; and a determination unit configured to determine whether or not a warning is given in accordance with whether or not a physical object recognized by the recognition unit is included in the predetermined area.