# US 2022/0284591 A1 · System and method of object detection based on image data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0284591 A1 - Document ID: US20220284591A1 - Status: Published application - Kind: A1 - Application number: 17/678,732 - Filed: 2022-02-23 - Published: 2022-09-08 - Earliest claimed: 2019-07-15 - Assignees: MUJIN, Inc. - Applicants: Mujin, Inc. - Inventors: Jinze YU; Jose Jeronimo MOREIRA RODRIGUES; Xutao YE - Web page: https://patentgrep.com/patent/US20220284591A1 ## CPC - G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components - G06T 7/13: Edge detection - G06T 7/174: involving the use of two or more images - G06T 7/181: involving edge growing; involving edge linking - G06T 7/33: using feature-based methods - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A system and method of detecting objects are provided. The method includes generating first edge information from first image data representing an object based on a first mode of image capture, generating second edge information from second image data representing the object based on a second mode of image capture, the second mode being different from the first mode, fusing the first edge information with the second edge information to generate fused edge information, generating an object detection hypothesis based on the fused edge information, and validating the object detection hypothesis based on the fused edge information, the first edge information, and/or the second edge information. ## Claims ### Claim 1 A method of detecting objects, the method comprising: generating first edge information from first image data representing an object based on a first mode of image capture; generating second edge information from second image data representing the object based on a second mode of image capture, the second mode being different from the first mode; fusing the first edge information with the second edge information to generate fused edge information; generating an object detection hypothesis based on the fused edge information; and validating the object detection hypothesis based on the fused edge information, the first edge information, and/or the second edge information. ### Claim 2 (depends on 1) The method of claim 1, wherein fusing the first edge information with the second edge information includes combining the first edge information with the second edge information into a common image space. ### Claim 3 (depends on 2) The method of claim 2, wherein combining the first edge information with the second edge information into the common image space includes: mapping the first edge information based on coordinates of detected edges associated with the first mode of image capture; and transforming the second edge information to correspond to an image space associated with the first mode of image capture. ### Claim 4 (depends on 3) The method of claim 3, wherein the image space associated with the first mode of image capture is a three-dimensional image space; and wherein transforming the second edge information to correspond to the image space associated with the first mode of image capture includes: projecting two-dimensional edge points from the second edge information onto the three-dimensional image space based on depth information associated with the first edge information corresponding to the two-dimensional locations of the two-dimensional edge points of the second edge information to generate projected three-dimensional points associated with the second edge information; and transforming the projected three-dimensional points to correspond to the three-dimensional image space. ### Claim 5 (depends on 4) The method of claim 4, wherein transforming the second edge information to correspond to the image space associated with the first mode of image capture further comprises: projecting the transformed three-dimensional points corresponding to the three-dimensional image space to two-dimensional coordinates of images associated with the first edge information. ### Claim 6 (depends on 1) The method of claim 1, wherein the image data representing the object based on the first mode of image capture is image data captured by a mode of two-dimensional image capture, and wherein detecting the first edge information includes one or more of: CANNY Edge Detection; Region Similarity Edge Detection; and Line Segment Detection. ### Claim 7 (depends on 1) The method of claim 1, wherein the image data representing the object based on the first mode of image capture is image data captured by a mode of three-dimensional image capture, and wherein detecting the first edge information includes: projecting the image data captured using the first mode into an image space associated with one or more objects to be detected; and extracting object edges from a projected 3D point cloud. ### Claim 8 A non-transitory computer readable medium encoded with instructions for making a computing device execute a method of detecting objects, the method comprising: generating first edge information from first image data representing an object based on a first mode of image capture; generating second edge information from second image data representing the object based on a second mode of image capture, the second mode being different from the first mode; fusing the first edge information with the second edge information to generate fused edge information; generating an object detection hypothesis based on the fused edge information; and validating the object detection hypothesis based on the fused edge information, the first edge information, and/or the second edge information. ### Claim 9 (depends on 8) The non-transitory computer readable medium of claim 8, wherein fusing the first edge information with the second edge information includes combining the first edge information with the second edge information into a common image space. ### Claim 10 (depends on 9) The non-transitory computer readable medium of claim 9, wherein combining the first edge information with the second edge information into the common image space includes: mapping the first edge information based on coordinates of detected edges associated with the first mode of image capture; and transforming the second edge information to correspond to an image space associated with the first mode of image capture. ### Claim 11 (depends on 10) The non-transitory computer readable medium of claim 10, wherein the image space associated with the first mode of image capture is a three-dimensional image space; and wherein transforming the second edge information to correspond to the image space associated with the first mode of image capture includes: projecting two-dimensional edge points from the second edge information onto the three-dimensional image space based on depth information associated with the first edge information corresponding to the two-dimensional locations of the two-dimensional edge points of the second edge information to generate projected three-dimensional points associated with the second edge information; and transforming the projected three-dimensional points to correspond to the three-dimensional image space. ### Claim 12 (depends on 11) The non-transitory computer readable medium of claim 11, wherein transforming the second edge information to correspond to the image space associated with the first mode of image capture further comprises: projecting the transformed three-dimensional points corresponding to the three-dimensional image space to two-dimensional coordinates of images associated with the first edge information. ### Claim 13 (depends on 8) The non-transitory computer readable medium of claim 8, wherein the first image data representing the object based on the first mode of image capture is image data captured by a mode of two-dimensional image capture, and wherein detecting the first edge information includes one or more of: CANNY Edge Detection; Region Similarity Edge Detection; and Line Segment Detection. ### Claim 14 (depends on 8) The non-transitory computer readable medium of claim 8, wherein the first image data representing the object based on the first mode of image capture is image data captured by a mode of three-dimensional image capture, and wherein detecting the first edge information includes: projecting the image data captured using the first mode into an image space associated with one or more objects to be detected; and extracting object edges from a projected 3D point cloud. ### Claim 15 An object detection system for detecting objects, the system comprising: a system interface configured for receiving first image data representing an object based on a first mode of image capture and receiving second image data representing the object based on a second mode of image capture, the second mode being different from the first mode; a processor, communicatively coupled to the system interface, the processor configured to perform an object detection method comprising: generating first edge information from the first image data representing an object based on a first mode of image capture; generating second edge information from the second image data representing an object based on a second mode of image capture, the second mode being different from the first mode; fusing the first edge information with the second edge information to generate fused edge information; generating an object detection hypothesis based on the fused edge information; and validating the object detection hypothesis based on the fused edge information, the first edge information, and/or the second edge information. ### Claim 16 (depends on 15) The object detection system of claim 15, wherein fusing the first edge information with the second edge information includes combining the first edge information with the second edge information into a common image space. ### Claim 17 (depends on 16) The object detection system of claim 16, wherein combining the first edge information with the second edge information into the common image space includes: mapping the first edge information based on coordinates of detected edges associated with the first mode of image capture; and transforming the second edge information to correspond to an image space associated with the first mode of image capture. ### Claim 18 (depends on 17) The object detection system of claim 17, wherein the image space associated with the first mode of image capture is a three-dimensional image space; and wherein transforming the second edge information to correspond to the image space associated with the first mode of image capture includes: projecting two-dimensional edge points from the second edge information onto the three-dimensional image space based on depth information associated with the first edge information corresponding to the two-dimensional locations of the two-dimensional edge points of the second edge information to generate projected three-dimensional points associated with the second edge information; and transforming the projected three-dimensional points to correspond to the three-dimensional image space. ### Claim 19 (depends on 18) The object detection system of claim 18, wherein transforming the second edge information to correspond to the image space associated with the first mode of image capture further comprises: projecting the transformed three-dimensional points corresponding to the three-dimensional image space to two-dimensional coordinates of images associated with the first edge information. ### Claim 20 (depends on 15) The object detection system of claim 15, wherein the first image data representing the object based on the first mode of image capture is image data captured by a mode of three-dimensional image capture, and wherein detecting the first edge information includes: projecting the image data captured using the first mode into an image space associated with one or more objects to be detected; and extracting object edges from a projected 3D point cloud.