# US 10,482,681 B2 · Recognition-based object segmentation of a 3-dimensional image > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,681 B2 - Document ID: US10482681B2 - Status: Granted - Kind: B2 - Application number: 15/019,011 - Filed: 2016-02-09 - Published: 2017-08-10 - Granted: 2019-11-19 - Earliest claimed: 2016-02-09 - Assignees: Intel Corporation - Applicants: Intel Corporation - Inventors: Gershom Kutliroff - Web page: https://patentgrep.com/patent/US10482681B2 ## CPC - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 7/12: Edge-based segmentation - G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60) - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2219/2008: Assembling, disassembling - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques are provided for segmentation of objects in a 3D image of a scene. An example method may include receiving, 3D image frames of a scene. Each of the frames is associated with a pose of a depth camera that generated the 3D image frames. The method may also include detecting the objects in each of the frames based on object recognition; associating a label with the detected object; calculating a 2D bounding box around the object; and calculating a 3D location of the center of the bounding box. The method may further include matching the detected object to an existing object boundary set, created from a previously received image frame, based on the label and the location of the center of the bounding box, or, if the match fails, creating a new object boundary set associated with the detected object. ## Claims ### Claim 1 A processor-implemented method for 3-Dimensional (3D) segmentation of objects, the method comprising: receiving, by a processor, a plurality of 3D image frames of a scene, wherein each of the 3D image frames is associated with a pose of a depth camera that generated the 3D image frames; detecting, by the processor, an object in each of the 3D image frames based on object recognition; associating, by the processor, a label with the detected object, the label generated from the object recognition; calculating, by the processor, a 2-Dimensional (2D) bounding box containing the detected object, and a 3D location of the center of the 2D bounding box; matching, by the processor, the detected object to an existing object boundary set created from a previously received 3D image frame, the matching based on the label and the 3D location of the center of the 2D bounding box; and in response to a failure of the matching, creating, by the processor, a new object boundary set associated with the detected object. ### Claim 2 (depends on 1) The method of claim 1, wherein the object boundary set comprises 3D positions of pixels in the 2D bounding box corresponding to the boundary of the object, and further comprises vectors associated with the pixels, the vectors specifying a ray from the position of the depth camera associated with the corresponding pose, to each of the pixels. ### Claim 3 (depends on 2) The method of claim 2, further comprising adjusting the object boundary set to remove duplicate pixels generated from different poses of the depth camera, the removal based on the distance of the pixels from the camera and further based on the direction of the associated vectors. ### Claim 4 (depends on 3) The method of claim 3, further comprising adjusting the object boundary set to remove pixels associated with an occluding object. ### Claim 5 (depends on 1) The method of claim 1, further comprising: detecting surface planes in the scene; calculating an intersection of the detected surface plane with the object boundary set; calculating a ratio of pixels in the intersection to pixels in the detected surface plane; and if the ratio is less than a threshold value, removing the pixels in the detected surface plane from the object boundary set. ### Claim 6 (depends on 5) The method of claim 5, wherein the detecting of surface planes further comprises: calculating normal vectors as the cross product of the difference of neighboring depth pixels of the 3D image frame; clustering the normal vectors based on their value and spatial proximity; and fitting a plane to each cluster based on a least-squares fit. ### Claim 7 (depends on 1) The method of claim 1, wherein each pose of the depth camera is estimated by one of: using a transformation of the camera based on an Iterative Closest Point (ICP) matching operation performed on depth pixels of the 3D image frames; or using a Simultaneous Localization and Mapping (SLAM) operation performed on Red-Green-Blue (RGB) pixels of the 3D image frames; or based on data provided by inertial sensors in the depth camera. ### Claim 8 (depends on 1) The method of claim 1, wherein the object recognition is based on at least one of template matching, classification using a bag-of-words vision model, and classification using a convolutional neural network. ### Claim 9 A system for 3-Dimensional (3D) segmentation of objects, the system comprising: an object detection circuit to: detect an object in each of a plurality of 3D image frames of a scene based on object recognition, wherein the plurality of 3D image frames are captured by a depth camera, each of the 3D image frames being associated with a pose of the depth camera; associate a label with the detected object, the label generated by the object detection circuit based on the object recognition; and calculate a 2-Dimensional (2D) bounding box containing the detected object and a 3D location of the center of the 2D bounding box; an object boundary set matching circuit to match the detected object to an existing object boundary set created from a previously received 3D image frame, the matching based on the label and the 3D location of the center of the 2D bounding box; and an object boundary set creation circuit to create, in response to a failure of the matching, a new object boundary set associated with the detected object. ### Claim 10 (depends on 9) The system of claim 9, wherein the object boundary set comprises 3D positions of pixels in the 2D bounding box corresponding to the boundary of the object, and further comprises vectors associated with the pixels, the vectors specifying a ray from the position of the depth camera associated with the corresponding pose, to each of the pixels. ### Claim 11 (depends on 10) The system of claim 10, further comprising a boundary adjustment circuit to adjust the object boundary set to remove duplicate pixels generated from different poses of the depth camera, the removal based on the distance of the pixels from the camera and further based on the direction of the associated vectors. ### Claim 12 (depends on 11) The system of claim 11, wherein the boundary adjustment circuit is further to adjust the object boundary set to remove pixels associated with an occluding object. ### Claim 13 (depends on 9) The system of claim 9, further comprising a surface plane removal circuit to: detect surface planes in the scene; calculate an intersection of the detected surface plane with the object boundary set; calculate a ratio of pixels in the intersection to pixels in the detected surface plane; and if the ratio is less than a threshold value, remove the pixels in the detected surface plane from the object boundary set. ### Claim 14 (depends on 13) The system of claim 13, wherein the surface plane removal circuit is further to: calculate normal vectors as the cross product of the difference of neighboring depth pixels of the 3D image frame; cluster the normal vectors based on their value and spatial proximity; and fit a plane to each cluster, based on a least-squares fit, to detect the surface planes in the scene. ### Claim 15 (depends on 9) The system of claim 9, wherein each pose of the depth camera is estimated by one of: using a transformation of the camera based on an Iterative Closest Point (ICP) matching operation performed on depth pixels of the 3D image frames; or using a Simultaneous Localization and Mapping (SLAM) operation performed on Red-Green-Blue (RGB) pixels of the 3D image frames; or based on data provided by inertial sensors in the depth camera. ### Claim 16 (depends on 9) The system of claim 9, wherein the object recognition is based on at least one of template matching, classification using a bag-of-words vision model, and classification using a convolutional neural network. ### Claim 17 (depends on 9) The system of claim 9, further comprising the depth camera. ### Claim 18 (depends on 9) A system-on-chip or chip set comprising the system of claim 9. ### Claim 19 At least one non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, result in the following operations for 3-Dimensional (3D) segmentation of objects, the operations comprising: receiving, a plurality of 3D image frames of a scene, wherein each of the 3D image frames is associated with a pose of a depth camera that generated the 3D image frames; detecting an object in each of the 3D image frames based on object recognition; associating a label with the detected object, the label derived from the object recognition; calculating a 2-Dimensional (2D) bounding box containing the detected object, and a 3D location of the center of the 2D bounding box; matching the detected object to an existing object boundary set created from a previously received 3D image frame, the matching based on the label and the 3D location of the center of the 2D bounding box; and creating in response to a failure of the matching, a new object boundary set associated with the detected object. ### Claim 20 (depends on 19) The computer readable storage medium of claim 19, wherein the object boundary set comprises 3D positions of pixels in the 2D bounding box corresponding to the boundary of the object, and further comprises vectors associated with the pixels, the vectors specifying a ray from the position of the depth camera associated with the corresponding pose, to each of the pixels. ### Claim 21 (depends on 20) The computer readable storage medium of claim 20, further comprising adjusting the object boundary set to remove duplicate pixels generated from different poses of the depth camera, the removal based on the distance of the pixels from the camera and further based on the direction of the associated vectors. ### Claim 22 (depends on 21) The computer readable storage medium of claim 21, further comprising adjusting the object boundary set to remove pixels associated with an occluding object. ### Claim 23 The computer readable storage medium of 19, further comprising: detecting surface planes in the scene; calculating an intersection of the detected surface plane with the object boundary set; calculating a ratio of pixels in the intersection to pixels in the detected surface plane; and if the ratio is less than a threshold value, removing the pixels in the detected surface plane from the object boundary set. ### Claim 24 (depends on 23) The computer readable storage medium of claim 23, wherein the detecting of surface planes further comprises: calculating normal vectors as the cross product of the difference of neighboring depth pixels of the 3D image frame; clustering the normal vectors based on their value and spatial proximity; and fitting a plane to each cluster based on a least-squares fit. ### Claim 25 (depends on 19) The computer readable storage medium of claim 19, wherein each pose of the depth camera is estimated by one of: using a transformation of the camera based on an Iterative Closest Point (ICP) matching operation performed on depth pixels of the 3D image frames; or using a Simultaneous Localization and Mapping (SLAM) operation performed on Red-Green-Blue (RGB) pixels of the 3D image frames; or based on data provided by inertial sensors in the depth camera, and wherein the object recognition is based on at least one of template matching, classification using a bag-of-words vision model, and classification using a convolutional neural network.