# US 10,872,228 B1 · Three-dimensional object detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,872,228 B1 - Document ID: US10872228B1 - Status: Granted - Kind: B1 - Application number: 16/142,144 - Filed: 2018-09-26 - Granted: 2020-12-22 - Earliest claimed: 2017-09-27 - Assignees: Apple Inc. - Applicants: Yin Zhou; Russell Y. Webb; Luca Ballan; Cuneyt Oncel Tuzel - Inventors: Yin Zhou; Russell Y. Webb; Luca Ballan; Cuneyt Oncel Tuzel - Web page: https://patentgrep.com/patent/US10872228B1 ## CPC - G01S 17/89: for mapping or imaging - G01S 7/4802: {using analysis of echo signal for target characterisation; Target signature; Target cross-section} - G01S 17/04: Systems determining the presence of a target - G01S 17/06: Systems determining position data of a target - G06V 10/82: using neural networks - G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/64: Three-dimensional [3D] objects - G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]} - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract A method for detecting objects in an environment includes obtaining, from one or more sensors, distance measurements from the one or more sensors to portions of the environment, generating a representation of a three-dimensional space using the distance measurements, identifying object features in the representation of the three-dimensional space using a neural network, comparing the object features to pre-defined three-dimensional templates to generate scores that represent correspondence of the object features to the pre-defined three-dimensional templates, and determining a location and a rotational orientation for a three-dimensional object based on the scores. ## Claims ### Claim 1 A method for detecting an object in an environment, the method comprising: obtaining, from one or more sensors, distance measurements from the one or more sensors to the object; generating a voxel representation of a three-dimensional space that includes the object; associating data values with voxels from the voxel representation of the three-dimensional space using the distance measurements, wherein each of the data values indicates whether presence of the object is detected in a respective one of the voxels, a first group of the data values for a first group of the voxels indicate that each of the first group of the voxels is a filled voxel in which presence of the object is detected, and a second group of the data values for a second group of the voxels in which the presence of the object is not detected indicate a distance to one of the filled voxels; identifying object features in the voxel representation of the three-dimensional space based on the data values using a neural network; comparing the object features to pre-defined three-dimensional templates to generate scores that represent correspondence of the object features to the pre-defined three-dimensional templates; and determining a location and a rotational orientation for a three-dimensional object based on the scores. ### Claim 2 (depends on 1) The method of claim 1, wherein the one or more sensors include LIDAR sensors. ### Claim 3 (depends on 1) The method of claim 1, wherein the voxel representation of the three-dimensional space is a volumetric representation in which the three-dimensional space is divided into the voxels, and the voxels have a predetermined spatial resolution. ### Claim 4 (depends on 1) The method of claim 1, further comprising: defining a surface that represents the object based on the distance measurements from the one or more sensors to the object, wherein associating the data values with the voxels from the voxel representation of the three-dimensional space is based on a position of the surface with respect to the voxels. ### Claim 5 (depends on 1) The method of claim 1, wherein: the pre-defined three-dimensional templates each include rotated bounding boxes that each have a length value, a width value, a height value, and a rotation value, and comparing the object features to the pre-defined three-dimensional templates is performed by a classifier layer using the rotated bounding boxes. ### Claim 6 (depends on 1) The method of claim 1, wherein: the pre-defined three-dimensional templates each include unrotated bounding boxes and separate rotation values, the unrotated bounding boxes each having a length value, a width value, and a height value, and comparing the object features to the pre-defined three-dimensional templates is performed by a first classifier layer using the unrotated bounding boxes and by a second classifier layer using the separate rotation values. ### Claim 7 (depends on 1) The method of claim 1, wherein the distance to one of the filled voxels is expressed as a number of voxels. ### Claim 8 A non-transitory computer-readable storage device including program instructions executable by one or more processors that, when executed, cause the one or more processors to perform operations for an object in an environment, the operations comprising: obtaining, from one or more sensors, distance measurements from the one or more sensors to the object; generating a voxel representation of a three-dimensional space that includes the object; associating data values with voxels from the voxel representation of the three-dimensional space using the distance measurements, wherein each of the data values indicates whether presence of the object is detected in a respective one of the voxels, a first group of the data values for a first group of the voxels indicate that each of the first group of the voxels is a filled voxel in which the presence of the object is detected, and a second group of the data values for a second group of the voxels in which the presence of the object is not detected indicate a distance to one of the filled voxels; identifying object features in the voxel representation of the three-dimensional space based on the data values using a neural network; comparing the object features to pre-defined three-dimensional templates to generate scores that represent correspondence of the object features to the pre-defined three-dimensional templates; and determining a location and a rotational orientation for a three-dimensional object based on the scores. ### Claim 9 (depends on 8) The non-transitory computer-readable storage device of claim 8, wherein the one or more sensors include LIDAR sensors. ### Claim 10 (depends on 8) The non-transitory computer-readable storage device of claim 8, wherein the voxel representation of the three-dimensional space is a volumetric representation in which the three-dimensional space is divided into the voxels, and the voxels have a predetermined spatial resolution. ### Claim 11 (depends on 8) The non-transitory computer-readable storage device of claim 8, further comprising: defining a surface that represents the object based on the distance measurements from the one or more sensors to the object, wherein associating the data values with the voxels from the voxel representation of the three-dimensional space is based on a position of the surface with respect to the voxels. ### Claim 12 (depends on 8) The non-transitory computer-readable storage device of claim 8, wherein: the pre-defined three-dimensional templates each include rotated bounding boxes that each have a length value, a width value, a height value, and a rotation value, and comparing the object features to the pre-defined three-dimensional templates is performed by a classifier layer using the rotated bounding boxes. ### Claim 13 (depends on 8) The non-transitory computer-readable storage device of claim 8, wherein: the pre-defined three-dimensional templates each include unrotated bounding boxes and separate rotation values, the unrotated bounding boxes each having a length value, a width value, and a height value, and comparing the object features to the pre-defined three-dimensional templates is performed by a first classifier layer using the unrotated bounding boxes and by a second classifier layer using the separate rotation values. ### Claim 14 (depends on 8) The non-transitory computer-readable storage device of claim 8, wherein the distance to one of the filled voxels is expressed as a number of voxels. ### Claim 15 A system for detecting an object in an environment, the system comprising: one or more sensors; a memory; and a processor configured to execute instructions stored in the memory, wherein the instructions, when executed, cause the processor to: obtain, from the one or more sensors, distance measurements from the one or more sensors to the object, generate a voxel representation of a three-dimensional space that includes the object, associate data values with voxels from the voxel representation of the three-dimensional space using the distance measurements, wherein each of the data values indicates whether presence of the object is detected in a respective one of the voxels, a first group of the data values for a first group of the voxels indicate that each of the first group of the voxels is a filled voxel in which the presence of the object is detected, and a second group of the data values for a second group of the voxels in which the presence of the object is not detected indicate a distance to one of the filled voxels, identify object features in the voxel representation of the three-dimensional space based on the data values using a neural network, compare the object features to pre-defined three-dimensional templates to generate scores that represent correspondence of the object features to the pre-defined three-dimensional templates, and determine a location and a rotational orientation for a three-dimensional object based on the scores. ### Claim 16 (depends on 15) The system of claim 15, wherein the one or more sensors include LIDAR sensors. ### Claim 17 (depends on 15) The system of claim 15, wherein the voxel representation of the three-dimensional space is a volumetric representation in which the three-dimensional space is divided into the voxels, and the voxels have a predetermined spatial resolution. ### Claim 18 (depends on 15) The system of claim 15, wherein the instructions further cause the processor to: define a surface that represents the object based on the distance measurements from the one or more sensors to the object, wherein instructions cause the processor to associate the data values with the voxels from the voxel representation of the three-dimensional space based on a position of the surface with respect to the voxels. ### Claim 19 (depends on 15) The system of claim 15, wherein: the pre-defined three-dimensional templates each include rotated bounding boxes that each have a length value, a width value, a height value, and a rotation value, and the instructions further cause the processor to compare the object features to the pre-defined three-dimensional templates by a classifier layer using the rotated bounding boxes. ### Claim 20 (depends on 15) The system of claim 15, wherein: the pre-defined three-dimensional templates each include unrotated bounding boxes and separate rotation values, the unrotated bounding boxes each having a length value, a width value, and a height value, and the instructions further cause the processor to compare the object features to the pre-defined three-dimensional templates by a first classifier layer using the unrotated bounding boxes and by a second classifier layer using the separate rotation values. ### Claim 21 (depends on 15) The system of claim 15, wherein the distance to one of the filled voxels is expressed as a number of voxels.