# US 2021/0279898 A1 · Object detection using multiple three dimensional scans > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0279898 A1 - Document ID: US20210279898A1 - Status: Published application - Kind: A1 - Application number: 17/326,559 - Filed: 2021-05-21 - Published: 2021-09-09 - Earliest claimed: 2018-07-13 - Assignees: Apple Inc. - Applicants: Apple, Inc. - Inventors: Vincent Chapdelaine-Couture; Mohamed Selim Ben Himane - Web page: https://patentgrep.com/patent/US20210279898A1 ## CPC - G06T 7/579: from motion - G06T 7/37: using transform domain methods - G06T 7/73: using feature-based methods - G06T 7/74: {involving reference images or patches} - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes - G06V 10/443: {by matching or filtering} - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract One exemplary implementation facilitates object detection using multiple scans of an object in different lighting conditions. For example, a first scan of the object can be created by capturing images of the object by moving an image sensor on a first path in a first lighting condition, e.g., bright lighting. A second scan of the object can then be created by capturing additional images of the object by moving the image sensor on a second path in a second lighting condition, e.g., dim lighting. Implementations determine a transform that associates the scan data from these multiple scans to one another and use the transforms to generate a 3D model of the object in a single coordinate system. Augmented content can be positioned relative to that object in the single coordinate system and thus will be displayed in the appropriate location regardless of the lighting condition in which the physical object is later detected. ## Claims ### Claim 1 A method comprising: at a device comprising a processor, a non-transitory computer-readable storage medium, and an image sensor: obtaining first scan data of a physical object in a first lighting condition using the image sensor, the first scan data comprising images from a plurality of image sensor positions defined in a first coordinate system; obtaining second scan data of the physical object in a second lighting condition using the image sensor, the second lighting condition different from the first lighting condition, the second scan data comprising images from a plurality of image sensor positions defined in a second coordinate system; determining, via the processor, a transform between the first coordinate system and the second coordinate system; and generating a three dimensional (3D) model of the physical object based on the first scan data, the second scan data, and the transform. ### Claim 2 (depends on 1) The method of claim 1, wherein determining the transform comprises matching a first image of the first scan data with a second image of the second scan data. ### Claim 3 (depends on 1) The method of claim 1, wherein determining the transform comprises matching multiple images of the first scan data with multiple images of the second scan data. ### Claim 4 (depends on 1) The method of claim 1, wherein determining the transform comprises determining an offset between poses of the image sensor associated with a first image of the first scan data and a second image of the second scan data. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the transform comprises aligning a first point cloud associated with the first scan data with a second point cloud associated with the second scan data. ### Claim 6 (depends on 1) The method of claim 1, wherein the 3D model is a point cloud of points associated with descriptors. ### Claim 7 (depends on 6) The method of claim 6, wherein the point cloud includes points having descriptors based on the first scan data and points having descriptors based on the second scan data. ### Claim 8 (depends on 6) The method of claim 6, wherein generating the 3D model comprises merging a first point having a descriptor based on the first scan data with a second point having a descriptor based on the second scan data. ### Claim 9 (depends on 8) The method of claim 8 further comprising determining to merge the first point and the second point based on proximity of the first point to the second point. ### Claim 10 (depends on 1) The method of claim 1, wherein generating the 3D model comprises representing the first scan data and the second scan data in a single coordinate system. ### Claim 11 (depends on 10) The method of claim 10 further comprising associating augmented content with the 3D model at a position relative to the single coordinate system. ### Claim 12 (depends on 1) The method of claim 1 further comprising: obtaining image data via the image sensor; detecting the physical object using the 3D model and the image data; and aligning the 3D model with the physical object using a single coordinate system. ### Claim 13 (depends on 12) The method of claim 12, wherein detecting the physical object comprises matching the image data with point cloud descriptors of the 3D model. ### Claim 14 (depends on 12) The method of claim 12, wherein detecting the physical object comprises determining a current pose of the image sensor relative to the 3D model. ### Claim 15 (depends on 12) The method of claim 12 further comprising displaying a computer generated reality (CGR) environment depicting the physical object based on the image data and augmented content, wherein the augmented content is positioned based on aligning the 3D model with the physical object using the single coordinate system. ### Claim 16 (depends on 1) The method of claim 1 further comprising determining to request a second scan based on testing the first scan data. ### Claim 17 A system comprising: a non-transitory computer-readable storage medium; an image sensor; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the system to perform operations comprising: obtaining first scan data of a physical object in a first lighting condition using the image sensor, the first scan data comprising images from a plurality of image sensor positions defined in a first coordinate system; obtaining second scan data of the physical object in a second lighting condition using the image sensor, the second lighting condition different from the first lighting condition, the second scan data comprising images from a plurality of image sensor positions defined in a second coordinate system; determining, via the processor, a transform between the first coordinate system and the second coordinate system, wherein determining the transform comprises determining an offset between poses of the image sensor associated with a first image of the first scan data and a second image of the second scan data; and generating a three dimensional (3D) model of the physical object based on the first scan data, the second scan data, and the transform. ### Claim 18 (depends on 17) The system of claim 17, wherein determining the transform comprises matching a first image of the first scan data with a second image of the second scan data. ### Claim 19 (depends on 17) The system of claim 17, wherein determining the transform comprises matching multiple images of the first scan data with multiple images of the second scan data. ### Claim 20 A non-transitory computer-readable storage medium, storing program instructions computer-executable on a computer to perform operations comprising: obtaining first scan data of a physical object in a first lighting condition using an image sensor, the first scan data comprising images from a plurality of image sensor positions defined in a first coordinate system; obtaining second scan data of the physical object in a second lighting condition using the image sensor, the second lighting condition different from the first lighting condition, the second scan data comprising images from a plurality of image sensor positions defined in a second coordinate system; determining, via a processor, a transform between the first coordinate system and the second coordinate system, wherein determining the transform comprises determining an offset between poses of the image sensor associated with a first image of the first scan data and a second image of the second scan data; and generating a three dimensional (3D) model of the physical object based on the first scan data, the second scan data, and the transform.