# US 12,118,807 B2 · Apparatus and method for three-dimensional object recognition > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,807 B2 - Document ID: US12118807B2 - Status: Granted - Kind: B2 - Application number: 17/280,841 - Filed: 2019-09-28 - Published: 2022-01-06 - Granted: 2024-10-15 - Earliest claimed: 2018-09-26 - Assignees: Sitesee Pty Ltd - Applicants: SiteSee Pty Ltd - Inventors: Lucio Piccoli; Laurie Opperman; Patrick Mahoney - Web page: https://patentgrep.com/patent/US12118807B2 ## CPC - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06V 20/653: {by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06T 7/11: Region-based segmentation - G06T 7/174: involving the use of two or more images - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 15/04: Texture mapping - G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation} - G06V 10/82: using neural networks - G06V 20/176: {Urban or other man-made structures} - G06V 20/647: {by matching two-dimensional images to three-dimensional objects} - G06T 2207/10032: Satellite or aerial image; Remote sensing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30184: Infrastructure - G06V 2201/12: Acquisition of 3D measurements of objects - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract The present application relates to a method for recognising at least one object in a three-dimensional scene, the method including, in an electronic processing device: determining a plurality of two-dimensional images of the scene, the images at least partially including the at least one object; determining a plurality of two-dimensional segmentations of the at least one object, the two-dimensional segmentations corresponding to the two dimensional images; generating a three-dimensional representation of the scene using the images; generating a mapping indicative of a correspondence between the images and the representation; and using the mapping to map the plurality of segmentations to the three dimensional representation, to thereby recognise the at least one object in the scene. ## Claims ### Claim 1 A method for recognizing at least one object in a three-dimensional scene, the method including performing the following steps in an electronic processing device: determining a plurality of two-dimensional images of the scene, the images at least partially including the at least one object; determining a plurality of two-dimensional segmentations of the at least one object, the two-dimensional segmentations corresponding to the two-dimensional images; generating a three-dimensional representation of the scene using the images; determining a pose estimation for each of the two-dimensional images; generating a plurality of two-dimensional representations of the three-dimensional representation using the pose estimations, each two-dimensional representation corresponding to a respective two-dimensional image; generating a mapping using the two-dimensional representations, the mapping being indicative of a correspondence between the images and the representation; and using the mapping to map the plurality of segmentations to the three-dimensional representation, to thereby recognize the at least one object in the scene; wherein each of the two-dimensional segmentations includes of a probability that each pixel in the corresponding two-dimensional image is indicative of the object and wherein the scene includes at least two objects, and the two-dimensional segmentation at least two probabilities per pixel, each probability corresponding to a respective object. ### Claim 2 (depends on 1) A method according to claim 1, wherein generating the plurality of two-dimensional representations includes generating a raster render of the three-dimensional representation at each of the plurality of pose estimations. ### Claim 3 (depends on 1) A method according to claim 1, wherein the three-dimensional representation includes a mesh, and wherein the mesh includes a plurality of vertices and a plurality of faces. ### Claim 4 (depends on 1) A method according to claim 1, wherein the method includes, in an electronic processing device: determining a plurality of face references indicative of the correspondence between each two-dimensional representation and respective faces of the rendered three-dimensional representation; and, generating the mapping using the face references. ### Claim 5 (depends on 1) A method according to claim 1, wherein the plurality of images of the scene are captured at a plurality of poses, wherein the plurality of images includes at least two adjacent images captured at different poses, both of the adjacent images being at least partially indicative of the same portion of the scene. ### Claim 6 (depends on 1) A method according to claim 1, wherein the plurality of images includes a plurality of adjacent images, and wherein the plurality of images includes adjacent images indicative of a plurality of views of the at least one object. ### Claim 7 (depends on 6) A method according to claim 6, wherein the method includes capturing the two-dimensional images using an image capture device, and wherein the image capture device includes an unmanned aerial vehicle. ### Claim 8 (depends on 1) A method according to claim 1, wherein the three-dimensional representation includes a three-dimensional reality model. ### Claim 9 (depends on 1) A method according to claim 1, wherein the at least one object includes at least one of: equipment; an antenna; non-antenna equipment; a cellular phone tower; ground; or a building. ### Claim 10 (depends on 1) A method according to claim 1, wherein the method includes, in an electronic processing device, generating a three-dimensional segmentation of the three-dimensional representation using the mapping and the two-dimensional segmentations. ### Claim 11 (depends on 1) A method according to claim 1, wherein the method includes, in an electronic processing device: determining at least one object parameter; and, recognizing the at least one object in the scene using the at least one object parameter, and wherein the at least one object parameter is indicative of at least one of: a position of the object; an orientation of the object; a size of the object; connectivity of objects; or, a distance between objects. ### Claim 12 (depends on 1) A method according to claim 1, wherein the segmentation includes at least one of supervised segmentation, unsupervised segmentation, or semi-supervised segmentation, and wherein the method includes, in an electronic processing device, performing the segmentation using a supervised segmentation algorithm. ### Claim 13 (depends on 12) A method according to claim 12, wherein the supervised segmentation algorithm includes a neural network, or wherein the supervised segmentation algorithm includes a convolution neural network (CNN). ### Claim 14 (depends on 12) A method according to claim 12, wherein the method includes, in an electronic processing device, training the segmentation algorithm. ### Claim 15 (depends on 14) A method according to claim 14, wherein the method includes, in an electronic processing device: determining a plurality of two-dimensional training images of a training scene, the training images at least partially including at least one training object, wherein the training object is indicative of the at least one object; determining a plurality of training pose estimations corresponding to each of the two-dimensional training images; generating a three-dimensional training representation of the training scene using the images; segmenting the three-dimensional training representation to recognize the training object; generating a plurality of two-dimensional segmented representations of the three-dimensional segmentation; and, training the segmentation algorithm using the plurality of two-dimensional images and segmented representations. ### Claim 16 (depends on 1) A method according to claim 1, wherein the method includes, in an electronic processing device: generating a three-dimensional mesh of the at least one object using the mapping, the two-dimensional segmentations and the three-dimensional representation, and wherein the method includes, in an electronic processing device post-processing the three-dimensional mesh. ### Claim 17 (depends on 1) A method according to claim 1 for training a supervised segmentation algorithm for recognizing at least one object in a scene, the method including, in an electronic processing device: determining a plurality of two-dimensional training images of a training scene, the training images at least partially including at least one training object, wherein the training object is indicative of the at least one object; determining a plurality of training pose estimations corresponding to each of the two-dimensional training images; generating a three-dimensional training representation of the training scene using the images; segmenting the three-dimensional training representation to recognize the training object; generating a plurality of two-dimensional segmented representations of the three-dimensional segmentation; and, training the segmentation algorithm using the plurality of two-dimensional images and segmented representations. ### Claim 18 An apparatus for recognizing at least one object in a three-dimensional scene, the apparatus including an electronic processing device that: determines a plurality of two-dimensional images of the scene, the images at least partially including the at least one object; determines a plurality of two-dimensional segmentations of the at least one object, the two-dimensional segmentations corresponding to the two-dimensional images; generates a three-dimensional representation of the scene using the images; determines a pose estimation for each of the two-dimensional images; generates a plurality of two-dimensional representations of the three-dimensional representation using the pose estimations, each two-dimensional representation corresponding to a respective two-dimensional image; generates a mapping using the two-dimensional representations, the mapping being indicative of a correspondence between the images and the representation; and uses the mapping to map the plurality of segmentations to the three-dimensional representation, to thereby recognize the at least one object in the scene; and wherein each of the two-dimensional segmentations includes of a probability that each pixel in the corresponding two-dimensional image is indicative of the object and wherein the scene includes at least two objects, and the two-dimensional segmentation at least two probabilities per pixel, each probability corresponding to a respective object. ### Claim 19 An object recognition system for recognizing at least one object in a three-dimensional scene, the system including: at least one image capture device for capturing a plurality of two-dimensional images; and, an electronic processing device that: determines a plurality of two-dimensional images of the scene, the images at least partially including the at least one object; determines a plurality of two-dimensional segmentations of the at least one object, the two-dimensional segmentations corresponding to the two-dimensional images; generates a three-dimensional representation of the scene using the images; determines a pose estimation for each of the two-dimensional images; generates a plurality of two-dimensional representations of the three-dimensional representation using the pose estimations, each two-dimensional representation corresponding to a respective two-dimensional image; generates a mapping using the two-dimensional representations, the mapping being indicative of a correspondence between the images and the representation; and uses the mapping to map the plurality of segmentations to the three-dimensional representation, to thereby recognize the at least one object in the scene; and wherein each of the two-dimensional segmentations includes of a probability that each pixel in the corresponding two-dimensional image is indicative of the object and wherein the scene includes at least two objects, and the two-dimensional segmentation at least two probabilities per pixel, each probability corresponding to a respective object.