# US 12,223,670 B2 · Using deep learning and structure-from-motion techniques to generate 3D point clouds from 2D data > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,670 B2 - Document ID: US12223670B2 - Status: Granted - Kind: B2 - Application number: 18/355,336 - Filed: 2023-07-19 - Published: 2023-11-09 - Granted: 2025-02-11 - Earliest claimed: 2020-02-11 - Assignees: State Farm Mutual Automobile Insurance Company - Applicants: State Farm Mutual Automobile Insurance Company - Inventors: Ryan Knuffman; Jeremy Carnahan - Web page: https://patentgrep.com/patent/US12223670B2 ## CPC - G06T 7/579: from motion - G06T 15/205: {Image-based rendering} - G06T 2207/10032: Satellite or aerial image; Remote sensing - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06T 2207/30184: Infrastructure - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A server includes a processor and a memory storing instructions that, when executed by the processor, cause the server to receive two-dimensional (2D) images, analyze the images using a trained deep network to generate points, process the labeled points to identify tie points, and combine the 2D dimensional images into a three-dimensional (3D) point cloud using structure-from-motion. A method for generating a semantically-segmented 3D point cloud from 2D data includes receiving 2D images, analyzing the images using a trained deep network to generate labeled points, processing the points to identify tie points, and combining the 2D images into a 3D point cloud using structure-from-motion. A non-transitory computer readable storage medium stores executable instructions that, when executed by a processor, cause a computer to receive 2D images, analyze the images using a trained deep network to generate labeled points, process the points to identify and combine tie points using structure-from-motion. ## Claims ### Claim 1 A computer system comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the computer system to: receive, via one or more processors, at least one two-dimensional image corresponding to an outdoor scene, the outdoor scene including one or more outdoor objects; and analyze, via one or more processors, the at least one two-dimensional image using a trained deep artificial neural network to generate a respective set of one or more labeled points, each of the one or more labeled points corresponding to a respective class label describing at least one of outdoor objects depicted in the two-dimensional image, wherein the respective class label indicates a type of outdoor object corresponding to the labeled point. ### Claim 2 (depends on 1) The computer system of claim 1, wherein the two-dimensional image corresponding to the outdoor scene is captured via a drone capture device. ### Claim 3 (depends on 1) The computer system of claim 1, the memory storing further instructions that, when executed by the one or more processors, cause the system to train the deep artificial neural network using a plurality of manually labeled training images. ### Claim 4 (depends on 1) The computer system of claim 1, the memory storing further instructions that, when executed by the one or more processors, cause the system to solve a system of equations. ### Claim 5 (depends on 1) The computer system of claim 1, the memory storing further instructions that, when executed by the one or more processors, cause the system to store the trained deep artificial neural network in an electronic storage device. ### Claim 6 (depends on 1) The computer system of claim 1, the memory storing further instructions that, when executed by the one or more processors, cause the system to insert the set of labeled points into a three-dimensional point cloud according to a plurality voting algorithm. ### Claim 7 (depends on 1) The computer system of claim 1, wherein each respective set of one or more labeled points are stored in a matrix. ### Claim 8 A computer-implemented method comprising: receiving at least one two-dimensional image corresponding to an outdoor scene, the outdoor scene including one or more outdoor objects; and analyzing the at least one two-dimensional image using a trained deep artificial neural network to generate a respective set of one or more labeled points, each of the one or more labeled points corresponding to a respective class label describing at least one of outdoor objects depicted in the two-dimensional image, wherein the respective class label indicates a type of outdoor object corresponding to the labeled point. ### Claim 9 (depends on 8) The computer-implemented method of claim 8, wherein the two-dimensional image corresponding to the outdoor scene is captured via a drone capture device. ### Claim 10 (depends on 8) The computer-implemented method of claim 8, further comprising: training the deep artificial neural network using a plurality of manually labeled training images. ### Claim 11 (depends on 8) The computer-implemented method of claim 8, further comprising: storing the trained deep artificial neural network in an electronic storage device. ### Claim 12 (depends on 8) The computer-implemented method of claim 8, further comprising: processing the set of labeled points to identify one or more tie points. ### Claim 13 (depends on 8) The computer-implemented method of claim 8, further comprising: inserting the set of labeled points into a three-dimensional point cloud according to a plurality voting algorithm. ### Claim 14 (depends on 8) The computer-implemented method of claim 8, wherein each respective set of one or more labeled points are stored in a matrix. ### Claim 15 A non-transitory computer readable storage medium storing executable instructions that, when executed by a processor, cause a computer to: receive at least one two-dimensional image corresponding to an outdoor scene, the outdoor scene including one or more outdoor objects; and analyze the at least one two-dimensional image using a trained deep artificial neural network to generate a respective set of one or more labeled points, each of the one or more labeled points corresponding to a respective class label describing at least one of outdoor objects depicted in the two-dimensional image, wherein the respective class label indicates a type of outdoor object corresponding to the labeled point. ### Claim 16 (depends on 15) The non-transitory computer readable storage medium of claim 15, storing further executable instructions that, when executed by a processor, cause a computer to: train the deep artificial neural network using a plurality of manually labeled training images. ### Claim 17 (depends on 15) The non-transitory computer readable storage medium of claim 15, storing further executable instructions that, when executed by a processor, cause a computer to: solve a system of equations. ### Claim 18 (depends on 15) The non-transitory computer readable storage medium of claim 15, storing further executable instructions that, when executed by a processor, cause a computer to: store the trained deep artificial neural network in an electronic storage device. ### Claim 19 (depends on 15) The non-transitory computer readable storage medium of claim 15, storing further executable instructions that, when executed by a processor, cause a computer to: insert the set of labeled points into a three-dimensional point cloud according to a plurality voting algorithm. ### Claim 20 (depends on 19) The non-transitory computer readable storage medium of claim 19, storing further executable instructions that, when executed by a processor, cause a computer to: transmit the three-dimensional point cloud to a user device.