# US 10,482,609 B2 · Optical flow determination system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,609 B2 - Document ID: US10482609B2 - Status: Granted - Kind: B2 - Application number: 15/478,784 - Filed: 2017-04-04 - Published: 2018-10-04 - Granted: 2019-11-19 - Earliest claimed: 2017-04-04 - Assignees: General Electric Company - Applicants: General Electric Company - Inventors: Ser Nam Lim; Mustafa Devrim Kaba; Mustafa Uzunbas; David Diwinsky - Web page: https://patentgrep.com/patent/US10482609B2 ## CPC - G06T 7/215: Motion-based segmentation - G06N 3/045: Combinations of networks - G06N 3/047: Probabilistic or stochastic networks - G06N 3/0475: Generative networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/094: Adversarial learning - G06T 7/277: involving stochastic approaches, e.g. using Kalman filters - G06T 2207/20084: Artificial neural networks [ANN] - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A generative adversarial network (GAN) system includes a generator sub-network configured to examine images of an object moving relative to a viewer of the object. The generator sub-network also is configured to generate one or more distribution-based images based on the images that were examined. The system also includes a discriminator sub-network configured to examine the one or more distribution-based images to determine whether the one or more distribution-based images accurately represent the object. A predicted optical flow of the object is represented by relative movement of the object as shown in the one or more distribution-based images. ## Claims ### Claim 1 A generative adversarial network (GAN) system comprising: a generator sub-network configured to examine camera-based images of interior surfaces of equipment that are moving relative to a camera that generated the images, the generator sub-network also configured to generate one or more distribution-based images based on the camera-based images that were examined; a discriminator sub-network configured to examine the one or more distribution-based images to determine whether the one or more distribution-based images accurately represent the interior surfaces of the equipment, wherein a predicted optical flow of the interior surfaces is represented by relative movement of the interior surfaces as shown in the one or more distribution-based images; and one or more processors configured to generate a three-dimensional (3D) point cloud using the predicted optical flow of the interior surfaces, the 3D point cloud depicting an interior of the equipment. ### Claim 2 (depends on 1) The system of claim 1, wherein the discriminator sub-network is configured to determine one or more loss functions indicative of errors in the predicted optical flow of the interior surfaces in the one or more distribution-based images. ### Claim 3 (depends on 2) The system of claim 2, wherein the generator sub-network is configured to be trained using the one or more loss functions. ### Claim 4 (depends on 3) The system of claim 3, wherein the generator sub-network is configured to be trained using the one or more loss functions by alternating back-propagating the one or more loss functions through the generator sub-network and back-propagating the one or more loss functions through the discriminator sub-network, wherein coefficients of the one or more loss functions are kept constant during back-propagating of the one or more loss functions through the discriminator sub-network. ### Claim 5 (depends on 3) The system of claim 3, wherein the generator sub-network is configured to generate one or more additional distribution-based images to represent the predicted optical flow of the interior surfaces of the equipment. ### Claim 6 (depends on 1) The system of claim 1, wherein the generator sub-network is configured to generate the one or more distribution-based images based on one or more distributions of pixel characteristics in the camera-based images that were examined by the generator sub-network. ### Claim 7 (depends on 1) The system of claim 1, wherein the camera-based images are obtained from the camera that is inserted into the equipment. ### Claim 8 A method comprising: examining camera-based images of interior surfaces of equipment that are moving relative to a camera that generated the camera-based images using a generator sub-network of a generative adversarial network (GAN) system; generating one or more distribution-based images using the generator sub-network based on the camera-based images that were examined by the generator sub-network; examining the one or more distribution-based images using a discriminator sub-network of the GAN system to determine whether the one or more distribution-based images accurately represent the interior surfaces of the equipment; responsive to determining that the one or more distribution-based images accurately represent the interior surfaces of the equipment using the discriminator sub-network, determining a predicted optical flow of the interior surfaces of the equipment abased on the one or more distribution-based images; and generating a three-dimensional (3D) point cloud using the predicted optical flow of the interior surfaces, the 3D point cloud depicting an interior of the equipment. ### Claim 9 (depends on 8) The method of claim 8, further comprising determining one or more loss functions indicative of errors in the predicted optical flow of the interior surfaces of the equipment in the one or more distribution-based images using the discriminator sub-network. ### Claim 10 (depends on 9) The method of claim 9, further comprising training the generator sub-network using the one or more loss functions. ### Claim 11 (depends on 10) The method of claim 10, wherein training the generator sub-network using the one or more loss functions includes alternating between back-propagating the one or more loss functions through the generator sub-network and back-propagating the one or more loss functions through the discriminator sub-network, wherein coefficients of the one or more loss functions are kept constant during back-propagating of the one or more loss functions through the discriminator sub-network. ### Claim 12 (depends on 9) The method of claim 9, further comprising generating one or more additional distribution-based images using the generator sub-network to represent the predicted optical flow of the interior surfaces of the equipment. ### Claim 13 (depends on 8) The method of claim 8, wherein the one or more distribution-based images are generated based on one or more distributions of pixel characteristics in the camera-based images that were examined by the generator sub-network. ### Claim 14 (depends on 8) The method of claim 8, further comprising: obtaining the camera-based images by inserting the camera into the interior of the equipment. ### Claim 15 A system comprising: a generator sub-network configured to examine camera-based images of a moving object, the generator sub-network also configured to generate one or more distribution-based images based on distributions of pixel characteristics in the camera-based images; a discriminator sub-network configured to examine the one or more distribution-based images to determine one or more errors between the one or more distribution-based images and the camera-based images of the moving object, the one or more errors indicative of whether the one or more distribution-based images accurately represent the moving object, wherein a predicted optical flow of the moving object is represented by relative movement of the object as shown in the one or more distribution-based images; and one or more processors configured to generate a three-dimensional (3D) point cloud representative of the moving object, the 3D point cloud generated using the predicted optical flow of the moving object. ### Claim 16 (depends on 15) The system of claim 15, wherein the generator sub-network is configured to be trained using the one or more errors. ### Claim 17 (depends on 16) The system of claim 16, wherein the generator sub-network is configured to generate one or more additional distribution-based images to represent the predicted optical flow of the moving object following training of the generator sub-network using the one or more errors. ### Claim 18 (depends on 15) The system of claim 15, wherein the one or more processors also are configured to change movement of a vehicle responsive to determining the predicted optical flow of the moving object.