# US 2021/0279852 A1 · Identifying flood damage to an indoor environment using a virtual representation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279852 A1
- Document ID: US20210279852A1
- Status: Published application
- Kind: A1
- Application number: 17/193,779
- Filed: 2021-03-05
- Published: 2021-09-09
- Earliest claimed: 2020-03-06
- Assignees: Yembo, Inc.
- Applicants: Yembo, Inc.
- Inventors: Anoop JAKKA; Siddharth MOHAN; Devin WALTMAN; Marc EDER; Maciej HALBER; Zachary RATTNER
- Web page: https://patentgrep.com/patent/US20210279852A1

## CPC

- G06Q 30/0205: {based on location or geographical consideration}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 16/29: Geographical information databases
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06Q 10/06395: {Quality analysis or management}
- G06Q 10/0875: Itemisation or classification of parts, supplies or services, e.g. bill of materials
- G06Q 30/0206: {Price or cost determination based on market factors}
- G06Q 40/08: Insurance
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/50: Depth or shape recovery
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V 20/64: Three-dimensional [3D] objects
- H04N 23/631: {Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters}
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G01S 17/06: Systems determining position data of a target
- G01S 17/89: for mapping or imaging
- G01S 19/42: Determining position
- G06F 18/24: Classification techniques
- G06Q 50/16: Real estate
- G06T 2207/20081: Training; Learning
- G06T 2219/004: Annotating, labelling
- Y02A 10/40: Controlling or monitoring, e.g. of flood or hurricane; Forecasting, e.g. risk assessment or mapping
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- Y02A: TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE

## Abstract

Disclosed is a method for using a virtual representation of an indoor environment to identify contents that have been damaged (e.g., by flooding). A virtual representation of a physical scene of an indoor environment is processed to identify a list of contents in the physical scene. The virtual representation may include 2-dimensional representations of the physical scene (e.g., images or video) or a 3-dimensional representation of the physical scene (e.g., 3D digital model). A reference line is determined in the virtual representation that is indicative of a maximum vertical extent of the damage in the physical scene. The position of the reference line is compared with the position of the identified contents in the virtual representation to determine contents that are likely to be damaged. For example, the contents that are at or below a plane represented by the reference line in the virtual representation may be identified as damaged.

## Claims

### Claim 1

A method for identifying damage in an indoor environment by analyzing a virtual model representative of the indoor environment, the method comprising:
obtaining a virtual representation of a physical scene in an indoor environment;
extracting data items from the virtual representation, wherein the data items correspond to contents in the physical scene;
determining a reference line in the virtual representation, the reference line indicative of a vertical extent of the damage in the physical scene; and
determining content damage information based on the reference line, the content damage information indicative of the contents damaged in the physical scene.

### Claim 2 (depends on 1)

The method of claim 1, wherein obtaining the virtual representation includes:
obtaining a collection of images of the physical scene.

### Claim 3 (depends on 1)

The method of claim 1, wherein obtaining the virtual representation includes:
generating a three-dimensional (3D) digital model of the physical scene.

### Claim 4 (depends on 3)

The method of claim 3, wherein generating the 3D digital model includes:
generating the 3D digital model based on depth data associated with the physical scene.

### Claim 5 (depends on 3)

The method of claim 3, wherein generating the 3D digital model includes:
generating the 3D digital model based on a collection of images or video data of the physical scene.

### Claim 6 (depends on 1)

The method of claim 1, wherein extracting the data items includes:
providing the virtual representation as an input to a first machine learning model to identify the data items, wherein the first machine learning model is trained to identify objects and structures in multiple physical scenes as the data items.

### Claim 7 (depends on 6)

The method of claim 6, wherein the first machine learning model is trained by:
obtaining physical scene data associated with a specified physical scene of the indoor environment, wherein the physical scene data includes an image, a video or a 3D digital model associated with the specified physical scene; and
training the first machine learning model with the physical scene data to predict a specified set of contents in the specified physical scene such that a cost function that is indicative of a difference between a reference set of contents and the specified set of contents is minimized.

### Claim 8 (depends on 6)

The method of claim 6, wherein the first machine learning model is configured to predict spatial localization data of the data items, the spatial localization data corresponding to location information of the contents in the physical scene.

### Claim 9 (depends on 1)

The method of claim 1, wherein the reference line indicates the vertical extent of flood water in the physical scene.

### Claim 10 (depends on 1)

The method of claim 1, wherein the reference line corresponds to a plane in the physical scene that separates a first set of contents damaged from a second set of contents not damaged in the physical scene.

### Claim 11 (depends on 1)

The method of claim 1, wherein determining the reference line includes:
providing the virtual representation as an input to a second machine learning model to obtain the reference line, wherein the second machine learning model is trained to identify reference lines in multiple physical scenes.

### Claim 12 (depends on 11)

The method of claim 11, wherein the second machine learning model is trained by:
obtaining physical scene data associated with a specified physical scene of a specified indoor environment, wherein the physical scene data includes an image, a video or a 3D digital model associated with the specified physical scene; and
training the second machine learning model with the physical scene data to predict a specified reference line in the specified physical scene such that a cost function that is indicative of a difference between an actual reference line and the predicted specified reference line is minimized.

### Claim 13 (depends on 1)

The method of claim 1, wherein determining the content damage information based on the reference line includes:
determining the content damage information based on a height of the reference line in the virtual representation.

### Claim 14 (depends on 13)

The method of claim 13, wherein determining the content damage information includes:
identifying a set of contents that is below a plane of the reference line as the contents damaged in the physical scene.

### Claim 15 (depends on 14)

The method of claim 14, wherein the set of contents correspond to contents in a volume of the physical scene beneath the reference line in the virtual representation.

### Claim 16 (depends on 1)

The method of claim 1 further comprising:
processing the virtual representation based on the content damage information to determine measurement information associated with the damage, the measurement information indicative of an extent of the damage.

### Claim 17 (depends on 16)

The method of claim 16 further comprising:
computing a cost associated with the damage based on the measurement information and the content damage information.

### Claim 18

A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause operations comprising:
obtaining a virtual representation of a physical scene in an indoor environment;
extracting data items from the virtual representation, wherein the data items correspond to contents in the physical scene;
determining a reference line in the virtual representation, the reference line indicative of a vertical extent of a damage in the physical scene; and
determining content damage information based on the reference line, the content damage information indicative of the contents damaged in the physical scene.

### Claim 19 (depends on 18)

The computer-readable medium of claim 18, wherein obtaining the virtual representation includes:
obtaining a collection of images of the physical scene or generating a 3D digital model of the physical scene.

### Claim 20 (depends on 18)

The computer-readable medium of claim 18, wherein extracting the data items includes:
providing the virtual representation as an input to a first machine learning model to identify the data items, wherein the first machine learning model is trained to identify objects and structures in multiple physical scenes as the data items.

### Claim 21 (depends on 18)

The computer-readable medium of claim 18, wherein the reference line indicates the vertical extent of flood water in the physical scene.

### Claim 22 (depends on 18)

The computer-readable medium of claim 18, wherein determining the reference line includes:
providing the virtual representation as an input to a second machine learning model to obtain the reference line, wherein the second machine learning model is trained to identify reference lines in multiple physical scenes.

### Claim 23 (depends on 18)

The computer-readable medium of claim 18, wherein determining the content damage information based on the reference line includes:
determining the content damage information based on a height of the reference line in the virtual representation.

### Claim 24

A system comprising:
a computer system comprising one or more processors programmed with computer program instructions that, when executed, cause operations comprising:
obtaining a virtual representation of a physical scene in an indoor environment;
extracting data items from the virtual representation, wherein the data items correspond to contents in the physical scene;
determining a reference line in the virtual representation, the reference line indicative of a vertical extent of a damage in the physical scene; and
determining content damage information based on the reference line, the content damage information indicative of the contents damaged in the physical scene.

### Claim 25 (depends on 24)

The system of claim 24, wherein obtaining the virtual representation includes:
obtaining a collection of images of the physical scene or generating a 3D digital model of the physical scene.

### Claim 26 (depends on 24)

The system of claim 24, wherein the reference line indicates the vertical extent of flood water in the physical scene.

### Claim 27 (depends on 24)

The system of claim 24, wherein determining the content damage information based on the reference line includes:
determining the content damage information based on a height of the reference line in the virtual representation.