# US 2021/0279811 A1 · Capacity optimized electronic model based prediction of changing physical hazards and inventory items

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279811 A1
- Document ID: US20210279811A1
- Status: Published application
- Kind: A1
- Application number: 17/193,957
- Filed: 2021-03-05
- Published: 2021-09-09
- Earliest claimed: 2020-03-06
- Assignees: Yembo, Inc.
- Applicants: Yembo, Inc.
- Inventors: Devin WALTMAN; Siddharth MOHAN; Anoop JAKKA; Marc EDER; Maciej HALBER; Zachary RATTNER
- Web page: https://patentgrep.com/patent/US20210279811A1

## CPC

- G06Q 30/0205: {based on location or geographical consideration}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 16/29: Geographical information databases
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06Q 10/06395: {Quality analysis or management}
- G06Q 10/0875: Itemisation or classification of parts, supplies or services, e.g. bill of materials
- G06Q 30/0206: {Price or cost determination based on market factors}
- G06Q 40/08: Insurance
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/50: Depth or shape recovery
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/00: Scenes; Scene-specific elements (control of digital cameras H04N23/60)
- G06V 20/64: Three-dimensional [3D] objects
- H04N 23/631: {Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters}
- H04N 23/80: Camera processing pipelines; Components thereof
- H04N 23/90: Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
- G01S 17/06: Systems determining position data of a target
- G01S 17/89: for mapping or imaging
- G01S 19/42: Determining position
- G06F 18/24: Classification techniques
- G06Q 50/16: Real estate
- G06T 2207/20081: Training; Learning
- G06T 2219/004: Annotating, labelling
- Y02A 10/40: Controlling or monitoring, e.g. of flood or hurricane; Forecasting, e.g. risk assessment or mapping
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- Y02A: TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE

## Abstract

The process of obtaining an insurance quote is currently based on information provided by filing online forms. Insurance companies use the consumer input data from online the forms to provide a quote for property insurance and for insurance underwriting. To complete the underwriting, the insurance companies have to make assumptions on various factors; for example, the structure and layout of a consumer's home, what type of content the consumer owns, its value, and its condition, among several others. It is desirable to get more accurate data on the consumer content, the structure and layout of the home, as well as other factors (e.g., hazards such as whether a house is on a canyon) in an automated fashion which would let insurance companies underwrite policies with a more accurate risk profile as well as save costs by not having to send an onsite inspector.

## Claims

### Claim 1

A non-transitory computer readable medium having instructions thereon, the instructions configured to cause a computer to execute a multi-stage electronic model for predicting physical hazards at a location, the physical hazards changing over time and from location to location, each stage of the multi-stage electronic model being optimized based on a required computing capability and a maximum allowable processing time for a given stage to minimize overall computing resources and processing time for the multi-stage electronic model, the instructions causing operations comprising:
receiving description data of the location, the description data generated via at least one of a camera, a user interface, an environment sensor, and an external location information database;
predicting, with a first stage of the multi-stage electronic model, a potential hazard type based on the received description data, the potential hazard type comprising one potential hazard type of a set of predetermined potential hazard types, the first stage having a first configuration optimized for potential hazard type predictions based on the received description data; and
predicting, with a second stage of the multi-stage electronic model, based on the predicted potential hazard type and the received description data, a physical hazard at the location, the second stage having a second configuration that is different from the first configuration, the second configuration associated with the predicted hazard type without regard for other potential hazard types, the second configuration optimized for predicting the physical hazard at the location once the hazard type is predicted by the first stage.

### Claim 2 (depends on 1)

The medium of claim 1, wherein predicting the potential hazard type comprises identifying objects and structures at the location by one or more of: object detection, semantic segmentation, instance segmentation, and panoptic segmentation, object tracking through multiple images, feature-matching, optical flow estimation, relative camera pose estimation, multi-view projection, non-maximum suppression, class-based suppression, and heuristic suppression.

### Claim 3 (depends on 1)

The medium of claim 1, wherein predicting the physical hazard comprises 3-dimensional reconstruction of the location using a Structure from Motion (SfM) algorithm, Multi-View Stereo (MVS), Simultaneous Localization and Mapping (SLAM), and/or depth estimation algorithms.

### Claim 4 (depends on 1)

The medium of claim 1, wherein the description data includes interior and/or exterior condition information about the location, wherein the interior and/or exterior condition information comprises a description of a geographical and/or spatial position of the location relative to potential natural hazards; and wherein the description of the geographical and/or spatial position of the location relative to potential natural hazards is determined at least in part based on information from an external location information database.

### Claim 5 (depends on 1)

The medium of claim 1, wherein the first and second stages of the multi-stage electronic model are trained with first and second training data, the first and second training data being different, the first training data comprising input-output training pairs associated with each potential hazard type in the set of predetermined potential hazard types, the second training data comprising input-output training pairs associated with only the predicted hazard type.

### Claim 6

A system configured for generating an inspection report utilizing a machine learning model, the system comprising one or more hardware processors configured by machine-readable instructions to:
receive description data of a location, the description data generated via at least one of a camera, a user interface, an environment sensor, and an external location information database;
generate an inventory list comprising a plurality of items at the location, utilizing the machine learning model, based on the description data, the inventory list further comprising inventory attributes associated with the plurality of items related to a context of the plurality of items at the location;
determine, utilizing the machine learning model, based on the description data, interior and/or exterior condition information for the location, the interior and/or exterior condition information describing a quality of structural components of the location and/or hazards in and/or around the location; and
generate, utilizing the machine learning model, based on the inventory list and the interior and/or exterior condition information, the inspection report for the location.

### Claim 7 (depends on 6)

The system of claim 6, wherein the description data comprises one or more media types, the one or more media types comprising at least one or more of video data, image data, audio data, text data, user interface/display data, and/or sensor data, and wherein receiving description data comprises receiving sensor data from one or more environment sensors, the one or more environment sensors comprising at least one of a GPS, an accelerometer, a gyroscope, a barometer, a microphone, Lidar, or a depth sensor.

### Claim 8 (depends on 6)

The system of claim 6, wherein generating the inspection report comprises determining the one or more inventory attributes, the inventory attributes comprising one or more of locations of the items at the location, a quality of items at the location, brand names of the items at the location, materials of the items at the location, damage to items at the location, age of the items at the location, condition of the items at the location, dimensions of the items at the location, or values of the items at the location.

### Claim 9 (depends on 6)

The system of claim 6, wherein the description data is captured by a mobile computing device associated with a user and transmitted to the one or more processors with or without user interaction.

### Claim 10 (depends on 6)

The system of claim 6, wherein generating the inspection report comprises causing the machine learning model to determine one or more insurance underwriting estimates using at least one or more inventory attributes, the quality of structural components of the location, and the hazards in and/or around the location as inputs for the machine learning model.

### Claim 11 (depends on 6)

The system of claim 6, wherein the one or more hardware processors are further configured to receive adjustments to the inventory list and/or the interior and/or exterior condition information, and use the adjustments and corresponding inventory attributes, quality of structural components of the location, and/or hazards in and/or around the location, as input to (1) update the inspection report, and/or (2) retrain the machine learning model.

### Claim 12 (depends on 6)

The system of claim 6, wherein the description data is time stamped, geo stamped, and/or user stamped.

### Claim 13 (depends on 6)

The system of claim 6, wherein the interior and/or exterior condition information comprises a description of a spatial position of the location relative to potential natural hazards.

### Claim 14 (depends on 13)

The system of claim 13, wherein the description of the spatial position of the location relative to potential natural hazards is determined at least in part based on information from the external location information database.

### Claim 15 (depends on 6)

The system of claim 6, wherein the inspection report is determined at least in part based on information from an external market information database.

### Claim 16

A method for generating an inspection report utilizing a machine learning model, the method comprising:
receiving description data of a location, the description data generated via at least one of a camera, a user interface, an environment sensor, and an external location information database;
generating an inventory list comprising a plurality of items at the location, utilizing the machine learning model, based on the description data, the inventory list further comprising inventory attributes associated with the plurality of items related to a context of the plurality of items at the location;
determining, utilizing the machine learning model, based on the description data, interior and/or exterior condition information for the location, the interior and/or exterior condition information describing a quality of structural components of the location and/or hazards in and/or around the location; and
generating, utilizing the machine learning model, based on the inventory list and the interior and/or exterior condition information, the inspection report for the location.

### Claim 17 (depends on 16)

The method of claim 16, wherein the description data comprises one or more media types, the one or more media types comprising at least one or more of video data, image data, audio data, text data, user interface/display data, and/or sensor data, and wherein receiving description data comprises receiving sensor data from one or more environment sensors, the one or more environment sensors comprising at least one of a GPS, an accelerometer, a gyroscope, a barometer, a microphone, Lidar, or a depth sensor.

### Claim 18 (depends on 16)

The method of claim 16, wherein generating the inspection report comprises determining the inventory attributes, the inventory attributes comprising one or more of locations of the items at the location, a quality of items at the location, brand names of the items at the location, materials of the items at the location, damage to items at the location, age of the items at the location, condition of the items at the location, dimensions of the items at the location, or values of the items at the location.

### Claim 19 (depends on 16)

The method of claim 16, wherein description data is captured by a mobile computing device associated with a user and transmitted to one or more processors configured to control the machine learning model with or without user interaction.

### Claim 20 (depends on 16)

The method of claim 16, wherein generating the inspection report comprises causing the machine learning model to determine one or more insurance underwriting estimates using at least one or more inventory attributes, the quality of structural components of the location, and the hazards in and/or around the location as inputs for the machine learning model.

### Claim 21 (depends on 16)

The method of claim 16, further comprising receiving adjustments to the inventory list and/or the interior and/or exterior condition information, and using the adjustments and corresponding inventory attributes, quality of structural components of the location, and/or hazards in and/or around the location, as input to (1) update the inspection report, and/or (2) retrain the machine learning model.

### Claim 22 (depends on 16)

The method of claim 16, wherein the description data is time stamped, geo stamped, and/or user stamped.

### Claim 23 (depends on 16)

The method of claim 16, wherein the interior and/or exterior condition information comprises a description of a spatial position of the location relative to potential natural hazards.

### Claim 24 (depends on 23)

The method of claim 23, wherein the description of the spatial position of the location relative to potential natural hazards is determined at least in part based on information from the external location information database.

### Claim 25 (depends on 16)

The method of claim 16, wherein the inspection report is determined at least in part based on information from an external market information database.