# US 2025/0148758 A1 · Joint asset and defect detection machine learning model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148758 A1
- Document ID: US20250148758A1
- Status: Published application
- Kind: A1
- Application number: 18/935,234
- Filed: 2024-11-01
- Published: 2025-05-08
- Earliest claimed: 2023-11-06
- Assignees: X Development LLC
- Applicants: X Development LLC
- Inventors: Xin-Jing Wang; Anthony Ha
- Web page: https://patentgrep.com/patent/US20250148758A1

## CPC

- G06V 10/764: using classification, e.g. of video objects
- G06F 18/2431: Multiple classes
- G06T 11/00: Two-dimensional [2D] image generation
- G06V 10/273: {removing elements interfering with the pattern to be recognised}
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V 20/176: {Urban or other man-made structures}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06T 2210/12: Bounding box
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

This disclosure describes a system, method, and computer storage medium for joint asset and defect detection. The approach includes receiving input data including an input image of a utility asset, the input image including one or more objects. Deep neural networks are configured to generate embeddings for classification labels of the one or more objects, each embedding corresponding to a classification label and including a mapping between the classification label and a subset of feature vectors. Defect classifiers are configured to determine a likelihood of an object from the one or more objects in the input image containing a type of defect. Each defect classifier is trained to determine a type of defect based on the embeddings for the one or more classification labels. The approach includes generating an output image that includes bounding boxes for the objects and an annotation corresponding a respective object from the objects.

## Claims

### Claim 1

A method for joint asset and defect detection, the method comprising:
receiving input data comprising an input image of a utility asset, the input image comprising one or more objects;
generating, by one or more deep neural networks, embeddings for one or more classification labels of the one or more objects in the input images, each embedding corresponding to a classification label and comprising a mapping between the classification label and a subset of feature vectors;
determining, by a plurality of defect classifiers, a likelihood of an object from the one or more objects in the input image containing a type of defect, wherein each defect classifier from the plurality of defect classifiers is trained to determine a type of defect based on the embeddings for the one or more classification labels; and
generating an output image comprising a plurality of bounding boxes for the one or more objects in the input image, and an annotation corresponding a respective object from the one or more objects in the input image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising generating, by one or more deep neural networks, the plurality of bounding boxes for the one or more objects in the input image, wherein each bounding box in the plurality of bounding boxes corresponds to an object from the one or more objects in the input image for the utility asset.

### Claim 3 (depends on 1)

The method of claim 1, further comprising generating, by one or more deep neural networks, asset label data for the one or more objects in the input image, wherein the asset label data comprises the one or more classification labels corresponding to the one or more objects in the input image, each classification label representing a type of utility asset.

### Claim 4 (depends on 1)

The method of claim 1, further comprising generating, by one or more deep neural networks, asset feature data for one or more objects in the input image, wherein the asset feature data comprises a plurality of feature vectors, the feature vectors representing features of the one or more objects in the input image, and wherein the subset of feature vectors comprises at least one of the plurality of feature vectors.

### Claim 5 (depends on 1)

The method of claim 1, wherein the corresponding bounding box for an object indicates a position of the respective object in the output image.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
generating, for input images of one or more utility assets, output images corresponding to the input images, wherein each output image from the output images comprises (i) bounding boxes, and (ii) annotations, for objects in the respective input image, and
generating, using the output images, a model representation of an electric grid comprising the one or more utility assets from the input images.

### Claim 7 (depends on 1)

The method of claim 1, wherein the annotation comprises a classification label and a likelihood associated with the classification label, the classification label indicating an asset type and defect status for the respective object, and the likelihood associated with the classification label represents a probability of the respective object in the input image matching the asset type and the defect status.

### Claim 8

A method for training a joint asset and defect detection model, the method comprising:
obtaining a plurality of training examples, wherein each training example in the plurality of training examples comprises (i) feature data from an image of a utility asset, (ii) an annotation, and (iii) classifier weightings corresponding to the annotation for the utility asset in the image, wherein the annotation comprises a label indicating a defect status of (i) defective, or (ii) non-defective;
generating a plurality of groupings for the plurality of training examples, wherein the plurality of training examples are divided among the plurality of groupings based on a count of unique training examples among the plurality of training examples, each grouping including training examples that share a count within a threshold value;
applying, to each grouping in the plurality of groupings, an activation function to the classifier weightings corresponding to the annotation for the training examples of the grouping;
generating, for each grouping in the plurality of groupings, a first additional class representing a subset of training examples that are not included in the grouping;
generating a first additional grouping, wherein the first additional grouping is an empty grouping;
sampling, for each grouping in the plurality of groupings, the feature data for training examples in the same grouping, wherein the sampled feature data is configured to be stored in the first additional class of the grouping;
generating, by a plurality of defect classifiers and based on the sampled feature data from the first additional classes of the plurality of groupings, a predicted annotation; and
updating one or more weights of at least one defect classifier in the plurality of defect classifiers based on a comparison of the predicted annotation and the annotation for a training example.

### Claim 9 (depends on 8)

The method of claim 8, wherein the annotation indicates a type and defect status of the utility asset in the image and the predicted annotation indicates a predicted type and a predicted defect status of the utility asset in the image.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
up-sampling, for each grouping in the plurality of groupings, a subset of training examples, each training example in the subset comprising a defect status with a label of defective, wherein up-sampling the grouping comprises sampling corresponding feature data of the subset of training examples at least one additional time when training the plurality of defect classifiers.

### Claim 11 (depends on 8)

The method of claim 8, further comprising:
normalizing, for each grouping in the plurality of groupings, the classifier weightings for training examples with a number of counts exceeding a threshold value in the grouping.

### Claim 12 (depends on 8)

The method of claim 8, wherein the utility asset is at least one of (i) a utility pole, (ii) a transformer, (iii) one or more wires, or (iv) other types of electrical grid distribution equipment.

### Claim 13 (depends on 8)

The method of claim 8, wherein the plurality of defect classifiers are configured to determine, based on embeddings generated for one or more classification labels of one or more objects, an updated bounding box for an output image,
wherein the updated bounding box has a higher likelihood of identifying an object in an input image than the respective bounding box for the object from the plurality of bounding boxes.

### Claim 14 (depends on 8)

The method of claim 8, wherein generating the predicted annotation for one or more objects in an input image comprises providing a training example to a model configured to perform asset-defect detection, wherein the training example comprises (i) a classification label for a respective object of the one or more objects indicating a type of utility asset and (ii) an annotation for the respective object of the one or more objects indicating a defect status of the utility asset.

### Claim 15

A system for joint asset and defect detection, the system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving input data comprising an input image of a utility asset, the input image comprising one or more objects;
generating, by one or more deep neural networks, embeddings for one or more classification labels of the one or more objects in the input images, each embedding corresponding to a classification label and comprising a mapping between the classification label and a subset of feature vectors;
determining, by a plurality of defect classifiers, a likelihood of an object from the one or more objects in the input image containing a type of defect, wherein each defect classifier from the plurality of defect classifiers is trained to determine a type of defect based on the embeddings for the one or more classification labels; and
generating an output image comprising the plurality of bounding boxes for the one or more objects in the input image, and an annotation corresponding a respective object from the one or more objects in the input image.

### Claim 16 (depends on 15)

The system of claim 15, the operations further comprising:
obtaining a plurality of training examples, wherein each training example in the plurality of training examples comprises (i) feature data from an image of a utility asset, (ii) an annotation, and (iii) classifier weightings corresponding to the annotation for the utility asset in the image, wherein the annotation comprises a label indicating a defect status of (i) defective, or (ii) non-defective;
generating a plurality of groupings for the plurality of training examples, wherein the plurality of training examples are divided among the plurality of groupings based on a count of unique training examples among the plurality of training examples, each grouping including training examples that share a count within a threshold value;
applying, to each grouping in the plurality of groupings, an activation function to the classifier weightings corresponding to the annotation for the training examples of the grouping;
generating, for each grouping in the plurality of groupings, a first additional class representing a subset of training examples that are not included in the grouping;
generating a first additional grouping, wherein the first additional grouping is an empty grouping;
sampling, for each grouping in the plurality of groupings, the feature data for training examples in the same grouping, wherein the sampled feature data is configured to be stored in the first additional class of the grouping;
generating, by a plurality of defect classifiers and based on the sampled feature data from the first additional classes of the plurality of groupings, a predicted annotation; and
updating one or more weights of at least one defect classifier in the plurality of defect classifiers based on a comparison of the predicted annotation and the annotation for a training example.

### Claim 17 (depends on 15)

The system of claim 15, wherein the operations further comprise:
generating, by one or more deep neural networks, the plurality of bounding boxes for the one or more objects in the input image, wherein each bounding box in the plurality of bounding boxes corresponds to an object from the one or more objects in the input image for the utility asset.

### Claim 18 (depends on 15)

The system of claim 15, wherein the operations further comprise:
generating, by one or more deep neural networks, asset label data for the one or more objects in the input image, wherein the asset label data comprises the one or more classification labels corresponding to the one or more objects in the input image, each classification label representing a type of utility asset.

### Claim 19 (depends on 15)

The system of claim 15, wherein the operations further comprise:
generating, by one or more deep neural networks, asset feature data for one or more objects in the input image, wherein the asset feature data comprises a plurality of feature vectors, the feature vectors representing features of the one or more objects in the input image, and wherein the subset of feature vectors comprises at least one of the plurality of feature vectors.

### Claim 20 (depends on 15)

The system of claim 15, wherein the annotation comprises a classification label and a likelihood associated with the classification label, the classification label indicating an asset type and defect status for the respective object, and the likelihood associated with the classification label represents a probability of the respective object in the input image matching the asset type and the defect status.
