# US 2021/0342790 A1 · Visualizing vehicle part replacement information using extended reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342790 A1
- Document ID: US20210342790A1
- Status: Published application
- Kind: A1
- Application number: 16/861,498
- Filed: 2020-04-29
- Published: 2021-11-04
- Earliest claimed: 2020-04-29
- Assignees: Capital One Services, LLC
- Applicants: Capital One Services, LLC
- Inventors: Geoffrey DAGLEY; Jason HOOVER; Micah PRICE; Olalekan AWOYEMI; Qiaochu TANG; Stephen WYLIE; Sunil VASISHT
- Web page: https://patentgrep.com/patent/US20210342790A1

## CPC

- G06Q 10/06395: {Quality analysis or management}
- G06F 16/7837: {using objects detected or recognised in the video content}
- G06F 16/787: using geographical or spatial information, e.g. location
- G06N 20/00: Machine learning
- G06Q 10/083: Shipping
- G06Q 10/0875: Itemisation or classification of parts, supplies or services, e.g. bill of materials
- G06Q 10/1097: {using calendar-based scheduling for task assignment}
- G06Q 10/20: Administration of product repair or maintenance
- G06Q 30/0631: {Recommending goods or services}
- G06Q 30/0633: {Managing shopping lists, e.g. compiling or processing purchase lists (shipping orders G06Q10/083; order filling G06Q10/087)}
- G06T 7/0004: {Industrial image inspection}
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 11/60: Creating or editing images; Combining images with text
- G07C 5/0808: {Diagnosing performance data (testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005)}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G07C 5/0825: {using optical means}
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE

## Abstract

A device having extended reality capabilities receives a video feed including one or more video frames depicting an object. The device provides the one or more video frames to a first machine learning model that identifies the object and one or more parts of the object that are depicted in the one or more video frames, and obtains positional tracking information that represents a position and an orientation associated with the object depicted in the one or more video frames relative to a coordinate space. The device identifies a plurality of parts from the one or more parts of the object that are depicted in the one or more video frames and determines, using one or more second machine learning models, information regarding the plurality of parts and information regarding one or more replacement parts.

## Claims

### Claim 1

A method, comprising:
capturing, by a device having extended reality capabilities, a video feed including one or more video frames depicting an object that is visible in a field of view of the device;
providing, by the device, the one or more video frames to a first machine learning model that identifies the object depicted in the one or more video frames and one or more parts of the object that are depicted in the one or more video frames;
obtaining, by the device, positional tracking information that represents a position and an orientation associated with the object depicted in the one or more video frames relative to a coordinate space that corresponds to the field of view of the device;
identifying, by the device, a part from the one or more parts of the object that are depicted in the one or more video frames;
determining, by the device and using one or more second machine learning models, information regarding the part and information regarding one or more replacement parts,
wherein the one or more second machine learning models have been trained to predict information regarding a historical life cycle of the part and recommend one or more replacement parts that have historically been used to replace the part; and
rendering, by the device, content, relative to the part, using the extended reality capabilities of the device,
wherein the content identifies the information regarding the part and the information regarding the one or more replacement parts,
wherein the information regarding the one or more replacement parts includes, for a replacement part of the one or more replacement parts, an option to automatically purchase the replacement part, an option to automatically schedule delivery of the replacement part, or an option to automatically schedule a service to install the replacement part.

### Claim 2 (depends on 1)

The method of claim 1, wherein the content includes two-dimensional overlays to be placed over images of the part that are depicted in the one or more video frames based on the positional tracking information.

### Claim 3 (depends on 1)

The method of claim 1, wherein the content includes information based on aggregate data relating to one or more quality control issues associated with a group of parts of which the part is a member.

### Claim 4 (depends on 1)

The method of claim 1, wherein the information regarding the part includes a remaining lifespan of the part.

### Claim 5 (depends on 1)

The method of claim 1, wherein the information regarding the one or more replacement parts includes a lifespan of the one or more replacement parts.

### Claim 6 (depends on 1)

The method of claim 1, wherein the information regarding the one or more replacement parts includes a premature failure rate of the one or more replacement parts,
wherein the premature failure rate is based on a comparison of a lifespan of the one or more replacement parts and historical failure data.

### Claim 7 (depends on 1)

The method of claim 1, wherein the content is first content, and
wherein the method further comprises:
identifying a plurality of parts from the one or more parts of the object that are depicted in the one or more video frames;
determining, using the one or more second machine learning models, information regarding the plurality of parts and information regarding one or more replacement parts for the plurality of parts; and
determining a total cost of owning the object based on the information regarding the plurality of parts and the information regarding one or more replacement parts for the plurality of parts; and
rendering second content, relative to the plurality of parts, using the extended reality capabilities of the device,
wherein the second content includes information regarding the total cost of owning the object.

### Claim 8

A device, comprising:
one or more memories; and
one or more processors, communicatively coupled to the one or more memories, configured to:
capture a video feed including one or more video frames depicting a vehicle that is visible in a field of view of the device;
provide the one or more video frames to a first machine learning model that identifies the vehicle depicted in the one or more video frames and one or more parts of the vehicle that are depicted in the one or more video frames;
obtain positional tracking information that represents a position and an orientation associated with the vehicle depicted in the one or more video frames relative to a coordinate space that corresponds to the field of view of the device;
identify a part from the one or more parts of the vehicle that are depicted in the one or more video frames;
determine, using one or more second machine learning models, information regarding the part and information regarding one or more replacement parts,
wherein the one or more second machine learning models have been trained to predict information regarding a historical life cycle of the part and recommend one or more replacement parts that have historically been used to replace the part; and
render content relative to the part,
wherein the content identifies the information regarding the part and the information regarding the one or more replacement parts,
wherein the information regarding the one or more replacement parts includes, for a replacement part of the one or more replacement parts, an option to automatically purchase the replacement part, an option to automatically schedule delivery of the replacement part, or an option to automatically schedule a service to install the replacement part.

### Claim 9 (depends on 8)

The device of claim 8, wherein the content includes information based on aggregate data relating to one or more quality control issues associated with a group of vehicles of which the vehicle is a member.

### Claim 10 (depends on 8)

The device of claim 8, wherein the information regarding the part includes a remaining lifespan of the part.

### Claim 11 (depends on 8)

The device of claim 8, wherein the information regarding the one or more replacement parts includes a premature failure rate of the one or more replacement parts,
wherein the premature failure rate is based on a comparison of a lifespan of the one or more replacement parts and historical failures.

### Claim 12 (depends on 8)

The device of claim 8, wherein the content is first content, and
wherein the one or more processors are further configured to:
render second content relative to the part,
wherein the second content includes instructions for replacing the part.

### Claim 13 (depends on 8)

The device of claim 8, wherein the one or more processors are further configured to:
obtain information regarding the vehicle via a machine-to-machine communication interface between the device and the vehicle.

### Claim 14

A non-transitory computer-readable medium storing instructions, the instructions comprising:
one or more instructions that, when executed by one or more processors, cause the one or more processors to:
receive a video feed including one or more video frames depicting an object that is visible in a field of view;
provide the one or more video frames to a first machine learning model that identifies the object depicted in the one or more video frames and one or more parts of the object that are depicted in the one or more video frames;
obtain positional tracking information that represents a position and an orientation associated with the object depicted in the one or more video frames relative to a coordinate space that corresponds to the field of view;
identify a plurality of parts from the one or more parts of the object that are depicted in the one or more video frames;
determine, using one or more second machine learning models, information regarding the plurality of parts and information regarding one or more replacement parts,
wherein the one or more second machine learning models have been trained to predict information regarding failure information for the plurality of parts and recommend one or more replacement parts to replace the part; and
render content relative to the plurality of parts,
wherein the content identifies the information regarding the plurality of parts and the information regarding the one or more replacement parts,
wherein the information regarding the one or more replacement parts includes, for a replacement part of the one or more replacement parts, an option to automatically purchase the replacement part, an option to automatically schedule delivery of the replacement part, or an option to automatically schedule a service to install the replacement part.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the content includes information based on aggregate data relating to one or more quality control issues associated with a group of objects of which the object is a member.

### Claim 16 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the information regarding the plurality of parts includes, for each part of the plurality of parts, an indication of a remaining lifespan.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein a color of the indication for each part of the plurality of parts varies based on the remaining lifespan of the part with which the indication is associated.

### Claim 18 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the information regarding the one or more replacement parts includes information identifying a lifespan of the one or more replacement parts.

### Claim 19 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the information regarding the one or more replacement parts includes a premature failure rate of the one or more replacement parts,
wherein the premature failure rate is based on a comparison of a lifespan of the one or more replacement parts and historical failure data.

### Claim 20 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the content is first content, and
wherein the one or more instructions, when executed by the one or more processors, further cause the one or more processors to:
determine, using the one or more second machine learning models, information regarding the plurality of parts and information regarding one or more replacement parts for the plurality of parts;
determine a total cost of owning the object based on the information regarding the plurality of parts and the information regarding one or more replacement parts for the plurality of parts; and
render second content relative to the plurality of parts,
wherein the second content includes information regarding the total cost of owning the object.
