# US 12,669,331 B2 · System and method for manufacturing and maintenance

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,669,331 B2
- Document ID: US12669331B2
- Status: Granted
- Kind: B2
- Application number: 18/001,417
- Filed: 2021-06-11
- Published: 2023-07-13
- Granted: 2026-06-30
- Earliest claimed: 2020-06-11
- Assignees: Interaptix Inc.
- Applicants: Interaptix Inc.
- Inventors: Bardia Bina; Dae Hyun Lee; Tyler James Doyle; Isaac Louis Gold Berman; Behzad Abghari
- Web page: https://patentgrep.com/patent/US12669331B2

## CPC

- G01C 21/165: {combined with non-inertial navigation instruments}
- G06V 20/20: in augmented reality scenes
- G01C 21/005: {with correlation of navigation data from several sources, e.g. map or contour matching (G01C21/30 takes precedence)}
- G01C 21/206: {specially adapted for indoor navigation}
- G05B 19/41875: {characterised by quality surveillance of production}
- G06Q 10/06395: {Quality analysis or management}
- G06Q 10/0875: Itemisation or classification of parts, supplies or services, e.g. bill of materials
- G06Q 10/20: Administration of product repair or maintenance
- G06Q 30/012: Providing warranty services
- G06Q 30/014: Providing recall services for goods or products
- G06Q 50/04: Manufacturing
- G06T 7/0004: {Industrial image inspection}
- G06T 7/74: {involving reference images or patches}
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G05B 2219/32368: Quality control
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/20101: Interactive definition of point of interest, landmark or seed
- G06T 2207/30164: Workpiece; Machine component
- G06T 2219/004: Annotating, labelling
- G07C 3/146: {during manufacturing process}
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G07C: TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE

## Abstract

A system and method for inspection maintenance and/or diagnosis of a variety of workpieces is provided. The system serves workers working on a workpiece, inspectors who are distal from the workers and/or can be used for remote training or for advanced diagnosis and/or repair. The system preferably includes a template of a set of one or more predefined required images of a workpiece required by an inspector to perform their inspection or diagnosis. The set of predefined required images is provided to the worker. The worker captures the images with an appropriate workpiece data capture device and provides them to the inspector for review. The inspector examines the provided images and either approves the workpiece based on their content, requests additional images for further examination and/or provides annotations and other information to the worker to address identified issues. The system maintains a database of all images and information.

## Claims

### Claim 1

A system for inspection of a workpiece, the system comprising:
a database configured to store at least one inspection template defining, for a workpiece, a set of data to be captured to inspect the workpiece, the set of data including a set of required images, each required image associated with a specified viewpoint relative to a model of the workpiece;
a workpiece data capture device communicating with the database, the workpiece data capture device configured to:
receive the at least one inspection template;
present, on a live view of the workpiece, a graphical capture guide corresponding to the specified viewpoint of a selected required image, the graphical capture guide defining a target camera pose relative to the workpiece;
determine a current camera pose of the workpiece data capture device relative to the workpiece;
present, on the live view, an alignment indication representing a relationship between the current camera pose and the target camera pose, including presentation of any inspection annotations referenced to the model; and
capture the selected required image responsive to alignment of the current camera pose with the target camera pose; and
an inspector station configured to:
present captured images in association with the model;
generate, based on a selected view of the model, an additional image capture request defining a corresponding specified viewpoint; and
generate at least one inspection annotation referenced to the model.

### Claim 2 (depends on 1)

The system of claim 1, wherein the at least one inspection template defines at least two manufacturing stages for which the inspection is to be performed and the data to be captured at each of the at least two manufacturing stages.

### Claim 3 (depends on 1)

The system of claim 1, wherein the data further includes one or more of: depth information, audio recordings, videos and text-based forms.

### Claim 4 (depends on 1)

The system of claim 1, wherein to present the captured images in association with the model, the inspector station is configured to:
display, in a model pane, the model; and
display, in a library pane, the set of data defined in the at least one inspection template for the workpiece.

### Claim 5 (depends on 4)

The system of claim 4, wherein, the inspector station is further configured to display in an image capture pane, in response to a selection of one of the captured images, the model and an overlay indicating a location where the one of the captured images was captured.

### Claim 6 (depends on 5)

The system of claim 5, wherein the overlay comprises a pyramid, wherein an apex of the pyramid indicates the location where the one of the captured images was captured, and a base of the pyramid indicates a plane of the one of the captured images at the location.

### Claim 7 (depends on 6)

The system of claim 6, wherein the inspector station is further configured to present, in an image viewing pane, a full resolution view of the one of the captured images at the location.

### Claim 8 (depends on 1)

The system of claim 1, wherein the inspector station is further configured to define a viewpoint fulcrum, define a depth relative to the viewpoint fulcrum to locate an annotation plane, and define the at least one inspection annotation on the annotation plane.

### Claim 9 (depends on 8)

The system of claim 8, wherein the workpiece data capture device is configured to present the inspection annotations as augmented reality elements on the live view.

### Claim 10 (depends on 1)

The system of claim 1, wherein the database is further configured to store, for each of the captured images, an associated camera pose relative to the model of the workpiece, and wherein the stored captured images and associated camera poses form a geometrically consistent data set usable for an automated inspection analysis.

### Claim 11

A method of inspecting a workpiece, the method comprising:
defining a set of required images to be captured to inspect the workpiece, each required image associated with a specified viewpoint relative to a model of the workpiece;
presenting, on a live view of the workpiece, a graphical capture guide corresponding to the specified viewpoint of a selected required image, the graphical capture guide defining a target camera pose relative to the workpiece;
determining a current camera pose of a workpiece data capture device relative to the workpiece;
presenting, on the live view, an alignment indication representing a relationship between the current camera pose and the target camera pose;
capturing, at the workpiece data capture device, the selected required image responsive to alignment of the current camera pose with the target camera pose;
presenting, at an inspector station, captured images in association with the model; and
generating inspection results, the inspection results comprising one of:
an additional image capture request defining a corresponding specified viewpoint; and
an approval of the workpiece.

### Claim 12 (depends on 11)

The method of claim 11, wherein presenting the captured images comprises:
displaying the model; and
displaying a library of data including the captured images.

### Claim 13 (depends on 12)

The method of claim 12, further comprising displaying, in response to a selection of one of the captured images from the library of data, the model and an overlay indicating a location where the one of the captured images was captured, the overlay comprising a pyramid, wherein an apex of the pyramid indicates the location where the one of the captured images was captured, and a base of the pyramid indicates a plane of the one of the captured images at the location.

### Claim 14 (depends on 13)

The method of claim 13, further comprising presenting a full resolution view of the one of the captured images at the location.

### Claim 15 (depends on 11)

The method of claim 11, further comprising:
Defining, relative to the model of the workpiece, a viewpoint fulcrum;
defining a depth relative to the viewpoint fulcrum to locate an annotation plane; and
defining an annotation on the annotation plane.

### Claim 16 (depends on 15)

The method of claim 15, further comprising displaying, at the workpiece data capture device, the annotation as an augmented reality element on the live view.

### Claim 17 (depends on 11)

The method of claim 11 further comprising:
storing, for each of the captured images, an associated camera pose relative to the model of the workpiece, the stored captured images and associated camera poses forming a geometrically consistent data set for an automated inspection analysis.