# US 2023/0410364 A1 · Semantic segmentation of inspection targets

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410364 A1
- Document ID: US20230410364A1
- Status: Published application
- Kind: A1
- Application number: 18/029,157
- Filed: 2021-09-29
- Published: 2023-12-21
- Earliest claimed: 2020-09-29
- Assignees: Kitov Systems Ltd
- Applicants: Kitov Systems Ltd
- Inventors: Ziv TSOREF; Nir AVRAHAMI; Tomer SHMUL
- Web page: https://patentgrep.com/patent/US20230410364A1

## CPC

- G01B 11/24: for measuring contours or curvatures
- G06T 7/75: {involving models}
- G01N 21/9515: {Objects of complex shape, e.g. examined with use of a surface follower device (measuring contours and curvatures G01B11/24)}
- G06T 7/0004: {Industrial image inspection}
- G06T 7/001: {using an image reference approach}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G01N 2021/9518: {using a surface follower, e.g. robot}
- G05B 2219/45066: Inspection robot
- G06T 2207/20081: Training; Learning
- G06T 2207/30108: Industrial image inspection
- G06T 2207/30164: Workpiece; Machine component
- G06T 2207/30244: Camera pose
- G06V 2201/06: Recognition of objects for industrial automation
- G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)

## Abstract

Automatic enrollment of an item of manufacture to a quality inspection system comprises using associations of enrollment images of an example of the item of manufacture to their corresponding camera poses. Enrollment images which show inspection targets (e.g., components of the item of manufacture) in views which are also suitable for use in visual inspection of further instances of the item of manufacture are identified. Their associated camera poses are selected and provided for use in inspection planning. In some embodiments, suitability of the camera pose is verified by performing inspection tests on the enrollment images.

## Claims

### Claim 1

A method of specifying visual inspection parameters for an item of manufacture, the method comprising:
accessing a plurality of enrollment images of an example of the item of manufacture;
for each of a plurality of regions appearing in a respective image of the plurality of enrollment images, classifying the region as imaging an identified inspection target having an inspection target type;
generating, using the regions and their classifications, a spatial model of the item of manufacture which indicates the spatial positioning of inspection targets and their respective inspection target types; and
calculating camera poses for use in obtaining images appropriate to inspection of the inspection targets, based on their respective modeled spatial positions and inspection target types.

### Claim 2 (depends on 1)

The method of claim 1, comprising identifying a change in an initial camera pose used to obtain at least one of the plurality of enrollment images, which said change potentially will provide an image with increased suitability for enrolling the identified inspection target, compared to the initial camera pose;
obtaining an auxiliary enrollment image using the changed camera pose; and
using the auxiliary enrollment image in the classifying.

### Claim 3 (depends on 1)

The method of claim 1, wherein the calculated camera poses include camera poses not used in the enrollment images used to generate the spatial model of the item of manufacture, the calculated camera poses being relatively more suitable as inspection images of the inspection targets than the camera poses used in obtaining the enrollment images.

### Claim 4 (depends on 1)

The method of claim 1, wherein the spatial model of the item of manufacture includes relative errors in the relative positioning of at least some surfaces of at least 1 cm.

### Claim 5 (depends on 1)

The method of claim 1, wherein the generating the spatial model includes using the classifications to identify regions in different images which correspond to the same portion of the spatial model.

### Claim 6 (depends on 1)

The method of claim 1, wherein the generating a spatial model comprises assigning geometric constraints to the identified inspection targets, based on the inspection target type classifications.

### Claim 7 (depends on 6)

The method of claim 6, wherein the generating uses the assigned geometric constraints for estimating surface angles of the example of the item of manufacture.

### Claim 8 (depends on 6)

The method of claim 6, wherein the generating uses the assigned geometric constraints for estimating orientations of the example of the item of manufacture.

### Claim 9 (depends on 6)

The method of claim 6, wherein the generating the spatial model includes using the assigned geometrical constraints to identify regions in different images which correspond to the same portion of the spatial model.

### Claim 10 (depends on 1)

The method of claim 1, wherein the enrollment images comprise 2-D images of the example of the item of manufacture.

### Claim 11 (depends on 1)

The method of claim 1, wherein the classifying comprises using a machine learning product to identify the inspection target type.

### Claim 12 (depends on 1)

The method of claim 1, comprising imaging to produce the enrollment images.

### Claim 13 (depends on 1)

The method of claim 1, comprising synthesizing a combined image from a plurality of the enrollment images, and performing the classifying and generating also using a region within the combined image spanning more than one of said plurality of the enrollment images.

### Claim 14 (depends on 1)

The method of claim 1, wherein the classifying comprises at least two stages of classifying for at least one of the inspection targets, and operations of the second stage of classifying are triggered by a result of the first stage of classifying.

### Claim 15 (depends on 14)

The method of claim 14, wherein the second stage of classifying classifies a region including at least a portion of, but different in size than another region classified in the first stage of classifying.

### Claim 16 (depends on 14)

The method of claim 14, wherein the second stage of classifying classifies a region to a more particular type belonging to a type identified in the first stage of classifying.

### Claim 17 (depends on 1)

The method of claim 1, wherein the generating also uses camera pose data indicative of camera poses from which the plurality of enrollment images were imaged.

### Claim at position 18

-42. (canceled)

### Claim 43

A system for specifying visual inspection parameters for an item of manufacture, the system comprising a processor and a memory storing instructions which instruct the processor to:
access a plurality of enrollment images of an example of the item of manufacture;
for each of a plurality of regions appearing in a respective image of the plurality of enrollment images, classify the region as imaging an identified inspection target having an inspection target type;
generate, using the regions and their classifications, a spatial model of the item of manufacture which indicates the spatial positioning of inspection targets and their respective inspection target types; and
calculate camera poses for use in obtaining images appropriate to inspection of the inspection targets, based on their respective modeled spatial positions and inspection target types.

### Claim 44 (depends on 43)

The system of claim 43, wherein the instructions instruct the processor to:
identify a change in an initial camera pose used to obtain at least one of the plurality of enrollment images, which said change potentially will provide an image with increased suitability for enrolling the identified inspection target, compared to the initial camera pose;
obtain an auxiliary enrollment image using the changed camera pose; and
use the auxiliary enrollment image in the classifying.

### Claim 45 (depends on 43)

The system of claim 43, wherein the instructions instruct the processor to classify the region through at least two stages of classification for at least one of the inspection targets, and operations of the second stage of classification are triggered by a result of the first stage of classification.
