# US 10,176,628 B2 · Method for creating a 3D representation and corresponding image recording apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,176,628 B2
- Document ID: US10176628B2
- Status: Granted
- Kind: B2
- Application number: 15/230,768
- Filed: 2016-08-08
- Published: 2017-02-09
- Granted: 2019-01-08
- Earliest claimed: 2015-08-08
- Assignees: Testo AG
- Applicants: Testo AG
- Inventors: Jan-Friso Evers-Senne; Philipp Sasse; Hellen Altendorf; Raphael Bartsch
- Web page: https://patentgrep.com/patent/US10176628B2

## CPC

- G06T 17/10: Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
- G06T 7/11: Region-based segmentation
- G06T 7/55: from multiple images
- G06T 15/205: {Image-based rendering}
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

In a method for creating a 3D representation of a recording object it is provided, with respect to recorded individual images (2, 3) from a recording object, to identify feature pixels (6, 7, 8, 9) respectively in a computer-aided manner by a feature detection, to calculate in a computer-implemented manner with respect to this feature pixels (6, 7, 8, 9), on the basis of correspondences in terms of content, respectively 3D points (11, 12) and camera poses (3, 4) of the individual images (2, 3), to fit at least one geometric primitive (14) into the calculated 3D points (11, 12) in a computer-implemented manner and to check in a plausibility check (15) whether a minimum discrepancy between the individual images (2, 3) results for the geometric primitive (14), and to output the geometric primitive (14) for which the discrepancy is minimized.

## Claims

### Claim 1

A method for creating a 3D representation with respect to a recording object, comprising
in a recording step (1), recording a sequence of individual images (2, 3) of the recording object from a plurality of camera poses (4),
in a feature detection step (5), applying a feature detection to the individual images (2, 3) in a computer-aided manner in order to identify image pixels as feature pixels (6, 7, 8, 9),
in a 3D calculation step (10) with respect to the feature pixels (6, 7, 8, 9), calculating the camera poses (4) and respective 3D points (11, 12) in a computer-aided manner,
in a fitting step (13) for the 3D points (11, 12) of the feature pixels (6, 7, 8, 9), fitting at least one geometric primitive (14) in a computer-aided manner,
in a plausibility check (15), at least one of checking or optimizing the at least one geometric primitive (14) as to whether or until a discrepancy between the individual images (2, 3) is determined,
in a representation step (16), outputting the geometric primitive (14) with a minimized discrepancy,
wherein one or more steps selected from the group of: the feature detection step (5), the 3D calculation step (10), the fitting step (13), or the plausibility check (15), are respectively performed until an end criterion is fulfilled,
wherein a functional dependence of parameters on a master parameter is kept available, and the master parameter is altered with respect to a renewed performance of one step or a plurality of steps from the group including the feature detection step (5), the 3D calculation step (10), the fitting step (13), or the plausibility check (15), and
a user input for selecting the master parameter as an external input to adjust a level of accuracy of the 3D representation.

### Claim 2 (depends on 1)

The method as claimed in claim 1, further comprising in the plausibility check (15)
in a segmentation step (17), segmenting the individual images (2, 3) in a computer-aided manner using the feature pixels (6, 7, 8, 9) as initial cells.

### Claim 3 (depends on 1)

The method as claimed in claim 1, further comprising
in a variation step (20), virtually projecting image pixels (21) of a segment (18, 19), on the basis of the associated camera pose (4) onto the at least one geometric primitive (14) in a computer-aided manner, and varying the at least one geometric primitive (14) in a computer-aided manner until the discrepancy between image pixels (21) of different individual images (2, 3) projected onto a common partial region (23) of the geometric primitive (14) is minimized.

### Claim 4 (depends on 1)

The method as claimed in claim 1, further comprising for determining the discrepancy, calculating a difference in at least one of color or brightness values of the image pixels (21) and summing the difference in terms of absolute value.

### Claim 5 (depends on 1)

The method as claimed in claim 1, further comprising in the fitting step (13), using one or more of: an at least one of parameterized or bounded cone envelope part, a cylinder lateral surface part, or a planar part as the geometric primitive (14).

### Claim 6 (depends on 1)

The method as claimed in claim 1, further comprising in the recording step (1), projecting a pattern onto the recording object.

### Claim 7 (depends on 1)

The method as claimed in claim 1, wherein a RANSAC method is applied in the fitting step (13).

### Claim 8 (depends on 1)

The method as claimed in claim 1, further comprising in a point generating step (27, 29), generating additional 3D points (28, 30) in a computer-aided dense reconstruction using at least one of a stereo algorithm or a rectification.

### Claim 9 (depends on 8)

The method as claimed in claim 8, wherein the point generating step (27, 29) is after the feature detection step (5).

### Claim 10 (depends on 1)

The method as claimed in claim 1, further comprising in an extrapolation step (31), calculating in a computer-aided manner an intersection of extensions of at least two respectively fitted, varied, or fitted and varied geometric primitives (14).

### Claim 11 (depends on 10)

The method as claimed in claim 10, further comprising in a connection step (33), connecting the at least two of the geometric primitives (14) at an intersection.

### Claim 12 (depends on 11)

The method as claimed in claim 11, wherein in at least one of the fitting step (13) or the variation step (20), two or more of the geometric primitives (14) are processed.

### Claim 13 (depends on 12)

The method as claimed in claim 12, wherein in at least one of the fitting step (13) or the variation step (20), the two or more of the geometric primitives (14) are processed independently of one another and in parallel with one another.

### Claim 14 (depends on 1)

The method as claimed in claim 1, wherein one step or a plurality of steps from a group including a segmentation step (17), a variation step (20), an extrapolation step (31), or a connection step (33) are respectively performed until an end criterion is fulfilled.

### Claim 15 (depends on 14)

The method as claimed in claim 14, wherein respectively a functional dependence of parameters on a master parameter is kept available, and the master parameter is altered with respect to a renewed performance of one step or a plurality of steps from the group including the segmentation step (17), the variation step (20), the connection step (33), or the extrapolation step (31).

### Claim 16 (depends on 14)

The method as claimed in claim 14, wherein at least two steps from the group including the recording step (1), the feature detection step (5), the 3D calculation step (10), the fitting step (13), the plausibility check (15), the segmentation step (17), the variation step (20), the extrapolation step (31), the connection step (33), the recording step (9) or the feature detection step (5), are performed in a manner overlapping in time.

### Claim 17

An image recording apparatus, comprising an image sensor designed for recording a sequence of individual images (2, 3), a computer configured to detect image features that applies a feature detection to the individual images (2, 3) and identifies image pixels as feature pixels (6, 7, 8, 9), a 3D calculator designed for calculating 3D points (11, 12) with respect to the feature pixels (6, 7, 8, 9) and camera poses (4) with respect to the individual images (2, 3), a fitting device designed for fitting at least one geometric primitive (14) for the 3D points (11, 12) of the feature pixels (6, 7, 8, 9), a plausibility device designed for at least one of checking or optimizing the at least one geometric primitive (14) until image pixels of different individual images (2, 3) that are assigned to common partial regions (23) of the at least one geometric primitive (14) on the basis of camera poses (4) have a minimized discrepancy, and the computer being further configured for (a) carrying out one or more steps selected from the group of: the feature detection step (5), the 3D calculation step (10), the fitting step (13), or the plausibility check (15), are respectively performed until an end criterion is fulfilled, (b) maintaining a functional dependence of parameters on a master parameter, and the master parameter is altered with respect to a renewed performance of one step or a plurality of steps from the group including the feature detection step (5), the 3D calculation step (10), the fitting step (13), or the plausibility check (15), (c) providing an external user input for selecting the master parameter as an external input to adjust a level of accuracy of the 3D representation, and (d) outputting the at least one geometric primitive (14) with the minimized discrepancy.

### Claim 18 (depends on 17)

The image recording apparatus as claimed in claim 17, wherein the plausibility device comprises: a segmentation device designed for segmenting the individual images (2, 3), wherein the feature pixels (6, 7, 8, 9) are at least one of provided or used as initial cells, and a variation device designed for at least one of virtually projecting image pixels (21) of a segment (18, 19) onto the at least one geometric primitive (14) on the basis of the calculated associated camera pose (4) or varying the at least one geometric primitive (14) until a discrepancy between image pixels (21) projected onto a common partial region (23) of the geometric primitive (14) is minimized.