# US 12,223,639 B2 · Photographing guide device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,639 B2
- Document ID: US12223639B2
- Status: Granted
- Kind: B2
- Application number: 17/417,218
- Filed: 2019-12-10
- Published: 2022-03-10
- Granted: 2025-02-11
- Earliest claimed: 2019-01-10
- Assignees: NEC Corporation
- Applicants: NEC Corporation
- Inventors: Jun Takada; Gaku Nakano
- Web page: https://patentgrep.com/patent/US12223639B2

## CPC

- G06T 7/001: {using an image reference approach}
- G06V 10/247: {by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids}
- G01H 17/00: Measuring mechanical vibrations or ultrasonic, sonic or infrasonic waves, not provided for in the other groups of this subclass
- G01M 99/00: Subject matter not provided for in other groups of this subclass
- G01N 21/88: Investigating the presence of flaws or contamination
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 7/0002: {Inspection of images, e.g. flaw detection}
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 7/337: {involving reference images or patches}
- G06T 7/50: Depth or shape recovery
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/579: from motion
- G06T 7/593: from stereo images
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/74: {involving reference images or patches}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/30184: Infrastructure
- G06V 2201/02: Recognising information on displays, dials, clocks
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G01H: MEASUREMENT OF MECHANICAL VIBRATIONS OR ULTRASONIC, SONIC OR INFRASONIC WAVES
- G01M: TESTING STATIC OR DYNAMIC BALANCE OF MACHINES OR STRUCTURES; TESTING OF STRUCTURES OR APPARATUS, NOT OTHERWISE PROVIDED FOR
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A photographing guide device includes a detection means and an output means. The detection means compares a candidate image of a structure captured from a capturing candidate position in a capturing candidate direction with a registered image of the structure captured from a given capturing position in a given capturing direction and stored in advance, thereby detecting a positional difference between the capturing candidate position and the given capturing position. The output means outputs information indicating the detected positional difference.

## Claims

### Claim 1

A photographing guide device comprising:
a memory storing program instructions; and
a processor coupled to the memory, wherein the processor is configured to execute the program instructions to:
extract a plurality of feature points from each of a candidate image of a structure captured from a capturing candidate position in a capturing candidate direction and a registered image of the structure captured from a given capturing position in a given capturing direction stored in advance;
associate each feature point extracted from the candidate image with a corresponding feature point extracted from the registered image, as a feature point pair, resulting in obtaining a plurality of feature point pairs;
when at least a predetermined number of the plurality of feature point pairs have been obtained, estimate a homography matrix of the candidate image and the registered image based on the plurality of feature point pairs;
detect a positional difference between the capturing candidate position and the given capturing position, and a direction difference between the capturing candidate direction and the given capturing direction, on a basis of the homography matrix; and
output information indicating the positional difference and the direction difference that have been detected, and indicating a synthetic image, wherein
detection of the positional difference includes transformation of the registered image on the basis of the homography matrix, and generation of the synthetic image in which the registered image after transformation and the candidate image are synthesized.

### Claim 2 (depends on 1)

The photographing guide device according to claim 1, wherein
detection of the positional difference includes generation of a crack image in which a crack on the structure is emphasized from the candidate image, and comparison of the generated crack image with a registered crack image that is generated from the registered image and stored in advance.

### Claim 3 (depends on 2)

The photographing guide device according to claim 2, wherein
detection of the positional difference includes estimation of a homography matrix of the crack image and the registered crack image, and detection of the positional difference on a basis of the homography matrix of the crack image and the registered crack image.

### Claim 4 (depends on 2)

The photographing guide device according to claim 2, wherein
detection of the positional difference includes estimation of a homography matrix of the crack image and the registered crack image, transformation of the registered crack image on a basis of the homography matrix of the crack image and the registered crack image, and generation of a synthetic image in which the registered crack image after transformation and the crack image are synthesized.

### Claim 5 (depends on 1)

The photographing guide device according to claim 1, wherein
detection of the positional difference includes comparison of a candidate three-dimensional shape obtained by measuring a three-dimensional shape of a peripheral environment from the capturing candidate position with a registered three-dimensional shape obtained by measuring the three-dimensional shape of the peripheral environment from the given capturing position and stored in advance.

### Claim 6 (depends on 5)

The photographing guide device according to claim 5, wherein
detection of the positional difference includes estimation of a geometric transformation matrix of the candidate three-dimensional shape and the registered three-dimensional shape, and detection of the positional difference on a basis of the geometric transformation matrix.

### Claim 7 (depends on 5)

The photographing guide device according to claim 5, wherein
detection of the positional difference includes estimation of a geometric transformation matrix of the candidate three-dimensional shape and the registered three-dimensional shape, transformation of the registered three-dimensional shape on a basis of the geometric transformation matrix, and generation of a synthetic image in which the registered three-dimensional shape after transformation and the candidate three-dimensional shape are synthesized.

### Claim 8

A photographing guide method performed by a computer and comprising:
extracting a plurality of feature points from each of a candidate image of a structure captured from a capturing candidate position in a capturing candidate direction and a registered image of the structure captured from a given capturing position in a given capturing direction stored in advance;
associating each feature point extracted from the candidate image with a corresponding feature point extracted from the registered image, as a feature point pair, resulting in obtaining a plurality of feature point pairs;
when at least a predetermined number of the plurality of feature point pairs have been obtained, estimating a homography matrix of the candidate image and the registered image based on the plurality of feature point pairs;
detecting a positional difference between the capturing candidate position and the given capturing position, and a direction difference between the capturing candidate direction and the given capturing direction, on a basis of the homography matrix; and
outputting information indicating the positional difference and the direction difference that have been detected, and indicating a synthetic image, wherein
detection of the positional difference includes transformation of the registered image on the basis of the homography matrix, and generation of the synthetic image in which the registered image after transformation and the candidate image are synthesized.

### Claim 9

A non-transitory computer-readable storage medium storing a program comprising instructions executable by a computer to perform processing comprising:
extracting a plurality of feature points from each of a candidate image of a structure captured from a capturing candidate position in a capturing candidate direction and a registered image of the structure captured from a given capturing position in a given capturing direction stored in advance;
associating each feature point extracted from the candidate image with a corresponding feature point extracted from the registered image, as a feature point pair, resulting in obtaining a plurality of feature point pairs;
when at least a predetermined number of the plurality of feature point pairs have been obtained, estimating a homography matrix of the candidate image and the registered image based on the plurality of feature point pairs;
detecting a positional difference between the capturing candidate position and the given capturing position, and a direction difference between the capturing candidate direction and the given capturing direction, on a basis of the homography matrix; and
outputting information indicating the positional difference and the direction difference that have been detected, and indicating a synthetic image, wherein
detection of the positional difference includes transformation of the registered image on the basis of the homography matrix, and generation of the synthetic image in which the registered image after transformation and the candidate image are synthesized.