# US 2019/0320154 A1 · Electronic system including image processing unit for reconstructing 3d surfaces and iterative triangulation method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0320154 A1
- Document ID: US20190320154A1
- Status: Published application
- Kind: A1
- Application number: 16/456,612
- Filed: 2019-06-28
- Published: 2019-10-17
- Earliest claimed: 2015-12-14
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Roderick KÖHLE; Francesco MICHIELIN; Dennis HARRES
- Web page: https://patentgrep.com/patent/US20190320154A1

## CPC

- H04N 13/128: Adjusting depth or disparity
- H04N 13/204: using stereoscopic image cameras (stereoscopic photography G03B35/00)
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An electronic system includes a circuitry configured to obtain a sequence of frames of an object under different viewing angles at consecutive time instances. For a first time instance, the circuitry generates a point cloud descriptive for an external surface of the object on basis of (i) a point cloud obtained for a second time instance preceding the first time instance and (ii) disparity information concerning a frame captured at the first time instance.

## Claims

### Claim 1

An electronic system, comprising
circuitry configured to
obtain a sequence of frames of an object under different viewing angles,
process the sequence of frames, wherein the circuitry for processing the sequence of frames is further configured to
receive preconditioned frames,
for each preconditioned frame, extract salient image points, wherein the salient image points identify small image areas of high contrast,
reduce a total number of salient image points by selecting only a predetermined number of stable salient image points, and
generate, for a first time instance, a point cloud descriptive for an external surface of the object based on (i) a point cloud obtained for a second time instance preceding the first time instance and (ii) disparity information concerning a frame captured at the first time instance,
display a 3D representation of the point cloud generated for the first time instance, and
update the displayed 3D representation while obtaining the sequence of frames.

### Claim 2 (depends on 1)

The electronic system according to claim 1, wherein
the disparity information concerns the frame captured at the first time instance and a frame captured at the second time instance.

### Claim 3 (depends on 1)

The electronic system according to claim 1, wherein
the frame captured at the first time instance includes one or more sub-frames captured at the first time instance from different positions.

### Claim 4 (depends on 1)

The electronic system according to claim 1, wherein
the circuitry is further configured to identify sparse correspondences between the first frame and a preceding frame for sparse salient image points in the first and the preceding frame, the preceding frame corresponding to a frame captured at the second time instance.

### Claim 5 (depends on 4)

The electronic system according to claim 4, wherein
the circuitry is further configured to calculate dense correspondences between two consecutive frames, a starting value being obtained by the preceding point cloud updated by information obtained about the identified sparse correspondences.

### Claim 6 (depends on 5)

The electronic system according to claim 5, wherein
the circuitry is further configured to calculate an actual pose of an image capturing unit relative to the object based on the dense correspondences dc(n) for the first frame fr(n) by using iterative methods.

### Claim 7 (depends on 1)

The electronic system according to claim 1, wherein
the circuitry is further configured to update depth information contained in the point cloud obtained at the second time instance by using information contained in the first frame.

### Claim 8 (depends on 7)

The electronic system according to claim 7, wherein
the circuitry uses a ray accumulator configured to temporarily store a ten coefficient vector.

### Claim 9 (depends on 1)

The electronic system according to claim 1, wherein the circuitry comprises,
an image capturing unit adapted to obtain the sequence of frames of the object under different viewing angles; and
a processing unit adapted to generate, for the first time instance, the point cloud descriptive for the external surface of the object based on (i) the point cloud obtained for the second time instance preceding the first time instance and (ii) disparity information concerning the frame captured at the first time instance.

### Claim 10

A 3D reconstruction method, comprising
capturing a sequence of frames of an object under different viewing angles at consecutive time instances;
processing the sequence of frames, wherein processing the sequence of frames includes
receiving preconditioned frames;
for each preconditioned frame, extracting salient image points, wherein the salient image points identify small image areas of high contrast:
reducing a total number of salient image points by selecting only a predetermined number of stable salient image points; and
generating, for a first time instance, a point cloud descriptive for an external surface of the object based on (i) a point cloud obtained for second time instance preceding the first time instance and (ii) disparity information concerning a frame captured at the first time instance;
displaying a 3D representation of the point cloud generated for the first time instance; and
updating the displayed 3D representation while capturing the sequence of frames.

### Claim 11 (depends on 10)

The 3D reconstruction method according to claim 10, wherein
the disparity information concerns the frame captured at the first time instance and a frame captured at the second time instance.

### Claim 12 (depends on 10)

The 3D reconstruction method according to claim 10, wherein
the frame captured at the first time instance includes one or more sub-frames captured at the first time instance from different positions.
