# US 2021/0341616 A1 · Sensor fusion system, synchronization control apparatus, and synchronization control method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0341616 A1
- Document ID: US20210341616A1
- Status: Published application
- Kind: A1
- Application number: 17/282,986
- Filed: 2019-09-27
- Published: 2021-11-04
- Earliest claimed: 2018-10-12
- Assignees: Sony Semiconductor Solutions Corporation
- Applicants: Sony Semiconductor Solutions Corporation
- Inventors: Masatake Saito
- Web page: https://patentgrep.com/patent/US20210341616A1

## CPC

- G01S 17/89: for mapping or imaging
- G01C 11/02: Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
- G01C 15/002: {Active optical surveying means (optical plumbing G01C15/105)}
- G01C 21/3602: {Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera}
- G01S 7/4814: {of transmitters alone}
- G01S 7/4816: {of receivers alone}
- G01S 7/4817: {relating to scanning}
- G01S 7/4863: Detector arrays, e.g. charge-transfer gates
- G01S 17/06: Systems determining position data of a target
- G01S 17/10: using transmission of interrupted, pulse-modulated waves (determination of distance by phase measurements G01S17/32)
- G01S 17/86: Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
- H04N 25/40: Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
- H04N 25/531: by controlling rolling shutters in CMOS SSIS
- H04N 25/767: Horizontal readout lines, multiplexers or registers
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The present disclosure relates to a sensor fusion system, a synchronization control apparatus, and a synchronization control method that make it possible to improve a performance in fusion recognition by synchronizing a ranging apparatus and an image-capturing apparatus. A sensor fusion system is provided that includes a ranging apparatus, an image-capturing apparatus, and a synchronization controller, the ranging apparatus including a light source module, a light source controller, and a light receiver, the light source module including a light source section, the image-capturing apparatus including a pixel section, an AD converter, and an exposure controller, the synchronization controller controlling the light source controller and the exposure controller such that a timing at which the light source section emits light to a specified region is synchronized with a timing at which a group of pixels corresponding to the specified region is exposed. The present disclosure is applicable to, for example, a sensor fusion system in which LiDAR and an imager are used in synchronization with each other.

## Claims

### Claim 1

A sensor fusion system, comprising:
a ranging apparatus that includes
a light source module that includes a light source section and a drive section, the light source section emitting light extending in a first direction, the drive section driving an emission orientation of the light such that the light is emitted in a second direction that is orthogonal to the first direction,
a light source controller that controls a timing at which the light source section emits the light,
a light receiver that has a specified light-receiving range, and receives the light that is emitted by the light source section to be reflected off an object,
a period-of-time measurement section that measures a period of time from the light source section emitting the light to the light receiver receiving the light reflected off the object, and
a distance calculator that calculates a distance to the object from the light receiver or the light source module on a basis of the measured period of time;
an image-capturing apparatus that includes
a pixel section that includes a plurality of two-dimensionally arranged pixels, and has an image-capturing range that includes at least a portion of an emission range to which the light is emitted by the light source section,
an AD converter that is arranged in the first direction, and performs AD conversion on signals output from a group of pixels from among the plurality of pixels, the group of pixels being a group of pixels arranged in the pixel section in the first direction, and
an exposure controller that controls exposure of the plurality of pixels such that the plurality of pixels is scanned in the second direction; and
a synchronization controller that controls the light source controller and the exposure controller such that a timing at which the light source section emits the light to a specified region is synchronized with a timing at which the group of pixels corresponding to the specified region is exposed.

### Claim 2 (depends on 1)

The sensor fusion system according to claim 1, wherein
the first direction is a vertical direction, and
the second direction is a horizontal direction.

### Claim 3 (depends on 1)

The sensor fusion system according to claim 1, further comprising
a setting section that sets a first effective region and a second effective region, the first effective region being the emission range to which the light is emitted by the light source section, the second effective region being the image-capturing range of the pixel section.

### Claim 4 (depends on 3)

The sensor fusion system according to claim 3, wherein
the setting section
sets a virtual region that includes the first effective region and the second effective region,
sets a position and a size of the first effective region in the virtual region, and
sets a position and a size of the second effective region in the virtual region.

### Claim 5 (depends on 4)

The sensor fusion system according to claim 4, wherein
on a basis of a coordinate system of the virtual region, the setting section sets a space in a group of points in the light receiver, the group of points being a group of points at which the light is received by the light receiver, and
on the basis of the coordinate system of the virtual region, the setting section sets a space between pixels of the plurality of pixels arranged in the pixel section.

### Claim 6 (depends on 5)

The sensor fusion system according to claim 5, wherein
the space in the group of points is larger than the space between the pixels of the plurality of pixels.

### Claim 7 (depends on 1)

The sensor fusion system according to claim 1, wherein
the drive section drives the emission orientation of the light such that the light is emitted in a single direction in a fixed manner, or such that the light is emitted back and forth alternately.

### Claim 8 (depends on 7)

The sensor fusion system according to claim 7, wherein
when the emission orientation of the light is driven such that the light is emitted back and forth alternately, the exposure controller switches between a timing of exposure performed when the driving is performed in a certain direction, and a timing of exposure performed when the driving is performed in a direction opposite to the certain direction.

### Claim 9 (depends on 8)

The sensor fusion system according to claim 8, wherein
one pixel is continuously used when a blanking period of time with respect to the image-capturing range is equal to or greater than a specified period of time due to the emission range and the image-capturing range being different, and when periods of time for which charges are accumulated do not overlap in the one pixel included in the pixel section upon reversing the emission orientation from the certain direction to the opposite direction.

### Claim 10 (depends on 9)

The sensor fusion system according to claim 9, wherein
a time phase is adjusted for the one pixel.

### Claim 11 (depends on 7)

The sensor fusion system according to claim 7, wherein
when the emission orientation of the light is driven such that the light is emitted back and forth alternately, the exposure controller switches between a timing of exposing a first region that corresponds to the driving performed in a certain direction, and a timing of exposing a second region that is different from the first region and corresponds to the driving performed in a direction opposite to the certain direction.

### Claim 12 (depends on 11)

The sensor fusion system according to claim 11, wherein
the first region includes an excessive pixel that is in excess due to the number of pixels of the plurality of pixels arranged in the pixel section being greater than the number of groups of points in the light receiver, the group of points being a group of points at which the light is received by the light receiver, and
the second region includes the excessive pixel.

### Claim 13 (depends on 12)

The sensor fusion system according to claim 12, wherein
at least one of the first region or the second region includes the pixel corresponding to the group of points in the light receiver, the group of points being a group of points at which the light is received by the light receiver.

### Claim 14 (depends on 1)

The sensor fusion system according to claim 1, wherein
the light source section includes a single light source that diffusely irradiates light in a one-dimensional direction, or includes a plurality of light sources arranged in parallel in the one-dimensional direction.

### Claim 15 (depends on 1)

The sensor fusion system according to claim 1, wherein
a control line from the exposure controller to the pixel section is arranged in the first direction.

### Claim 16 (depends on 1)

The sensor fusion system according to claim 1, wherein
a scanning scheme of the ranging apparatus is horizontal back-and-forth scanning, and
an ADC scheme of the image-capturing apparatus is a row-parallel ADC scheme.

### Claim 17 (depends on 16)

The sensor fusion system according to claim 16, wherein
the exposure controller controls exposure performed by a rolling shutter scheme.

### Claim 18 (depends on 1)

The sensor fusion system according to claim 1, wherein
the ranging apparatus is LiDAR, and
the image-capturing apparatus includes an image sensor.

### Claim 19

A synchronization control apparatus that controls synchronization of a ranging apparatus and an image-capturing apparatus,
the ranging apparatus including
a light source module that includes a light source section and a drive section, the light source section emitting light extending in a first direction, the drive section driving an emission orientation of the light such that the light is emitted in a second direction that is orthogonal to the first direction,
a light source controller that controls a timing at which the light source section emits the light,
a light receiver that has a specified light-receiving range, and receives the light that is emitted by the light source section to be reflected off an object,
a period-of-time measurement section that measures a period of time from the light source section emitting the light to the light receiver receiving the light reflected off the object, and
a distance calculator that calculates a distance to the object from the light receiver or the light source module on a basis of the measured period of time,
the image-capturing apparatus including
a pixel section that includes a plurality of two-dimensionally arranged pixels, and has an image-capturing range that includes at least a portion of an emission range to which the light is emitted by the light source section,
an AD converter that is arranged in the first direction, and performs AD conversion on signals output from a group of pixels from among the plurality of pixels, the group of pixels being a group of pixels arranged in the pixel section in the first direction, and
an exposure controller that controls exposure of the plurality of pixels such that the plurality of pixels is scanned in the second direction,
the synchronization control apparatus comprising
a synchronization controller that controls the light source controller and the exposure controller such that a timing at which the light source section emits the light to a specified region is synchronized with a timing at which the group of pixels corresponding to the specified region is exposed.

### Claim 20

A synchronization control method, comprising
controlling a light source controller and an exposure controller by a synchronization control apparatus that controls synchronization of a ranging apparatus and an image-capturing apparatus,
the ranging apparatus including
a light source module that includes a light source section and a drive section, the light source section emitting light extending in a first direction, the drive section driving an emission orientation of the light such that the light is emitted in a second direction that is orthogonal to the first direction,
a light source controller that controls a timing at which the light source section emits the light,
a light receiver that has a specified light-receiving range, and receives the light that is emitted by the light source section to be reflected off an object,
a period-of-time measurement section that measures a period of time from the light source section emitting the light to the light receiver receiving the light reflected off the object, and
a distance calculator that calculates a distance to the object from the light receiver or the light source module on a basis of the measured period of time,
the image-capturing apparatus including
a pixel section that includes a plurality of two-dimensionally arranged pixels, and has an image-capturing range that includes at least a portion of an emission range to which the light is emitted by the light source section,
an AD converter that is arranged in the first direction, and performs AD conversion on signals output from a group of pixels from among the plurality of pixels, the group of pixels being a group of pixels arranged in the pixel section in the first direction, and
an exposure controller that controls exposure of the plurality of pixels such that the plurality of pixels is scanned in the second direction,
the controlling the light source controller and the exposure controller is performed such that a timing at which the light source section emits the light to a specified region is synchronized with a timing at which the group of pixels corresponding to the specified region is exposed.
