# US 2020/0404318 A1 · Image processing device and image processing method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0404318 A1
- Document ID: US20200404318A1
- Status: Published application
- Kind: A1
- Application number: 16/979,159
- Filed: 2018-12-04
- Published: 2020-12-24
- Earliest claimed: 2018-03-16
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: JUNICHI MATSUMOTO
- Web page: https://patentgrep.com/patent/US20200404318A1

## CPC

- H04N 19/513: Processing of motion vectors
- H04N 19/52: by predictive encoding
- G06T 3/00: Geometric image transformations in the plane of the image
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence)
- H04N 19/521: {for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors}
- H04N 19/61: in combination with predictive coding
- H04N 23/60: Control of cameras or camera modules
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A motion vector correction unit including a correction method determination unit 40 and an inter-prediction unit 42 performs scaling processing on a motion vector of, for example, a peripheral block of a current predictive block in a fisheye image shot using a fisheye lens, on the basis of fisheye information, for example, a projection method of the fisheye lens and a radius of the fisheye image. The scaling processing performs scaling by transforming the motion vector of the peripheral block to a motion vector of an image having no fisheye distortion, and transforms the motion vector after the scaling to the motion vector in the fisheye image. By reducing a difference between a motion vector of the current predictive block and a predictive motion vector by performing the scaling processing, encoding efficiency of the fisheye image can be improved.

## Claims

### Claim 1

An image processing device comprising:
a motion vector correction unit that performs scaling processing on a motion vector used to calculate a predictive motion vector of a current predictive block in a fisheye image shot using a fisheye lens, on a basis of fisheye information regarding the fisheye lens and the fisheye image.

### Claim 2 (depends on 1)

The image processing device according to claim 1, wherein
the motion vector correction unit performs the scaling processing on the motion vector, on a basis of a projection method of the fisheye lens and a radius of the fisheye image.

### Claim 3 (depends on 2)

The image processing device according to claim 2, wherein
the motion vector correction unit performs the scaling processing on the motion vector using a correction factor that corrects fisheye distortion that varies according to a distance from a center of the fisheye lens.

### Claim 4 (depends on 3)

The image processing device according to claim 3, wherein
the motion vector correction unit performs scaling by transforming the motion vector in the fisheye image to a motion vector in an image having no fisheye distortion using the correction factor and the radius of the fisheye image, and transforms the motion vector after the scaling to the motion vector in the fisheye image.

### Claim 5 (depends on 3)

The image processing device according to claim 3, wherein
the motion vector correction unit performs the scaling processing using a correction factor table corresponding to the fisheye lens used to shoot, from correction factor tables for respective projection methods of the fisheye lens, each of the correction factor tables indicating the correction factor that is normalized.

### Claim 6 (depends on 5)

The image processing device according to claim 5, wherein
the motion vector correction unit retains the correction factor tables for the respective projection methods in advance.

### Claim 7 (depends on 3)

The image processing device according to claim 3, wherein
the correction factor is calculated, on a basis of a characteristic formula of the projection method of the fisheye lens and a characteristic formula of a central projection method for obtaining the image having no fisheye distortion.

### Claim 8 (depends on 1)

The image processing device according to claim 1, wherein
the motion vector used to calculate the predictive motion vector is a motion vector of a peripheral block of the current predictive block.

### Claim 9 (depends on 1)

The image processing device according to claim 1, further comprising:
a lossless encoder that incorporates fisheye information regarding the fisheye lens used to shoot, into an encoded stream of the fisheye image.

### Claim 10 (depends on 9)

The image processing device according to claim 9, wherein
the lossless encoder incorporates information indicating whether or not the scaling processing is performed, into the encoded stream.

### Claim 11 (depends on 9)

The image processing device according to claim 9, wherein
the fisheye information includes information indicating a projection method of the fisheye lens and information indicating a radius of the fisheye image.

### Claim 12 (depends on 11)

The image processing device according to claim 11, wherein
the fisheye information includes information indicating a projection method of an optical image, instead of the information indicating the radius of the fisheye image.

### Claim 13 (depends on 1)

The image processing device according to claim 1, further comprising:
a lossless decoder that decodes an encoded stream of the fisheye image, wherein
the motion vector correction unit performs the scaling processing using fisheye information regarding the fisheye lens used to shoot, the fisheye information being obtained by decoding the encoded stream by the lossless decoder.

### Claim 14 (depends on 13)

The image processing device according to claim 13, wherein,
in a case where the information obtained by decoding the encoded stream by the lossless decoder indicates that the scaling processing is to be performed, the motion vector correction unit performs the scaling processing using the fisheye information.

### Claim 15

An image processing method comprising:
performing, by a motion vector correction unit, scaling processing on a motion vector used to calculate a predictive motion vector of a current predictive block in a fisheye image shot using a fisheye lens, on a basis of fisheye information regarding the fisheye lens and the fisheye image.
