# US 11,308,588 B2 · Image processing method, device, and virtual reality display apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,588 B2
- Document ID: US11308588B2
- Status: Granted
- Kind: B2
- Application number: 16/494,588
- Filed: 2019-01-28
- Published: 2021-10-28
- Granted: 2022-04-19
- Earliest claimed: 2018-05-29
- Assignees: Beijing Boe Optoelectronics Technology Co., Ltd.; Boe Technology Group Co., Ltd.
- Applicants: Beijing Boe Optoelectronics Technology Co., Ltd.; Boe Technology Group Co., Ltd.
- Inventors: Minglei Chu; Hao Zhang; Lili Chen; Chenru Wang; Yali Liu; Yukun Sun; Guixin Yan; Zhanshan Ma; Ziqiang Guo; Zehua Dong; Bingxin Liu
- Web page: https://patentgrep.com/patent/US11308588B2

## CPC

- G06T 5/80: Geometric correction
- G02B 27/022: {Viewing apparatus (G02B27/04, G02B27/06, G02B27/08 take precedence)}
- G06T 7/62: of area, perimeter, diameter or volume
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

The present disclosure provides an image processing method, device, and virtual reality display apparatus for performing anti-distortion processing on images. The image processing method, for an electronic device including a lens unit, includes: determining a to-be-buffered image size according to a size of an input image and anti-distortion parameters of the lens unit; generating, based on the input image and the anti-distortion parameters, an anti-distortion buffer image having an image size according to the to-be-buffered image size.

## Claims

### Claim 1

An image processing method for an electronic device including a lens unit, comprising:
determining a to-be-buffered image size according to a size of an input image and anti-distortion parameters of the lens unit; and
generating, based on the input image and the anti-distortion parameters, an anti-distortion buffer image having an image size according to the to-be-buffered image size,
wherein the determining a to-be-buffered image size according to a size of an input image and anti-distortion parameters of the lens unit comprises:
determining an anti-distortion grid for the input image based on the anti-distortion parameters, and coordinate values of four vertices of an anti-distortion image on the anti-distortion grid determined by using a center of the input image as an origin, and determining absolute values of the coordinate values of the four vertices in a column direction respectively;
determining a minimum absolute value Y among absolute values of the coordinate values of the four vertices in the column direction; and
determining the to-be-buffered image size according to the minimum absolute value Y.

### Claim 2 (depends on 1)

The image processing method according to claim 1, wherein generating an anti-distortion buffer image having the image size comprises:
selecting one row of pixels from the input image to perform an anti-distortion operation, the anti-distortion operation including determining pixel data of the anti-distortion image corresponding to the one row of pixels according to the anti-distortion parameters, and writing pixel data of the anti-distortion image to the anti-distortion buffer image.

### Claim 3 (depends on 2)

The image processing method according to claim 2, wherein the selecting one row of pixels from the input image to perform an anti-distortion operation comprises: performing an anti-distortion operation progressively on each row of pixels in the input image.

### Claim 4 (depends on 1)

The image processing method according to claim 1, wherein the anti-distortion parameters include an object-image relationship between heights of a plurality of pixels of the anti-distortion image and heights of a plurality of corresponding pixels in a virtual image formed by the anti-distortion image passing through the lens unit, and the object-image relationship is determined based on optical parameters of the lens unit.

### Claim 5 (depends on 4)

The image processing method according to claim 4, wherein the optical parameters of the lens unit include a focal length of the lens unit, a distance between a display position of the input image and the lens unit, and a distance between a user viewing position and the lens unit.

### Claim 6 (depends on 5)

The image processing method according to claim 5, wherein heights of a plurality of pixels in the anti-distortion image are respective distances from the plurality of pixels in the anti-distortion image to a mapped point on the input image of a lens center, heights of a plurality of corresponding pixels in the virtual image are respective distances from the plurality of corresponding pixels in the virtual image to a mapped point in the virtual image of the lens center.

### Claim 7 (depends on 1)

The image processing method according to claim 1, wherein the size of the input image is W*H, a size of the to-be-buffered image in a row direction is W, and a size of the to-be-buffered image in a column direction is k*H*(1−|Y|)+1, where k is a real number greater than or equal to 1.

### Claim 8 (depends on 2)

The image processing method according to claim 2, wherein the determining pixel data of an anti-distortion image corresponding to the one row of pixels according to the anti-distortion parameters, and writing pixel data of the anti-distortion image to the anti-distortion buffer image comprises:
for each pixel in each row of pixel data in the input image:
determining a vector from a mapped point on the input image of the lens center to the pixel;
determining a size of the virtual image based on the object-image relationship and the size of the input image;
determining an image height of a corresponding pixel in the virtual image of the pixel according to the vector and the size of the virtual image;
determining, based on the object-image relationship, an object height of a corresponding pixel in the anti-distortion image of the pixel according to the image height of a corresponding pixel in the virtual image;
writing pixel data of the pixel to the to-be-buffered image according to the object height of the corresponding pixel in the anti-distortion image.

### Claim 9 (depends on 8)

The image processing method according to claim 8, wherein the writing pixel data of the pixel to the anti-distortion buffer image according to the object height of the corresponding pixel in the anti-distortion image comprises:
determining a corresponding pixel in the to-be-buffered image of the corresponding pixel according to the object height of the corresponding pixel in the anti-distortion image, and storing a grayscale value of the pixel in the corresponding pixel in anti-distortion buffer image.

### Claim 10 (depends on 8)

The image processing method according to claim 8, wherein the determining, for each pixel in each row of pixel data in the input image, a vector from a mapped point on the input image of the lens center to the pixel comprises:
determining a distance and a direction from the mapped point on the input image of the lens center to the pixel.

### Claim 11 (depends on 1)

The image processing method according to claim 1, further comprising:
outputting a first row of pixel data of the anti-distortion buffer image for displaying and clearing already-displayed pixel data, and
shift up each row of pixel data that is not yet displayed by one row in the anti-distortion buffer image.

### Claim 12 (depends on 11)

The image processing method according to claim 11, further comprising:
after the anti-distortion operation is performed on all the pixels of the input image, progressively outputting remaining image data in the anti-distortion buffer image for displaying.

### Claim 13

An image processing device, comprising:
a processor; and
a first memory, wherein the first memory stores instructions which, when executed by the processor, causes the processor to perform the an image processing method, wherein the method comprises:
determining a to-be-buffered image size according to a size of an input image and anti-distortion parameters of the lens unit; and
generating, based on the input image and the anti-distortion parameters, an anti-distortion buffer image having an image size according to the to-be-buffered image size,
wherein the determining a to-be-buffered image size according to a size of an input image and anti-distortion parameters of the lens unit comprises:
determining an anti-distortion grid for the input image based on the anti-distortion parameters, and coordinate values of four vertices of an anti-distortion image on the anti-distortion grid determined by using a center of the input image as an origin, and determining absolute values of the coordinate values of the four vertices in a column direction respectively;
determining a minimum absolute value Y among absolute values of the coordinate values of the four vertices in the column direction; and
determining the to-be-buffered image size according to the minimum absolute value Y; and
the image processing device further comprises a second memory configured to store a portion of the anti-distortion image.

### Claim 14 (depends on 13)

A virtual reality display apparatus, comprising:
a sensor configured to collect sensor data for determining a current state of the virtual reality display apparatus;
a display unit configured to receive an input image determined based on the sensor data and perform anti-distortion processing on the input image to generate pixel data of an anti-distortion image, and perform display driving according to pixel data of the anti-distortion image, wherein the display unit includes the image processing device according to claim 13; and
a lens unit configured to image an image driven for displaying by the display unit.

### Claim 15 (depends on 14)

The virtual reality display apparatus according to claim 14, wherein the sensor includes one or more of a speed sensor, an acceleration sensor, a geomagnetic sensor, a touch sensor, and a distance sensor.

### Claim 16 (depends on 14)

The virtual reality display apparatus according to claim 14, wherein the display unit includes one or more of a central processing unit, a graphics processing unit, an FPGA, an ASIC, and a CPLD.

### Claim 17 (depends on 14)

The virtual reality display apparatus according to claim 14, wherein the lens unit includes one or more of a convex lens, a Fresnel lens, and a concave lens.