# US 2021/0063735 A1 · Method and device for controlling image display in a vr system, and vr head mounted device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0063735 A1
- Document ID: US20210063735A1
- Status: Published application
- Kind: A1
- Application number: 16/631,136
- Filed: 2019-08-01
- Published: 2021-03-04
- Earliest claimed: 2018-12-29
- Assignees: Goertek Inc.
- Applicants: Goertek Inc.
- Inventors: Lei CAI; Tianrong DAI
- Web page: https://patentgrep.com/patent/US20210063735A1

## CPC

- G02B 27/0101: {characterised by optical features (G02B27/0172 takes precedence)}
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G06F 3/01: Input arrangements or combined input and output arrangements for interaction between user and computer (G06F3/16 takes precedence)
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06T 3/18: Image warping, e.g. rearranging pixels individually
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 15/00: Three-dimensional [3D] image rendering
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- H04N 13/261: with monoscopic-to-stereoscopic image conversion
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20: Special algorithmic details
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 13/344: with head-mounted left-right displays
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method and device for controlling image display in a VR system, and a VR head mounted device. The method comprises: monitoring a synchronization signal of an image frame in a VR system and acquiring an original 2D image; sampling sensor data at a preset time point before a next synchronization signal arrives to obtain latest pose information of a tracked object; converting the original 2D image into a corresponding 3D image, and calculating a motion vector corresponding to each pixel point of the 3D image according to the latest pose information and pose information corresponding to the 3D image; performing position transformation on pixel points of the original 2D image based on the motion vector, and filling at pixel points of a vacant area appearing after the position transformation to obtain a target frame; and triggering display of the target frame when the next synchronization signal arrives.

## Claims

### Claim 1

A method for controlling image display in a VR system, comprising:
monitoring a synchronization signal of an image frame in the VR system and acquiring an original 2D image;
sampling sensor data at a preset time point before a next synchronization signal arrives to obtain latest pose information of a tracked object, wherein the pose information includes information indicating rotation of the tracked object and information indicating translation of the tracked object;
converting the original 2D image into a corresponding 3D image, and calculating a motion vector corresponding to each pixel point of the 3D image according to the latest pose information and pose information corresponding to the 3D image;
performing position transformation with respect to pixel points of the original 2D image based on the motion vector, and filling at pixel points of a vacant area appearing after the position transformation to obtain a target frame; and
triggering display of the target frame when the next synchronization signal arrives.

### Claim 2 (depends on 1)

The method according to claim 1, wherein calculating a motion vector corresponding to each pixel point of the 3D image according to the latest pose information and pose information corresponding to the 3D image comprises:
calculating a corresponding original position of each pixel point of the 3D image in a 3D space corresponding to the original 2D image, by using an inverse matrix of a matrix used in 3D to 2D spatial transformation;
calculating a corresponding new position of each pixel point in the 3D space according to an offset between the latest pose information and the pose information corresponding to the 3D image; and
calculating to obtain the motion vector corresponding to each pixel point of the 3D image by using the original position and the new position of each pixel point in the 3D space.

### Claim 3 (depends on 2)

The method according to claim 2, wherein calculating a corresponding original position of each pixel point of the 3D image corresponding to the original 2D image in a 3D space by using an inverse matrix of a matrix used in 3D to 2D spatial transformation comprises:
acquiring horizontal position information, vertical position information, and depth information of each pixel point of the original 2D image, and obtaining a position vector of each pixel point of the original 2D image; and
calculating a corresponding original position of each pixel point of the 3D image in a 3D space corresponding to the original 2D image, by using an inverse matrix of a matrix used in 3D to 2D spatial transformation and the position vector of each pixel point.

### Claim 4 (depends on 1)

The method according to claim 1, wherein performing position transformation with respect to pixel points of the original 2D image based on the motion vector, and filling at pixel points of a vacant area appearing after the position transformation to obtain a target frame comprises:
selecting a part of the pixel points from the pixel points of the original 2D image as key pixel points; and
performing position transformation with respect to each selected key pixel point based on a size and a direction indicated by the motion vector, and filling at pixel points of a vacant area appearing after the position transformation to obtain a target frame.

### Claim 5 (depends on 4)

The method according to claim 4, wherein selecting a part of the pixel points from the pixel points of the original 2D image as key pixel points comprises:
dividing the original 2D image into a plurality of regular grids, and selecting pixel points corresponding to grid vertices as key pixel points.

### Claim 6 (depends on 5)

The method according to claim 5, wherein filling at pixel points of a vacant area appearing after the position transformation comprises:
determining a vacant area in an area enclosed by grid vertices after the position transformation; and
filling pixel points of the vacant area by interpolation.

### Claim 7 (depends on 1)

The method according to claim 1, wherein sampling sensor data to obtain latest pose information of a tracked object comprises:
sampling data of an inertial measurement unit (IMU) of a VR system to obtain latest pose information of a user's head.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the original 2D image is divided into 200×100 regular grids.

### Claim 9

A device for controlling image display in a VR system, comprising:
an acquisition module, for monitoring a synchronization signal of an image frame in a VR system and acquiring an original 2D image;
a sampling module, for sampling sensor data at a preset time point before a next synchronization signal arrives to obtain latest pose information of an tracked object, wherein the pose information includes information indicating rotation of the tracked object and information indicating translation of the tracked object;
a vector calculation module for converting the original 2D image into a corresponding 3D image, and calculating a motion vector corresponding to each pixel point of the 3D image according to the latest pose information and pose information corresponding to the 3D image;
a target frame generation module, for performing position transformation with respect to pixel points of the original 2D image based on the motion vector, and filling at pixel points of a vacant area appearing after the position transformation to obtain a target frame; and
a triggering module, for triggering display of the target frame when the next synchronization signal arrives.

### Claim 10 (depends on 9)

The device according to claim 9, wherein the vector calculation module is specifically for
calculating a corresponding original position of each pixel point of the 3D image in a 3D space corresponding to the original 2D image, by using an inverse matrix of a matrix used in 3D to 2D spatial transformation;
calculating a corresponding new position of each pixel point in the 3D space according to an offset between the latest pose information and the pose information corresponding to the 3D image; and
calculating to obtain the motion vector corresponding to each pixel point of the 3D image by using the original position and the new position of each pixel point in the 3D space.

### Claim 11 (depends on 10)

The device according to claim 10, wherein the vector calculation module is specifically for
acquiring horizontal position information, vertical position information, and depth information of each pixel point of the original 2D image, and obtaining a position vector of each pixel point of the original 2D image; and
calculating a corresponding original position of each pixel point of the 3D image corresponding to the original 2D image in a 3D space, by using an inverse matrix of a matrix used in 3D to 2D spatial transformation and the position vector of each pixel point.

### Claim 12 (depends on 9)

The device according to claim 9, wherein the target frame generating module is specifically for
selecting a part of the pixel points from the pixel points of the original 2D image as key pixel points; and
performing position transformation with respect to each selected key pixel point based on a size and a direction indicated by the motion vector, and filling at pixel points of a vacant area appearing after the position transformation to obtain a target frame.

### Claim 13 (depends on 9)

The device according to claim 9, wherein the target frame generating module is specifically for dividing the original 2D image into a plurality of regular grids, and selecting pixel points corresponding to grid vertices as key pixel points.

### Claim 14 (depends on 13)

The device according to claim 13, wherein the target frame generating module is specifically for determining a vacant area in an area enclosed by grid vertices after the position transformation; and filling pixel points of the vacant area by interpolation.

### Claim 15 (depends on 13)

The device according to claim 13, wherein the target frame generating module is specifically dividing the original 2D image into 200×100 regular grids.

### Claim 16 (depends on 1)

A VR head mounted device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected by an internal bus, the memory stores program instructions executable by the processor which enable to implement the method according to claim 1 when executed by the processor.

### Claim 17 (depends on 16)

The VR head mounted device according to claim 16, wherein calculating a motion vector corresponding to each pixel point of the 3D image according to the latest pose information and pose information corresponding to the 3D image comprises:
calculating a corresponding original position of each pixel point of the 3D image in a 3D space corresponding to the original 2D image, by using an inverse matrix of a matrix used in 3D to 2D spatial transformation;
calculating a corresponding new position of each pixel point in the 3D space according to an offset between the latest pose information and the pose information corresponding to the 3D image; and
calculating to obtain the motion vector corresponding to each pixel point of the 3D image by using the original position and the new position of each pixel point in the 3D space.

### Claim 18 (depends on 17)

The VR head mounted device according to claim 17, wherein calculating a corresponding original position of each pixel point of the 3D image corresponding to the original 2D image in a 3D space by using an inverse matrix of a matrix used in 3D to 2D spatial transformation comprises:
acquiring horizontal position information, vertical position information, and depth information of each pixel point of the original 2D image, and obtaining a position vector of each pixel point of the original 2D image; and
calculating a corresponding original position of each pixel point of the 3D image in a 3D space corresponding to the original 2D image, by using an inverse matrix of a matrix used in 3D to 2D spatial transformation and the position vector of each pixel point.

### Claim 19 (depends on 16)

The VR head mounted device according to claim 16, wherein performing position transformation with respect to pixel points of the original 2D image based on the motion vector, and filling at pixel points of a vacant area appearing after the position transformation to obtain a target frame comprises:
selecting a part of the pixel points from the pixel points of the original 2D image as key pixel points; and
performing position transformation with respect to each selected key pixel point based on a size and a direction indicated by the motion vector, and filling at pixel points of a vacant area appearing after the position transformation to obtain a target frame.

### Claim 20 (depends on 19)

The VR head mounted device according to claim 19, wherein selecting a part of the pixel points from the pixel points of the original 2D image as key pixel points comprises:
dividing the original 2D image into a plurality of regular grids, and selecting pixel points corresponding to grid vertices as key pixel points;
and wherein filling at pixel points of a vacant area appearing after the position transformation comprises:
determining a vacant area in an area enclosed by grid vertices after the position transformation; and
filling pixel points of the vacant area by interpolation.
