# US 2016/0252955 A1 · Image processing device, image processing method, computer program, and image display system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0252955 A1
- Document ID: US20160252955A1
- Status: Published application
- Kind: A1
- Application number: 15/026,766
- Filed: 2014-11-04
- Published: 2016-09-01
- Earliest claimed: 2013-12-25
- Assignees: Sony Corporation
- Applicants: Sony Corporation
- Inventors: Yuki Yamamoto; Kenta Kawamoto; Yoichi Hirota
- Web page: https://patentgrep.com/patent/US20160252955A1

## CPC

- G06F 3/012: {Head tracking input arrangements}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06F 3/147: using display panels
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 11/20: Drawing from basic elements
- G06T 11/60: Creating or editing images; Combining images with text
- G09G 5/003: {Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto (specific for a CRT G09G1/165; for a flat panel G09G3/2092)}
- H04N 5/74: Projection arrangements for image reproduction, e.g. using eidophor
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G09G 3/003: {to produce spatial visual effects}
- G09G 2370/12: Use of DVI or HDMI protocol in interfaces along the display data pipeline
- G09G 2370/16: Use of wireless transmission of display information
- H04N 5/145: {Movement estimation (for video coding H04N19/51)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

An image following a movement of a head of a user who observes an image is generated. At a drawing device 300, posture angle data to be received at a certain time after a time series of posture angle data that have already been received are predicted, and rendering processing is performed on an image based on the predicted posture angle data. A prediction algorithm for the posture angle data is roughly divided into two steps: (1) estimating a predicted time to receive the posture angle data; and (2) predicting the posture angle data at the predicted time. By rendering the image at the predicted time from the predicted posture angle data, a sense of delay felt when the image following the movement of the head of the user is presented is reduced.

## Claims

### Claim 1

An image processing device comprising:
a receiving unit that receives time series data from an external device;
a predicted time calculation unit that calculates a predicted time from a time when the latest one of previously received data is generated in the external device to a time when a next image is displayed;
a time series data prediction unit that predicts time series data at the predicted time; and
a drawing processing unit that renders an image based on the time series data predicted by the prediction unit.

### Claim 2 (depends on 1)

The image processing device according to claim 1, wherein
the time series data prediction unit includes a configuration to interpolate between two pieces of time series data, and extrapolates a portion ahead of the two pieces of time series data to predict time series data ahead by the predicted time.

### Claim 3 (depends on 1)

The image processing device according to claim 1, further comprising a sending unit that sends the rendered image to a display device.

### Claim 4 (depends on 1)

The image processing device according to claim 1, further comprising a display unit that displays the rendered image.

### Claim 5 (depends on 1)

The image processing device according to claim 1, wherein
the external device tracks a position or a posture of a part or all of a body of a user who observes an image,
the receiving unit receives, from the external device, posture information as the time series data, and
the drawing processing unit renders, based on the posture information, an image following a movement of a part or all of the body of the user.

### Claim 6 (depends on 5)

The image processing device according to claim 5, wherein
the predicted time calculation unit calculates the predicted time based on a total delay amount from when the posture information is sent from the external device to when an image that is based on the posture information is displayed.

### Claim 7 (depends on 6)

The image processing device according to claim 6, wherein
the total delay amount includes a transmission delay that occurs when the posture information is transmitted from the external device, a rendering delay that occurs when the image is rendered based on the posture information, and a display delay that occurs when the rendered image is displayed.

### Claim 8 (depends on 6)

The image processing device according to claim 6, wherein
the predicted time calculation unit divides the total delay amount into a constant delay that is measurable and a variant delay that is unmeasurable to calculate the total delay amount.

### Claim 9 (depends on 8)

The image processing device according to claim 8, wherein
the variant delay includes a variation in a transmission delay.

### Claim 10 (depends on 5)

The image processing device according to claim 5, wherein
the receiving unit receives, from the external device, a quaternion representing the position or the posture of a part or all of the body of the user as the time series data.

### Claim 11 (depends on 10)

The image processing device according to claim 10, wherein
the time series data prediction unit includes a configuration to perform a linear interpolation on two quaternions, and extrapolates a portion ahead of the two quaternions to predict a quaternion ahead by the predicted time.

### Claim 12 (depends on 10)

The image processing device according to claim 10, wherein
the time series data prediction unit includes a configuration to perform a spherical linear interpolation on two quaternions, and extrapolates a portion ahead of the two quaternions to predict a quaternion ahead by the predicted time.

### Claim 13

An image processing device comprising:
a receiving unit that receives time series data from an external device;
a predicted time calculation unit that calculates a predicted time from a time added, in the external device, to the latest one of previously received data to a time when a next image is displayed;
a time series data prediction unit that predicts time series data at the predicted time; and
a drawing processing unit that renders an image based on the time series data predicted by the prediction unit.

### Claim 14

An image processing device comprising:
a receiving unit that receives time series data from an external device;
a predicted time calculation unit that calculates a predicted time from the latest time of sending previously received data from the external device to a time when a next image is displayed;
a time series data prediction unit that predicts time series data at the predicted time; and
a drawing processing unit that renders an image based on the time series data predicted by the prediction unit.

### Claim 15

An image processing method comprising:
a receiving step of receiving time series data from an external device;
a predicted time calculation step of calculating a predicted time from a time when the latest one of previously received data is generated in the external device to a time when a next image is displayed;
a time series data prediction step of predicting time series data at the predicted time; and
a drawing processing step of rendering an image based on the time series data predicted in the prediction step.

### Claim 16

A computer program described in a computer-readable form so as to cause a computer to function as:
a receiving unit that receives time series data from an external device;
a predicted time calculation unit that calculates a predicted time from a time when the latest one of previously received data is generated in the external device to a time when a next image is displayed;
a time series data prediction unit that predicts time series data at the predicted time; and
a drawing processing unit that renders an image based on the time series data predicted by the prediction unit.

### Claim 17

An image display system comprising:
an external device that sends time series data;
a drawing device that receives the time series data and renders an image; and
a display device that displays the image rendered by the drawing device, wherein
the drawing device predicts a predicted time from a time when the latest one of previously received data is generated in the external device to a time when a next image is displayed, predicts time series data at the predicted time, and renders the image based on the predicted time series data.

### Claim 18 (depends on 17)

The image display system according to claim 17, wherein
the external device is a tracking device that tracks a posture of a part or all of a body of a user who observes a display image on the display unit, and sends posture information, and
the drawing device renders, based on the received posture information, an image following a movement of a part or all of the body of the user.

### Claim 19 (depends on 17)

The image display system according to claim 17, wherein
the tracking device sends a quaternion representing a position or a posture of a part or all of a body of a user, and
the drawing device predicts a quaternion ahead by the predicted time by extrapolating a portion ahead of two quaternions to perform a spherical linear interpolation on the two quaternions, and renders the image based on the predicted quaternion.
