# US 10,311,647 B2 · Three dimensional image generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,311,647 B2
- Document ID: US10311647B2
- Status: Granted
- Kind: B2
- Application number: 15/226,315
- Filed: 2016-08-02
- Published: 2016-11-24
- Granted: 2019-06-04
- Earliest claimed: 2014-02-05
- Assignees: Gravity Sketch Limited
- Applicants: Gravity Sketch LIMITED
- Inventors: Daniela Maria Paredes Fuentes; Guillaume Couche; Pierre-Yves Paslier; Oluwaseyi Sosanya
- Web page: https://patentgrep.com/patent/US10311647B2

## CPC

- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures
- G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 3/60: Rotation of whole images or parts thereof
- G06T 15/08: Volume rendering
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Apparatus, method and computer program for use in generating a three dimensional image. A three dimensional image volume comprising a three dimensional image is displayed using a visualization device in a way that the position and orientation of an input surface is visible within the three dimensional image volume. User input indicative of a two dimensional image provided in the plane of the input surface is received via the input surface. First positional data indicative of a relative position and orientation of the input surface in relation to the three dimensional image volume is then determined. Subsequently, the three dimensional image is modified to include a three dimensional representation of the two dimensional image, the three dimensional representation having a position and orientation within the three dimensional image volume determined from the first positional data.

## Claims

### Claim 1

Apparatus for use in generating a three dimensional image, the apparatus comprising:
an input device comprising an input surface;
a visualization device,
a processing system; and
a memory,
wherein the memory comprises computer readable instructions which, when executed by the processing system, cause the apparatus to:
display, using the visualization device, a three dimensional image volume comprising a three dimensional image, the position and orientation of the input surface being visible within the three dimensional image volume;
receive first user input via the input surface, the first user input being indicative of a two dimensional image provided in a plane of the input surface;
determine first positional data indicative of a relative position and orientation of the input surface in relation to the three dimensional image volume;
modify the three dimensional image to include a three dimensional representation of the two dimensional image, the three dimensional representation having a position and orientation within the three dimensional image volume determined from the first positional data;
receive second user input indicative that at least one of a position or an orientation of the three dimensional image in relation to the input surface is to be changed;
in response to the second user input, change the at least one of the position or the orientation of the three dimensional image in relation to the input surface so as to display, using the visualization device:
a first part of the three dimensional image as if located on a near side of the input surface as viewed by a user; and
a second part of the three dimensional image as if located on a far side of the input surface as viewed by the user; and
during the display of the three dimensional image with the first part displayed as if located on the near side of the input surface as viewed by the user and the second part displayed as if located on the far side of the input surface as viewed by the user, receive third user input via the input surface,
wherein the third user input is indicative of a further two dimensional image provided in the plane of the input surface, which plane intersects the three dimensional image such that the first part of the three dimensional image is located at an opposite side of the plane to the second part of the three dimensional image.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the visualization device comprises an augmented reality display device configured to display the three dimensional image volume while enabling the user to see the input surface.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the visualization device comprises a virtual reality display device configured to display both the three dimensional image volume and a virtual rendering of the input surface, the virtual rendering of the input surface being positioned and orientated based on a physical position and orientation of the input surface.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein:
the visualization device comprises a first position detecting part;
the apparatus comprises a second position detecting part, separate from the first position detecting part, the first and second position detecting parts configured to cooperate to enable determination of a relative position and orientation of the visualization device in relation to the second position detecting part; and
the memory is configured to store second positional data indicative of a relative position and orientation, in relation to the second position detecting part, at which the three dimensional image volume is to be displayed,
wherein the computer readable instructions, when executed by the processing system, cause the apparatus to:
retrieve, from memory, stored second positional data;
determine, using the first and second position detecting parts, third positional data indicative of a relative position and orientation of the visualization device in relation to the second position detecting part; and
calculate, using the second and third positional data, a relative position and orientation for the three dimensional image volume in relation to the visualization device for use in the displaying of the three dimensional image volume.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the computer readable instructions, when executed by the processing system, cause the apparatus to modify the stored second positional data in accordance with the second user input.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the second user input is received via the input surface.

### Claim 7 (depends on 5)

The apparatus of claim 5, wherein the apparatus comprises one or more control elements configured to receive the second user input.

### Claim 8 (depends on 5)

The apparatus of claim 5, wherein the second position detecting part and the input surface are physically connected whereby to have a constant relative position and orientation.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the second user input is indicative of one or more of:
a rotation of the three dimensional image volume about an axis in the plane of the image surface;
a rotation of the three dimensional image volume about an axis perpendicular to the plane of the image surface;
a translation of the three dimensional image volume in a direction in the plane of the image surface;
a translation of the three dimensional image volume along a direction perpendicular to the plane of the image surface; and
a change in a relative size of the three dimensional image volume.

### Claim 10 (depends on 4)

The apparatus of claim 4, wherein the computer readable instructions, when executed by the processing system, cause the apparatus to:
receive fourth user input identifying a two dimensional image within the three dimensional image; and
modify the second positional data such that the three dimensional image volume is displayed with the two dimensional image identified by the fourth user input aligned with the input surface.

### Claim 11 (depends on 4)

The apparatus of claim 4, wherein the apparatus comprise a third position detecting part connected to the input surface, and the second and third position detecting parts are arranged to cooperate to enable detection of a relative position and orientation of the input surface in relation to the second position detecting part, and the first positional data is determined using the second and third position detecting parts.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the input surface is connected to a stabilization mechanism.

### Claim 13 (depends on 4)

The apparatus of claim 4, wherein the apparatus comprises a plurality of second position detecting parts, and the computer readable instructions, when executed by the processing system, cause the apparatus to:
use third positional data determined using one of the second position detecting parts to display the three dimensional image;
receive fifth user input identifying an other one of the second position detecting parts; and
use third positional data determined using the other one of the second position detecting parts to display the three dimensional image.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the computer readable instructions, when executed by the processing system, cause the apparatus to calculate second positional data based on the relative positions of the one and the other one of the second position detecting parts.

### Claim 15 (depends on 1)

The apparatus of claim 1, wherein the computer readable instructions, when executed by the processing system, cause the apparatus to
during the display of the three dimensional image with the first part displayed as if located on the near side of the input surface as viewed by the user and the second part displayed as if located on the far side of the input surface as viewed by the user, display the three dimensional image within the image volume such that the first part is distinguishable from the second part.

### Claim 16 (depends on 1)

The apparatus of claim 1, wherein the computer readable instructions, when executed by the processing system, cause the apparatus to:
identify a third part of the three dimensional image which is to be displayed as if located on, or within a predetermined distance of, the input surface; and
display the three dimensional image within the three dimensional image volume such that the third part is distinguishable from parts of the three dimensional image other than the third part.

### Claim 17 (depends on 1)

The apparatus of claim 1, wherein the apparatus comprises an interface configured to receive input image data for display using the visualization device.

### Claim 18 (depends on 1)

The apparatus of claim 1, wherein the apparatus comprises an interface configured to output image data for utilization by an external system.

### Claim 19 (depends on 1)

The apparatus of claim 1, wherein the apparatus comprises a plurality of visualization devices, and wherein the apparatus is configured to display the same three dimensional image volume using each of the plurality of visualization devices.

### Claim 20

A method for use in generating a three dimensional image, the method comprising:
displaying, using a visualization device, a three dimensional image volume comprising a three dimensional image, the position and orientation of an input surface of an input device being visible within the three dimensional image volume;
receiving first user input via the input surface, the first user input being indicative of a two dimensional image provided in a plane of the input surface;
determining first positional data indicative of a relative position and orientation of the input surface in relation to the three dimensional image volume;
modifying the three dimensional image to include a three dimensional representation of the two dimensional image, the three dimensional representation having a position and orientation within the three dimensional image volume determined from the first positional data;
receiving second user input indicative that at least one of a position or an orientation of the three dimensional image in relation to the input surface is to be changed;
in response to the second user input, changing the at least one of the position or the orientation of the three dimensional image in relation to the input surface so as to display, using the visualization device:
a first part of the three dimensional image as if located on a near side of the input surface as viewed by a user; and
a second part of the three dimensional image as if located on a far side of the input surface as viewed by the user; and
during the display of the three dimensional image with the first part displayed as if located on the near side of the input surface as viewed by the user and the second part displayed as if located on the far side of the input surface as viewed by the user, receiving third user input via the input surface,
wherein the third user input is indicative of a further two dimensional image provided in the plane of the input surface, which plane intersects the three dimensional image such that the first part of the three dimensional image is located at an opposite side of the plane to the second part of the three dimensional image.

### Claim 21

A non-transitory computer-readable storage medium comprising a set of computer-readable instructions stored thereon, which, when executed by a processing system, cause the processing system to control performance of a method for use in generating a three dimensional image, the method comprising:
displaying, using a visualization device, a three dimensional image volume comprising a three dimensional image, the position and orientation of an input surface of an input device being visible within the three dimensional image volume;
receiving first user input via the input surface, the first user input being indicative of a two dimensional image provided in a plane of the input surface;
determining first positional data indicative of a relative position and orientation of the input surface in relation to the three dimensional image volume;
modifying the three dimensional image to include a three dimensional representation of the two dimensional image, the three dimensional representation having a position and orientation within the three dimensional image volume determined from the first positional data;
receiving second user input indicative that at least one of a position or an orientation of the three dimensional image in relation to the input surface is to be changed;
in response to the second user input, changing the at least one of the position or the orientation of the three dimensional image in relation to the input surface so as to display, using the visualization device:
a first part of the three dimensional image as if located on a near side of the input surface as viewed by a user; and
a second part of the three dimensional image as if located on a far side of the input surface as viewed by the user; and
during the display of the three dimensional image with the first part displayed as if located on the near side of the input surface as viewed by the user and the second part displayed as if located on the far side of the input surface as viewed by the user, receiving third user input via the input surface,
wherein the third user input is indicative of a further two dimensional image provided in the plane of the input surface, which plane intersects the three dimensional image such that the first part of the three dimensional image is located at an opposite side of the plane to the second part of the three dimensional image.