# US 2024/0257481 A1 · Rerendering a position of a hand to decrease a size of a hand to create a realistic virtual/augmented reality environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0257481 A1
- Document ID: US20240257481A1
- Status: Published application
- Kind: A1
- Application number: 18/629,845
- Filed: 2024-04-08
- Published: 2024-08-01
- Earliest claimed: 2015-09-08
- Assignees: Ultrahaptics IP Two Limited
- Applicants: Ultrahaptics IP Two Limited
- Inventors: Alex Marcolina; David Holz
- Web page: https://patentgrep.com/patent/US20240257481A1

## CPC

- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G02B 27/0176: {characterised by mechanical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/012: {Head tracking input arrangements}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T 7/74: {involving reference images or patches}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 20/20: in augmented reality scenes
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G06T 2207/30196: Human being; Person
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The technology disclosed relates to a method of realistic rendering of a real object as a virtual object in a virtual space using an offset in the position of the hand in a three-dimensional (3D) sensory space. An offset between expected positions of the eye(s) of a wearer of a head mounted device and a sensor attached to the head mounted device for sensing a position of at least one hand in a three-dimensional (3D) sensory space is determined. A position of the hand in the three-dimensional (3D) sensory space can be sensed using a sensor. The sensed position of the hand can be transformed by the offset into a re-rendered position of the hand as would appear to the wearer of the head mounted device if the wearer were looking at the actual hand. The re-rendered hand can be depicted to the wearer of the head mounted device.

## Claims

### Claim 1

A system including:
a processor; and
a memory storing computer instructions that, when executed by the processor, perform operations comprising:
determining an offset between an eye of a user and an actual position of a sensor;
determining one or more control object properties;
obtaining a re-rendered position of a control object, the re-rendered position of the control object being a result of an application of a transformation matrix to a position of the control object determined in dependence on the one or more control object properties, the transformation matrix being based on the actual position of the sensor and for decreasing a rendered size of the control object (i) in accordance with the offset between the eye of the user and the actual position of the sensor and (ii) with respect to the control object being rendered without consideration of the offset; and
providing, for display to the user, a control object image using the re-rendered position of the control object.

### Claim 2 (depends on 1)

The system of claim 1, further configured to perform:
obtaining additional re-rendered positions of the control object, the additional re-rendered positions being based on the offset being applied to sensed additional positions of the control object, thereby providing the control object image having a motion or gesture of the control object.

### Claim 3 (depends on 2)

The system of claim 2, further configured to perform:
obtaining an indication of quality of tracking for the sensor, the indication of quality of tracking indicating whether the sensor is tracking a position or motion of the control object; and
adding to the control object image a visual indication corresponding to the indication of quality of tracking, thereby providing the user with a visual cue of how well the sensor is tracking the control object.

### Claim 4 (depends on 3)

The system of claim 3, wherein the indication of quality of tracking includes an indication that the sensor is able to recognize the control object in captured images.

### Claim 5 (depends on 3)

The system of claim 3, wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the control object image when the sensor is unable to recognize the control object in captured images.

### Claim 6 (depends on 1)

The system of claim 1, wherein the control object image comprises a non-rigid transformation based on an error indication applied after the transformation matrix is applied to the position of the control object.

### Claim 7

A method comprising:
determining an offset between an eye of a user and an actual position of a sensor;
determining one or more control object properties;
obtaining a re-rendered position of a control object, the re-rendered position of the control object being a result of an application of a transformation matrix to a position of the control object determined in dependence on the one or more control object properties, the transformation matrix being based on the actual position of the sensor and for decreasing a rendered size of the control object (i) in accordance with the offset between the eye of the user and the actual position of the sensor and (ii) with respect to the control object being rendered without consideration of the offset; and
providing, for display to the user, a control object image using the re-rendered position of the control object.

### Claim 8 (depends on 7)

The method of claim 7, further including:
obtaining additional re-rendered positions of the control object, the additional re-rendered positions being based on the offset being applied to sensed additional positions of the control object, thereby providing the control object image having a motion or gesture of the control object.

### Claim 9 (depends on 8)

The method of claim 8, further including:
obtaining an indication of quality of tracking for the sensor, the indication of quality of tracking indicating whether the sensor is tracking a position or motion of the control object; and
adding to the control object image a visual indication corresponding to the indication of quality of tracking, thereby providing the user with a visual cue of how well the sensor is tracking the control object.

### Claim 10 (depends on 9)

The method of claim 9, wherein the indication of quality of tracking includes an indication that the sensor is able to recognize the control object in captured images.

### Claim 11 (depends on 9)

The method of claim 9, wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the control object image when the sensor is unable to recognize the control object in captured images.

### Claim 12 (depends on 8)

The method of claim 8, wherein:
the control object image comprises a non-rigid transformation based on an error indication applied after the transformation matrix is applied to the position of the control object.

### Claim 13

A non-transitory computer-readable recording medium having instructions recorded thereon, the instructions, when executed by a processor, perform a method comprising:
determining an offset between an eye of a user and an actual position of a sensor;
determining one or more control object properties;
obtaining a re-rendered position of a control object, the re-rendered position of the control object being a result of an application of a transformation matrix to a position of the control object determined in dependence on the one or more control object properties, the transformation matrix being based on the actual position of the sensor and for decreasing a rendered size of the control object (i) in accordance with the offset between the eye of the user and the actual position of the sensor and (ii) with respect to the control object being rendered without consideration of the offset; and
providing, for display to the user, a control object image using the re-rendered position of the control object.

### Claim 14 (depends on 13)

The non-transitory computer-readable recording medium of claim 13, wherein the method further comprises:
obtaining additional re-rendered positions of the control object, the additional re-rendered positions being based on the offset being applied to sensed additional positions of the control object, thereby providing the control object image having a motion or gesture of the control object.

### Claim 15 (depends on 14)

The non-transitory computer-readable recording medium of claim 14, wherein the method further comprises:
obtaining an indication of quality of tracking for the sensor, the indication of quality of tracking indicating whether the sensor is tracking a position or motion of the control object; and
adding to the control object image a visual indication corresponding to the indication of quality of tracking, thereby providing the user with a visual cue of how well the sensor is tracking the control object.

### Claim 16 (depends on 15)

The non-transitory computer-readable recording medium of claim 15, wherein the indication of quality of tracking includes an indication that the sensor is able to recognize the control object in captured images.

### Claim 17 (depends on 15)

The non-transitory computer-readable recording medium of claim 15, wherein the visual indication corresponding to the indication of quality of tracking includes a colored outline circumscribed about the control object image when the sensor is unable to recognize the control object in captured images.

### Claim 18 (depends on 15)

The non-transitory computer-readable recording medium of claim 15, wherein the visual indication corresponding to the indication of quality of tracking includes coloring the control object image a different color when the sensor is able to recognize the control object in captured images than when the sensor is unable to recognize the control object in captured images.

### Claim 19 (depends on 14)

The non-transitory computer-readable recording medium of claim 14, wherein:
the control object image comprises a non-rigid transformation based on an error indication applied after the transformation matrix is applied to the position of the control object.

### Claim 20 (depends on 14)

The non-transitory computer-readable recording medium of claim 14, wherein the method further comprises:
sensing an indication of whether the control object has grabbed a virtual object; and
adding to the control object image a visual indication corresponding to the indication of whether the control object has grabbed the virtual object, thereby providing the user with a visual cue of whether the virtual object is grabbed.
