# US 2021/0333883 A1 · Hybrid wireless streaming solution, switching from 2d to 3d based on an intuitive hand gesture in virtual reality or augmented reality (ar) head-mounted displays (hmd)

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0333883 A1
- Document ID: US20210333883A1
- Status: Published application
- Kind: A1
- Application number: 16/857,761
- Filed: 2020-04-24
- Published: 2021-10-28
- Earliest claimed: 2020-04-24
- Assignees: Sony Interactive Entertainment Inc.
- Applicants: Sony Interactive Entertainment Inc.
- Inventors: Jeffrey R. Stafford
- Web page: https://patentgrep.com/patent/US20210333883A1

## CPC

- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- H04N 21/816: {involving special video data, e.g 3D video}
- A63F 13/20: Input arrangements for video game devices
- A63F 13/26: having at least one additional display device, e.g. on the game controller or outside a game booth
- A63F 13/42: by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
- A63F 13/50: Controlling the output signals based on the game progress
- G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
- H04N 21/4122: {additional display device, e.g. video projector (digital output for controlling a plurality of local displays G06F3/1423)}
- H04N 21/47: End-user applications
- H04N 21/4781: {Games}
- H04N 21/4854: {for modifying image parameters, e.g. image brightness, contrast}
- G02B 2027/0187: {slaved to motion of at least a part of the body of the user, e.g. head, eye}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS

## Abstract

A computer game can be viewed on a TV through an augmented reality (AR) head-mounted display (HMD). The wearer can make a gesture such as a first to cause the HMD to present 2D rectangles and 3D objects representing locations in the room that the game can be “presented” in. The wearer may then open his hand and “cast” toward one of the room locations, and the game is transferred to the HMD for presentation in the associated rectangle (in 2D) or 3D object (in 3D).

## Claims

### Claim 1

An assembly, comprising:
at least one processor configured with instructions executable to:
present a computer simulation on at least a first display;
receive user-generated input; and
responsive to the user-generated input, switch presenting the computer simulation to a head-mounted display (HMD) different from the first display.

### Claim 2 (depends on 1)

The assembly of claim 1, wherein the computer simulation comprises a computer game and the first display comprises a TV.

### Claim 3 (depends on 1)

The assembly of claim 1, wherein the input comprises at least one image of at least one gesture in free space.

### Claim 4 (depends on 1)

The assembly of claim 1, wherein the instructions are executable to:
present on the HMD at least one graphic corresponding to a location in a space at which the computer simulation can be emulated to be presented.

### Claim 5 (depends on 4)

The assembly of claim 4, wherein the instructions are executable to present on the HMD the graphic responsive to first input.

### Claim 6 (depends on 5)

The assembly of claim 5, wherein the graphic comprises a three-dimensional (3D) object and the instructions are executable to:
responsive to the user-generated input being associated with the 3D object, switch a presentation mode from an image-based mode to an object-based mode; and
stream objects in the object-based mode to the HMD for presentation of the objects on the HMD in in a portion of the HMD indicated by the 3D object.

### Claim 7 (depends on 6)

The assembly of claim 6, wherein the instructions are executable to:
pause video of the computer simulation between receiving the user-generated input and commencing streaming the objects to the HMD while continuing to play audio of the computer simulation.

### Claim 8 (depends on 4)

The assembly of claim 4, wherein the graphic comprises a two-dimensional (2D) object and the instructions are executable to:
compress information in an image-based mode; and
stream the information to the HMD for presentation of the information on the HMD in a portion of the HMD indicated by the 2D object.

### Claim 9 (depends on 1)

The assembly of claim 1, wherein the processor is implemented by a dongle.

### Claim 10 (depends on 6)

The assembly of claim 6, wherein the instructions are executable to:
stream objects in the object-based mode to the HMD based at least in part on a direction of gaze such that only objects corresponding to the direction of gaze are streamed.

### Claim 11

An assembly comprising:
at least one video display;
at least one head-mounted display (HMD); and
at least one processor configured with instructions for:
presenting a computer simulation on the video display; and
responsive to at least one hand gesture in free space, shifting presentation of the computer simulation from the video display to the HMD.

### Claim 12 (depends on 11)

The assembly of claim 11, wherein the hand gesture comprises a firsthand gesture and the instructions are executable for:
responsive to a secondhand gesture, present on the HMD visual indication of a location in a space to which presentation of the computer simulation may be emulated to be shifted.

### Claim 13 (depends on 12)

The assembly of claim 12, wherein the secondhand gesture is received prior to the firsthand gesture, and the instructions are executable for:
shifting presentation of the computer simulation from the video display to the HMD responsive to the firsthand gesture indicating the location in the space.

### Claim 14 (depends on 11)

The assembly of claim 11, wherein the computer simulation comprises a computer game and the video display comprises a TV.

### Claim 15 (depends on 12)

The assembly of claim 12, wherein the visual indication comprises a three-dimensional (3D) object and the instructions are executable for:
responsive to the firsthand gesture being associated with the location in the space, switching a presentation mode from an image-based mode to an object-based mode; and
streaming objects in the object-based mode to the HMD for presentation of the objects on the HMD in in a portion of the HMD indicated by the 3D object.

### Claim 16 (depends on 15)

The assembly of claim 15, wherein the instructions are executable for:
pausing the computer simulation between imaging the firsthand gesture and commencing streaming the objects to the HMD.

### Claim 17 (depends on 12)

The assembly of claim 12, wherein the visual indication comprises a two-dimensional (2D) object and the instructions are executable for:
compressing information in an image-based mode based at least on part on a direction of gaze; and
streaming the information to the HMD for presentation of the information on the HMD in a portion of the HMD indicated by the 2D object.

### Claim 18 (depends on 11)

The assembly of claim 11, wherein the processor is implemented by a dongle.

### Claim 19

A method, comprising:
presenting a video simulation on a video display;
identifying at least a first gesture; and
responsive to the first gesture, presenting the video simulation on at least one head-mounted display (HMD).

### Claim 20 (depends on 19)

The method of claim 19, comprising:
responsive to the first gesture, stop presenting the video simulation on the video display; and
responsive to receiving a second gesture prior to the first gesture, presenting on the HMD an indication of at least one location at which the video presentation may be presented.
