# US 2022/0044480 A1 · System for three dimensional extension with multiscreen extended reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0044480 A1
- Document ID: US20220044480A1
- Status: Published application
- Kind: A1
- Application number: 16/986,735
- Filed: 2020-08-06
- Published: 2022-02-10
- Earliest claimed: 2020-08-06
- Assignees: AT&T Intellectual Property I, L.P.
- Applicants: AT&T Intellectual Property I, L.P.
- Inventors: Eric Zavesky; James Pratt
- Web page: https://patentgrep.com/patent/US20220044480A1

## CPC

- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06T 1/0021: {Image watermarking}
- G06T 15/20: Perspective computation
- G06V 10/94: Hardware or software architectures specially adapted for image or video understanding
- G06V 20/42: {of sport video content}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Aspects of the subject disclosure may include, for example, receiving a content item for displaying in a display system, the display system having a viewing plane, identifying an object in the content item, initiating an immersive experience for a user, wherein the immersive experience is based on the identifying an object and wherein the immersive experience includes a virtual object based on the object and presented in a virtual environment with the user and appearing to the user to move from the viewing plane of the display system into the virtual environment with the user, and providing data defining the immersive experience to extended reality hardware for interaction by the user. Other embodiments are disclosed.

## Claims

### Claim 1

A device, comprising:
a processing system including a processor; and
a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising:
receiving a content item for displaying in a display system, the display system having a viewing plane;
identifying an object in the content item;
initiating an immersive experience for a user, wherein the immersive experience is based on the identifying of the object and wherein the immersive experience includes a virtual object presented in a virtual environment with the user and appearing to the user to move from the viewing plane of the display system into the virtual environment with the user, wherein the virtual object is based on the object; and
providing data defining the immersive experience to extended reality hardware for interaction by the user.

### Claim 2 (depends on 1)

The device of claim 1, wherein the operations further comprise:
detecting a visual element in the content item;
based on the visual element, detecting the object; and
based on the detecting the object, computing features and movement of the virtual object as an extension of the object in the content item.

### Claim 3 (depends on 2)

The device of claim 2, wherein the operations further comprise:
based on the visual element, retrieving from storage a model of the object to reduce computational requirements for computing the features and movement of the virtual object; and
based on the model, determining the features and movement of the virtual object.

### Claim 4 (depends on 2)

The device of claim 2, wherein the detecting a visual element in the content item comprises:
identifying a digital watermark in a content stream forming the content item; and
determining, from the digital watermark, features of the object.

### Claim 5 (depends on 1)

The device of claim 1, wherein the receiving a content item for displaying in a display system comprises receiving data defining a two-dimensional video presentation for display on the display system.

### Claim 6 (depends on 5)

The device of claim 5, wherein the data defining a two-dimensional video presentation comprises data defining a sports program and wherein the object comprises a game piece of the sports program, the game piece having an apparent trajectory and an apparent velocity in the sports program, and wherein virtual object comprises a virtual representation of the game piece appearing in the immersive experience to move from the viewing plane of the display system into the virtual environment with the apparent trajectory and the apparent velocity of the game piece as viewed in the sports program.

### Claim 7 (depends on 6)

The device of claim 6, wherein the operations further comprise:
identifying one or more objects in the virtual environment with the user; and
modifying the apparent trajectory, the apparent velocity of the virtual representation of the game piece based on the identifying the one or more objects.

### Claim 8 (depends on 1)

The device of claim 1, wherein the operations further comprise:
identifying a scene change in the content item; and
updating appearance of the virtual object in the virtual environment based on the scene change.

### Claim 9

A machine-readable medium, comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising:
receiving information defining a two-dimensional video content item;
displaying the two-dimensional video content item on a two-dimensional display system having a viewing plane;
identifying an object in the two-dimensional video content item, the object being viewable by a user in a video presentation on the two-dimensional display system;
initiating an immersive experience for displaying to a user on extended reality hardware operated in conjunction with the two-dimensional display system, wherein the immersive experience includes presenting on the extended reality hardware a virtual object presented in a virtual environment with the user and appearing to the user to move from the viewing plane of the two-dimensional display system into the virtual environment with the user, wherein the virtual object is based on the object;
providing, to the extended reality hardware, data defining the immersive experience;
receiving user interaction with the virtual object in the virtual environment;
modifying an aspect of the virtual object responsive to the user interaction with the virtual object; and
providing, to the extended reality hardware, updated data defining a modified aspect of the virtual object.

### Claim 10 (depends on 9)

The machine-readable medium of claim 9, wherein the operations further comprise:
identifying one or more physical objects in a physical environment with the user, the virtual environment with the user of the immersive experience being overlaid on the physical environment with the user;
identifying an interaction between the virtual object and the one or more physical objects;
modifying a further aspect of the virtual object responsive to the interaction between the virtual object and the one or more physical objects; and
providing, to the extended reality hardware, updated data defining the modified further aspect of the virtual object.

### Claim 11 (depends on 10)

The machine-readable medium of claim 10, wherein the identifying an interaction between the virtual object and the one or more physical objects comprises:
determining a motion of the virtual object in the virtual environment that will position the virtual object inside a physical object of the one or more physical objects; and
altering the motion of the virtual object responsive to the determining the motion.

### Claim 12 (depends on 10)

The machine-readable medium of claim 10, wherein the identifying the one or more physical objects in a physical environment with the user comprises:
viewing the physical environment with a computer vision system;
locating the one or more physical objects;
characterizing the one or more physical objects; and
storing a three-dimensional map of the physical environment.

### Claim 13 (depends on 9)

The machine-readable medium of claim 9, wherein the identifying an object in the two-dimensional video content item comprises:
detecting a visual element in the two-dimensional video content item;
decoding the visual element to determine decoded information about the object; and
based on the decoded information, identifying the object in the two-dimensional video content item.

### Claim 14 (depends on 9)

The machine-readable medium of claim 9, wherein the providing the data defining the immersive experience comprises:
providing data defining a three-dimensional extended reality immersive experience to an extended reality headset of a user positioned before the two-dimension display system, the data defining a three-dimensional extended reality immersive experience operative to create images for the user on the extended reality headset corresponding to the two-dimensional video content item displayed on the two-dimensional display system so that objects moving in the two-dimensional video content item displayed on the two-dimensional display system appear to move toward the user in the three-dimensional extended reality immersive experience viewed by the user in the extended reality headset.

### Claim 15

A method, comprising:
receiving, by a processing system including a processor, a video content item for display to a user on a two-dimensional display system, the two-dimensional display system having a viewing plane;
identifying, by the processing system, a moving object in the video content item when the video content item is displayed on the two-dimensional display system, wherein the moving object has visual properties, and wherein the moving object appears to move toward the viewing plane of the two-dimensional display system;
generating, by the processing system, a virtual object having visual properties matching the visual properties of the moving object; and
providing, by the processing system, data defining the virtual object to extended reality hardware viewable by the user together with the two-dimensional display system, the extended reality hardware displaying a three-dimensional virtual environment to the user in a physical environment spaced from the viewing plane of the two-dimensional display system, the data defining the virtual object causing the virtual object to appear to move from the viewing plane of the two-dimensional display system the two toward the user in the three-dimensional virtual environment.

### Claim 16 (depends on 15)

The method of claim 15, wherein the identifying the moving object in the video content item comprises:
receiving, by the processing system, metadata, wherein the metadata is associated with the video content item; and
determining, by the processing system, information about the moving object responsive to the metadata and operates to reduce an amount of time required to identify the moving object in the video content item.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
identifying, by the processing system, one or more physical objects, including the user, in the physical environment;
identifying, by the processing system, an interaction between the virtual object and the one or more physical objects;
modifying, by the processing system, a motion of the virtual object in response to the interaction between the virtual object and the one or more physical objects; and
providing, by the processing system, updated data defining the virtual object to extended reality hardware, wherein the updated data reflects the modifying the motion of the virtual object.

### Claim 18 (depends on 17)

The method of claim 17, comprising:
providing, by the processing system, haptic responses to the user, the haptic responses based on and interaction between the virtual object and the user.

### Claim 19 (depends on 15)

The method of claim 15, comprising:
synchronizing, by the processing system, the three-dimensional virtual environment displayed to the user by the extended reality hardware and the video content item displayed to the user on the two-dimensional display system, wherein the synchronizing comprises synchronizing in time and in point of view.

### Claim 20 (depends on 15)

The method of claim 15, comprising:
displaying, by the processing system, the video content item on the two-dimensional display system and the three-dimensional virtual environment on the extended reality hardware to a plurality of viewers including the user, wherein the plurality of viewers are located in the physical environment to view the two-dimensional display system and wherein the extended reality hardware comprises a plurality of extended reality headsets, each respective extended reality headset worn by a respective viewer of the plurality of viewers.