Multi-screen interactions in virtual and augmented reality

A computer program product for providing multi-screen interaction in an interactive application, the computer program product comprising:

• a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation, the operation comprising:
• establishing an application session using a primary user device, the application session comprising a virtual environment relating to a physical environment in which the primary user device is located, wherein the primary user device is configured to display one or more graphical images relating to a primary virtual camera view of the virtual environment;
• determining a physical location of a secondary user device relative to the primary user device based on identifying a virtual glyph using the secondary user device, wherein the virtual glyph comprises a computer generated symbol generated using the primary user device, wherein the virtual glyph is generated to be visible at both the primary user device and the secondary user device, and wherein the virtual glyph is configured to be visible to the secondary user device through a transparent display of the primary user device;
• determining a secondary virtual camera view of the virtual environment, wherein the secondary virtual camera view relates to the primary virtual camera view and is based on the determined physical location of the secondary user device relative to the primary user device;
• transmitting image data relating to the secondary virtual camera view from the primary user device to the secondary user device, wherein the secondary user device is configured to display a first graphical image relating to the secondary virtual camera view based on the image data;
• and
• responsive to receiving an interaction request from the secondary user device, controlling, by the primary user device, the application session based on the interaction request,
• wherein the secondary user device is further configured to:
• generate a second graphical image relating to the secondary virtual camera view, based on data received from an Inertial Measurement Unit (IMU) at the secondary user device, and
• display the second graphical image.

A system, comprising:

• a processor;
• and
• a memory storing a program, which, when executed on the processor, performs an operation, the operation comprising:
• establishing an application session using a primary user device, the application session comprising a virtual environment relating to a physical environment in which the primary user device is located, wherein the primary user device is configured to display one or more graphical images relating to a primary virtual camera view of the virtual environment;
• determining a physical location of a secondary user device relative to the primary user device based on identifying a virtual glyph using the secondary user device, wherein the virtual glyph comprises a computer generated symbol generated using the primary user device, wherein the virtual glyph is generated to be visible at both the primary user device and the secondary user device, and wherein the virtual glyph is configured to be visible to the secondary user device through a transparent display of the primary user device;
• determining a secondary virtual camera view of the virtual environment, wherein the secondary virtual camera view relates to the primary virtual camera view and is based on the determined physical location of the secondary user device relative to the primary user device;
• transmitting image data relating to the secondary virtual camera view from the primary user device to the secondary user device, wherein the secondary user device is configured to display a first graphical image relating to the secondary virtual camera view based on the image data;
• and
• responsive to receiving an interaction request from the secondary user device, controlling, by the primary user device, the application session based on the interaction request,
• wherein the secondary user device is further configured to:
• generate a second graphical image relating to the secondary virtual camera view, based on data received from an Inertial Measurement Unit (IMU) at the secondary user device, and
• display the second graphical image.