US 2019/0102135 A1Published application

Scalable interaction with multi-displays

AssigneeSummit Technology Laboratory

InventorsDuy-Quoc Lai, Mehdi Rahimzadeh

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems and methods for multiple users to interact with a multi-display using multiple modalities. The multi-display allows a lucid transition between personal non-private work environment and shared work environment for multiple groups of users in an open workspace. This provides users with freedom to use large amounts of space (with or without whiteboard) and the aggregate compute and storage resources available in an open workspace in any configuration suitable for their work dynamics and applications. It allows users to explore and manipulate data using a branch explore merge paradigm via a combination of personal display spaces to create shared display spaces and segregation of personal displays thereof on-demand using interaction modalities like hand gestures, laser pointers and even personal devices. The result is a paradigm where the displays are used as mediums for interacting with the data.

Claims21

Dependent claims

A user interactive display system comprising:

a. a display surface (101), wherein the display surface is shared by a plurality of display devices;
b. one or more interaction devices (130);
c. the plurality of display devices (102), wherein each display device (104) is operatively connected to one or more computing devices in a network of computing devices (106), wherein the display devices are configured to display one or more images on the display surface, wherein each display device is configured to display a portion of the one or more images;
d. one or more sensors (103), wherein each sensor (105) is operatively connected to one or more of computing devices in the network of computing devices, wherein the sensors are jointly disposed to observe one or more interactions of the one or more interaction devices and jointly disposed to observe the display surface;
and
e. the network of computing devices, wherein one or more of the computing devices are configured to execute corresponding instructions, wherein the corresponding instructions are computer-readable instructions comprising:
i. coordinating one or more of the computing devices to jointly display the one or more images using the plurality of display devices;
ii. receiving sensor data from the sensors connected to the computing devices, if any;
iii. processing the sensor data to detect one or more interactions from the one or more interaction devices;
iv. communicating and receiving one or more interactions to and from one or more of the computing devices;
v. determining a reaction to a user interaction;
and
vi. executing the reaction to the user interaction;
wherein the plurality of display devices (102) together project the one or more images on the display surface (101), wherein a user (120) controlling one of the interaction devices (130) executes an interaction with the display surface, whereupon one or more sensors (105) detect the interaction, whereupon one or more of the computing devices process the sensor data to determine a user interaction, and whereupon one or more of the computing devices determines and executes a reaction to the user intent.

11 more elements

A user interactive display system comprising:

a. a display surface (101), wherein the display surface is shared by a plurality of display devices;
b. one or more interaction devices (130);
c. the plurality of display devices (102), wherein each display device (104) is operatively connected to a computing device in a network of computing devices (106), wherein the display devices are configured to display one or more images on the display surface, wherein each display device is configured to display a portion of the one or more images;
d. one or more sensors (103), wherein each sensor (105) is operatively connected to a computing device in a network of computing devices, capable of observing an interaction of one or more of the interaction devices, wherein the sensors are jointly disposed to observe the display surface;
and
e. the network of computing devices, wherein one or more of the computing devices are configured to execute corresponding instructions, wherein the corresponding instructions are computer-readable instructions comprising:
i. coordinating with the other computing devices to jointly display the one or more images using the plurality of display devices;
ii. receiving sensor data from the sensors connected to the computing devices, if any;
iii. processing the sensor data to detect an interaction of the interaction device;
iv. communicating interactions to other computing devices;
v. receiving interactions from other computing devices;
vi. determining a reaction to a user interaction;
and
vii. executing the reaction to the user interaction;
wherein the plurality of display devices (102) together project the one or more images on the display surface (101), wherein a user (120) controlling one of the interaction devices (130) executes an interaction with the display surface, whereupon one or more sensors (105) detect the interaction, whereupon one or more of the computing devices process the sensor data to determine a user interaction, and whereupon one or more of the computing devices determines and executes a reaction to the user intent.

12 more elements

A user interactive display system comprising:

a. a display surface (101), wherein the display surface is shared by a plurality of display devices;
b. one or more interaction devices (130);
c. the plurality of display devices (102), wherein each display device (104) is operatively connected to a computing device in a network of computing devices (106), wherein the display devices are configured to display an image on the display surface, wherein each display device has a display field, wherein the display field comprises the shape and position of the portion of the display surface that the display device is configured to display upon, wherein each display device is configured to display a portion of the image in their display field;
d. one or more sensors (103), wherein each sensor (105) is operatively connected to a computing device in the network of computing devices and capable of observing one or more interactions of the one or more interaction devices, wherein each sensor has a field of view, wherein the interaction must be within the field of view of the sensor to be observed, wherein the fields of view of the one or more sensors are jointly disposed to observe the display surface;
e. a communications network (107), capable of sending communications between the plurality of display devices, the one or more sensors, and the network of computing devices;
and
f. the network of computing devices (106), wherein one or more of the computing devices are configured to execute corresponding instructions, wherein the corresponding instructions are computer-readable instructions comprising:
i. detecting the plurality of display devices connected to one or more of the computing devices;
ii. for each display device connected to one or more of the computing devices:
A. detecting the neighboring display devices;
B. communicating with other computing devices to determine which computing devices are connected to the neighboring display devices, wherein the computing devices of the neighboring display devices comprise the neighboring computing devices;
C. determining the display field the display device controls;
D. determining the portion of the image the display device displays based on the display field that the display device controls;
E. registering the portion of the image to the display surface;
F. displaying the image;
and
G. communicating with the neighboring computing devices to match the features of the portion of the image to the portions of the image displayed by the neighboring display devices;
iii. detecting one or more sensors connected to the computing device;
iv. determining the portion of the display surface observed by each sensor connected to the computing device;
v. receiving sensor data from the one or more sensors connected to the computing device;
vi. processing the sensor data to detect an interaction of the interaction device;
vii. communicating interactions to one or more of the computing devices;
viii. receiving interactions from one or more of the computing devices;
ix. sending data to one or more of the computing devices;
x. receiving data from one or more of the computing devices;
xi. determining a reaction to a user interaction;
and
xii. executing the reaction to the user interaction;
wherein the plurality of display devices (102) together project a display image on the display surface (101), wherein a user (120) controlling one of the interaction devices (130) executes an interaction with the display surface, whereupon one or more of the sensors (105) detects the interaction, whereupon one or more of the computing devices process the sensor data to determine a user intent from the interaction and execute a reaction to the user intent.

26 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/151,060Filed 3 Oct 2018
Priority: 3 Oct 2017
Published: 4 Apr 2019
Term ends: Not estimatedThe source records don’t support an estimate.

Classification23 codes

G06F 3/1446
Controlling a plurality of local displays, e.g. CRT and flat panel display: display composed of modules, e.g. video walls
G06F 3/011
Input arrangements or combined input and output arrangements for interaction between user and computer: arrangements for interaction with the…
G06F 3/033
Arrangements for converting the position or the displacement of a member into a coded form: pointing devices displaced or positioned by the user ,…
G06F 3/0386
Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry: for light pen