# US 2017/0257610 A1 · Device and method for orchestrating display surfaces, projection devices, and 2d and 3d spatial interaction devices for creating interactive environments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0257610 A1
- Document ID: US20170257610A1
- Status: Published application
- Kind: A1
- Application number: 15/511,238
- Filed: 2015-09-15
- Published: 2017-09-07
- Earliest claimed: 2014-09-16
- Assignees: Ingenuity I/O
- Applicants: Ingenuity I/O
- Inventors: STÉPHANE VALES; VINCENT PEYRUQUÉOU; ALEXANDRE LEMORT
- Web page: https://patentgrep.com/patent/US20170257610A1

## CPC

- H04N 9/3185: {Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14)}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06F 3/1423: {controlling a plurality of local displays, e.g. CRT and flat panel display}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G09B 9/302: {the image being transformed by computer processing, e.g. updating the image to correspond to the changing point of view}
- G09B 9/32: by projected image ({G09B9/307,} G09B9/36 take precedence)
- G09G 3/001: {using specific devices not provided for in groups G09G3/02 - G09G3/36, e.g. using an intermediate record carrier such as a film slide; Projection systems; Display of non-alphanumerical information, solely or in combination with alphanumerical information, e.g. digital display on projected diapositive as background (slide projectors per se G03B23/00 = 42 HP)}
- H04N 9/3194: {including sensor feedback}
- G09G 2354/00: Aspects of interface with display user
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)

## Abstract

A device to manage the projection of images onto a plurality of media, and to geometrically designate and model a plurality of selected areas on display surfaces. The display areas form a visual environment of a user. The designations and models result in an environmental geometric model. A controller interprets information provided by at least one spatial interaction device of the user in the environmental geometric model. The controller generates images to be projected onto the various display areas by at least one image projector in accordance with the actions of the user as detected by the spatial interaction devices.

## Claims

### Claim at position 1

1-10. (canceled)

### Claim 11

A device to manage display and interaction on a plurality of physical surfaces, comprising:
a plurality of display areas chosen on the display surfaces to project images, said plurality of display areas forming a visual environment of at least one user;
a controller to designate and geometrically model said plurality of display areas to provide a geometric environment model, to interpret information provided by at least one spatial interaction device of said at least one user in the geometric environment model, and to generate images to be projected onto said plurality of display areas; and
at least one image projector to project the generated images onto said plurality of display areas as a function of actions of said at least one user detected by said at least one spatial interaction device.

### Claim 12 (depends on 1)

A display device comprising the management device according to claim 1, wherein said plurality of physical surfaces is a plurality of passive display surfaces;
wherein said at least one image projector is configured to project the generated images onto said plurality of passive display surfaces; and wherein said at least one spatial interaction device is configured to detect gestural instructions of said at least one use.

### Claim 13

A method for managing display and interaction on a plurality of areas chosen on display surfaces, comprising the steps of receiving projected images by the display surfaces from at least one image projector and generating a global geometric environment model comprising data characterizing a position and dimensions of each display surface facing an image projector; and wherein an orientation or a distance of each display surface in relation to the image projector is unknown initially.

### Claim 14 (depends on 13)

The method according to claim 13, further comprising a step of modeling each display surface by at least one of the following sub-steps of: a direct geometric measurement in a space, a geometric measurement with an aid of a three-dimensional modeling system, and a visual calibration.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the visual calibration sub-step comprises an automated visual calibration with an aid of a computer vision system coupled to the image projector displaying sequences of visual patterns to detect and calibrate projection planes.

### Claim 16 (depends on 14)

The method according to claim 14, wherein the visual calibration sub-step comprises modeling a virtual projection plane as a function of an orientation and a focal length of the image projector, the virtual projection plane is normal to a projection axis of the image projector and positioned at a distance dependent on the focal length of the image projector.

### Claim 17 (depends on 13)

The method according to claim 13, further comprising a step of integrating 3D spatial interaction devices into the global geometric environment model by determining geometric transformations to interpret information that the 3D spatial interaction devices provide in a same three-dimensional coordinate system as used in the global geometric environment model of the display surfaces.

### Claim 18 (depends on 17)

The method according to claim 17, wherein the integrating step comprises sub-steps of:
calculating a transformation function between a coordinate system of the 3D spatial interaction devices and a coordinate system of the global geometric environment model, in accordance with positions of at least two points in the two coordinate systems or a position of one point of a vector; and
generating a correspondence function to map between information of the 3D spatial interaction devices and the display surfaces.

### Claim 19 (depends on 13)

The method according to claim 13, further comprising a step of integrating at least one 2D spatial interaction device into the global geometric environment model by determining transformation from 2D coordinates received from said at least one 2D spatial interaction device into 3D coordinates of the global geometric environment model.

### Claim 20 (depends on 13)

The method according to claim 13, further comprising a step of generating the projected images displayed on the display surfaces in real time as a function of actions of a user detected by spatial interaction devices.

### Claim 21 (depends on 20)

The method according to claim 20, wherein the generating step comprises sub-steps of mathematically projecting spatialized information of the spatial interaction devices on the display surfaces; and utilizing the spatialized information to locate physical entities and to project information on or around the physical entities.
