# US 9,880,728 B2 · Methods and systems for controlling a virtual interactive surface and interactive display systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,880,728 B2 - Document ID: US9880728B2 - Status: Granted - Kind: B2 - Application number: 14/759,787 - Filed: 2013-12-30 - Published: 2015-12-10 - Granted: 2018-01-30 - Earliest claimed: 2013-01-08 - Assignees: Ayotle - Applicants: Ayotle SAS - Inventors: José Alonso Ybanez Zepeda - Web page: https://patentgrep.com/patent/US9880728B2 ## CPC - G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)} - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06F 2203/04101: 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems and methods for controlling a virtual interactive surface in a scene include a surface for displaying information and one or several virtual sensors positioned with respect to the surface. The virtual sensor includes data representing a geometric form, data representing a position in the scene, and data representing one or several trigger conditions. A data representation of the scene is captured and a determination whether a virtual sensor trigger condition is fulfilled is made, based on an analysis of the captured data representation in an area corresponding to the geometric form and position of the virtual sensor. ## Claims ### Claim 1 A method for controlling a virtual interactive surface in a scene, wherein the virtual interactive surface comprises a surface for displaying information and one or several virtual sensors positioned with respect to the surface, wherein a virtual sensor comprises data representing a geometric form, data representing a position in the scene and data representing one or several trigger conditions, the method comprising: capturing a first data representation of the scene; generating point cloud data representing surfaces of objects of the captured scene, the point cloud data comprising data representing non-empty points of a first volume area corresponding to the captured first data representation and respective associated position in the first volume area; and determining whether one or several of the virtual sensor trigger conditions is/are fulfilled based on an analysis of the captured first data representation in a volume area corresponding to the geometric form and position of each virtual sensor, wherein for a first virtual sensor said analysis comprises comparing a number of non-empty points whose position falls in the volume area corresponding to the geometric form and position of said first virtual sensor is greater to a triggering threshold. ### Claim 2 (depends on 1) The method according to claim 1, wherein the first data representation of the scene includes an image data representation of the scene. ### Claim 3 (depends on 2) The method according to claim 2, further comprising: capturing a second data representation of the scene; and configuring the geometric form and position of each virtual sensor based on an image generated from the captured second data representation. ### Claim 4 (depends on 1) The method according to claim 1, further comprising: capturing a second data representation of the scene; and configuring the geometric form and position of each virtual sensor based on an image generated from the captured second data representation. ### Claim 5 (depends on 1) The method according to claim 1, comprising analyzing point cloud data representing points in the volume area which correspond to the geometric form and position of the virtual sensor, wherein the analyzing comprises: detecting the points of the point cloud which are positioned in the volume area corresponding to the geometric form and position of the virtual sensor: determining an intensity value representing intensity amounts for some of the detected points; and comparing the determined intensity value to a predetermined threshold. ### Claim 6 (depends on 1) The method according to claim 1, further comprising defining and configuring one or several virtual sensors in the scene using the first data representation of the scene. ### Claim 7 (depends on 1) The method according to claim 1, further comprising: executing one or several command actions in response to determining whether one or several of the virtual sensor trigger condition is/are fulfilled. ### Claim 8 (depends on 7) The method according to claim 7, wherein the one or several commands comprises modifying the information displayed on the surface. ### Claim 9 (depends on 1) The method according to claim 1, wherein for a second virtual sensor said analysis comprises determining a reference point from the non-empty points whose position falls in a second volume area which corresponds to the geometric form and position of said second virtual sensor, said method comprising triggering a signal for selecting a region of the surface based on the relative position of the reference point with respect to each dimension of the second volume area. ### Claim 10 (depends on 1) The method according to claim 1, wherein said one or several virtual sensors comprises a first virtual sensor for triggering a signal for launching an application on a computer, a second virtual sensor for triggering a signal for selecting a region of the surface, and a third virtual sensor for triggering a signal for launching an action on the computer. ### Claim 11 (depends on 1) The method according to claim 1, wherein said one or several virtual sensors comprises a first virtual sensor for triggering a signal for launching on a computer one of an application and an action, a second virtual sensor for triggering a signal for selecting a region of the surface, and a third virtual sensor for activating and deactivating the first and the second virtual sensors. ### Claim 12 (depends on 1) The method according to claim 1, wherein the volume areas of the virtual sensors are rectangular parallelepipeds disposed parallel to the surface. ### Claim 13 (depends on 12) The method according to claim 12, wherein the volume areas of one virtual sensor contains the volume area of another virtual sensors. ### Claim 14 (depends on 12) The method according to claim 12, wherein the volume areas of one virtual sensor are arranged separately. ### Claim 15 A system for controlling a virtual interactive surface in a scene, wherein the virtual interactive surface comprises a surface for displaying information and one or several virtual sensors positioned with respect to the surface, wherein a virtual sensor comprises data representing a geometric form, data representing a position in the scene and data representing one or several trigger conditions, the system comprising: a scene capture sub-system configured to capture a scene and generate a first data representation of the scene; a virtual sensor sub-system in operative communication with the scene capture sub-system, the virtual sensor sub-system comprising an interface for receiving data representing the first data representation of the scene, a processor, and a memory operatively coupled to the processor, wherein the virtual sensor sub-system is configured to perform a method for detecting activation of each virtual sensor in the scene, the method comprising: receiving point cloud data representing surfaces of objects of the captured scene and comprising data representing non-empty points of a first volume area corresponding to the captured first data representation and respective associated position in the first volume area; and determining whether one or several of the virtual sensor trigger condition is/are fulfilled based on an analysis of the captured first data representation in an area corresponding to the geometric form and position of the virtual sensor, wherein for a first virtual sensor said analysis comprises comparing a number of non-empty points whose position falls in the volume area corresponding to the geometric form and position of said first virtual sensor is greater to a triggering threshold. ### Claim 16 (depends on 15) A system according to claim 15, wherein the virtual sensor sub-system includes functionality to define and configure one or several virtual sensors in the scene using the first data representation of the scene. ### Claim 17 (depends on 16) A system according to claim 16, further comprising: a command sub-sub-system in operative communication with the virtual sensor sub-system, wherein the command sub-system is configured to perform a method for: executing one or several command actions in response to determining whether one or several of the virtual sensor trigger condition is/are fulfilled. ### Claim 18 (depends on 15) A system according to claim 15, further comprising: a command sub-sub-system in operative communication with the virtual sensor sub-system, wherein the command sub-system is configured to perform a method for: executing one or several command actions in response to determining whether one or several of the virtual sensor trigger condition is/are fulfilled. ### Claim 19 (depends on 11) An interactive display system comprising: a virtual interactive surface in a scene, wherein the virtual interactive surface comprises a surface for displaying information and one or several virtual sensors positioned with respect to the surface, wherein a virtual sensor comprises data representing a geometric form, data representing a position in the scene and data representing one or several trigger conditions; a system for controlling the virtual interactive surface according to claim 11; a display sub-system adapted to display information on the surface, operatively connected to the system for controlling the virtual interactive surface. ### Claim 20 (depends on 19) An interactive display system according to claim 19, wherein one or several virtual sensors comprise data representing a form adapted to cover the surface for displaying information. ### Claim 21 (depends on 19) An interactive display system according to claim 19, wherein the virtual interactive surface comprises several virtual sensors, each having a different trigger condition. ### Claim 22 A non-transitory computer-readable storage medium storing a computer program that, when executed, causes a system comprising a processor operatively coupled with a memory, to perform a method for controlling a virtual interactive surface in a scene, wherein the virtual interactive surface comprises a surface for displaying information and one or several virtual sensors positioned with respect to the surface, wherein a virtual sensor comprises data representing a geometric form, data representing a position in the scene and data representing one or several trigger conditions, the method comprising: receiving point cloud data representing surfaces of objects of the captured scene and comprising data representing non-empty points of a first volume area corresponding to the captured first data representation and respective associated position in the first volume area; and determining whether one or several of the virtual sensor trigger conditions is/are fulfilled based on an analysis of data representing a captured first data representation of the scene in a volume area corresponding to the geometric form and position of each virtual sensor, wherein for a first virtual sensor said analysis comprises comparing a number of non-empty points whose position falls in the volume area corresponding to the geometric form and position of said first virtual sensor is greater to a triggering threshold.