# US 2015/0338998 A1 · System and methods for providing a three-dimensional touch screen > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2015/0338998 A1 - Document ID: US20150338998A1 - Status: Published application - Kind: A1 - Application number: 14/719,139 - Filed: 2015-05-21 - Published: 2015-11-26 - Earliest claimed: 2014-05-22 - Assignees: Ubi Interactive Inc. - Applicants: Ubi interactive inc. - Inventors: Anup Koyadan Chathoth; Chao Zhang - Web page: https://patentgrep.com/patent/US20150338998A1 ## CPC - G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)} - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0482: Interaction with lists of selectable items, e.g. menus - G06F 3/0488: using a touch-screen or digitiser, e.g. input of commands through traced gestures - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - H04N 9/31: Projection devices for colour picture display {, e.g. using electronic spatial light modulators [ESLM]} - G06F 2203/04101: 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems and techniques for providing a three-dimensional touch screen are described. Example systems include a touch screen manager that is configured to display a screen via a display device, such as by projecting on a wall or other surface a screen generated by a computing device, such as a desktop, laptop, or tablet computer. The manager is further configured to receive position information from a sensor that determines the positions of objects, such as the finger or hand of a user, within a three-dimensional space in front of the display surface. The manager then converts the received position information into user interface events, including gestures (e.g., pinch, swipe), mouse-type events (e.g., click, drag), or the like. The user interface events are provided to an application or other module executing on the computing device. ## Claims ### Claim 1 A method for managing a three-dimensional touch screen, the method comprising: providing a three-dimensional touch screen that facilitates user interaction in three dimensions by: displaying a screen via a display device; receiving position information from a visual sensor, the position information reflecting the position of an object in a three-dimensional space located adjacent to the displayed screen; generating a user interface event based on the received position information; and providing the generated user interface event to an application program. ### Claim 2 (depends on 1) The method of claim 1, wherein the displaying a screen includes: causing a projector to display the screen on a wall, causing a display monitor to display the screen, and/or displaying the screen on a non-flat surface. ### Claim 3 (depends on 1) The method of claim 1, wherein the receiving position information from a visual sensor includes: receiving the position information from a range camera device, the position information including an x, y, z position of the object in the three-dimensional space. ### Claim 4 (depends on 1) The method of claim 1, wherein the generating a user interface event based on the received position information includes: generating a touch event that is one of a press, swipe, pinch, or expand. ### Claim 5 (depends on 1) The method of claim 1, wherein the generating a user interface event based on the received position information includes: generating a pointer mouse event that is one of a click, hover, select, or double-click. ### Claim 6 (depends on 1) The method of claim 1, wherein the generating a user interface event based on the received position information includes: generating an event based on multiple objects tracked in the three-dimensional space, the multiple objects including at least two fingers or two hands. ### Claim 7 (depends on 1) The method of claim 1, further comprising: presenting a setup user interface that instructs a user regarding one or more of: placement of the visual sensor, whether the display device is supported, whether the displayed screen is within range of the visual sensor, and whether lighting conditions are acceptable. ### Claim 8 (depends on 7) The method of claim 7, wherein the presenting a setup user interface includes: displaying an indication of a field of coverage of the visual sensor, so that the user can position the visual sensor at a location where the entire display screen is visible to the visual sensor. ### Claim 9 (depends on 7) The method of claim 7, wherein the presenting a setup user interface includes: instructing the user to reposition the visual sensor so that the visual sensor can support a selected interaction mode. ### Claim 10 (depends on 7) The method of claim 7, wherein the presenting a setup user interface includes: selecting a portion of the display to make interactive. ### Claim 11 (depends on 1) The method of claim 1, further comprising: calibrating the three-dimensional touch screen by: detecting position and orientation of the displayed screen based on a displayed pattern; and detecting shape, size, and geometry of a surface on which the screen is displayed. ### Claim 12 (depends on 11) The method of claim 11, wherein the calibrating the three-dimensional touch screen includes: building a three-dimensional model of a non-flat surface on which the screen is displayed. ### Claim 13 (depends on 11) The method of claim 11, wherein the calibrating the three-dimensional touch screen includes: performing a two-step calibration process when a display surface is not visible to the visual sensor, by calibrating the display to the visual spectrum and then using a different object to perform background calibration. ### Claim 14 (depends on 11) The method of claim 11, wherein the calibrating the three-dimensional touch screen includes: performing the calibration concurrently as a user is adjusting position of the visual sensor with respect to a display. ### Claim 15 (depends on 11) The method of claim 11, wherein the calibrating the three-dimensional touch screen includes: performing a partial calibration by displaying a resizable object that is configured to be moved and resized by a user to define an area of the displayed screen to make interactive. ### Claim 16 (depends on 11) The method of claim 11, wherein the calibrating the three-dimensional touch screen includes: determining, based on the position of the visual sensor with respect to its display, that a current physical setup is substantially similar to a previously used physical setup; and in response to the detecting, performing auto-calibration without user input and/or skipping calibration and relying on calibration information determined with respect to the previously used physical setup. ### Claim 17 (depends on 1) The method of claim 1, further comprising: automatically selecting an interaction mode based on operating conditions of the three-dimensional touch screen, the operational conditions including one or more of current resolution, current lighting conditions, distance between the displayed screen and the visual sensor, and an object being held by a user of the touch screen. ### Claim 18 (depends on 17) The method of claim 17, wherein the automatically selecting an interaction mode includes: selecting a finger-based interaction mode; and tracking a finger of a user by providing position information that reflects the position of the finger in the three-dimensional space. ### Claim 19 (depends on 17) The method of claim 17, wherein the automatically selecting an interaction mode includes: selecting a hand-based integration mode; and tracking a hand of a user by providing position information that reflects the position of the hand in the three-dimensional space. ### Claim 20 (depends on 17) The method of claim 17, wherein the automatically selecting an interaction mode includes: selecting a pen-based integration mode; and tracking a pen operated by a user by providing position information that reflects the position of the finger in the three-dimensional space. ### Claim 21 (depends on 1) The method of claim 1, wherein the receiving position information from a visual sensor includes: receiving position information that reflects the position of a pen device that includes a light source attached to a pressure sensitive tip, the tip when pressed configured to cause the light source to emit an infrared light signal that is tracked by the visual sensor. ### Claim 22 (depends on 21) The method of claim 21, wherein the pen includes a button that, when pressed, causes the light source to emit the infrared light signal. ### Claim 23 (depends on 21) The method of claim 21, wherein the light source is an infrared light emitting diode. ### Claim 24 (depends on 21) The method of claim 21, wherein the generating a user interface event based on the received position information includes: generating the user interface event based on the light signal emitted by the pen. ### Claim 25 (depends on 21) The method of claim 21, further comprising: switching interaction modes between pen-based interaction mode and hand/finger-based interaction mode. ### Claim 26 (depends on 25) The method of claim 25, wherein the switching interaction modes includes: automatically switching to the pen-based interaction mode when the pen is within a threshold distance of a surface on which the screen is displayed, wherein the pen is configured to detect its proximity to the surface and send a message triggering the pen-based interaction mode. ### Claim 27 (depends on 25) The method of claim 25, wherein the switching interaction modes includes: switching modes in response to a user selection, the user selection including one of: a gesture, a wireless signal transmitted by the pen, a selection of a menu item displayed on the screen. ### Claim 28 (depends on 1) The method of claim 1, further comprising: determining that a user interface event cannot be generated due to current conditions or action of the user; and in response, providing visual or audible feedback to a user to modify the condition or their action. ### Claim 29 (depends on 1) A non-transitory computer-readable medium including contents that are configured, when executed, to cause a computing system to perform a method for managing a three-dimensional touch screen, the method comprising: performing the method of claim 1. ### Claim 30 A system for managing a three-dimensional touch screen, the system comprising: a first computing device comprising a processor and a memory; and wherein the memory stores a code module that is configured, when executed by the processor, to provide a three-dimensional touch screen by: displaying a screen via a display device; receiving position information from a visual sensor, the position information representing the position of an object in a three-dimensional space located adjacent to the displayed screen; generating a user interface event based on the received position information; and providing the generated user interface event to an application program. ### Claim 31 (depends on 30) The system of claim 30, further comprising: a second computing device that is a laptop or tablet computer that executes the application program; and wherein first computing device is configured to: wirelessly receive screen data from the first device; provide the screen data to the code module for display via the display device; and wirelessly transmit the generated user interface event to the second computing system.