# US 2018/0024697 A1 · Projecting a structured light pattern onto a surface and detecting and responding to interactions with the same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2018/0024697 A1 - Document ID: US20180024697A1 - Status: Published application - Kind: A1 - Application number: 15/654,904 - Filed: 2017-07-20 - Published: 2018-01-25 - Earliest claimed: 2016-07-20 - Assignees: Heptagon Micro Optics Pte. Ltd. - Applicants: Heptagon Micro Optics Pte. Ltd. - Inventors: Chi Zhang; Florin Cutu - Web page: https://patentgrep.com/patent/US20180024697A1 ## CPC - G06F 3/0426: {tracking fingers with respect to a virtual keyboard projected or printed on the surface (virtual keyboards on touch screens G06F3/04886)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus - G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)} - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure describes projecting a structured light pattern projected onto a surface and detecting and responding to interactions with the same. The techniques described here can, in some cases, facilitate recognizing that an object such as a user's hand is adjacent the plane of a projection surface and can distinguish the object from the projection surface itself. Movement of the object then can be interpreted, for example, as a specified type of gesture that can trigger a specified type of operation to occur. ## Claims ### Claim 1 A method comprising: (a) projecting a structured light pattern onto a projection surface, the structured light pattern being composed of light of a particular wavelength; (b) acquiring an image based on light of the particular wavelength reflected from a vicinity of the projection surface; (c) identifying, by one or more processors, regions of the acquired image for which there is a change in intensity data relative to corresponding regions of a reference image and for which disparity data indicates the regions of the acquired image correspond to a feature that is within a specified distance of the projection surface; (d) determining, by the one or more processors, that the feature is a particular type of object; (e) repeating (b), (c) and (d) one or more times; (f) performing frame-to-frame tracking to determine a path of movement of the object; and (g) determining, by the one or more processors, that the object's path of movement represents a specified type of gesture. ### Claim 2 (depends on 1) The method of claim 1 including performing image morphology filtering with respect to the identified regions of the acquired image. ### Claim 3 (depends on 1) The method of claim 1 including performing shape descriptor filtering with respect to the identified regions of the acquired image. ### Claim 4 (depends on 1) The method of claim 1 wherein, in (c), the acquired image is compared to the reference image to identify regions of the acquired image for which the change in intensity is equal to or greater than a predetermined threshold value. ### Claim 5 (depends on 1) The method of claim 1 including generating a disparity map for the acquired image by assigning one of two possible disparity values to pixels in the acquired image. ### Claim 6 (depends on 5) The method of claim 5 wherein (c) includes identifying regions of the acquired image for which the disparity value is a particular one of the two possible disparity values. ### Claim 7 (depends on 1) The method of claim 1 wherein the particular wavelength is in the infra-red or near infra-red portions of the spectrum. ### Claim 8 (depends on 1) The method of claim 1 further including projecting a visible pattern onto the projection surface at the same time as the structured light pattern is projected onto the projection surface. ### Claim 9 (depends on 1) The method of claim 1 further including: in response to determining that the object's path of movement represents a specified type of gesture, causing another apparatus to perform a specified operation based on the type of gesture. ### Claim 10 (depends on 9) The method of claim 9 wherein causing another apparatus to perform a specified operation based on the type of gesture includes at least one of the following: causing a display monitor to display an alphanumeric character; causing a light to be turned on, off or dimmed; causing a computing device to transition to a next slide of a presentation. ### Claim 11 An apparatus comprising: a first projector operable to project a structured light pattern onto a projection surface, the structured light pattern being composed of light of a particular wavelength; an image sensor operable to acquire images based on light of the particular wavelength reflected from a vicinity of the projection surface; and one or more processors operable collectively to: (a) identify regions of an acquired image for which there is a change in intensity data relative to corresponding regions of a reference image and for which disparity data indicates the regions of the acquired image correspond to a feature that is within a specified distance of the projection surface; (b) determine that the feature is a particular type of object; (c) repeat (a) and (b) one or more times for at least one other acquired image; (d) perform frame-to-frame tracking to determine a path of movement of the object; and (e) determine that the object's path of movement represents a specified type of gesture. ### Claim 12 (depends on 11) The apparatus of claim 11 wherein the one or more processors are operable to perform image morphology filtering with respect to the identified regions of the acquired image. ### Claim 13 (depends on 11) The apparatus of claim 11 wherein the one or more processors are operable to perform shape descriptor filtering with respect to the identified regions of the acquired image. ### Claim 14 (depends on 11) The apparatus of claim 11 wherein the one or more processors are operable, in (a), to compare the acquired image to the reference image to identify regions of the acquired image for which the change in intensity is equal to or greater than a predetermined threshold value. ### Claim 15 (depends on 11) The apparatus of claim 11 wherein the one or more processors are operable to generate a disparity map for the acquired image by assigning one of two possible disparity values to pixels in the acquired image. ### Claim 16 (depends on 15) The apparatus of claim 15 wherein the one or more processors are operable to identify regions of the acquired image for which the disparity value is a particular one of the two possible disparity values. ### Claim 17 (depends on 11) The apparatus of claim 11 wherein the particular wavelength is in the infra-red or near infra-red portions of the spectrum. ### Claim 18 (depends on 11) The apparatus of claim 11 further including a second light projector operable to project a visible pattern onto the projection surface at the same time as the structured light pattern is projected onto the projection surface. ### Claim 19 (depends on 11) The apparatus of claim 11 further including a device, wherein the one or more processors are operable, in response to determining that the object's path of movement represents a specified type of gesture, to cause the device to perform a specified operation based on the type of gesture. ### Claim 20 (depends on 11) The apparatus of claim 11 wherein the one or more processors are operable, in response to determining that the object's path of movement represents a specified type of gesture, to perform at least one of the following: cause a display monitor to display an alphanumeric character; cause a light to be turned on, off or dimmed; cause a computing device to transition to a next slide of a presentation.