# US 10,481,740 B2 · Projecting a structured light pattern onto a surface and detecting and responding to interactions with the same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,481,740 B2 - Document ID: US10481740B2 - Status: Granted - Kind: B2 - Application number: 15/661,193 - Filed: 2017-07-27 - Published: 2018-02-01 - Granted: 2019-11-19 - Earliest claimed: 2016-08-01 - Assignees: ams Sensors Singapore Pte. Ltd. - Applicants: ams Sensors Singapore Pte. Ltd. - Inventors: Florin Cutu; Chi Zhang - Web page: https://patentgrep.com/patent/US10481740B2 ## CPC - G06F 3/0425: {using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected (tracking a projected light spot to determine a position on a display surface G06F3/0386)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06T 7/11: Region-based segmentation - G06T 7/136: involving thresholding - G06T 7/155: involving morphological operators - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/248: {involving reference images or patches} - G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light - G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16) - G06T 2207/10048: Infrared image - G06T 2207/20036: Morphological image processing - G06T 2207/20092: Interactive image processing based on input by user - G06T 2207/30241: Trajectory - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract The present disclosure describes projecting a structured light pattern onto a surface and detecting and responding to interactions with the same. A method includes-acquiring an image based on light reflected from a vicinity of the projection surface, and identifying regions of the acquired image that correspond to a feature that is within a specified distance of the projection surface by identifying regions of the acquired image for which intensity data differs relative to other regions of the acquired image and which fit a specified homographic relationship with respect to corresponding regions of a reference image. ## Claims ### Claim 1 A method comprising: (a) projecting a structured light pattern onto a projection surface, the structured light pattern being composed of light of a particular wavelength; (b) acquiring an image based on light of the particular wavelength reflected from a vicinity of the projection surface; (c) identifying, by one or more processors, regions of the acquired image that correspond to a feature that is within a specified distance of the projection surface by identifying regions of the acquired image for which intensity data differs relative to other regions of the acquired image and which fit a specified homographic relationship with respect to corresponding regions of a reference image, and generating a binary image in which regions of the acquired image that are determined to correspond to the feature within the specified distance of the projection surface are assigned a first value, and regions of the acquired image that are determined not to correspond to the feature within the specified distance of the projection surface are assigned a second different value; (d) use the binary image to determine, by the one or more processors, that the feature is a particular type of object; (e) repeating (b), (c) and (d) one or more times; (f) performing frame-to-frame tracking to determine a path of movement of the object; and (g) determining, by the one or more processors, that the object's path of movement represents a specified type of gesture. ### Claim 2 (depends on 1) The method of claim 1 including determining the homographic relationship between a first set of coordinates in the reference image and a second set of coordinates in the acquired image. ### Claim 3 (depends on 1) The method of claim 1 including performing image morphology filtering with respect to the identified regions of the acquired image. ### Claim 4 (depends on 1) The method of claim 1 including performing shape descriptor filtering with respect to the identified regions of the acquired image. ### Claim 5 (depends on 1) The method of claim 1 wherein, in (c), regions of the acquired image are identified for which intensity data differs relative to other regions of the acquired image by at least a predetermined threshold value. ### Claim 6 (depends on 1) The method of claim 1 wherein the particular wavelength is in the infra-red or near infra-red portions of the spectrum. ### Claim 7 (depends on 1) The method of claim 1 further including projecting a visible pattern onto the projection surface at the same time as the structured light pattern is projected onto the projection surface. ### Claim 8 (depends on 1) The method of claim 1 further including: in response to determining that the object's path of movement represents a specified type of gesture, causing another apparatus to perform a specified operation based on the type of gesture. ### Claim 9 (depends on 8) The method of claim 8 wherein causing another apparatus to perform a specified operation based on the type of gesture includes at least one of the following: causing a display monitor to display an alphanumeric character; causing a light to be turned on, off or dimmed; causing a computing device to transition to a next slide of a presentation. ### Claim 10 An apparatus comprising: a first projector operable to project a structured light pattern onto a projection surface, the structured light pattern being composed of light of a particular wavelength; an image sensor operable to acquire images based on light of the particular wavelength reflected from a vicinity of the projection surface; and one or more processors operable collectively to: (a) identify regions of an acquired image that correspond to a feature that is within a specified distance of the projection surface by identifying regions of the acquired image for which intensity data differs relative to other regions of the acquired image and which fit a specified homographic relationship with respect to corresponding regions of a reference image, and generate a binary image in which regions of the acquired image that are determined to correspond to the feature within the specified distance of the projection surface are assigned a first value, and regions of the acquired image that are determined not to correspond to the feature within the specified distance of the projection surface are assigned a second different value; (b) use the binary image to determine that the feature is a particular type of object; (c) repeat (a) and (b) one or more times for at least one other acquired image; (d) perform frame-to-frame tracking to determine a path of movement of the object; and (e) determine that the object's path of movement represents a specified type of gesture. ### Claim 11 (depends on 10) The apparatus of claim 10 wherein the one or more processors are operable to perform image morphology filtering with respect to the identified regions of the acquired image. ### Claim 12 (depends on 10) The apparatus of claim 10 wherein the one or more processors are operable to perform shape descriptor filtering with respect to the identified regions of the acquired image. ### Claim 13 (depends on 10) The apparatus of claim 10 wherein the one or more processors are operable, in (a), to identify regions of the acquired image for which intensity data differs relative to other regions of the acquired image by at least a predetermined threshold value. ### Claim 14 (depends on 10) The apparatus of claim 10 wherein the particular wavelength is in the infra-red or near infra-red portions of the spectrum. ### Claim 15 (depends on 10) The apparatus of claim 10 further including a second light projector operable to project a visible pattern onto the projection surface at the same time as the structured light pattern is projected onto the projection surface. ### Claim 16 (depends on 10) The apparatus of claim 10 further including a device, wherein the one or more processors are operable, in response to determining that the object's path of movement represents a specified type of gesture, to cause the device to perform a specified operation based on the type of gesture. ### Claim 17 (depends on 10) The apparatus of claim 10 wherein the one or more processors are operable, in response to determining that the object's path of movement represents a specified type of gesture, to perform at least one of the following: cause a display monitor to display an alphanumeric character; cause a light to be turned on, off or dimmed; cause a computing device to transition to a next slide of a presentation.