# US 12,010,466 B2 · Visual recognition based method and system for projecting patterned light, method and system applied to oral inspection, and machining system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,010,466 B2 - Document ID: US12010466B2 - Status: Granted - Kind: B2 - Application number: 17/559,070 - Filed: 2021-12-22 - Published: 2022-12-22 - Granted: 2024-06-11 - Earliest claimed: 2021-06-22 - Assignees: Industrial Technology Research Institute - Applicants: Industrial Technology Research Institute - Inventors: Cheng-Han Tsai; Kuo-Feng Hung - Web page: https://patentgrep.com/patent/US12010466B2 ## CPC - H04N 9/3185: {Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14)} - H04N 9/3194: {including sensor feedback} - G06V 10/245: {by locating a pattern; Special marks for positioning} - G06V 10/98: Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns - H04N 23/56: provided with illuminating means - H04N 23/61: based on recognised objects - B25J 19/023: {including video camera means} - G06V 40/168: {Feature extraction; Face representation} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) ## Abstract A visual recognition based method for projecting patterned light includes: projecting a calibration image onto a projection screen by a projection module; capturing the calibration image by an image-capturing module to obtain a calibration information between the projection module and the image-capturing module; capturing an object by the image-capturing module to obtain a to-be-recognized image of the object; detecting the object in the to-be-recognized image and acquiring a plurality of feature points associated with a plurality of feature areas of the object in the to-be-recognized image; retrieving a plurality of target feature points corresponding to a target object from the feature points; obtaining a projection coordinate of the target feature points\ according to the calibration information and providing the projection coordinate to the projection module; and projecting a projection pattern with shape corresponding to the target object onto the object by the projection module according to the projection coordinate. ## Claims ### Claim 1 A visual recognition based method for projecting patterned light comprising: projecting a calibration image onto a projection screen by a projection module; capturing the calibration image by an image-capturing module to obtain a calibration information between the projection module and the image-capturing module; capturing an object by the image-capturing module to obtain a to-be-recognized image of the object; detecting the object in the to-be-recognized image, and acquiring a plurality of feature points associated with a plurality of feature areas of the object in the to-be-recognized image; retrieving a plurality of target feature points corresponding to a target object from the feature points; obtaining a projection coordinate of the target feature points according to the calibration information, and providing the projection coordinate to the projection module; and projecting a projection pattern with shape corresponding to the target object onto the object by the projection module according to the projection coordinate. ### Claim 2 (depends on 1) The visual recognition based method for projecting patterned light according to claim 1, wherein the calibration image comprises a plurality of graphic labels respectively disposed in corners of the calibration image. ### Claim 3 (depends on 2) The visual recognition based method for projecting patterned light according to claim 2, wherein the step of obtaining the calibration information between the projection module and the image-capturing module comprises: iteratively performing following steps with different projection distances: projecting the calibration image onto the projection screen by the projection module at a projection distance for the image-capturing module to capture the calibration image; identifying the graphic labels according to the captured calibration image; obtaining coordinates of a plurality of alignment points which the identified graphic labels correspond to; obtaining coordinates of a plurality of reference points of a standard screen; and transforming the coordinates of the alignment points and the coordinates of the reference points to obtain a calibration matrix corresponding to the projection distance; and a plurality of calibration matrixes corresponding to the different projection distances as the calibration information. ### Claim 4 (depends on 3) The visual recognition based method for projecting patterned light according to claim 3, wherein the reference points are corners of the standard screen. ### Claim 5 (depends on 3) The visual recognition based method for projecting patterned light according to claim 3, wherein the step of obtaining the coordinates of the alignment points which the identified graphic labels correspond to comprises: respectively obtaining a plurality of depth values of the graphic labels by the image-capturing module; calculating an average depth information based on the depth values; wherein when a difference between each of the depth values of the graphic labels and the average depth information is less than a threshold value, the coordinates of the alignment points are obtained. ### Claim 6 (depends on 5) The visual recognition based method for projecting patterned light according to claim 5, wherein in the step of obtaining the coordinates of the alignment points which the identified graphic labels correspond to, the depth values of the graphic labels and the average depth information are projected onto the projection screen by the projection module. ### Claim 7 (depends on 1) The visual recognition based method for projecting patterned light according to claim 1, wherein the object is a face, a workpiece, a material basket or a product to be assembled. ### Claim 8 (depends on 1) The visual recognition based method for projecting patterned light according to claim 1, wherein the target object is a mouth, and the projection pattern is in the shape of the mouth. ### Claim 9 (depends on 1) The visual recognition based method for projecting patterned light according to claim 1, further comprising: when detecting the object in the to-be-recognized image fails, the brightness of the to-be-recognized image is adjusted. ### Claim 10 (depends on 1) The visual recognition based method for projecting patterned light according to claim 1, wherein the step of obtaining the projection coordinate of the target feature points according to the calibration information, and providing the projection coordinate to the projection module comprises: searching for a reference area according to the feature areas and retrieving a reference feature point from the reference area; obtaining a depth value corresponding to the reference feature point; and transforming coordinates of the target feature points into the projection coordinate using the calibration information corresponding to the depth value. ### Claim 11 (depends on 1) The visual recognition based method for projecting patterned light according to claim 1, wherein the image-capturing module is a depth camera. ### Claim 12 A visual recognition based system for projecting patterned light, provided with a calibration mode and a projection mode, comprising: a projection module configured for projecting a calibration image onto a projection screen in the calibration mode; an image-capturing module configured for capturing the calibration image in the calibration mode, and capturing an object in the projection mode to obtain a to-be-recognized image of the object; and a processor coupled to the projection module and the image-capturing module and configured for, in the calibration mode, obtaining a calibration information between the projection module and the image-capturing module according to the captured calibration image, and, in the projection mode, detecting the object in the to-be-recognized image, acquiring a plurality of feature points associated with a plurality of feature areas of the object in the to-be-recognized image, retrieving a plurality of target feature points corresponding to a target object from the feature points, obtaining a projection coordinate of the target feature points according to the calibration information and providing the projection coordinate to the projection module, and commanding the projection module to project a projection pattern with shape corresponding to the target object onto the object according to the projection coordinate. ### Claim 13 (depends on 12) The visual recognition based system for projecting patterned light according to claim 12, wherein the projection module and the image-capturing module are all-in-one. ### Claim 14 (depends on 12) The visual recognition based system for projecting patterned light according to claim 12, wherein the calibration image comprises a plurality of graphic labels respectively disposed in corners of the calibration image. ### Claim 15 (depends on 14) The visual recognition based system for projecting patterned light according to claim 14, wherein the projection module projects he calibration image onto the projection screen at a projection distance for the image-capturing module to capture the calibration image, and the processor identifies the graphic labels according to the captured calibration image, obtains coordinates of a plurality of alignment points which the identified graphic labels correspond to, obtains coordinates of a plurality of reference points of a standard screen, and transforms the coordinates of the alignment points and the coordinates of the reference points to obtain a calibration matrix corresponding to the projection distance; wherein in the calibration mode, the projection module projects the calibration image onto the projection screen at different projection distances, so as to obtain a plurality of calibration matrixes corresponding to the different projection distances as the calibration information. ### Claim 16 (depends on 15) The visual recognition based system for projecting patterned light according to claim 15, wherein the reference points are corners of the standard screen. ### Claim 17 (depends on 15) The visual recognition based system for projecting patterned light according to claim 15, wherein the processor respectively obtains a plurality of depth values of the graphic labels from the image-capturing module, and calculates an average depth information based on the depth values; wherein when a difference between each of the depth values of the graphic labels and the average depth information is less than a threshold value, the processor obtains the coordinates of the alignment points. ### Claim 18 (depends on 17) The visual recognition based system for projecting patterned light according to claim 17, wherein the projection module projects the depth values of the graphic labels and the average depth information are projected onto the projection screen. ### Claim 19 (depends on 12) The visual recognition based system for projecting patterned light according to claim 12, wherein the object is a face, a workpiece, a material basket or a product to be assembled. ### Claim 20 (depends on 12) The visual recognition based system for projecting patterned light according to claim 12, wherein the target object is a mouth, and the projection pattern is in the shape of the mouth. ### Claim 21 (depends on 12) The visual recognition based system for projecting patterned light according to claim 12, wherein when detecting the object in the to-be-recognized image fails, the processor adjusts the brightness of the to-be-recognized image. ### Claim 22 (depends on 12) The visual recognition based system for projecting patterned light according to claim 12, wherein the processor searches for a reference area according to the feature areas and retrieves a reference feature point from the reference area, obtains a depth value corresponding to the reference feature point, and transforms coordinates of the target feature points into the projection coordinate using the calibration information corresponding to the depth value. ### Claim 23 (depends on 12) The visual recognition based system for projecting patterned light according to claim 12, wherein the image-capturing module is a depth camera. ### Claim 24 A method applied to oral inspection comprising: projecting a calibration image onto a projection screen by a projection module; capturing the calibration image by an image-capturing module to obtain a calibration information between the projection module and the image-capturing module; capturing a human's face by the image-capturing module to obtain a to-be-recognized image of the human's face; detecting the human's face in the to-be-recognized image, and acquiring a plurality of feature points associated with a plurality of facial features of the human's face in the to-be-recognized image; retrieving a plurality of mouth feature points corresponding to a human's mouth from the feature points; obtaining a projection coordinate of the mouth feature points according to the calibration information, and providing the projection coordinate to the projection module; and projecting a projection pattern with shape corresponding to the human's mouth onto the human's face by the projection module according to the projection coordinate. ### Claim 25 (depends on 24) The method applied to oral inspection according to claim 24, wherein the step of obtaining the projection coordinate of the mouth feature points according to the calibration information, and providing the projection coordinate to the projection module comprises: searching for a nose area according to the facial features and retrieving a nose feature point from the nose area; obtaining a depth value corresponding to the nose feature point; and transforming coordinates of the mouth feature points into the projection coordinate using the calibration information corresponding to the depth value. ### Claim 26 (depends on 24) The method applied to oral inspection according to claim 24, wherein the image-capturing module is a depth camera. ### Claim 27 (depends on 25) The method applied to oral inspection according to claim 25, wherein the nose feature point is a feature point of a nose tip. ### Claim 28 A system applied to oral inspection, provided with a calibration mode and a projection mode, comprising: a projection module configured for projecting a calibration image onto a projection screen in the calibration mode; an image-capturing module configured for capturing the calibration image in the calibration mode, and capturing a human's face in the projection mode to obtain a to-be-recognized image of the human's face; and a processor coupled to the projection module and the image-capturing module and configured for, in the calibration mode, obtaining a calibration information between the projection module and the image-capturing module according to the captured calibration image, and, in the projection mode, detecting the human's face in the to-be-recognized image, acquiring a plurality of feature points associated with a plurality of facial features of the human's face in the to-be-recognized image, retrieving a plurality of mouth feature points corresponding to a human's mouth from the feature points, obtaining a projection coordinate of the mouth feature points according to the calibration information and providing the projection coordinate to the projection module, and commanding the projection module to project a projection pattern with shape corresponding to the human's mouth onto the human's face according to the projection coordinate. ### Claim 29 (depends on 28) The system applied to oral inspection according to claim 28, wherein the projection module and the image-capturing module are all-in-one. ### Claim 30 (depends on 28) The system applied to oral inspection according to claim 28, wherein the processor searches for a nose area according to the facial features and retrieves a nose feature point from the nose area, obtains a depth value corresponding to the nose feature point, and transforms coordinates of the nose feature points into the projection coordinate using the calibration information corresponding to the depth value. ### Claim 31 (depends on 30) The system applied to oral inspection according to claim 30, wherein the nose feature point is a feature point of a nose tip. ### Claim 32 (depends on 28) The system applied to oral inspection according to claim 28, wherein the image-capturing module is a depth camera. ### Claim 33 A machining system, provided with a calibration mode and a projection mode, comprising: a robotic arm configured for machining a workpiece along a machining path; a projection module configured for projecting a calibration image onto a projection screen in the calibration mode; an image-capturing module configured for capturing the calibration image in the calibration mode, and capturing the workpiece in the projection mode to obtain a to-be-recognized image of the workpiece; and a processor coupled to the projection module and the image-capturing module and configured for, in the calibration mode, obtaining a calibration information between the projection module and the image-capturing module according to the captured calibration image, and, in the projection mode, detecting the workpiece in the to-be-recognized image, acquiring a plurality of feature points associated with a plurality of feature areas of the workpiece in the to-be-recognized image, retrieving a plurality of target feature points corresponding to the machining path according to the feature points, obtaining a projection coordinate of the target feature points according to the calibration information and providing the projection coordinate to the projection module, and commanding the projection module to project a machining path pattern onto the workpiece according to the projection coordinate. ### Claim 34 (depends on 33) The machining system according to claim 33, wherein the feature points correspond to any point on the contour of the workpiece. ### Claim 35 (depends on 33) The machining system according to claim 33, wherein the processor searches for a central area of the workpiece according to the feature areas and retrieves a central point corresponding to the central area, obtains a depth value corresponding to the central point from the image-capturing module, and transforms coordinates of the target feature points into the projection coordinate using the calibration information corresponding to the depth value.