# US 11,745,352 B2 · Vision sensor system, control method, and non-transitory computer readable storage medium > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,745,352 B2 - Document ID: US11745352B2 - Status: Granted - Kind: B2 - Application number: 17/042,934 - Filed: 2019-05-09 - Published: 2021-01-21 - Granted: 2023-09-05 - Earliest claimed: 2018-05-29 - Assignees: OMRON Corporation - Applicants: OMRON Corporation - Inventors: Toyoo Iida - Web page: https://patentgrep.com/patent/US11745352B2 ## CPC - B25J 9/1697: {Vision controlled systems} - B25J 9/1692: {Calibration of manipulator} - B25J 13/089: {Determining the position of the robot with reference to its environment} - B25J 19/023: {including video camera means} - G01B 11/22: for measuring depth - G01B 11/2513: {with several lines being projected in more than one direction, e.g. grids, patterns} - G01B 11/2518: {Projection by scanning of the object} - G06T 5/50: using two or more images, e.g. averaging or subtraction - G06T 5/77: Retouching; Inpainting; Scratch removal - G01B 2210/52: Combining or merging partially overlapping images to an overall image - G05B 2219/39016: Simultaneous calibration of manipulator and camera - G05B 2219/39039: Two cameras detect same reference on workpiece to define its position in space - G05B 2219/39045: Camera on end effector detects reference pattern - G05B 2219/39057: Hand eye calibration, eye, camera on hand, end effector - G05B 2219/40613: Camera, laser scanner on end effector, hand eye manipulator, local - G06T 2200/04: involving 3D image data - G06T 2207/10016: Video; Image sequence - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2207/30164: Workpiece; Machine component - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) ## Abstract On the basis of a measured shape, which is a three-dimensional shape of a subject measured by means of a measuring unit using an image captured when an image capturing unit is disposed in a first position and a first attitude, a movement control unit determines a second position and a second attitude for capturing an image of the subject again, and sends an instruction to a movement mechanism. The three-dimensional shape is represented by means of height information from a reference surface. The movement control unit extracts, from the measured shape, a deficient region having deficient height information, and determines the second position and the second attitude on the basis of the height information around the deficient region of the measured shape. The position and attitude of the image capturing unit can be determined in such a way as to make it easy to eliminate the effects of shadows. ## Claims ### Claim 1 A vision sensor system comprising: a camera that captures an image of a subject; a movement mechanism that disposes the camera in an instructed position with an instructed attitude; a first processor that measures a three-dimensional shape of a surface of the subject using an image captured by the camera; and a second processor that determines a second position and a second attitude for capturing an image of the subject again on the basis of a measured shape which is a three-dimensional shape of the subject measured by means of the first processor using an image captured when the camera is disposed in a first position and a first attitude, and sends an instruction to the movement mechanism, wherein the three-dimensional shape is represented by means of height information from a reference surface, and the second processor extracts a deficient region which is deficient in the height information from the measured shape, and determines the second position and the second attitude on the basis of the height information around the deficient region in the measured shape, wherein the second processor determines the second position and the second attitude so that height information of a target segment that links two points around the deficient region in the measured shape is able to be acquired. ### Claim 2 (depends on 1) The vision sensor system according to claim 1, wherein the second processor obtains a first angle between the target segment and a first plane perpendicular to an optical axis of the camera disposed in the first position and the first attitude, determines a difference between the first angle and a second angle which is determined in advance in accordance with characteristics of the camera and the subject as an angle of movement from the first attitude to the second attitude, and determines an attitude, rotated by the angle of movement from the first attitude along a second plane including the target segment and perpendicular to the reference surface in a direction in which an angle between the target segment and the optical axis of the camera approaches 90 degrees, as the second attitude. ### Claim 3 (depends on 2) The vision sensor system according to claim 2, wherein a segment obtained by projecting the target segment onto the reference surface is parallel to a direction determined in advance. ### Claim 4 (depends on 2) The vision sensor system according to claim 2, wherein, in the measured shape, one of the two points is a point at which the height information is maximum around the deficient region, and the other of the two points is a point at which the height information is minimum around the deficient region. ### Claim 5 (depends on 2) The vision sensor system according to claim 2, wherein the second processor determines the second position so that an optical axis of the camera passes through a reference point which is a point having the larger height information out of the two points. ### Claim 6 (depends on 2) The vision sensor system according claim 2, wherein the second processor determines the second position so that an optical axis of the camera passes through a reference point which is a point having the smaller height information out of the two points. ### Claim 7 (depends on 2) The vision sensor system according to claim 2, wherein the second processor determines the second position so that an optical axis of the camera passes through a reference point which is a midpoint between the two points. ### Claim 8 (depends on 1) The vision sensor system according to claim 1, wherein, in the measured shape, one of the two points is a point at which the height information is maximum around the deficient region, and the other of the two points is a point at which the height information is minimum around the deficient region. ### Claim 9 (depends on 1) The vision sensor system according to claim 1, wherein the second processor determines the second position and the second attitude so that height information of a target region overlapping the deficient region in an approximate plane approximating at least three points around the deficient region in the measured shape is able to be acquired. ### Claim 10 (depends on 9) The vision sensor system according to claim 9, wherein the second processor obtains a first angle between the target region and a first plane perpendicular to an optical axis of the camera disposed in the first position and the first attitude, determines a difference between the first angle and a second angle which is determined in advance in accordance with characteristics of the camera and the subject as an angle of movement from the first attitude to the second attitude, and determines an attitude, rotated by the angle of movement from the first attitude along a second plane including a normal line of the target region and perpendicular to the reference surface in a direction in which an angle between the target region and the optical axis of the camera approaches 90 degrees, as the second attitude. ### Claim 11 (depends on 10) The vision sensor system according to claim 10, wherein the second processor determines the second position so that an optical axis of the camera passes through a reference point within the target region. ### Claim 12 (depends on 9) The vision sensor system according to claim 9, wherein the second processor determines the second position so that an optical axis of the camera passes through a reference point within the target region. ### Claim 13 (depends on 1) The vision sensor system according to claim 1, wherein the second processor determines the second position so that an optical axis of the camera passes through a reference point which is a point having the larger height information out of the two points. ### Claim 14 (depends on 13) The vision sensor system according to claim 13, wherein the second processor determines the second position so that a distance between the reference point and the first position and a distance between the reference point and the second position become equal to each other. ### Claim 15 (depends on 1) The vision sensor system according claim 1, wherein the second processor determines the second position so that an optical axis of the camera passes through a reference point which is a point having the smaller height information out of the two points. ### Claim 16 (depends on 15) The vision sensor system according to claim 15, wherein the second processor determines the second position so that a distance between the reference point and the first position and a distance between the reference point and the second position become equal to each other. ### Claim 17 (depends on 1) The vision sensor system according to claim 1, wherein the second processor determines the second position so that an optical axis of the camera passes through a reference point which is a midpoint between the two points. ### Claim 18 A control method of controlling a vision sensor system provided with an image capturing unit that captures an image of a subject, a movement mechanism that disposes the image capturing unit in an instructed position with an instructed attitude, and a measuring unit that measures a three-dimensional shape of a surface of the subject using an image captured by the image capturing unit, wherein the three-dimensional shape is represented by means of height information from a reference surface, and the control method comprises: a step of extracting a deficient region which is deficient in the height information from a measured shape which is a three-dimensional shape of the subject measured by means of the measuring unit using an image captured when the image capturing unit is disposed in a first position and a first attitude; and a step of determining a second position and a second attitude for capturing an image of the subject again on the basis of the height information around the deficient region in the measured shape, wherein the step of determining the second position and the second attitude for capturing the image of the subject again on the basis of the height information around the deficient region in the measured shape comprises: determining the second position and the second attitude so that height information of a target segment that links two points around the deficient region in the measured shape is able to be acquired. ### Claim 19 A non-transitory computer readable storage medium storing a program for supporting a vision sensor system provided with an image capturing unit that captures an image of a subject, a movement mechanism that disposes the image capturing unit in an instructed position with an instructed attitude, and a measuring unit that measures a three-dimensional shape of a surface of the subject using an image captured by the image capturing unit, wherein the three-dimensional shape is represented by means of height information from a reference surface, and the program causes a computer to execute: a step of extracting a deficient region which is deficient in the height information from a measured shape which is a three-dimensional shape of the subject measured by means of the measuring unit using an image captured when the image capturing unit is disposed in a first position and a first attitude; and a step of determining a second position and a second attitude for capturing an image of the subject again on the basis of the height information around the deficient region in the measured shape, wherein the step of determining the second position and the second attitude for capturing the image of the subject again on the basis of the height information around the deficient region in the measured shape comprises: determining the second position and the second attitude so that height information of a target segment that links two points around the deficient region in the measured shape is able to be acquired.