# US 10,319,111 B2 · Image projection device for 3D measurement and calibration method for calibration of camera and projector > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,319,111 B2 - Document ID: US10319111B2 - Status: Granted - Kind: B2 - Application number: 14/571,707 - Filed: 2014-12-16 - Published: 2015-07-02 - Granted: 2019-06-11 - Earliest claimed: 2013-12-27 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Yohsuke Kaji - Web page: https://patentgrep.com/patent/US10319111B2 ## CPC - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 3/18: Image warping, e.g. rearranging pixels individually - G06T 5/80: Geometric correction - H04N 9/3182: {Colour adjustment, e.g. white balance, shading or gamut (white balance per se H04N9/73; control of amplitude of colour signals H04N9/68; colour control circuits for displays in general G09G5/02, G09G3/2003, G09G3/3607)} - H04N 9/3185: {Geometric adjustment, e.g. keystone or convergence (optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14)} - H04N 9/3194: {including sensor feedback} - G06T 2207/10024: Color image - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract There is provided an image projection device including a camera, a projector configured to project an image, a camera calibration unit configured to perform calibration of the camera, a projector calibration unit configured to perform calibration of the projector, and an image correction unit configured to correct the image projected from the projector based on a result of the calibration. The projector calibration unit performs ray tracing of a known checker pattern on which structural light is projected by the projector, and estimates parameters by acquiring correspondence relation between a lattice point of the checker pattern and projector coordinates. ## Claims ### Claim 1 An image projection device, comprising: a camera; a projector configured to: project a first image; and project structural light on a checker pattern; and circuitry configured to: calibrate the camera; ray trace the checker pattern; determine first distortion parameters of the projector based on evaluation of a distortion amount, wherein the distortion amount is based on a group of lattice points of the checker pattern, wherein the group of lattice points of the checker pattern constitutes a straight line on projector coordinates; determine a first correspondence relation between a first lattice point of the group of lattice points of the checker pattern and the projector coordinates, and a second correspondence relation between a second lattice point of the group of lattice points on a second image of the checker pattern after distortion correction and world coordinates; determine internal parameters of the projector and external parameters of the projector, based on the first correspondence relation; and determine a first perspective projection matrix that includes the internal parameters and the external parameters of the projector by first nonlinear optimization, based on the second correspondence relation; and generate a corrected image based on correction of the first image projected from the projector, wherein the first image projected from the projector is corrected based on at least one of the determined internal parameters, the determined external parameters, or the determined first distortion parameters. ### Claim 2 (depends on 1) The image projection device according to claim 1, wherein the circuitry is further configured to determine the internal parameters of the projector and the external parameters of the projector based on lens distortion of the projector, and wherein the determination of the internal parameters of the projector and the external parameters of the projector is independent of a calibration result of the camera. ### Claim 3 (depends on 1) The image projection device according to claim 1, wherein the circuitry is further configured to determine the first distortion parameters of the projector by a Levenberg-Marquardt (LM) method, with an inclination difference between two candidates of straight lines as the distortion amount, and wherein the group of lattice points constitutes the straight line on each of the two candidates of straight lines. ### Claim 4 (depends on 1) The image projection device according to claim 1, wherein the circuitry is further configured to determine the first perspective projection matrix based on the first nonlinear optimization of a re-projection error between detection image coordinates of the camera and re-projection image coordinates by a Levenberg-Marquardt (LM) method. ### Claim 5 (depends on 1) The image projection device according to claim 1, wherein the circuitry is further configured to determine the first correspondence relation with sub-pixel accuracy. ### Claim 6 (depends on 1) The image projection device according to claim 1, wherein the circuitry is further configured to determine LocalHomography of calibration of a plurality of projector pixels limited to surroundings of the first lattice point of the checker pattern. ### Claim 7 (depends on 6) The image projection device according to claim 6, wherein the circuitry is further configured to determine the LocalHomography of calibration of the plurality of projector pixels with a limit to a surrounding area of the first lattice point, and wherein the surrounding area of the first lattice point excludes influences of lens distortion of the projector. ### Claim 8 (depends on 7) The image projection device according to claim 7, wherein the circuitry is further configured to determine the LocalHomography of calibration of the plurality of projector pixels based on a decoding error of the structural light. ### Claim 9 (depends on 1) The image projection device according to claim 1, wherein the circuitry is further configured to: detect the first lattice point from a third image obtained based on photography of the checker pattern with the camera, determine second distortion parameters of the camera based on the evaluation of the distortion amount based on the group of lattice points, and determine internal parameters of the camera and external parameters of the camera based on the second correspondence relation. ### Claim 10 (depends on 9) The image projection device according to claim 9, wherein the circuitry is further configured to determine the second distortion parameters of the camera by a Levenberg-Marquardt (LM) method, with an inclination difference between two candidates of straight lines as the distortion amount, and wherein the group of lattice points constitutes the straight line on each of the two candidates of straight lines. ### Claim 11 (depends on 9) The image projection device according to claim 9, wherein the circuitry is further configured to determine a second perspective projection matrix including the internal parameters of the camera and the external parameters of the camera by second nonlinear optimization, based on the second correspondence relation. ### Claim 12 (depends on 11) The image projection device according to claim 11, wherein the circuitry is further configured to determine the second perspective projection matrix based on the second nonlinear optimization of a re-projection error between detection image coordinates of the camera and re-projection image coordinates by a Levenberg-Marquardt (LM) method. ### Claim 13 (depends on 1) The image projection device according to claim 1, wherein the circuitry is further configured to ray trace the world coordinates obtained by projection of a pixel on the projector coordinates with ray influenced by lens distortion. ### Claim 14 (depends on 1) The image projection device according to claim 1, wherein the circuitry is further configured to reverse ray trace a pixel on the projector coordinates, irradiated with ray, to project the world coordinates based on lens distortion. ### Claim 15 (depends on 14) The image projection device according to claim 14, wherein the circuitry is further configured to: ray trace a gray code projected by the projector on the checker pattern with the world coordinates, calculate a corresponding point of the world coordinates and the projector coordinates in a state with projector distortion, determine a distortion correction function, determine a third correspondence relation between the world coordinates and distortionless projection coordinates, and formulate a reverse ray tracing process. ### Claim 16 (depends on 15) The image projection device according to claim 15, wherein the circuitry is further configured to: determine a reverse distortion correction function D′ to correct distortionless projector coordinates to the projector coordinates in the state with projector distortion; and formulate a ray tracing process to ray trace the world coordinates projected on a projector pixel. ### Claim 17 (depends on 1) The image projection device according to claim 1, wherein the circuitry is further configured to determine at least one of the first distortion parameters, the internal parameters of the projector, the external parameters of the projector, internal parameter of the camera, or external parameter of the camera, with high accuracy based on a number of photographed images and depth information of a depth from the camera to a checker board, and wherein the check pattern in on the checker board. ### Claim 18 A calibration method, comprising: in an image projection device including a camera and a projector: projecting structural light on a checker pattern: ray tracing the checker pattern; determining distortion parameters of the projector by evaluating a distortion amount, wherein the distortion amount is based on a group of lattice points of the checker pattern, wherein the group of lattice points of the checker pattern constitutes a straight line on projector coordinates; determining a first correspondence relation between a first lattice point of the group of lattice points of the checker pattern and the projector coordinates and a second correspondence relation between a second lattice point of the group of lattice points on an image of the checker pattern after distortion correction and world coordinates; determining internal parameters of the projector and external parameters of the projector based on the first correspondence relation, wherein the determination of the internal parameters and the external parameters of the projector is independent of camera calibration parameters; and determining a perspective projection matrix that includes the internal parameters and the external parameters of the projector by nonlinear optimization, based on the second correspondence relation.