# US 12,340,502 B2 · Projector assembly system and method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,340,502 B2 - Document ID: US12340502B2 - Status: Granted - Kind: B2 - Application number: 17/615,699 - Filed: 2020-06-02 - Published: 2022-07-28 - Granted: 2025-06-24 - Earliest claimed: 2019-06-03 - Assignees: InspecVision Limited - Applicants: InspecVision Limited - Inventors: Jan Antonis - Web page: https://patentgrep.com/patent/US12340502B2 ## CPC - G06T 7/001: {using an image reference approach} - G09G 3/002: {to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT} - G01B 11/25: by projecting a pattern, e.g. {one or more lines,} moiré fringes on the object (G01B11/255 takes precedence {; image analysis for depth or shape recovery G06T7/50}) - G06T 2207/20081: Training; Learning - G06T 2207/20224: Image subtraction - G06T 2207/30108: Industrial image inspection - G06T 2207/30164: Workpiece; Machine component - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS ## Abstract A method of assembling a plurality of objects comprises: creating a pixel map of a current scene in a work zone by mapping pixels of a camera to pixels of a projector; comparing the current scene pixel map with a corresponding previous scene pixel map; creating a projector pixel set depending on the differences between the current scene pixel map and the previous scene pixel map; and causing the projector to illuminate the work zone using the projector pixel set. The method may include a training mode in which the correct assembly steps are learned, a guidance mode in which a user is guided through the assembly steps, and a verification mode in which the actual assembly steps are verified. ## Claims ### Claim 1 A method of assembling a plurality of objects in a work zone using a camera and a projector, wherein said camera is configured to capture images defined by camera pixels, and said projector is configured to project images defined by projector pixels, the method comprising: in a training mode, in respect of a first step of a sequence of assembly steps: arranging at least one object correctly in said work zone; causing said projector to illuminate a first current scene in said work zone by projecting at least one projector image onto said work zone, wherein said at least one projector image is defined by projector pixels, wherein said first current scene comprises said at least one object correctly arranged in said work zone; causing said camera to capture at least one captured image of said first current scene in said work zone illuminated by said at least one projector image, wherein said at least one captured image is defined by camera pixels; mapping the camera pixels of said at least one captured image to the projector pixels of said at least one projector image to create a first current scene pixel map that defines a correspondence between said camera pixels and said projector pixels for said first step of said sequence of assembly steps; comparing said first current scene pixel map with a corresponding other reference scene pixel map for said work zone to identify at least one difference between said first current scene pixel map and said reference scene pixel map; creating a projector pixel set for said first step, said projector pixel set representing said at least one difference identified between said first current scene pixel map and said reference scene pixel map; and in said training mode, in respect of each subsequent step of said sequence of assembly steps: arranging at least one other object correctly in said work zone, and/or rearranging said at least one object in said work zone; causing said projector to illuminate a respective current scene in said work zone by projecting at least one projector image onto said work zone, wherein said at least one projector image is defined by projector pixels; causing said camera to capture at least one captured image of said respective current scene in said work zone illuminated by said at least one projector image, wherein said at least one captured image is defined by camera pixels; mapping the camera pixels of said at least one captured image to the projector pixels of said at least one projector image to create a respective current scene pixel map that defines a correspondence between said camera pixels and said projector pixels for the respective subsequent step of said sequence of assembly steps; comparing said respective current scene pixel map with a preceding scene pixel map for a preceding step of said sequence of assembly steps to identify at least one difference between said respective current scene pixel map and said preceding scene pixel map; creating a projector pixel set for the respective subsequent step, said projector pixel set representing said at least one difference identified between said respective current scene pixel map and said preceding scene pixel map; and in a guidance mode or in a verification mode, in respect of said first step: causing said projector to illuminate said work zone using said projector pixel set for said first step to indicate, by illumination of said work zone, said at least one difference identified between said first current scene pixel map and said reference scene pixel map, and in said guidance mode or in said verification mode, in respect of each subsequent step of said sequence of assembly steps: causing said projector to illuminate said work zone using said projector pixel set for the respective subsequent step to indicate, by illumination of said work zone, said at least one difference identified between said respective current scene pixel map and said preceding scene pixel map. ### Claim 2 (depends on 1) The method of claim 1, wherein creating said first current scene pixel map and said respective current scene pixel map involves causing the projector to project a light pattern with a known structure to said work zone, causing said camera to capture the projected structured light pattern, and mapping said camera pixels with said projector pixels using said light pattern structure. ### Claim 3 (depends on 2) The method of claim 2, wherein causing the projector to project said light pattern with a known structure to said work zone involves causing said projector to project a sequence of images to said work zone. ### Claim 4 (depends on 1) The method of claim 1, wherein comparing said first current scene pixel map with said reference scene pixel map involves determining the or each difference between said first current scene pixel map with said reference scene pixel map, and wherein said comparing said respective current scene pixel map with said preceding scene pixel map involves determining the or each difference between said respective current scene pixel map with said preceding scene pixel map. ### Claim 5 (depends on 1) The method of claim 1, wherein comparing said first current scene pixel map with said reference scene pixel map involves subtracting said current scene pixel map from said reference scene pixel map, or subtracting said reference scene pixel map from said first current scene pixel map, and wherein said comparing said respective current scene pixel map with preceding scene pixel map involves subtracting said respective current scene pixel map from said preceding scene pixel map, or subtracting said preceding reference scene pixel map from said respective current scene pixel map. ### Claim 6 (depends on 1) The method of claim 1, further including, in said verification mode, for said first step and said respective subsequent step, directly or indirectly determining a size of said projector pixel set, and causing said projector to illuminate said work zone using said projector pixel set if said size exceeds a threshold value. ### Claim 7 An assembly system comprising: at least one camera configured to capture images defined by camera pixels; at least one projector configured to project images defined by projector pixels; and a computer-implemented controller, said controller being configured to, in a training mode, in respect of a first step of a sequence of assembly steps: cause said projector to illuminate a first current scene in said work zone by projecting at least one projector image onto said work zone, wherein said at least one projector image is defined by projector pixels, wherein said first current scene comprises said at least one object correctly arranged in said work zone; cause said camera to capture at least one captured image of said first current scene in said work zone illuminated by said at least one projector image, wherein said at least one captured image is defined by camera pixels; map the camera pixels of said at least one captured image to the projector pixels of said at least one projector image to create a first current scene pixel map that defines a correspondence between said camera pixels and said projector pixels for said first step of said sequence of assembly steps; compare said first current scene pixel map with a reference scene pixel map for said work zone to identify at least one difference between said first current scene pixel map and said reference scene pixel map; and create a projector pixel set for said first step, said projector pixel set representing said at least one difference identified between said first current scene pixel map and said reference scene pixel map, said controller being further configured to, in said training mode, in respect of each subsequent step of said sequence of assembly steps: cause said projector to illuminate a respective current scene in said work zone by projecting at least one projector image onto said work zone, wherein said at least one projector image is defined by projector pixels; cause said camera to capture at least one captured image of said respective current scene in said work zone illuminated by said at least one projector image, wherein said at least one captured image is defined by camera pixels; map the camera pixels of said at least one captured image to the projector pixels of said at least one projector image to create a respective current scene pixel map that defines a correspondence between said camera pixels and said projector pixels for the respective subsequent step of said sequence of assembly steps; compare said respective current scene pixel map with a preceding scene pixel map for a preceding step of said sequence of assembly steps to identify at least one difference between said respective current scene pixel map and said preceding scene pixel map; and create a projector pixel set for the respective subsequent step, said projector pixel set representing said at least one difference identified between said respective current scene pixel map and said preceding scene pixel map, said controller being further configured to, in a guidance mode or in a verification mode, in respect of said first step: cause said projector to illuminate said work zone using said projector pixel set for said first step to indicate, by illumination of said work zone, said at least one difference identified between said first current scene pixel map and said reference scene pixel map, said controller being further configured to, in said guidance more or in said verification mode, in respect of each subsequent step of said sequence of assembly steps: cause said projector to illuminate said work zone using said projector pixel set for the respective subsequent step to indicate, by illumination of said work zone, said at least one difference identified between said respective current scene pixel map and said preceding scene pixel map. ### Claim 8 (depends on 7) The assembly system of claim 7, wherein said controller is configured to create said first current scene pixel map and said respective scene pixel map by causing the projector to project a light pattern with a known structure to said work zone, causing said camera to capture the projected structured light pattern, and mapping said camera pixels with said projector pixels using said light pattern structure. ### Claim 9 (depends on 7) The assembly system of claim 7, wherein said controller is configured to cause the projector to project said light pattern with a known structure to said work zone by causing said projector to project a sequence of images to said work zone. ### Claim 10 (depends on 7) The assembly system of claim 7, wherein said controller is configured to compare said first current scene pixel map with said reference scene pixel map by determining the or each difference between said first current scene pixel map and said reference scene pixel map, and to compare said respective current scene pixel map with said preceding scene pixel map by determining the or each difference between said respective current scene pixel map and said preceding scene pixel map. ### Claim 11 (depends on 7) The assembly system of claim 7, wherein said controller is configured to compare said first current scene pixel map with said reference scene pixel map by subtracting said first current scene pixel map from said reference scene pixel map, or subtracting said reference scene pixel map from said first current scene pixel map, and to compare said respective current scene pixel map with preceding scene pixel map by subtracting said respective current scene pixel map from said preceding scene pixel map, or subtracting said preceding reference scene pixel map from said respective current scene pixel map. ### Claim 12 (depends on 7) The assembly system of claim 7, wherein said controller is configured to, in said verification mode, in respect of said first step and said respective subsequent step, directly or indirectly determine a size of said projector pixel set, and to cause said projector to illuminate said work zone using said projector pixel set if said size exceeds a threshold value.