# US 12,008,785 B2 · Detection, 3D reconstruction and tracking of multiple rigid objects moving in relation to one another > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,008,785 B2 - Document ID: US12008785B2 - Status: Granted - Kind: B2 - Application number: 17/596,245 - Filed: 2020-05-28 - Published: 2022-10-06 - Granted: 2024-06-11 - Earliest claimed: 2019-06-05 - Assignees: Conti Temic microelectronic GmbH - Applicants: Conti Temic microelectronic GmbH - Inventors: Axel Roth - Web page: https://patentgrep.com/patent/US12008785B2 ## CPC - G06T 7/73: using feature-based methods - G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments - G06T 7/579: from motion - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06T 2207/30241: Trajectory - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract The disclosure relates to a method and a system for the detection, 3D reconstruction and tracking of multiple rigid objects moving relative to one another from a series of images from at least one camera and can be used, in particular, in the context of a camera-based environment detection system for assisted or automated driving. ## Claims ### Claim 1 A method for detection, 3D reconstruction and tracking of multiple rigid objects moving relative to one another from a series of images from at least one camera, comprising: selecting, by processor circuitry, images at determined capture times (key cycles) from the series of images from the at least one camera; collectively optimizing, by the processor circuitry, all parameters of a model for describing the rigid objects based on the selected images from the key cycles on a basis of sparsely selected pixels in the key cycles, wherein the model comprises parameters for describing a number, a 3D structure, and positions of the rigid objects in the key cycles and an assignment of the sparsely selected pixels to the rigid objects; by minimizing, by the processor circuitry, an error function, wherein the error function comprises a photometric error Ephoto according to intensities of images of multiple key cycles, and a first a priori energy term Ecomp according to the number of the rigid objects; and outputting, by the processor circuitry, the number, the 3D structure and a trajectory of the rigid objects from the series of images. ### Claim 2 (depends on 1) The method according to claim 1, wherein the error function comprises a second a priori energy term Egeo according to the 3D structure of the rigid objects. ### Claim 3 (depends on 1) The method according to claim 1, wherein the error function comprises the following model parameters: an inverse depth for each sparsely selected pixel per movement model; a number of movement models, wherein a movement model is assigned to each currently assumed rigid object; a position per movement model and active key cycle; and an assignment probability of each sparsely selected pixel to each movement model. ### Claim 4 (depends on 3) The method according to claim 3, wherein the error function additionally comprises the following model parameters: a sensor characteristic for each selected image, and a focal length and a principal point for each of the at least one camera so that the collective optimization of all parameters occasions an autocalibration of the at least one camera. ### Claim 5 (depends on 1) The method according to claim 1, wherein a direct image alignment is performed by the processor circuitry with one or more image pyramid levels for tracking of individual objects. ### Claim 6 (depends on 1) The method according to claim 1, wherein, alternately, trajectory and structural parameters are optimized by the processor circuitry by a photometric bundle adjustment based on object-specific key cycles and an affiliation of pixels to a movement model is optimized by the processor circuitry, in order to optimize the error function. ### Claim 7 (depends on 6) The method according to claim 6, wherein a number of movement models is subsequently optimized by the processor circuitry, wherein in an event of a movement model being added to the error function or of a movement model being removed from the error function, a new assignment of the sparsely selected pixels to movement models is conducted by the processor circuitry, and the optimization of the error function is restarted by the processor circuitry. ### Claim 8 (depends on 1) The method according to claim 1, wherein the at least one camera executes a movement with respect to an object which corresponds to a stationary rigid background. ### Claim 9 (depends on 1) The method according to claim 1, wherein a multiplicity of series of images is captured with a camera system comprising multiple synchronized cameras, wherein the parameters of the model comprise the positions of each rigid object with regard to the camera system, wherein pixels are selected from all cameras of the camera system, wherein pixels are selected from images of the key cycles of at least one camera of the camera system, wherein observations in the at least one camera of the camera system and at least one key cycle of the key cycles are included as energy terms of the photometric error for selected pixels, and wherein a collective optimization of all parameters of the model is performed by the processor circuitry in order to minimize the error function. ### Claim 10 (depends on 9) The method according to claim 9, wherein a multi-camera direct image alignment is performed by the processor circuitry with one or more image pyramid levels for tracking of individual objects, wherein the collective optimization of all the parameters of the model is performed by the processor circuitry in order to minimize the error function, wherein the parameters of the model comprise the positions of each rigid object with regard to the camera system, wherein observations in the at least one camera of the camera system of the cycle to be optimized are included as energy terms of the photometric error for selected pixels. ### Claim 11 (depends on 10) The method according to claim 10, wherein the parameters of the model comprise further intrinsic photometric, further intrinsic geometric and/or further extrinsic camera parameters of the at least one camera of the camera system so that the collective optimization of all parameters of the model occasions an autocalibration of the at least one camera of the camera system. ### Claim 12 A device for detection, 3D reconstruction and tracking of multiple rigid objects moving relative to one another from a series of images from at least one camera received by an input unit, comprising the input unit, a selection unit, an optimization unit, and an output unit, wherein the selection unit comprises processor circuitry that is configured to select images at determined capture times (key cycles) from the series of images from the at least one camera; the optimization unit comprises the processor circuitry that is configured to collectively optimize all model parameters of a model for describing the rigid objects moving relative to one another based on the selected images from the key cycles on a basis of sparsely selected pixels in the key cycles, wherein the model comprises parameters for describing a number, a 3D structure, and positions of the rigid objects in the key cycles and an assignment of the sparsely selected pixels to the rigid objects; by minimizing an error function, wherein the error f unction comprises a photometric error Ephoto and a first a priori energy term Ecomp according to the number of the rigid objects; and the output unit comprises the processor circuitry that is configured to output the number, the 3D structure and a trajectory of the rigid objects from the series of images.