# US 10,768,294 B2 · System and method for multimodal, motion-aware radar imaging > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,768,294 B2 - Document ID: US10768294B2 - Status: Granted - Kind: B2 - Application number: 15/924,710 - Filed: 2018-03-19 - Published: 2019-09-19 - Granted: 2020-09-08 - Earliest claimed: 2018-03-19 - Assignees: Mitsubishi Electric Research Laboratories, Inc. - Applicants: Mitsubishi Electric Research Laboratories, Inc. - Inventors: Petros T Boufounos; Hassan Mansour; Rakshith Sharma Srinivasa; Jeroen van Baar - Web page: https://patentgrep.com/patent/US10768294B2 ## CPC - G01S 13/89: for mapping or imaging - G01S 7/415: {Identification of targets based on measurements of movement associated with the target} - G01S 13/72: for two-dimensional [2D] tracking, e.g. combination of angle and range tracking, track-while-scan radar - G01S 13/867: {Combination of radar systems with cameras} - G01S 13/887: {for detection of concealed objects, e.g. contraband or weapons} - G01S 13/9005: {with optical processing of the SAR signals} - G01S 13/9029: {specially adapted for moving target detection within a single SAR image or within multiple SAR images taken at the same time} - G06F 18/25: Fusion techniques - G06T 3/16: Spatio-temporal transformations, e.g. video cubism - G01S 13/9064: {Inverse SAR [ISAR]} - G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract A radar imaging system to reconstruct a radar reflectivity image of a scene including an object moving with the scene, includes an optical sensor to track the object over a period of time including multiple time steps to produce, for each of the multiple time steps, a deformation of a nominal shape of the object, and an electromagnetic sensor to acquire snapshots of the scene over the multiple time steps to produce a set of radar reflectivity image of the object with deformed shapes defined by the corresponding deformations of the nominal shape of the object. The system also includes a processor configured to determine, for each of the multiple time steps using the deformation determined for the corresponding time step, a transformation between the radar reflectivity image of the object acquired by the electromagnetic sensor at the corresponding time step and a radar reflectivity image of the object in the prototypical pose, and to combine the radar reflectivity images of the object with deformed shapes transformed with the corresponding transformations to produce the radar reflectivity image of the object in the prototypical pose. ## Claims ### Claim 1 A radar imaging system to reconstruct a radar reflectivity image of a scene including an object moving with the scene, comprising: an optical sensor to track the object over a period of time including multiple time steps to produce, for each of the multiple time steps, a deformation of a nominal shape of the object, wherein the nominal shape of the object is a shape of the object arranged in a prototypical pose; at least one electromagnetic sensor to acquire snapshots of the scene over the multiple time steps to produce a set of radar reflectivity image of the object with deformed shapes defined by the corresponding deformations of the nominal shape of the object, wherein at least two different radar reflectivity images include the object with different deformed shapes; and a processor to determine, for each of the multiple time steps using the deformation determined for the corresponding time step, a transformation between the radar reflectivity image of the object acquired by the electromagnetic sensor at the corresponding time step and a radar reflectivity image of the object in the prototypical pose; and to combine the radar reflectivity images of the object with deformed shapes transformed with the corresponding transformations to produce the radar reflectivity image of the object in the prototypical pose. ### Claim 2 (depends on 1) The radar imaging system of claim 1, wherein the processor determines each transformation as a subsampled permutation that permutes locations of some points of the radar reflectivity image of the object in the prototypical pose and removes different points of the radar reflectivity image of the object in the prototypical pose in dependence of the deformation of the nominal shape of the object in the prototypical pose. ### Claim 3 (depends on 2) The radar imaging system of claim 2, wherein the transformation is the discretization of a blurred or filtered subsampled rearranging of coordinates in a continuous space. ### Claim 4 (depends on 2) The radar imaging system of claim 2, the subsampled permutation preserves intensities of the points in permuted locations. ### Claim 5 (depends on 1) The radar imaging system of claim 1, further comprising: a tracking system including the optical sensor to produce each deformation to include an optical transformation between points of an optical reflectivity image including the object in the deformed shape and points of a prototypical optical reflectivity image including the object in the nominal shape, wherein the processor determines the transformation as a function of the optical transformation. ### Claim 6 (depends on 1) The radar imaging system of claim 1, wherein the nominal shape of the object arranged in the prototypical pose is selected as a shape of the object in one of an optical or radar reflectivity image, such that the transformation between the radar image of the object having the selected shape and the radar reflectivity image of the object in the prototypical pose is an identity transformation. ### Claim 7 (depends on 1) The radar imaging system of claim 1, further comprising: an input interface to accept information indicative of selection of the nominal shape of the object arranged in the prototypical pose; and an output interface to render the radar reflectivity image of the object in a selected pose, including the prototypical pose. ### Claim 8 (depends on 1) The radar imaging system of claim 1, wherein the processor determines the radar reflectivity image of the object in the prototypical pose that, when transformed by the transformation determined for each time step and acquired by the radar acquisition function, is consistent with the measurements at all of the multiple time steps. ### Claim 9 (depends on 8) The radar system of claim 8, wherein, for the determining the radar reflectivity image of the object in the prototypical pose, the processor regularizes using one or combination of sparsity in basis or dictionary, smoothness under a basis or dictionary transformation, and a total variation of the reflectivity image. ### Claim 10 (depends on 1) The radar imaging system of claim 1, wherein the processor adjusts each transformation with a local error correction and determines concurrently the radar image of the object in the prototypical pose and each local error correction. ### Claim 11 (depends on 10) The radar imaging system of claim 10, wherein the processor determines concurrently the radar image of the object in the prototypical pose and each local error correction using one or combination of alternating minimization, projections, and constrained regularization. ### Claim 12 (depends on 1) The radar imaging system of claim 1, wherein the object moving in the scene is a person. ### Claim 13 (depends on 1) The radar imaging system of claim 1, further comprising: a plurality of electromagnetic sensors with a fixed aperture size, wherein the processor determines the radar reflectivity image of the object in the prototypical pose by combining measurements of each electromagnetic sensor. ### Claim 14 (depends on 13) The radar imaging system of claim 13, wherein the electromagnetic sensors are moving according to known motions, and wherein the processor adjusts the transformation of the radar reflectivity image of the object acquired by the electromagnetic sensor at the corresponding time step based on the know motion of the electromagnetic sensors for the corresponding time step. ### Claim 15 (depends on 1) The radar imaging system of claim 1, wherein the electromagnetic sensor includes one or combination of a mmWave radar, a Thz imaging sensor, and a backscatter X-ray sensor, wherein the optical sensor includes one or combination of an optical camera, a depth camera, and an infrared camera. ### Claim 16 (depends on 1) The radar imaging system of claim 1, wherein a resolution of the radar reflectivity image of the object in the prototypical pose is greater than resolutions of the radar reflectivity images of the object with deformed shapes. ### Claim 17 A method for reconstructing a radar reflectivity image of a scene including an object moving with the scene, wherein the method uses a processor operatively connected to an optical sensor and an electromagnetic sensor, wherein the processor is coupled with stored instructions implementing the method, wherein the instructions, when executed by the processor carry out steps of the method, comprising: tracking, using the optical sensor, the object over a period of time including multiple time steps to produce, for each of the multiple time steps, a deformation of a nominal shape of the object, wherein the nominal shape of the object is a shape of the object arranged in a prototypical pose; acquiring, using the electromagnetic sensor, snapshots of the scene over the multiple time steps to produce a set of radar reflectivity images of the object with deformed shapes defined by the corresponding deformations of the nominal shape of the object, wherein at least two different radar reflectivity images include the object with different deformed shapes; determining, for each of the multiple time steps using the deformation determined for the corresponding time step, a transformation between the radar reflectivity image of the object acquired by the electromagnetic sensor at the corresponding time step and a radar reflectivity image of the object in the prototypical pose; and combining the radar reflectivity images of the object with deformed shapes transformed with the corresponding transformations to produce the radar reflectivity image of the object in the prototypical pose. ### Claim 18 (depends on 17) The method of claim 17, wherein the processor determines each transformation as a subsampled permutation that permutes locations of some points of the radar reflectivity image of the object in the prototypical pose and removes different points of the radar reflectivity image of the object in the prototypical pose in dependence of the deformation of the nominal shape of the object in the prototypical pose. ### Claim 19 (depends on 17) The method of claim 17, wherein the object moving in the scene is a person. ### Claim 20 A non-transitory computer readable storage medium embodied thereon a program executable by a processor for performing a method, wherein the processor is operatively connected to an optical sensor and an electromagnetic sensor, the method comprising: tracking, using the optical sensor, the object over a period of time including multiple time steps to produce, for each of the multiple time steps, a deformation of a nominal shape of the object, wherein the nominal shape of the object is a shape of the object arranged in a prototypical pose; acquiring, using the electromagnetic sensor, snapshots of the scene over the multiple time steps to produce a set of radar reflectivity image of the object with deformed shapes defined by the corresponding deformations of the nominal shape of the object, wherein at least two different radar reflectivity images include the object with different deformed shapes; determining, for each of the multiple time steps using the deformation determined for the corresponding time step, a transformation between the radar reflectivity image of the object acquired by the electromagnetic sensor at the corresponding time step and a radar reflectivity image of the object in the prototypical pose; and combining the radar reflectivity images of the object with deformed shapes transformed with the corresponding transformations to produce the radar reflectivity image of the object in the prototypical pose.