# US 2024/0257394 A1 · Method, system and computer readable medium for calibration of cooperative sensors

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0257394 A1
- Document ID: US20240257394A1
- Status: Published application
- Kind: A1
- Application number: 18/040,181
- Filed: 2021-07-30
- Published: 2024-08-01
- Earliest claimed: 2020-08-01
- Assignees: Curium Pte. Ltd.
- Applicants: Curium Pte. Ltd.
- Inventors: Ali Hasnain; Kutluhan Buyukburc; Pradeep Anand Ravindranath
- Web page: https://patentgrep.com/patent/US20240257394A1

## CPC

- G06N 3/08: Learning methods
- G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 20/00: Machine learning
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/762: using clustering, e.g. of similar faces in social networks
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V 20/64: Three-dimensional [3D] objects
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present application provides methods, systems and computer readable media for calibration of cooperative sensors. In an embodiment, there is provided a method for calibration of cooperative sensors. The method comprises: obtaining a first set of sensor data for an environment from a first sensor; obtaining a second set of sensor data for the environment from a second sensor that is cooperative with the first sensor; identifying one or more objects from the first set of sensor data and the second set of sensor data; generating a first point cloud data (PCD) representation for the one or more objects identified from the first set of sensor data; generating a second point cloud data (PCD) representation for the one or more objects identified from the second set of sensor data; identifying one or more common objects that are present in both the first PCD representation and the second PCD representation; identifying feature point pairs for each object in the one or more common objects, wherein each feature point pair of the feature point pairs comprises one or more feature points extracted from the first PCD representation and/or the second PCD representation corresponding to a same or similar feature of the object; and for each feature point pair of the feature point pairs, minimizing a distance between feature points in the feature point pair so as to form an extrinsic calibration matrix for calibrating the second sensor based on the first sensor.

## Claims

### Claim at position 1

1-46. (canceled)

### Claim 47

A method for calibration of cooperative sensors, the method comprising:
obtaining a first set of sensor data for an environment from a first sensor;
obtaining a second set of sensor data for the environment from a second sensor that is cooperative with the first sensor;
identifying one or more objects from the first set of sensor data and the second set of sensor data, wherein the one or more objects comprise one or more dynamic objects;
generating a first point cloud data (PCD) representation for the one or more objects identified from the first set of sensor data;
generating a second point cloud data (PCD) representation for the one or more objects identified from the second set of sensor data;
identifying one or more common objects that are present in both the first PCD representation and the second PCD representation;
identifying feature point pairs for each object in the one or more common objects, wherein each feature point pair of the feature point pairs comprises one or more feature points extracted from the first PCD representation and/or the second PCD representation corresponding to a same or similar feature of the object; and
for each feature point pair of the feature point pairs, minimizing a distance between feature points in the feature point pair so as to form an extrinsic calibration matrix for calibrating the second sensor based on the first sensor.

### Claim 48 (depends on 47)

The method according to claim 47, wherein the obtaining of the first set of sensor data comprises obtaining one or more frames of the first sensor, and wherein the obtaining of the second set of sensor data comprises obtaining one or more frames of the second sensor.

### Claim 49 (depends on 47)

The method according to claim 47, wherein the first sensor has a first field of view, the second sensor has a second field of view, and the first field of view overlaps with the second field of view.

### Claim 50 (depends on 47)

The method according to claim 47, wherein the one or more objects further comprise one or more static objects.

### Claim 51 (depends on 47)

The method according to claim 47, further comprising:
identifying the one or more common objects that are present in both the first PCD representation and the second PCD representation using prior knowledge or applying coarse sensor fusion to identify a common field of view for the first sensor and the second sensor, and
projecting the one or more common objects into a frame of reference of the second sensor for calibrating the second sensor based on the first sensor.

### Claim 52 (depends on 47)

The method according to claim 47, further comprising:
obtaining a pose data from one of the first sensor and the second sensor, the pose data indicating a pose of the one of the first sensor and the second sensor;
transforming the first PCD representation and the second PCD representation into a common frame of reference based on the pose, wherein the transforming includes applying a pose correction to one of the first sensor and the second sensor that does not provide the pose data; and
identifying the one or more common objects in the common frame of reference.

### Claim 53 (depends on 47)

The method according to claim 47, wherein prior to the identifying of one or more common objects that are present in both the first PCD representation and the second PCD representation, the method further comprises:
obtaining a first pose data from the first sensor, the first pose data indicating a first pose of the first sensor;
obtaining a second pose data from the second sensor, the second pose data indicating a second pose of the second sensor;
transforming the first PCD representation and the second PCD representation into a common frame of reference based on the first pose and the second pose; and
identifying the one or more common objects in the common frame of reference.

### Claim 54 (depends on 47)

The method according to claim 47, wherein prior to the generating of the first PCD representation for the one or more objects identified from the first set of sensor data, the method comprises:
segmenting the one or more objects from the first set of sensor data and the second set of sensor data based on one or more of the following:
a machine/deep learning approach,
a Computer Vision approach, and
prior object knowledge or position and geometry of the one or more objects.

### Claim 55 (depends on 47)

The method according to claim 47, further comprising:
segmenting while identifying the one or more objects from the first set of sensor data and the second set of sensor data based on one or more of the following:
a machine/deep learning approach,
a Computer Vision approach, and
prior object knowledge or position and geometry of the one or more objects.

### Claim 56 (depends on 47)

The method according to claim 47, wherein the first sensor or the second sensor is one of the following:
a camera sensor,
a Light Detection and Ranging (LiDAR) sensor,
a Radio Detection and Ranging (RADAR) sensor,
an ultrasonic sensor,
a proximity or distance sensor, and
a range sensor.

### Claim 57 (depends on 47)

The method according to claim 47, wherein the feature points are extracted from the first PCD representation and/or the second PCD representation using a deep learning approach, wherein the extracted feature points comprise uniformly sampled representation and/or Centroid representation, and wherein the deep learning approach comprises one of PointNet and STN-based approaches.

### Claim 58 (depends on 47)

The method according to claim 47, wherein the identifying of feature point pairs for each object in the one or more common objects is based on one or more of the following:
an Iterative Closest Point (ICP) algorithm,
a k-nearest neighbors (KNN) algorithm, and
a density-based clustering algorithm.

### Claim 59

A system for calibration of cooperative sensors, the system comprising:
at least one processor; and
a memory including computer program code for execution by the at least one processor, the computer program code instructs the at least one processor to:
obtain a first set of sensor data for an environment from a first sensor;
obtain a second set of sensor data for the environment from a second sensor that is cooperative with the first sensor;
identify one or more objects from the first set of sensor data and the second set of sensor data, wherein the one or more objects comprise one or more dynamic objects;
generate a first point cloud data (PCD) representation for the one or more objects identified from the first set of sensor data;
generate a second point cloud data (PCD) representation for the one or more objects identified from the second set of sensor data;
identify one or more common objects that are present in both the first PCD representation and the second PCD representation;
identify feature point pairs for each object in the one or more common objects, wherein each feature point pair of the feature point pairs comprises one or more feature points extracted from the first PCD representation and/or the second PCD representation corresponding to a same or similar feature of the object; and
for each feature point pair of the feature point pairs, minimize a distance between feature points in the feature point pair so as to form an extrinsic calibration matrix for calibrating the second sensor based on the first sensor.

### Claim 60 (depends on 59)

The system according to claim 59, wherein the first set of sensor data comprises one or more frames of the first sensor, and the second set of sensor data comprises one or more frames of the second sensor.

### Claim 61 (depends on 59)

The system according to claim 59, wherein the first sensor has a first field of view, the second sensor has a second field of view, and the first field of view overlaps with the second field of view.

### Claim 62 (depends on 59)

The system according to claim 59, wherein the one or more objects further comprise one or more static objects.

### Claim 63 (depends on 59)

The system according to claim 59, wherein the system is further configured to:
identify the one or more common objects that are present in both the first PCD representation and the second PCD representation using prior knowledge or applying coarse sensor fusion to identify a common field of view for the first sensor and the second sensor, and
project the one or more common objects into a frame of reference of the second sensor for calibrating the second sensor based on the first sensor.

### Claim 64 (depends on 59)

The system according to claim 59, wherein the system is further configured to:
obtain a pose data from at least one of the first sensor and the second sensor, the pose data indicating a pose of the at least one of the first sensor and the second sensor;
transform the first PCD representation and the second PCD representation into a common frame of reference based on the pose, wherein the transforming includes applying a pose correction to one of the first sensor and the second sensor that does not provide the pose data; and
identify the one or more common objects in the common frame of reference.

### Claim 65 (depends on 59)

The system according to claim 59, wherein prior to the identifying of one or more common objects that are present in both the first PCD representation and the second PCD representation, the system is further configured to:
obtain a first pose data from the first sensor, the first pose data indicating a first pose of the first sensor;
obtain a second pose data from the second sensor, the second pose data indicating a second pose of the second sensor;
transform the first PCD representation and the second PCD representation into a common frame of reference based on the first pose and the second pose; and
identify the one or more common objects in the common frame of reference.

### Claim 66 (depends on 59)

The system according to claim 59, wherein prior to the generating of the first PCD representation for the one or more objects identified from the first set of sensor data, the system is configured to:
segment the one or more objects from the first set of sensor data and the second set of sensor data based on one or more of the following:
a machine/deep learning approach,
a Computer Vision approach, and
prior object knowledge or position and geometry of the one or more objects.

### Claim 67 (depends on 59)

The system according to claim 59, wherein the system is further configured to:
segment while identify the one or more objects from the first set of sensor data and the second set of sensor data based on one or more of the following:
a machine/deep learning approach,
a Computer Vision approach, and
prior object knowledge or position and geometry of the one or more objects.

### Claim 68 (depends on 59)

The system according to claim 59, wherein the first sensor or the second sensor is one of the following:
a camera sensor,
a Light Detection and Ranging (LiDAR) sensor,
a Radio Detection and Ranging (RADAR) sensor,
an ultrasonic sensor,
a proximity or distance sensor, and
a range sensor.

### Claim 69 (depends on 59)

The system according to claim 59, wherein the feature points are extracted from the first PCD representation and/or the second PCD representation using a deep learning approach, wherein the extracted feature points comprise uniformly sampled representation and/or Centroid representation, and wherein the deep learning approach comprises one of PointNet and STN-based approaches.

### Claim 70 (depends on 59)

The system according to claim 59, wherein during the identifying of feature point pairs for each object in the set of object of interest, the system is configured to identify feature point pairs for each object in the one or more common objects based on one or more of the following:
an Iterative Closest Point (ICP) algorithm,
a k-nearest neighbors (KNN) algorithm, and
a density-based clustering algorithm.

### Claim 71 (depends on 59)

The system according to claim 59, wherein the system is one of the following:
a semi-autonomous vehicle,
a fully autonomous vehicle,
an autonomous robot,
a drone,
a ship,
a plane,
an Internet of Things (IoT) system,
an Industry 4.0 system, and
a medical device.

### Claim 72

A non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:
obtain a first set of sensor data for an environment from a first sensor;
obtain a second set of sensor data for the environment from a second sensor that is cooperative with the first sensor;
identify one or more objects from the first set of sensor data and the second set of sensor data, wherein the one or more objects comprise one or more dynamic objects;
generate a first point cloud data (PCD) representation for the one or more objects identified from the first set of sensor data;
generate a second point cloud data (PCD) representation for the one or more objects identified from the second set of sensor data;
identify one or more common objects that are present in both the first PCD representation and the second PCD representation;
identify feature point pairs for each object in the one or more common objects, wherein each feature point pair of the feature point pairs comprises one or more feature points extracted from the first PCD representation and/or the second PCD representation corresponding to a same or similar feature of the object; and
for each feature point pair of the feature point pairs, minimize a distance between feature points in the feature point pair so as to form an extrinsic calibration matrix for calibrating the second sensor based on the first sensor.
