# US 2025/0204992 A1 · Method and system for 3d surface model fitting via robot arm controlled laser grid and camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0204992 A1
- Document ID: US20250204992A1
- Status: Published application
- Kind: A1
- Application number: 18/999,620
- Filed: 2024-12-23
- Published: 2025-06-26
- Earliest claimed: 2023-12-21
- Assignees: Edda Technology, Inc.
- Applicants: Edda Technology, Inc.
- Inventors: Yifan Wang; Guo-Qing Wei; Li Fan; Xiaolan Zeng; Jianzhong Qian
- Web page: https://patentgrep.com/patent/US20250204992A1

## CPC

- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- B25J 9/1679: {characterised by the tasks executed}
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/75: {involving models}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30204: Marker
- G06T 2210/41: Medical
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present teaching relates to generating a point cloud of a target organ using a camera and a laser emitter inserted into a patient's body near the target organ and tracked by a tracker. The laser emitter emits, at tracked moving locations, laser beams that hit the target organ at multiple points which are captured by the camera in an image. A three-dimensional (3D) coordinate for each point is determined based on a laser line equation, connecting the laser emitter and the point, and an image line equation, connecting the camera and the point. The 3D coordinates for the multiple points are used to generate the point cloud.

## Claims

### Claim 1

A method, comprising:
deploying a tracker in a tracker coordinate system in a surgery for operating on a target organ of a patient;
inserting a camera and a laser emitter into the patient near the target organ so that laser beams embittered by the laser emitter at different tracked moving positions hit the surface of the target organ at respective points which are captured by the camera in an image;
with respect to each of the respective points,
deriving a laser line equation of a laser line connecting the laser emitter and the respective point,
deriving an image line equation of an image line connecting the camera and the respective point, wherein the laser line equation and the image line equation are expressed in the tracker coordinate system, and
obtaining a three-dimensional (3D) coordinate of the respective point in the tracker coordinate system based on the laser line equation and the image line equation; and
generating a point cloud based on the 3D coordinates in the tracker coordinate system for the respective points representing a depth map of the captured part of the surface of the target organ.

### Claim 2 (depends on 1)

The method of claim 1, wherein
the camera is coupled with a camera marker with a first spatial relation;
the laser emitter is coupled with a laser emitter marker with a second spatial relation, wherein
the camera marker and the laser emitter marker are outside of the patient's body and are tracked by the tracker in the tracker coordinate system,
a tracked 3D location of the camera can be determined from tracked position of the camera marker based on the first spatial relation, and
a tracked 3D location of the laser emitter can be determined from tracked position of the laser emitter marker based on the second spatial relation.

### Claim 3 (depends on 2)

The method of claim 2, wherein the step of deriving a laser line equation of a laser line comprises:
determining an initial laser line equation in a laser emitter coordinate system for the laser line based on the tracked 3D location of the laser emitter and operating parameters of the laser emitter; and
converting the initial laser line equation in the laser emitter coordinate system to the laser line equation in the tracker coordinate system based on a laser-tracker transformation matrix obtained via calibration.

### Claim 4 (depends on 2)

The method of claim 2, wherein the step of deriving an image line equation of an image line comprises:
determining a two-dimensional (2D) coordinate of a pixel in the image corresponding to a projection of the respective point;
obtaining an initial image line equation based on the 2D coordinate, the tracked 3D location of the camera, and operating parameters of the camera in a camera coordinate system; and
converting the initial image line equation in the camera coordinate system to the image line equation in the tracker coordination system based on a camera-tracker transformation matrix obtained via calibration.

### Claim 5 (depends on 1)

The method of claim 1, wherein the 3D coordinate of the respective point in the tracker coordinate system is obtained by:
identifying a meeting coordinate in the tracker coordinate system that satisfies both the laser line equation and the image line equation; and
providing the meeting coordinate in the tracker coordinate system as the 3D coordinate of the respective point.

### Claim 6 (depends on 1)

The method of claim 1, wherein the 3D coordinate of the respective point in the tracker coordinate system is obtained by:
identifying a first point on the laser line satisfying the laser line equation and a second point on the image line satisfying the image line equation so that a distance between the first and the second points represents a minimized distance between the laser line and the image line;
selecting an approximate point between the first and the second point along a line connecting the first and the second point; and
obtaining a 3D coordinate of the approximate point in the tracker coordinate system as the 3D coordinate of the respective point.

### Claim 7 (depends on 1)

The method of claim 1, wherein the respective points include:
additional hits on the surface by beams from a diffractive optical element (DOE) placed in front of the laser emitter, wherein the DOE yields multiple beams for each of the laser beam from the laser emitter that passes through the DOE;
additional laser beams emitted by the laser emitter at the different tracked moving positions; and
the combination thereof.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
retrieving a 3D model for the target model previously constructed based on data related to the patient; and
fitting the 3D model to the point cloud to obtain a fitted 3D model in the tracker coordinate system that aligns with the target organ and fits the surface of the target organ represented by the point cloud.

### Claim 9 (depends on 8)

The method of claim 8, further comprising transforming the fitted 3D model in the tracker coordinate system to a transformed fitted 3D model in a robot coordinate system.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
receiving a specified location on the surface of the target organ during the surgery;
identifying, from the transformed fitted 3D model, a 3D coordinate in the robot coordinate system for the specified location; and
controlling a robot arm holding a surgical instrument to reach the specified location on the surface of the target organ based on the 3D coordinate in the robot coordinated system.

### Claim 11

A system, comprising:
a tracker deployed in a tracker coordinate system in a surgery for operating on a target organ of a patient;
a camera and a laser emitter inserted into the patient near the target organ so that laser beams embittered by the laser emitter at different tracked moving positions hit the surface of the target organ at respective points which are captured by the camera in an image;
a three-dimensional (3D) point cloud determiner implemented by a processor and configured for, with respect to each of the respective points,
deriving a laser line equation of a laser line connecting the laser emitter and the respective point,
deriving an image line equation of an image line connecting the camera and the respective point, wherein the laser line equation and the image line equation are expressed in the tracker coordinate system, and
obtaining a 3D coordinate of the respective point in the tracker coordinate system based on the laser line equation and the image line equation, and
generating a point cloud based on the 3D coordinates in the tracker coordinate system for the respective points representing a depth map of the captured part of the surface of the target organ.

### Claim 12 (depends on 11)

The system of claim 11, wherein
the camera is coupled with a camera marker with a first spatial relation;
the laser emitter is coupled with a laser emitter marker with a second spatial relation, wherein
the camera marker and the laser emitter marker are outside of the patient's body and are tracked by the tracker in the tracker coordinate system,
a tracked 3D location of the camera can be determined from tracked position of the camera marker based on the first spatial relation, and
a tracked 3D location of the laser emitter can be determined from tracked position of the laser emitter marker based on the second spatial relation.

### Claim 13 (depends on 12)

The system of claim 12, wherein the step of deriving a laser line equation of a laser line comprises:
determining an initial laser line equation in a laser emitter coordinate system for the laser line based on the tracked 3D location of the laser emitter and operating parameters of the laser emitter; and
converting the initial laser line equation in the laser emitter coordinate system to the laser line equation in the tracker coordinate system based on a laser-tracker transformation matrix obtained via calibration.

### Claim 14 (depends on 12)

The system of claim 12, wherein the step of deriving an image line equation of an image line comprises:
determining a two-dimensional (2D) coordinate of a pixel in the image corresponding to a projection of the respective point;
obtaining an initial image line equation based on the 2D coordinate, the tracked 3D location of the camera, and operating parameters of the camera in a camera coordinate system; and
converting the initial image line equation in the camera coordinate system to the image line equation in the tracker coordination system based on a camera-tracker transformation matrix obtained via calibration.

### Claim 15 (depends on 11)

The system of claim 11, wherein the 3D coordinate of the respective point in the tracker coordinate system is obtained by:
identifying a meeting coordinate in the tracker coordinate system that satisfies both the laser line equation and the image line equation; and
providing the meeting coordinate in the tracker coordinate system as the 3D coordinate of the respective point.

### Claim 16 (depends on 11)

The system of claim 11, wherein the 3D coordinate of the respective point in the tracker coordinate system is obtained by:
identifying a first point on the laser line satisfying the laser line equation and a second point on the image line satisfying the image line equation so that a distance between the first and the second points represents a minimized distance between the laser line and the image line;
selecting an approximate point between the first and the second point along a line connecting the first and the second point; and
obtaining a 3D coordinate of the approximate point in the tracker coordinate system as the 3D coordinate of the respective point.

### Claim 17 (depends on 11)

The system of claim 11, wherein the respective points include:
additional hits on the surface by beams from a diffractive optical element (DOE) placed in front of the laser emitter, wherein the DOE yields multiple beams for each of the laser beam from the laser emitter that passes through the DOE;
additional laser beams emitted by the laser emitter at the different tracked moving positions; and
the combination thereof.

### Claim 18 (depends on 11)

The system of claim 11, further comprising a point cloud model-fitting unit implemented by a processor and configured for:
retrieving a 3D model for the target model previously constructed based on data related to the patient; and
fitting the 3D model to the point cloud to obtain a fitted 3D model in the tracker coordinate system that aligns with the target organ and fits the surface of the target organ represented by the point cloud.

### Claim 19 (depends on 18)

The system of claim 18, wherein the point cloud model-fitting unit is further configured for transforming the fitted 3D model in the tracker coordinate system to a transformed fitted 3D model in a robot coordinate system.

### Claim 20 (depends on 19)

The system of claim 19, further comprising a surgical tool robot arm controller implemented by a processor and configured for:
receiving a specified location on the surface of the target organ during the surgery;
identifying, from the transformed fitted 3D model, a 3D coordinate in the robot coordinate system for the specified location; and
controlling a robot arm holding a surgical instrument to reach the specified location on the surface of the target organ based on the 3D coordinate in the robot coordinated system.
