# US 11,896,320 B2 · Optical tracking device with built-in structured light module

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,896,320 B2
- Document ID: US11896320B2
- Status: Granted
- Kind: B2
- Application number: 18/099,805
- Filed: 2023-01-20
- Published: 2023-05-25
- Granted: 2024-02-13
- Earliest claimed: 2020-04-24
- Assignees: Smith & Nephew, Inc.; Smith & Nephew Orthopaedics AG; Smith & Nephew Asia Pacific Pte. Limited
- Applicants: Smith & Nephew, Inc.; Smith & Nephew Asia Pacific Pte. Limited; Smith & Nephew Orthopaedics AG
- Inventors: Gaëtan Marti; Maurice Hälg; Ranjith Steve Sivagnanaselvam
- Web page: https://patentgrep.com/patent/US11896320B2

## CPC

- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 34/10: Computer-aided planning, simulation or modelling of surgical operations
- A61B 34/30: Surgical robots
- A61B 34/32: operating autonomously
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- A61B 90/92: coded with colour
- A61B 90/96: using barcodes
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/246: using feature-based methods, e.g. the tracking of corners or segments
- G06T 7/285: using a sequence of stereo image pairs
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06T 7/85: {Stereo camera calibration}
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 30/40: for processing medical images, e.g. editing
- H04N 5/2226: {Determination of depth image, e.g. for foreground/background separation (determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10)}
- H04N 23/56: provided with illuminating means
- A61B 34/25: {User interfaces for surgical systems}
- A61B 2017/00203: {with speech control or speech recognition}
- A61B 2017/00207: {with hand gesture control or hand gesture recognition}
- A61B 2017/00973: {pedal-operated}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/2051: {Electromagnetic tracking systems}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2034/2057: {Details of tracking cameras}
- A61B 2034/2063: {Acoustic tracking systems, e.g. using ultrasound}
- A61B 2034/2065: {Tracking using image or pattern recognition}
- A61B 2034/256: {having a database of accessory information, e.g. including context sensitive help or scientific articles}
- A61B 2090/309: {using white LEDs}
- A61B 2090/364: {Correlation of different images or relation of image positions in respect to the body}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/367: {creating a 3D dataset from 2D images using position information}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/3945: {Active visible markers, e.g. light emitting diodes}
- A61B 2090/3979: {infrared}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- G06T 2207/10012: Stereo images
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/10024: Color image
- G06T 2207/10048: Infrared image
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30004: Biomedical image processing
- G06T 2207/30204: Marker
- G06T 2207/30244: Camera pose
- G06T 2210/41: Medical
- H04N 5/2624: {for obtaining an image which is composed of whole input images, e.g. splitscreen}
- Y02A 90/10: Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- Y02A: TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE

## Abstract

A system is disclosed that includes an optical tracking device and a surgical computing device. The optical tracking device includes a structured light module and an optical module that includes an image sensor and is spaced from the structured light module at a known distance. The surgical computing device includes a display device, a non-transitory computer readable medium including instructions, and processor(s) configured to execute the instructions to generate a depth map from a first image captured by the image sensor during projection of a pattern into a surgical environment by the structured light module. The pattern is projected in a near-infrared (NIR) spectrum. The processor(s) are further configured to execute the stored instructions to reconstruct a 3D surface of anatomical structure(s) based on the generated depth map. Additionally, the processor(s) are configured to execute the stored instructions to output the reconstructed 3D surface to the display device.

## Claims

### Claim 1

A system comprising:
an optical tracking device comprising a structured light module and an optical module comprising an image sensor and spaced from the structured light module at a known distance; and
a surgical computing device comprising a display device, a non-transitory computer-readable medium comprising instructions stored thereon, and a processor coupled to the non-transitory computer-readable medium and configured to execute the stored instructions to:
generate a depth map from a first image captured by the image sensor during projection of a pattern into a surgical environment by the structured light module, wherein the pattern is projected in a near-infrared (NIR) spectrum;
reconstruct a three-dimensional surface of one or more anatomical structures in the surgical environment based on the generated depth map;
output the reconstructed three-dimensional surface to the display device; and
process a second image captured by the image sensor in a visible light spectrum to determine tracking information comprising at least a three-dimensional position of each of one or more fiducials within the surgical environment;
wherein the first and second images are captured in a common coordinate system.

### Claim 2 (depends on 1)

The system of claim 1, wherein the optical module comprises first and second optical modules comprising first and second image sensors, respectively, in a stereo configuration.

### Claim 3 (depends on 2)

The system of claim 2, wherein the first and second image sensors are spaced at a known distance and rigidly coupled to a casing of the optical tracking device.

### Claim 4 (depends on 2)

The system of claim 2, wherein the first and second image sensors are fixed on a baseline of the optical tracking device and the processor is further configured to execute the stored instructions to calibrate a first pose of the first image sensor with respect to a second pose of the second image sensor.

### Claim 5 (depends on 2)

The system of claim 2, wherein the structured light module is configured to project one or more of a plurality of sequential patterns, a plurality of stripes, a random pattern, a pseudo-random pattern, or a plurality of points.

### Claim 6 (depends on 1)

The system of claim 1, wherein the processor is further configured to execute the stored instructions to output an indication of one or more objects to the display device along with the reconstructed three-dimensional surface, wherein the one or more objects are associated with the one or more fiducials and located based on the tracking information.

### Claim 7 (depends on 1)

The system of claim 1, further comprising an infrared light (IR) emitting diode (LED) light source surrounding, at least in part, the optical module.

### Claim 8 (depends on 1)

The system of claim 1, wherein the processor is further configured to execute the stored instructions to generate a plurality of three-dimensional surfaces of the one or more anatomical structures from a plurality of depth maps generated from a plurality of images captured by the image sensor during projection of the pattern into the surgical environment by the structured light module.

### Claim 9 (depends on 8)

The system of claim 8, wherein the processor is further configured to execute the stored instructions to stitch the plurality of three-dimensional surfaces to reconstruct the three-dimensional surface of the one or more anatomical structures.

### Claim 10 (depends on 8)

The system of claim 8, wherein the processor is further configured to execute the stored instructions to densify the reconstructed three-dimensional surface of the one or more anatomical structures based on the plurality of three-dimensional surfaces.

### Claim 11 (depends on 8)

The system of claim 8, wherein the processor is further configured to execute the stored instructions to generate a three-dimensional sequence of movement of the one or more anatomical structures occurring between capture of one or more of the plurality of images.

### Claim 12 (depends on 1)

The system of claim 1, wherein the optical module further comprises a lens and an NIR filter.