# US 2023/0410445 A1 · Augmented-reality surgical system using depth sensing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410445 A1
- Document ID: US20230410445A1
- Status: Published application
- Kind: A1
- Application number: 18/365,844
- Filed: 2023-08-04
- Published: 2023-12-21
- Earliest claimed: 2021-08-18
- Assignees: Augmedics Ltd.
- Applicants: Augmedics Ltd.
- Inventors: Nissan Elimelech; Stuart Wolf; Nitzan Krasney; Monica Marie Kuhnert
- Web page: https://patentgrep.com/patent/US20230410445A1

## CPC

- A61B 90/36: {Image-producing devices or illumination devices not otherwise provided for}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- A61B 34/20: Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
- A61B 90/37: {Surgical systems with images on a monitor during operation}
- G02B 27/0172: {characterised by optical features}
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06T 7/0016: {involving temporal comparison}
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06T 7/55: from multiple images
- G06T 7/593: from stereo images
- G06T 7/62: of area, perimeter, diameter or volume
- G06T 7/73: using feature-based methods
- H04N 13/128: Adjusting depth or disparity
- H04N 13/156: Mixing image signals
- H04N 13/167: Synchronising or controlling image signals
- H04N 13/239: using two two-dimensional [2D] image sensors having a relative position equal to or related to the interocular distance (H04N13/243 takes precedence)
- H04N 13/344: with head-mounted left-right displays
- H04N 13/383: for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
- H04N 23/11: for generating image signals from visible and infrared light wavelengths
- A61B 2017/00216: {with eye tracking or head position tracking control}
- A61B 2034/105: {Modelling of the patient, e.g. for ligaments or bones}
- A61B 2034/107: {Visualisation of planned trajectories or target regions}
- A61B 2034/2055: {Optical tracking systems}
- A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image}
- A61B 2090/367: {creating a 3D dataset from 2D images using position information}
- A61B 2090/371: {with simultaneous use of two cameras}
- A61B 2090/372: {Details of monitor hardware}
- A61B 2090/373: {using light, e.g. by using optical scanners}
- A61B 2090/502: {Headgear, e.g. helmet, spectacles}
- G02B 2027/0134: {of stereoscopic type}
- G02B 2027/0138: {comprising image capture systems, e.g. camera}
- G02B 2027/014: {comprising information/image processing systems}
- G02B 2027/0141: {characterised by the informative content of the display}
- G02B 2027/0178: {Eyeglass type (eyeglass details G02C)}
- G06T 2200/04: involving 3D image data
- G06T 2207/10012: Stereo images
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30008: Bone
- G06T 2207/30012: Spine; Backbone
- G06T 2207/30204: Marker
- G06T 2210/12: Bounding box
- G06T 2210/41: Medical
- H04N 13/246: Calibration of cameras
- H04N 13/257: Colour aspects
- H04N 13/271: wherein the generated image signals comprise depth maps or disparity maps
- H04N 13/279: the virtual viewpoint locations being selected by the viewers or determined by tracking
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Disclosed herein are systems, devices, and methods for image-guided surgery. Some systems include a head-mounted unit, having a see-through augmented-reality display and a depth sensor, which is configured to generate depth data with respect to a region of interest (ROI) of a body of a patient that is viewed through the display by a user wearing the head-mounted unit. A processor, which is configured to receive a three-dimensional (3D) tomographic image of the body of the patient, computes a depth map of the ROI based on the depth data generated by the depth sensor, to compute a transformation over the ROI so as to register the tomographic image with the depth map, and to apply the transformation in presenting a part of the tomographic image on the display in registration with the ROI viewed through the display.

## Claims

### Claim 1

A head-mounted system for image-guided surgery, comprising:
a head-mounted unit, comprising a see-through augmented-reality display and a depth sensor, which is configured to generate depth data with respect to a region of interest (ROI) of a body of a patient that is viewed through the display by a user wearing the head-mounted unit; and
a processor, which is configured to:
receive a three-dimensional (3D) tomographic image of the body of the patient;
compute a depth map of the ROI based on the depth data generated by the depth sensor;
compute a transformation over the ROI so as to register the tomographic image with the depth map; and
apply the transformation in presenting a part of the tomographic image on the display in registration with the ROI viewed through the display,
wherein the depth map includes a spine of the patient, which is exposed in a surgical procedure,
wherein the processor is configured to compute the transformation by registering the spine in the depth map with the spine appearing in the tomographic image,
wherein the tomographic image comprises a CT scan of the patient which was performed with an array of radiopaque fiducial markers fixed to the body of the patient, and
wherein the processor is configured to identify respective 3D coordinates of the fiducial markers in the depth map and to register the CT scan with the ROI viewed through the display by matching the fiducial markers in the CT to the respective 3D coordinates.

### Claim 2 (depends on 1)

The system of claim 1, wherein the ROI comprises a bone of the body to which an anchoring device is coupled, and wherein the processor is further configured to:
identify a location of the anchoring device in the depth map;
update the depth map in a course of a surgery;
detect a change in the location of the anchoring device in the updated depth map; and
take a corrective action responsively to the change.

### Claim 3 (depends on 2)

The system of claim 2, wherein the corrective action comprises modifying a presentation on the display responsively to the change in the location of the anchoring device.

### Claim 4 (depends on 1)

The system of claim 1, wherein the processor is configured to:
process the depth data so as to detect a position of a marker that is fixed to the body of the patient,
recognize a location of the head-mounted unit relative to the body based on the detected position, and
position the image presented on the display responsively to the recognized location.

### Claim 5 (depends on 1)

The system of claim 1, wherein the processor is configured to:
process the depth data so as to identify a change in an anatomical structure in the body of the patient during a surgical procedure; and
modify the image presented on the display responsively to the identified change.

### Claim 6 (depends on 1)

The system of claim 1, wherein the processor is configured to:
process the depth data so as to identify an implant inserted into the body of the patient during a surgical procedure; and
modify the image presented on the display responsively to the identified implant.

### Claim 7 (depends on 1)

The system of claim 1, wherein the 3D tomographic image is a computed tomographic image.

### Claim 8 (depends on 1)

The system of claim 1, wherein the depth sensor comprises a pattern projector and a camera configured for structured-light depth mapping.

### Claim 9 (depends on 8)

The system of claim 8, wherein the pattern projector comprises a laser dot pattern projector configured to apply structured light comprising dots arranged in a pattern.

### Claim 10 (depends on 9)

The system of claim 9, wherein the pattern is a pseudorandom pattern of dots.

### Claim 11 (depends on 10)

The system of claim 10, wherein the depth map is calculated based on a local disparity of images of the pattern captured by the camera relative to an undistorted reference pattern, together with a known offset between the pattern projector and the camera.

### Claim 12 (depends on 1)

The system of claim 1, wherein the depth sensor comprises a pair of cameras configured for stereoscopic depth mapping.

### Claim 13 (depends on 1)

The system of claim 1, wherein the depth sensor comprises a beam projector and one or more detectors configured for time-of-flight measurement.

### Claim 14 (depends on 1)

The system of claim 1, wherein the head-mounted unit comprises a head strap.

### Claim 15 (depends on 1)

The system of claim 1, wherein the head-mounted unit comprises a visor.

### Claim 16

A head-mounted system for image-guided surgery comprising:
a see-through augmented-reality display disposed so as to be viewable by a user over a region of interest (ROI) of a body of a patient;
a depth sensor configured to generate depth data with respect to the ROI; and
a processor and a memory for storing instructions that, when executed by the processor cause the system to:
receive a three-dimensional (3D) tomographic image of the body of the patient;
determine a depth map of the ROI based at least in part on the depth data;
determine a transformation over the ROI so as to register the 3D tomographic image with the depth map; and
display at least a part of the 3D tomographic image on the see-through augmented-reality display in registration with the ROI based at least in part on the transformation.

### Claim 17 (depends on 1)

The system of claim 1, wherein:
the 3D tomographic image is a computed tomographic image;
the depth sensor comprises a pattern projector and a camera configured for structured-light depth mapping; and
the pattern projector comprises a laser dot pattern projector configured to apply structured light comprising dots arranged in a pattern.

### Claim 18 (depends on 17)

The system of claim 17, wherein the pattern is a pseudorandom pattern of dots.

### Claim 19 (depends on 18)

The system of claim 18, wherein the depth map is calculated based on a local disparity of images of the pattern captured by the camera relative to an undistorted reference pattern, together with a known offset between the pattern projector and the camera.

### Claim 20 (depends on 16)

The system of claim 16, further comprising a head-mounted unit comprising the see-through augmented-reality display.
