# US 11,994,664 B2 · Augmented reality laser capture microdissection machine > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,994,664 B2 - Document ID: US11994664B2 - Status: Granted - Kind: B2 - Application number: 17/295,353 - Filed: 2019-10-31 - Published: 2022-01-20 - Granted: 2024-05-28 - Earliest claimed: 2019-01-09 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Jason Hipp; Martin Stumpe - Web page: https://patentgrep.com/patent/US11994664B2 ## CPC - G02B 21/32: Micromanipulators structurally combined with microscopes - G02B 21/365: {Control or image processing arrangements for digital or video microscopes (G02B21/361, G02B21/362 take precedence)} - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06N 20/00: Machine learning - A61B 2090/365: {augmented reality, i.e. correlating a live optical image with another image} - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) ## Abstract An augmented reality (AR) subsystem including one or more machine learning models, automatically overlays an augmented reality image, e.g., a border or outline, that identifies cells of potential interest, in the field of view of the specimen as seen through the eyepiece of an LCM microscope. The operator does not have to manually identify the cells of interest for subsequent LCM, e.g, on a workstation monitor, as in the prior art. The operator is provided with a switch, operator interface tool or other mechanism to select the identification of the cells, that is, indicate approval of the identification of the cells, while they view the specimen through the eyepiece. Activation of the switch or other mechanism invokes laser excising and capture of the cells of interest via a known and conventional LCM subsystem. ## Claims ### Claim 1 An augmented reality laser capture microdissection machine for capturing cells of interest from a sample, comprising, in combination; a) a microscope having an eyepiece and a camera configured to capture images of a field of view of the microscope as seen through the eyepiece; b) an augmented reality subsystem configured to receive the images from the camera, the subsystem including a machine learning model stored on a non-transitory machine readable medium identifying cells of potential interest in the images and a optics module projecting into a view of the microscope as seen through the eyepiece an outline of cells of potential interest identified by the machine learning model; c) a laser capture and microdissection subsystem configured for excising the cells of interest from the sample with one or more lasers and placing such cells of interest on a suitable medium; and d) an operator-activated input mechanism configured to provide input to the laser capture and microdissection subsystem whereby operator activation of the input mechanism while viewing the specimen and the outline of cells identified by the machine leaning model at the eyepiece invokes the laser capture and microdissection subsystem so as to excise the cells of interest from the sample within the outline and place them on the medium. ### Claim 2 (depends on 1) The machine of claim 1, wherein the operator-activated mechanism is configured so as to permit the operator to activate the mechanism while viewing the specimen though the eyepiece. ### Claim 3 (depends on 2) The machine of claim 2, wherein the mechanism is activated by a voice command. ### Claim 4 (depends on 2) The machine of claim 2, wherein the mechanism is activated by a foot switch. ### Claim 5 (depends on 2) The machine of claim 2, wherein the mechanism is activated by a mouse or keyboard stroke. ### Claim 6 (depends on 1) The machine of claim 1, wherein the machine learning model comprises a deep convolutional neural network. ### Claim 7 (depends on 1) The machine of claim 1, wherein the augmented reality subsystem includes a multitude of machine learning models for different tissue types and magnification levels. ### Claim 8 A method of capturing cells of interest from a sample using a microscope having an eyepiece, comprising the steps of: a) projecting an augmented reality image into a field of view of the microscope as seen through the eyepiece, the augmented reality image identifying cells of potential interest for laser capture microdissection; and b) invoking a laser capture and microdissection subsystem coupled to the microscope so as to excise the cells of potential interest, as presented in the augmented reality image, from the sample in response to an operator instruction. ### Claim 9 (depends on 8) The method of claim 8, wherein the augmented reality image comprises a border surrounding cells of potential interest. ### Claim 10 (depends on 8) The method of claim 8, wherein the operator instruction comprises a voice command. ### Claim 11 (depends on 8) The method of claim 8, wherein the operator instruction comprises activation of a foot switch. ### Claim 12 (depends on 8) The method of claim 8, wherein the operator instructions comprises a mouse click or keyboard stroke. ### Claim 13 (depends on 8) The method of claim 8, wherein the method comprises the step of repeating steps a) and b) as an operator of the microscope changes magnification or performs zoom operations while viewing the sample with the microscope. ### Claim 14 (depends on 8) The method of claim 8, wherein step b) wherein the operator instruction is received while the operator of the microscope is viewing the specimen through the eyepiece of the microscope. ### Claim 15 In a machine including a microscope having an eyepiece and a laser capture and microdissection subsystem configured to excise cells of interest from a sample and place them on a suitable medium, the improvement comprising: providing said machine with an augmented reality subsystem comprising: a) a camera for capturing images of a field of view of the microscope; b) a machine learning model for identifying cells of potential interest within the images; c) an augmented reality (AR) image generation unit and an optics module for overlaying an enhancement onto the field of view as seen through the eyepiece in a form of an outline of potential cells of interest for laser capture microdissection (LCM); and d) an operator activated input mechanism for invoking the laser capture and microdissection subsystem to excise the potential cells of interest in accordance with the enhancement from the sample and place the cells of interest in a suitable medium.