# US 2023/0410309 A1 · Methods, computer programs, and systems for automated microinjection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410309 A1
- Document ID: US20230410309A1
- Status: Published application
- Kind: A1
- Application number: 18/198,501
- Filed: 2023-05-17
- Published: 2023-12-21
- Earliest claimed: 2022-05-17
- Assignees: Overture Life, Inc.
- Applicants: Overture Life, Inc.
- Inventors: Adrián ALVAREZ FERNANDEZ; Santiago MUNNE; Luis MOLLINEDO HERRERA; Gloria CALDERON OYA; Nuno Luis COSTA BORGES; Jesús RAMOS MEMBRIVE; Sergi MAS SABATES
- Web page: https://patentgrep.com/patent/US20230410309A1

## CPC

- G06T 7/0014: {using an image reference approach}
- G06V 20/52: Surveillance or monitoring of activities, e.g. for recognising suspicious objects (recognising microscopic objects G06V20/69)
- C12N 15/89: using microinjection
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 7/30: Determination of transform parameters for the alignment of images, i.e. image registration
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 20/693: {Acquisition}
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06V 20/698: {Matching; Classification}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G16H 30/40: for processing medical images, e.g. editing
- G01N 15/1425: {using an analyser being characterised by its control arrangement}
- G01N 15/1429: Signal processing
- G01N 15/1433: using image recognition
- G01N 2015/1006: {for cytology}
- G06F 18/00: Pattern recognition
- G06N 3/09: Supervised learning
- G06T 2207/10056: Microscopic image
- G06T 2207/20036: Morphological image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2207/30044: Fetus; Embryo
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06V 2201/034: of medical instruments
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- C12N: MICROORGANISMS OR ENZYMES; COMPOSITIONS THEREOF; PROPAGATING, PRESERVING, OR MAINTAINING MICROORGANISMS; MUTATION OR GENETIC ENGINEERING; CULTURE MEDIA (microbiological testing media C12Q1/00)
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Provided herein are methods, computer programs, and systems for automated microinjection, for example, automated Intracytoplasmic Sperm Injection (ICSI). Methods described herein include creating, by a processing unit, a first dataset of an oocyte and a holding device and a second dataset of an injection pipette; detecting the oocyte and the holding device in the first dataset and the injection pipette in the second dataset; selecting the image of the first dataset and of the second dataset where an equatorial plane of the oocyte/holding device and of the injection pipette has an improved focusing parameter; selecting images of the first and second datasets and labeling the pixels associated with the oocyte and to the injection pipette; detecting a tip of the injection pipette; detecting different morphological structures of the oocyte using artificial intelligence computer vision algorithms on the first dataset; creating an injection trajectory for the injection pipette to perform the ICSI using the detected morphological structures; detecting when the oocyte is rupturing and when the spermatozoa has been released from the injection pipette into the cytoplasm of oocyte.

## Claims

### Claim 1

A method, comprising:
a) receiving a first set of images, wherein each image of the first set of images independently contains an oocyte immobilized by a holding device, wherein the oocyte in each image of the first set of images is the same oocyte, and wherein the holding device in each image of the first set of images is the same holding device;
b) labeling, by an image detection algorithm, a plurality of pixels associated with the oocyte in each image of the first set of images; and
c) determining, using artificial intelligence, and based on the labeled plurality of pixels associated with the oocyte, the image in which the oocyte is most in focus in comparison to the other images of the first set of images.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
d) receiving a second set of images, wherein each image of the second set of images independently contains an injection pipette, wherein the injection pipette in each image of the second set of images is the same injection pipette, wherein in each image of the second set of images the injection pipette is positioned for injection of a spermatozoon into the oocyte.
e) labeling, by an image detection algorithm, a plurality of pixels associated with the injection pipette in each image of the second set of images; and
f) determining, using artificial intelligence, and based on the labeled plurality of pixels associated with the injection pipette, the image in which the injection pipette is most in focus in comparison to the other images of the second set of images.

### Claim 3 (depends on 1)

The method of claim 1, wherein each image of the first set of images is acquired by an imaging device, wherein each image of the first set of images has a visual plane and the visual plane of each image of the first set of images is parallel, wherein the oocyte moves in an axis perpendicular to an optical sensor plane, wherein each position along the axis perpendicular to the sensor plane is independently associated with a given oocyte position, wherein the sensor plane is parallel to the visual plane of each image of the first set of images, wherein each image of the first set of images is independently associated with an oocyte position, wherein one oocyte position is most effective, wherein the most effective oocyte position is the position associated with the image of the first set of images where the oocyte is most in focus in comparison to the other images of the first set of images.

### Claim 4 (depends on 2)

The method of claim 2, wherein each image of the second set of images is acquired by an imaging device along the axis perpendicular to the sensor plane, wherein each position along the axis perpendicular to the sensor plane is independently associated with a given injection pipette position, wherein each image of the second set of images is independently associated with an injection pipette position, wherein one injection pipette position is most effective, wherein the most effective injection pipette position is the position associated with the image of the second set of images where the injection pipette is most in focus in comparison to the other images of the second set of images.

### Claim 5 (depends on 2)

The method of claim 2, further comprising aligning the oocyte and the injection pipette based on: (i) the image of the first set of images where the oocyte is most in focus in comparison to the other images of the first set of images; and (ii) the image of the second set of images where the injection pipette is most in focus in comparison to the other images of the second set of images.

### Claim 6 (depends on 1)

The method of claim 1, further comprising identifying a morphological structure of the oocyte based on the labeled plurality of pixels associated with the oocyte.

### Claim 7 (depends on 6)

The method of claim 6, wherein the identifying the morphological structure of the oocyte based on the labeled plurality of pixels is by an artificial neural network.

### Claim 8 (depends on 6)

The method of claim 6, wherein the identifying the morphological structure of the oocyte based on the labeled plurality of pixels is by a computer vision algorithm.

### Claim 9 (depends on 1)

The method of claim 1, further comprising detecting a background of the oocyte in each image of the first set of images.

### Claim 10 (depends on 2)

The method of claim 2, further comprising identifying a tip of the injection pipette based on the labeled plurality of pixels associated with the injection pipette.

### Claim 11 (depends on 2)

The method of claim 2, wherein each image of the first set of images and each image of the second set of images are acquired from a lower side of the oocyte to an upper side of the oocyte.

### Claim 12 (depends on 6)

The method of claim 6, further comprising determining an injection trajectory into the oocyte for the injection pipette based on the identified morphological structure of the oocyte and the identified tip of the injection pipette.

### Claim 13 (depends on 12)

The method of claim 12, wherein the injection trajectory is determined by:
identifying a center of the morphological structure of the oocyte;
determining where the injection trajectory crosses a zona pellucida of the oocyte;
determining a distance that the injection trajectory must penetrate into cytoplasm of the oocyte to be effective using the identified center of the morphological structure; and
determining whether the injection trajectory crosses a polar body of the oocyte.

### Claim 14 (depends on 13)

The method of claim 13, wherein the morphological structure is the zona pellucida.

### Claim 15 (depends on 13)

The method of claim 13, wherein the morphological structure is the polar body.

### Claim 16 (depends on 13)

The method of claim 13, wherein the morphological structure is a perivitelline space.

### Claim 17 (depends on 13)

The method of claim 13, wherein the morphological structure is the cytoplasm.

### Claim 18 (depends on 12)

The method of claim 12, further comprising executing, by the injection pipette, an intracytoplasmic sperm injection (ICSI) on the oocyte at the injection trajectory, wherein the spermatozoon is injected from the injection pipette into the oocyte.

### Claim 19 (depends on 18)

The method of claim 18, further comprising activating the injection pipette to pierce the zona pellucida when the injection pipette crosses a zona pellucida of the oocyte.

### Claim 20 (depends on 19)

The method of claim 19, further comprising deactivating the injection pipette when the injection pipette crosses a perivitelline space of the oocyte.

### Claim 21 (depends on 20)

The method of claim 20, further comprising reactivating the injection pipette to puncture the oolemma, thereby releasing the spermatozoon inside the oocyte.

### Claim 22 (depends on 18)

The method of claim 18, further comprising:
receiving a third set of images, wherein each image of the third set of images independently contains the oocyte during execution of the ICSI, wherein the oocyte comprises an oolemma;
labeling, by an image detection algorithm, a plurality of pixels associated with rupturing or relaxing of the oolemma in each image of the third set of images; and
labeling, by an image detection algorithm, a plurality of pixels associated with the oolemma not rupturing or relaxing of the oolemma in each image of the third set of images.

### Claim 23 (depends on 22)

The method of claim 22, further comprising calculating the optical flow among consecutive images of the third set of images.

### Claim 24 (depends on 22)

The method of claim 22, further comprising training a classification algorithm using the labeled plurality of pixels associated with the oolemma rupturing or relaxing and the labeled plurality of pixels associated with the oolemma not rupturing or relaxing to classify the images in two classes.

### Claim 25 (depends on 18)

The method of claim 18, further comprising detecting release of the spermatozoon from the injection pipette by:
receiving a fourth set of images, wherein each image of the fourth set of images independently contains the spermatozoon during execution of the ICSI on the oocyte at the injection trajectory, wherein the spermatozoon in each image of the fourth set of images is the same spermatozoon, and wherein the injection pipette in each image of the fourth set of images is the same injection pipette;
labeling, by an image detection algorithm, a plurality of pixels associated with the spermatozoon in each image of the fourth set of images;
predicting a location of the spermatozoon by training a detection algorithm using the fourth set of images; and
predicting a location of a tip of the injection pipette by training a detection algorithm using the fourth set of images.

### Claim 26 (depends on 1)

A system comprising:
a processing unit comprising at least one memory and one or more processors configured to execute the method of claim 1.

### Claim 27 (depends on 1)

A computer program product comprising a non-transitory computer-readable medium having computer-executable code encoded therein, the computer-executable code adapted to be executed to implement the method of claim 1.