# US 2023/0267378 A1 · Systems, methods, and storage media for identifying an object in a photograph using a machine learning system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267378 A1
- Document ID: US20230267378A1
- Status: Published application
- Kind: A1
- Application number: 17/848,989
- Filed: 2022-06-24
- Published: 2023-08-24
- Earliest claimed: 2022-02-24
- Assignees: Earthsnap, Inc.
- Applicants: Earthsnap, Inc.
- Inventors: Eric Ralls
- Web page: https://patentgrep.com/patent/US20230267378A1

## CPC

- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- G06F 16/128: {Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion (error detection or correction of the data by redundancy in operations or in hardware G06F11/14, G06F11/16)}
- G06F 16/955: using information identifiers, e.g. uniform resource locators [URL]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems, methods, and storage media for operating a machine learning system for identifying an object in a photograph are disclosed. The system is configured to prepare a plurality of data files for training the machine learning system by associating a label with each of the plurality of data files, splitting the plurality of data files into different data sets, including full, training, testing, and validation data sets, creating a final machine learning model based on metrics and/or artifacts associated with training on the full data set, and deploying the final machine learning model to a machine learning system endpoint. The system is further configured to identify at least one object in a user uploaded photograph based on invoking a first or a second trained model, the first or second trained model associated with the final machine learning model.

## Claims

### Claim 1

A method of operating a machine-learning system, the method comprising:
preparing a plurality of data files for training by associating a label with each of the plurality of data files, identifying and fixing errors in the plurality of data files, and uploading the plurality of data files to a storage device;
obtaining a versioning snapshot of the plurality of data files uploaded to a storage device;
providing metadata associated with the plurality of data files to the machine-learning system;
splitting the plurality of data files into a plurality of data sets, including at least a training data set, a testing data set, and a validation data set;
creating record IO data files comprising a plurality of training set record IO files, a plurality of testing set record IO files, a plurality of validation set record IO files, and a plurality of full set record IO files;
merging the plurality of record IO files for each data set of the plurality of data sets;
passing the merged record IO files to the storage device;
identifying hyperparameter settings for use in association with training the merged full set record IO files by (1) selecting a portion of the merged testing IO files and a portion of the merged validation IO files, (2) applying a plurality of hyperparameters to the selected portion of the merged testing IO files and the selected portion of the merged validation IO files across a plurality of training jobs, and (3) reviewing one or more metrics related to the plurality of training jobs;
storing artifacts related to the plurality of training jobs at the storage device;
using the identified hyperparameter settings and the artifacts to train the merged full set record IO files;
storing one or more metrics and artifacts associated with training the merged full set record IO files;
using the one or more metrics and artifacts associated with training the merged full set record IO files to create a final machine learning model;
obtaining the final machine learning model from a machine learning system registry; and
deploying the final machine learning model to one or more machine learning system endpoints.

### Claim 2 (depends on 1)

The method of claim 1, wherein the label comprises one of a predetermined category and a species associated with one of the predetermined categories;
wherein, when the label comprises a predetermined category, the final machine learning model comprises a first trained model and the first trained model comprises a parent model;
wherein, when the label comprises, a species associated with one of the predetermined categories, the final machine learning model comprises a second trained model and the second trained model comprises a child model;
the method further comprising:
when the label comprises a predetermined category, identifying the predetermined categories associated with the plurality of data files;
when the label comprises a predetermined category, training the machine-learning system on the same number of data files for each identified category;
wherein the identifying the predetermined categories associated with the plurality of data files and the training the machine-learning system on the same number of data files for each identified category occurs after providing metadata associated with the plurality of data files to the machine-learning system and before splitting the plurality of data files into the training data set, the testing data set, and the validation data set.

### Claim 3 (depends on 1)

The method of claim 1, wherein the predetermined categories comprise eight categories;
wherein the eight categories comprise or are otherwise associated with plants, mammals, insects, amphibians, fish, birds, tree bark, and snails; and
wherein the species comprise one of a plant species, a mammal species, an insect species, an amphibian species, a fish species, a bird species, a tree bark species, and a snail species.

### Claim 4 (depends on 1)

The method of claim 1, wherein the plurality of data files uploaded to the storage device comprises images of at least one of a plurality of the predetermined categories, and the species associated with one of the predetermined categories;
wherein the storage device comprises a cloud-based storage system;
wherein uploading the plurality of data files to a storage device comprises uploading the plurality of data files to a folder on the cloud-based storage system;
wherein the folder on the cloud-based storage system comprises a folder name;
wherein the folder name comprises the label;
wherein an output of the training jobs comprises identification information related to an object in the images; and
wherein the one or more metrics comprises a percentage of the identification information which accurately identifies the object in the images.

### Claim 5 (depends on 4)

The method of claim 4, wherein, when the plurality of data files uploaded to the storage device comprise images of a plurality of the predetermined categories, the plurality of data files comprise at least 100 images for each of the plurality of predetermined categories included in the plurality of data files;
wherein training the machine-learning system on the same number of data files for each category comprises:
determining a fewest number of data files associated with a single predetermined category across all predetermined categories identified in the plurality of data files,
for all predetermined categories identified in the plurality of data files having a number of data files greater than the fewest number of data files, randomly selecting an amount of data files to equal the fewest number of data files, and
training the machine-learning system on the fewest number of data files associated with the single predetermined category, and the randomly selected data files for each of the predetermined categories identified as having a number of data files greater than the fewest number; and
wherein the artifacts comprise assets to reproduce the model.

### Claim 6 (depends on 5)

The method of claim 5, wherein using the fewest number of data files across all predetermined categories identified in the plurality of data files to train the machine-learning system comprises decreasing a bias in the machine learning system towards any particular category of the identified categories; and
wherein the one or more metrics are reviewed with a machine learning system report.

### Claim 7 (depends on 4)

The method of claim 4, wherein identifying and fixing errors in the plurality of data files comprises identifying and fixing mismatches in the plurality of data files;
wherein the metadata comprises the label and a uniform resource identifier;
wherein the training data set comprises 80% of the plurality of data files, the testing data set comprises 10% of the plurality of data files, the validation data set comprises 10% of the plurality of data files, and a full data set comprises 100% of the plurality of data files; and
after creating record IO data files, reporting information about each data set, wherein the information about each data set comprises one or more of a JavaScript Object Notation (JSON) file, a total image count in each data set, and a corrupted images count in each data set.

### Claim 8

A method of identifying an object in a photograph, the method comprising:
sending an initial request to identify the object in the photograph from a user device to an application programming interface (API);
generating, at the API, a pre-signed Uniform Resource Locator (URL) associated with a storage system;
generating metadata associated with the initial request;
sending an initial response to the initial request;
uploading the photograph from a user device directly to a location associated with the pre-signed URL;
sending a second request to identify the object in the photograph from the user device to the API;
providing a second response to the second request;
obtaining the photograph from the storage system;
using the photograph to invoke a first trained model from a machine learning system endpoint;
using the photograph to invoke a second trained model from the machine learning system endpoint;
obtaining from the second trained model a list of one or more most likely identification probabilities associated with the object;
updating the metadata associated with the object at the API;
sending a third request from the user device to the API, wherein the third request comprises a request to receive information associated with the list of one or more most likely identification probabilities; and
displaying at the user device one or more of the list of one or more most likely identification probabilities associated with the object, and information associated with the one or more of the list of one or more most likely identification probabilities associated with the object.

### Claim 9 (depends on 8)

The method of claim 8, wherein the initial response comprises one of a response comprising the pre-signed URL and at least a portion of the metadata, or a response denying the request;
wherein the storage system comprises a cloud-based image storage system;
wherein the metadata comprises one or more of an image path, geographic location information associated with the photograph, user identification information, timing information, and a status of the request;
wherein the second request includes at least a portion of the metadata;
wherein the second response informs the user device from the API that the image is being processed;
obtaining the photograph from the storage system comprises using the metadata at the API to locate the photograph on the cloud-based image storage system;
wherein using the photograph to invoke a first trained model from a machine learning system endpoint comprises invoking the first trained model by the API;
wherein the first trained model comprises a parent model;
wherein the second trained model comprises a child model; and
wherein using the photograph to invoke the second trained model from the machine learning system endpoint comprises using the parent model to invoke the child model.

### Claim 10 (depends on 9)

The method of claim 9, wherein the response denying the request is provided when at least one of the API fails to identify at least one photograph due to an account setting associated with the photograph preventing identifying the photograph, and a security setting preventing identifying the photograph;
wherein the image path comprises a data string identifying a location where the photograph is stored;
wherein the user identification information comprises one or more of a data string associated with a user, a timestamp, and a data string associated with the request;
wherein the status of the request comprises a first status, a second status, a third status, and a fourth status;
wherein the cloud-based image storage system comprises Amazon Simple Storage Service (S3);
wherein at least one of the metadata and the photograph are located in a cache associated with the cloud-based image storage system;
wherein the parent model identifies a category associated with the object;
wherein the child model identifies at least one species associated with the category associated with the object;
wherein the list of one or more most likely identification probabilities is not received when the machine learning system endpoint is unavailable; and
wherein updating the metadata associated with the object comprises:
changing a status of the request to the third status,
providing a period of time associated with the period between sending the initial request to identify the object in the photograph and invoking the first trained model from the machine learning system endpoint, and
providing a period of time associated with the period between sending the initial request to identify the object in the photograph and invoking the second trained model from the machine learning system endpoint.

### Claim 11 (depends on 10)

The method of claim 10, wherein the account setting comprises a non-premium user account;
wherein the first status identifies that the object has not been identified in the photograph and is set after the initial request;
wherein the second status identifies that identification of the object in the photograph is in progress, and is set after the second request;
wherein the third status comprises one of identification of the object is complete after receiving the list of one or more most likely identification probabilities and identification of the object is failed when the list of one or more most likely identification probabilities is not received;
wherein the fourth status comprises an accepted status when the object is selected and a suggested status when the object name is suggested; and
wherein the child model comprises one of a plant child model, a mammal child model, an insect child model, an amphibian child model, a fish child model, a bird child model, and a tree bark child model.

### Claim 12 (depends on 8)

The method of claim 8, wherein the sending the second request to identify the object in the photograph from the user device to the API comprises sending a plurality of second requests to identify the object in the photograph from the user device to the API;
wherein each of the plurality of second requests to identify the object in the photograph after a first of the plurality of second requests to identify the object in the photograph is sent at a predetermined period of time after the prior of the plurality of second requests to identify the object in the photograph,
and further comprising;
ceasing sending the plurality of second requests to identify the object in the photograph when the list of one or more most likely identification probabilities associated with the object is obtained.

### Claim 13 (depends on 12)

The method of claim 12, wherein the plurality of second requests to identify the object in the photograph from the user device to the API comprises a polling operation and hash information related to the metadata; and
wherein the predetermined period of time comprises a two second interval.

### Claim 14 (depends on 8)

The method of claim 8, wherein the list of one or more most likely identification probabilities associated with the object comprises the information associated with the list of one or more most likely identification probabilities, and a list of species; and
wherein the information associated with the list of one or more most likely identification probabilities comprises an object name for each of the most likely identification probabilities, an object description for each of the most likely identification probabilities, a hash value associated with each of the most likely identification probabilities, and a photograph for each of the most likely identification probabilities; and
wherein the list of species includes a hash value associated with the species.

### Claim 15

A non-transient computer-readable storage medium having instructions embodied thereon, the instructions being executable by one or more processors to perform a method for operating a machine-learning system, the method comprising:
preparing a plurality of data files for training by associating a label with each of the plurality of data files, identifying and fixing errors in the plurality of data files, and uploading the plurality of data files to a storage device;
obtaining a versioning snapshot of the plurality of data files uploaded to a storage device;
providing metadata associated with the plurality of data files to the machine-learning system;
splitting the plurality of data files into a plurality of data sets, including at least a training data set, a testing data set, and a validation data set;
creating record IO data files comprising a plurality of training set record IO files, a plurality of testing set record IO files, a plurality of validation set record IO files, and a plurality of full set record IO files;
merging the plurality of record IO files for each data set of the plurality of data sets;
passing the merged record IO files to the storage device;
identifying hyperparameter settings for use in association with training the merged full set record IO files by (1) selecting a portion of the merged testing IO files and a portion of the merged validation IO files, (2) applying a plurality of hyperparameters to the selected portion of the merged testing IO files and the selected portion of the merged validation IO files across a plurality of training jobs, and (3) reviewing one or more metrics related to the plurality of training jobs;
storing artifacts related to the plurality of training jobs at the storage device;
using the identified hyperparameter settings and the artifacts to train the merged full set record IO files;
storing one or more metrics and artifacts associated with training the merged full set record IO files;
using the one or more metrics and artifacts associated with training the merged full set record IO files to create a final machine learning model;
obtaining the final machine learning model from a machine learning system registry; and
deploying the final machine learning model to one or more machine learning system endpoints.

### Claim 16 (depends on 15)

The computer-readable storage medium of claim 15, wherein the label comprises one of a predetermined category and a species associated with one of the predetermined categories;
wherein, when the label comprises a predetermined category, the final machine learning model comprises a first trained model and the first trained model comprises a parent model;
wherein, when the label comprises a species associated with one of the predetermined categories, the final machine learning model comprises a second trained model and the second trained model comprises a child model;
the method further comprising:
when the label comprises a predetermined category, identifying the predetermined categories associated with the plurality of data files;
when the label comprises a predetermined category, training the machine-learning system on the same number of data files for each identified category;
wherein the identifying the predetermined categories associated with the plurality of data files and the training the machine-learning system on the same number of data files for each identified category occurs after providing metadata associated with the plurality of data files to the machine-learning system and before splitting the plurality of data files into the training data set, the testing data set, and the validation data set.

### Claim 17 (depends on 15)

The computer-readable storage medium of claim 15, wherein the predetermined categories comprise eight categories;
wherein the eight categories comprise or are otherwise associated with plants, mammals, insects, amphibians, fish, birds, tree bark, and snails; and
wherein the species comprise one of a plant species, a mammal species, an insect species, an amphibian species, a fish species, a bird species, a tree bark species, and a snail species.

### Claim 18

A non-transient computer-readable storage medium having instructions embodied thereon, the instructions being executable by one or more processors to perform a method for identifying an object in a photograph, the method comprising:
sending an initial request to identify the object in the photograph from a user device to an application programming interface (API);
generating, at the API, a pre-signed uniform resource locator (URL) associated with a storage system;
generating metadata associated with the initial request;
sending an initial response to the initial request;
uploading the photograph from a user device directly to a location associated with the pre-signed URL;
sending a second request to identify the object in the photograph from the user device to the API;
providing a second response to the second request;
obtaining the photograph from the storage system;
using the photograph to invoke a first trained model from a machine learning system endpoint;
using the photograph to invoke a second trained model from the machine learning system endpoint;
obtaining from the second trained model a list of one or more most likely identification probabilities associated with the object;
updating the metadata associated with the object at the API;
sending a third request from the user device to the API, wherein the third request comprises a request to receive information associated with the list of one or more most likely identification probabilities; and
displaying at the user device one or more of the list of one or more most likely identification probabilities associated with the object, and information associated with the one or more of the list of one or more most likely identification probabilities associated with the object.

### Claim 19 (depends on 18)

The computer-readable storage medium of claim 18, wherein the initial response comprises one of a response comprising the pre-signed URL and at least a portion of the metadata, or a response denying the request;
wherein the storage system comprises a cloud-based image storage system;
wherein the metadata comprises one or more of an image path, geographic location information associated with the photograph, user identification information, timing information, and a status of the request;
wherein the second request includes at least a portion of the metadata;
wherein the second response informs the user device from the API that the image is being processed;
wherein the method obtaining the photograph from the storage system comprises using the metadata at the API to locate the photograph on the cloud-based image storage system;
wherein using the photograph to invoke a first trained model from a machine learning system endpoint comprises invoking the first trained model by the API;
wherein the first trained model comprises a parent model;
wherein the second trained model comprises a child model; and
wherein using the photograph to invoke the second trained model from the machine learning system endpoint comprises using the parent model to invoke the child model.

### Claim 20 (depends on 19)

The computer-readable storage medium of claim 19, wherein the response denying the request is provided when at least one of the API fails to identify at least one photograph due to an account setting associated with the photograph preventing identifying the photograph, and a security setting preventing identifying the photograph;
wherein the image path comprises a data string identifying a location where the photograph is stored;
wherein the user identification information comprises one or more of a data string associated with a user, a timestamp, and a data string associated with the request;
wherein the status of the request comprises a first status, a second status, a third status, and a fourth status;
wherein the cloud-based image storage system comprises Amazon Simple Storage Service (S3);
wherein at least one of the metadata and the photograph are located in a cache associated with the cloud-based image storage system;
wherein the parent model identifies a category associated with the object;
wherein the child model identifies at least one species associated with the category associated with the object;
wherein the list of one or more most likely identification probabilities is not received when the machine learning system is unavailable; and
wherein updating the metadata associated with the object comprises:
changing a status of the request to the third status,
providing a period of time associated with the period between sending the initial request to identify the object in a photograph and invoking the first trained model from the machine learning system endpoint, and
providing a period of time associated with the period between sending an initial request to identify the object in a photograph and invoking the second trained model from the machine learning system endpoint.
