# US 2016/0260023 A1 · Digital object library management system for machine learning applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0260023 A1
- Document ID: US20160260023A1
- Status: Published application
- Kind: A1
- Application number: 14/635,711
- Filed: 2015-03-02
- Published: 2016-09-08
- Earliest claimed: 2015-03-02
- Assignees: Northrop Grumman Systems Corporation
- Applicants: Northrop Grumman Systems Corporation
- Inventors: Scott B. Miserendino, JR.; Donald D. Steiner; Ryan V. Peters; Guy B. Fairbanks
- Web page: https://patentgrep.com/patent/US20160260023A1

## CPC

- G06N 20/00: Machine learning
- G06F 18/213: Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F 21/562: {Static detection}
- G06N 3/12: using genetic models
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Digital object library management systems and methods for machine learning applications are taught herein. Such a method includes populating a digital object library with a number of machine readable digital objects, modifying the digital objects to include additional machine readable data about the digital objects or other digital objects and the relationships among existing digital objects, generating lists of objects for use in construction and verification of machine learning models used to classify unknown objects into one or more categories, building queries to generate object lists, initiating model generation, in which a machine learning model used to classify unknown objects into one or more categories is generated, initiating model evaluation, storing models, object lists, evaluation results, and associations among these objects, generating a visual display of object metadata, lists, relational information, and evaluation results and running distributable algorithms across the library of digital objects.

## Claims

### Claim 1

A method for managing training and testing of machine learning models using a digital object library management system, graphical user interface, distributed analytic modules, and one or more clusters of computers, the method comprising:
populating a digital object library with a number of machine readable digital objects;
modifying the digital objects to include additional machine readable data about the digital objects or other digital objects and the relationships among existing digital objects;
generating lists of objects for use in construction and verification of machine learning models used to classify unknown objects into one or more categories;
building queries to generate object lists;
initiating model generation, wherein one or more machine learning models used to classify unknown objects into one or more categories are generated;
initiating model evaluation;
storing models, object lists, evaluation results, and association among these objects;
generating a visual display of object metadata, lists, relational information, and evaluation results; and
running distributable algorithms across the library of digital objects.

### Claim 2 (depends on 1)

The method of claim 1, wherein modifying the digital object metadata is performed using distributable algorithms.

### Claim 3 (depends on 1)

The method of claim 1, wherein the system is used to manage the generation of classifiers for the purpose of distinguishing benign computer readable files from malicious computer readable files.

### Claim 4 (depends on 1)

The method of claim 1, wherein the model generation is performed on multiple computers simultaneously.

### Claim 5 (depends on 1)

The method of claim 1, wherein the model evaluation is performed on multiple computers simultaneously.

### Claim 6 (depends on 1)

The method of claim 1, wherein the distributable algorithms include Map/Reduce.

### Claim 7 (depends on 1)

The method of claim 1, wherein the models used for classification are generated by supervised machine learning algorithms.

### Claim 8 (depends on 1)

The method of claim 1, wherein a cluster of computers is used for training and testing the models and a separate cluster is used for data storage.

### Claim 9 (depends on 1)

The method of claim 1, wherein the modifying the digital objects includes receiving user input about the digital objects.

### Claim 10 (depends on 1)

The method of claim 1, wherein the building queries to generate the object lists includes receiving user selections to build the queries.

### Claim 11 (depends on 1)

The method of claim 1, wherein the modifying the digital objects includes generating one or more metadata digital objects that include the additional machine readable data about the digital objects or other digital objects and the relationships among existing digital objects.

### Claim 12

A system for managing digital object libraries used for training and testing of machine learning models comprising:
a digital object library management system (DOLMS) server that is used to manage the storage, creation and modification of digital objects used for training and testing machine learning models, wherein the DOLMS server further includes:
a digital object repository (DOR);
graphical user interface (GUI) software;
a database; and
distributed analytic modules;
a test and training cluster of computers used to train and test the machine learning models; and
a storage cluster of computers used to store the digital objects in one or more digital object libraries.

### Claim 13 (depends on 12)

The system of claim 12 wherein test and training cluster include a separate test cluster and training cluster.

### Claim 14 (depends on 13)

The system of claim 13 wherein the test cluster includes a virtual test cluster including a plurality of test virtual machines.

### Claim 15 (depends on 12)

The system of claim 12 wherein the DOR receives digital objects uploaded based on user selections.

### Claim 16 (depends on 15)

The system of claim 15 wherein the DOR stores the digital objects in one or more digital object libraries in the storage cluster.

### Claim 17 (depends on 12)

The system of claim 12 wherein the GUI software generates GUIs displayed to a user and through which the user makes selections to affect the training and testing of the machine learning models, configure the DOR, manage objects in DOR, and search the DOR.

### Claim 18 (depends on 12)

The system of claim 12 wherein the distributed analytic modules operate over the digital objects in the storage cluster in order to generate new information regarding one or more digital objects.

### Claim 19

A tangible computer readable medium including instructions for executing a method for managing training and testing of machine learning models using a digital object library management system, graphical user interface, distributed analytic modules, and one or more clusters of computers, by:
populating a digital object library with a number of machine readable digital objects;
modifying the digital objects to include additional machine readable data about the digital objects or other digital objects and the relationships among existing digital objects;
generating lists of objects for use in construction and verification of machine learning models used to classify unknown objects into one or more categories;
building queries to generate object lists;
initiating model generation, wherein one or more machine learning models used to classify unknown objects into one or more categories are generated;
initiating model evaluation;
storing models, object lists, evaluation results, and association among these objects;
generating a visual display of object metadata, lists, relational information, and evaluation results; and
running distributable algorithms across the library of digital objects.
