# US 2024/0202600 A1 · Machine learning model administration and optimization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202600 A1
- Document ID: US20240202600A1
- Status: Published application
- Kind: A1
- Application number: 18/542,676
- Filed: 2023-12-16
- Published: 2024-06-20
- Earliest claimed: 2022-12-16
- Assignees: C3.ai, Inc.
- Applicants: C3.ai, Inc.
- Inventors: Louis Poirier; Sina Pakazad; John Abelt; Aliakbar Panahi; Michael Haines; Romain Juban; Yushi Homma; Riyad Muradov
- Web page: https://patentgrep.com/patent/US20240202600A1

## CPC

- G06F 16/345: {Summarisation for human users}
- G06F 16/3326: {using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages}
- G06F 16/334: Query execution (filtering based on additional data G06F16/335)
- G06F 16/3347: {using vector based model}
- G06F 16/335: Filtering based on additional data, e.g. user or group profiles (filtering in web context G06F16/9535, G06F16/9536)
- G06F 16/338: Presentation of query results
- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06N 3/045: Combinations of networks
- G06N 3/0475: Generative networks
- G06N 3/092: Reinforcement learning
- G06N 5/04: Inference or reasoning models
- G06N 20/00: Machine learning
- G06N 3/063: using electronic means
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for a model inference service system that provides a technical solution for deploying and updating trained machine-learning models with support for specific use case deployments and implementations at scale with efficient processing. The model inference service system includes a hierarchical model registry for versioning models and model dependencies for each versioned model, a model inference service for rapidly deploying model instances in run-time environments, and a model processing system for managing multiple instances of deployed models. Changes to deployed models are captured as new versions in the hierarchical model registry.

## Claims

### Claim 1

A method comprising:
receiving a request associated with a machine learning application, wherein the request includes application information, user information, and execution information;
selecting, by one or more processing devices, a baseline model and one or more child model records from a hierarchical structure based on the request, wherein the baselines model and the one more child model records include model metadata with parameters describing dependencies, access control, and deployment configurations;
assembling a versioned model of the baseline model using the one more child model records and associated dependencies; and
deploying the versioned model in a configured run-time instantiation for use by the application based on the associated metadata.

### Claim 2 (depends on 1)

The method of claim 1, wherein selecting comprises:
determining compatibility between the application information and execution information of the request with dependencies and deployment configurations from model metadata, and further determining access control of the model metadata and the user information of the request.

### Claim 3 (depends on 1)

The method of claim 1,
wherein the child model records comprise intermediate representations of the baseline model with changed parameters from a previous instantiation of the baseline model.

### Claim 4 (depends on 1)

The method of claim 1,
wherein the baseline model is pre-trained on a general domain dataset, and
wherein the one or more child model records comprise intermediate representations with changed parameters of the baseline model trained on an enterprise specific dataset.

### Claim 5 (depends on 1)

The method of claim 1, wherein the deployment configurations determine a set of computing requirements for the run-time instance of the versioned model.

### Claim 6 (depends on 1)

The method of claim 1, wherein assembling the versioned model further comprises: pre-loading a set of model configurations comprising at least one or more of: model weights, adapter instructions.

### Claim 7 (depends on 1)

The method of claim 1, wherein the hierarchical structure comprises a catalogue of different baseline models that are pre-trained with different domain specific datasets, and child model records associated with each different baseline model are generated based on an intermediate record.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
receiving multiple requests received for one or more additional instances of the versioned model;
deploying multiple instances of the versioned model;
capturing changes to the versioned model as new model records with new model metadata in the hierarchical repository.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
monitoring utilization of one or more additional model processing units for the multiple instances of the versioned model; and
executing one or more load-balancing operations to terminate execution of the one or more additional instances of the versioned model based on a threshold condition of the computing environment.

### Claim 10 (depends on 1)

The method of claim 1, wherein deploying the versioned model further comprises the machine learning application executing instructions to transmit control system commands for one or more industrial devices.

### Claim 11

A system comprising:
memory storing instructions that, when executed by the one or more processors, cause the system to perform:
a model inference service for instantiating different versioned model to service a machine-learning application,
wherein a model registry comprises a hierarchical structure with a baselines model and child model records that include model metadata with parameters describing dependencies and deployment configurations to assemble the different versioned model, wherein each versioned model is assembled with the baseline model using the one more child model records and associated dependencies,
wherein the model inference service concurrently deploys multiple run-time instances with different versions of the model for different user sessions, and
wherein the model registry is updated with new model records based on the changes to the baseline model from multiple run-time instances.

### Claim 12 (depends on 11)

The system of claim 11, wherein the versioned model for each user session of the different users is based at least on the users access control privileges of each user session.

### Claim 13 (depends on 11)

The system of claim 11,
wherein the hierarchical repository comprises a catalogue of additional baseline models pretrained on datasets from different domains, and
wherein the additional model records associated with each additional baseline model is fine-tuned using local enterprise datasets.

### Claim 14 (depends on 11)

The system of claim 11, the instantiating different versioned are capable of multiple generative tasks including conversational, summarizing, computational, predictive, visualization.

### Claim 15 (depends on 11)

The system of claim 11, wherein the machine-learning application utilizes the versioned model, and wherein deploying the versioned model further comprises the machine learning application executing instructions to transmit control system commands for one or more industrial devices.

### Claim 16

A method comprising:
storing a plurality of model configuration records in a hierarchical structure of a model registry;
receiving a model request; and
retrieving, based on the model request, one or more model configuration records from the hierarchical structure of the model registry.

### Claim 17 (depends on 16)

The method of claim 16, wherein one or more versioned models are selected and replaced at run-time.

### Claim 18 (depends on 16)

The method of claim 16, wherein each of the selected one or more models are pre-trained on customer-specific data subsequent to being trained on the domain-specific dataset.

### Claim 19 (depends on 16)

The method of claim 16, further comprising:
compressing at least a portion of the plurality of model parameters of the model, thereby generating a compressed model;
deploying the compressed model to an edge device of an enterprise network;
decompressing the compressed model at run-time.

### Claim 20 (depends on 16)

The method of claim 16, wherein the compressing comprises a quantization of at least a portion of the plurality of model parameters, and the decompressing comprises a dequantization of the plurality of quantized model parameters.
