# US 2023/0274086 A1 · Computer implemented methods for the automated analysis or use of data, including use of a large language model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274086 A1
- Document ID: US20230274086A1
- Status: Published application
- Kind: A1
- Application number: 18/301,615
- Filed: 2023-04-17
- Published: 2023-08-31
- Earliest claimed: 0001-01-01
- Assignees: Unlikely Artificial Intelligence Limited
- Applicants: Unlikely Artificial Intelligence Limited
- Inventors: William TUNSTALL-PEDOE; Robert HEYWOOD; Seth WARREN; Paul DARIAS; Duncan REYNOLDS; Ayush SHAH; Luci KRNIC; Ziyi ZHU
- Web page: https://patentgrep.com/patent/US20230274086A1

## CPC

- G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30)
- G06F 16/3344: {using natural language analysis}
- G06F 40/30: Semantic analysis
- G06F 40/56: Natural language generation
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods are provided, such as a method of interacting with a large language model (LLM), including the step of a processing system using a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, to provide new context data for the LLM, in order to improve the output, such as continuation text output, generated by the LLM in response to a prompt; and such as a method of interacting with a LLM, including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user. Related computer systems are provided.

## Claims

### Claim 1

A method for ensuring that an LLM generates original text, comprising the steps of:
(i) providing a first database of previous text that the LLM should not generate;
(ii) performing a beam search;
(iii) checking potential continuations against the first database;
(iv) when a potential continuation matches non-original text in the first database, adjusting the continuation to no longer match non-original text in the first database;
the method further including a method for adding citations to the text generated by the LLM comprising the steps of:
(v) providing a second database of text used to train the LLM which includes sources associated with each section of text stored;
(vi) checking sections of the continuation generated by the LLM against the second database; and
(vii) retrieving sources where the sections of the continuation match text contained within the second database.

### Claim 2 (depends on 1)

The method of claim 1 where adjusting the continuation includes the step of selecting a different sequence from the beam search.

### Claim 3 (depends on 1)

The method of claim 1, further comprising the step of providing a classifier that can distinguish between prompts that require original text as a continuation and those that do not and applying the classifier to the prompt.

### Claim 4 (depends on 1)

The method of claim 1, where checking potential continuations against the first database includes close but imperfect matches to the previous text.

### Claim 5 (depends on 1)

The method of claim 1, further comprising the steps of:
providing a classifier to determine whether a section of the continuation is appropriate for citing references against;
utilising the classifier when matching sections of the continuation.

### Claim 6 (depends on 1)

The method of claim 1, further comprising the step of:
adjusting the continuation to appropriately cite at least some of the sources.

### Claim 7 (depends on 1)

The method of claim 1, including a method of interacting with the large language model (LLM), including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output (e.g. text output) generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user.

### Claim 8 (depends on 7)

The method of claim 7, in which the processing system translates at least some of the continuation data into the structured, machine-readable representation of data to analyse the translated continuation data.

### Claim 9 (depends on 7)

The method of claim 7, in which the processing system translates the continuation data into the structured, machine-readable representation of data to analyse the translated continuation data.

### Claim 10 (depends on 8)

The method of claim 8, in which the processing system analyses the translated continuation data for at least one of factual inaccuracies, contradictions and scope, and, where necessary or beneficial, provides corrected output for use in the continuation output provided by the LLM, through the processing system providing an updated prompt, that is presented to a user.

### Claim 11 (depends on 8)

The method of claim 8, in which the processing system analyses the translated continuation data and, where necessary or beneficial, provides corrected output for use in the continuation output that is presented to a user.

### Claim 12 (depends on 7)

The method of claim 7, in which the processing system analyses the internal, logical self-consistency, including a correct time ordering of events, of the continuation output and, where necessary or beneficial, provides logically self-consistent output, for use in the continuation output that is provided to a user.

### Claim 13 (depends on 7)

The method of claim 7, in which the processing system analyses the internal, logical self-consistency of the continuation output and, where necessary or beneficial, provides logically self-consistent output, for use in the continuation output that is provided to a user.

### Claim 14 (depends on 7)

The method of claim 7, in which the processing system analyses the correspondence of the continuation output generated by the LLM to how people understand the real world or reason in the real world and, where necessary or beneficial, provides reasoned output, corresponding to how people understand the real world or reason in the real world, for use in the continuation output that is provided to a user.

### Claim 15 (depends on 7)

The method of claim 7,
(a) in which the processing system analyses the continuation output for bias and, where necessary or beneficial, provides reduced bias or bias-free output, for use in the continuation output that is provided to a user; or
(b) in which the processing system analyses the continuation output for the absence of relevant dynamic or real-time information and, where necessary or beneficial, provides any missing or useful dynamic or real-time information for use in the continuation output that is provided to a user; or
(c) in which the processing system analyses the continuation output for any reasoned text and, where necessary or beneficial, provides improved reasoned text for use in the continuation output that is provided to a user; or
(d) in which the LLM is answering a question and in which the processing system analyses the answer to that question in the continuation output and where necessary or beneficial, provides an improved answer for use in the continuation output that is provided to a user; or
(e) in which a classifier operates to identify when a prompt is likely to result in a continuation output where accuracy is important, and/or when accuracy is important in the continuation output, and to then use the processing system to improve the factual accuracy and/or factual scope of that continuation output; or
(f) in which the continuation output from the LLM is a partial continuation, namely an output made before the LLM has stopped generating or whilst the LLM is still generating; or
(g) in which a classifier operates to identify when a prompt is likely to result in a continuation where accuracy is important; or
(h) when used to improve one or more of: the accuracy of the output of the LLM; factual accuracy and/or factual scope of the output; the internal, logical self-consistency (e.g. correct time ordering of events) of the output; correspondence of the output generated by the LLM to how people understand the real world or reason in the real world; bias reduction or removal in the output; inclusion of dynamic or real-time information; or
(i) when used to improve one or more parameters of the output of the LLM: level of formality, level of brevity (e.g. briefer when the language is to be spoken), suitability for speaking via a text to speech system, other style language, level of certainty; or
(j) in which the LLM is a generative AI based system; or
(k) in which the LLM is an autoregressive language model; or
(l) in which the LLM is an autoregressive language model, which is a Generative Pre-trained Transformer; or
(m) when used for: generation of program code; or
(n) when used for: the solution of any problem that can be described in natural language; or
(o) when used for any of the following: the generation of poetry, lyrics, creative writing, the generation of other forms of writing such as essays, summaries of knowledge, summaries of longer texts, essays, scientific papers; internet search.

### Claim 16

A method for ensuring that an LLM generates original text, comprising the steps of:
(i) providing a database of previous text that the LLM should not generate;
(ii) performing a beam search;
(iii) checking potential continuations against the database;
(iv) when a potential continuation matches non-original text in the database, adjusting the continuation to no longer match non-original text in the database;
the method further including a method for adding citations to the text generated by the LLM comprising the steps of:
(v) using the database, the database including text used to train the LLM which includes sources associated with each section of text stored;
(vi) checking sections of the continuation generated by the LLM against the database; and
(vii) retrieving sources where the sections of the continuation match text contained within the database.

### Claim 17 (depends on 16)

The method of claim 16 where adjusting the continuation includes the step of selecting a different sequence from the beam search.

### Claim 18 (depends on 16)

The method of claim 16, further comprising the step of providing a classifier that can distinguish between prompts that require original text as a continuation and those that do not and applying the classifier to the prompt.

### Claim 19 (depends on 16)

The method of claim 16, where checking potential continuations against the database includes close but imperfect matches to the previous text.

### Claim 20 (depends on 16)

The method of claim 16, further comprising the steps of:
Providing a classifier to determine whether a section of the continuation is appropriate for citing references against;
Utilising the classifier when matching sections of the continuation.

### Claim 21 (depends on 16)

The method of claim 16, further comprising the step of:
Adjusting the continuation to appropriately cite at least some of the sources.

### Claim 22 (depends on 16)

The method of claim 16, including a method of interacting with the large language model (LLM), including the step of providing continuation data generated by the LLM to a processing system that uses a structured, machine-readable representation of data that conforms to a machine-readable language, such as a universal language, in which the processing system is configured to analyse the continuation output (e.g. text output) generated by the LLM in response to a prompt to enable an improved version of that continuation output to be provided to a user.

### Claim 23 (depends on 22)

The method of claim 22, in which the processing system translates at least some of the continuation data into the structured, machine-readable representation of data to analyse the translated continuation data.

### Claim 24 (depends on 22)

The method of claim 22, in which the processing system translates the continuation data into the structured, machine-readable representation of data to analyse the translated continuation data.

### Claim 25 (depends on 23)

The method of claim 23, in which the processing system analyses the translated continuation data for at least one of factual inaccuracies, contradictions and scope, and, where necessary or beneficial, provides corrected output for use in the continuation output provided by the LLM, through the processing system providing an updated prompt, that is presented to a user.

### Claim 26 (depends on 23)

The method of claim 23, in which the processing system analyses the translated continuation data and, where necessary or beneficial, provides corrected output for use in the continuation output that is presented to a user.

### Claim 27 (depends on 22)

The method of claim 22, in which the processing system analyses the internal, logical self-consistency, including a correct time ordering of events, of the continuation output and, where necessary or beneficial, provides logically self-consistent output, for use in the continuation output that is provided to a user.

### Claim 28 (depends on 22)

The method of claim 22, in which the processing system analyses the internal, logical self-consistency of the continuation output and, where necessary or beneficial, provides logically self-consistent output, for use in the continuation output that is provided to a user.

### Claim 29 (depends on 22)

The method of claim 22, in which the processing system analyses the correspondence of the continuation output generated by the LLM to how people understand the real world or reason in the real world and, where necessary or beneficial, provides reasoned output, corresponding to how people understand the real world or reason in the real world, for use in the continuation output that is provided to a user.

### Claim 30 (depends on 22)

The method of claim 22,
(a) in which the processing system analyses the continuation output for bias and, where necessary or beneficial, provides reduced bias or bias-free output, for use in the continuation output that is provided to a user; or
(b) in which the processing system analyses the continuation output for the absence of relevant dynamic or real-time information and, where necessary or beneficial, provides any missing or useful dynamic or real-time information for use in the continuation output that is provided to a user; or
(c) in which the processing system analyses the continuation output for any reasoned text and, where necessary or beneficial, provides improved reasoned text for use in the continuation output that is provided to a user; or
(d) in which the LLM is answering a question and in which the processing system analyses the answer to that question in the continuation output and where necessary or beneficial, provides an improved answer for use in the continuation output that is provided to a user; or
(e) in which a classifier operates to identify when a prompt is likely to result in a continuation output where accuracy is important, and/or when accuracy is important in the continuation output, and to then use the processing system to improve the factual accuracy and/or factual scope of that continuation output; or
(f) in which the continuation output from the LLM is a partial continuation, namely an output made before the LLM has stopped generating or whilst the LLM is still generating; or
(g) in which a classifier operates to identify when a prompt is likely to result in a continuation where accuracy is important; or
(h) when used to improve one or more of: the accuracy of the output of the LLM; factual accuracy and/or factual scope of the output; the internal, logical self-consistency (e.g. correct time ordering of events) of the output; correspondence of the output generated by the LLM to how people understand the real world or reason in the real world; bias reduction or removal in the output; inclusion of dynamic or real-time information; or
(i) when used to improve one or more parameters of the output of the LLM: level of formality, level of brevity (e.g. briefer when the language is to be spoken), suitability for speaking via a text to speech system, other style language, level of certainty; or
(j) in which the LLM is a generative AI based system; or
(k) in which the LLM is an autoregressive language model; or
(l) in which the LLM is an autoregressive language model, which is a Generative Pre-trained Transformer; or
(m) when used for: generation of program code; or
(n) when used for: the solution of any problem that can be described in natural language; or
(o) when used for any of the following: the generation of poetry, lyrics, creative writing, the generation of other forms of writing such as essays, summaries of knowledge, summaries of longer texts, essays, scientific papers; internet search.
