Text chunking using context disruption
AssigneeDell Products L.P.
InventorsClaudio Romero, Renam Castro da Silva, Renan Carlos de Faria
Life of this patent
Estimating the term…
Abstract
Chunking text in generative artificial intelligence (GenAI) systems including retrieval augmented generation (RAG) system. When ingesting sources into a RAG, the textual data extracted from the sources is chunked or split. A chunking operation chunks or splits the textual data at locations where the predictability or probability of a text piece does not benefit from contextual information provided by prior or contextual pieces of text. This results in more coherent chunks and improves the performance of the Gen AI system.
Claims20
A method comprising:
- generating a list of text pieces from textual data that is generated from one or more sources;
- generating pairs of text pieces from the list of text pieces, wherein each of the pairs includes a context text piece and a current text piece;
- for each pair of text pieces:
- determining a first probability by a model of the current text piece in the pair of text pieces without considering the context text piece;
- determining a second probability by the model of the current text piece that considers the context text piece;
- performing a probability function using the first probability and the second probability to determine whether the context text piece reduces self-information of the current text piece compared to not knowing the context text piece;
- and
- appending the current text piece to a current text unit when the probability function is satisfied,
- wherein a set of text units constitutes chunks for a generative artificial intelligence system.
A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
- generating a list of text pieces from textual data that is generated from one or more sources;
- generating pairs of text pieces from the list of text pieces, wherein each of the pairs includes a context text piece and a current text piece;
- for each pair of text pieces:
- determining a first probability by a model of the current text piece in the pair of text pieces without considering the context text piece;
- determining a second probability by the mode of the current text piece that considers the context text piece;
- performing a probability function using the first probability and the second probability to determine whether the context text piece reduces self-information of the current text piece compared to not knowing the context text piece;
- and
- appending the current text piece to a current text unit when the probability function is satisfied,
- wherein a set of text units constitutes chunks for a generative artificial intelligence system.
Dependencies are inferred from the claim text.