Privacy preserving insights and distillation of large language model backed experiences

A data processing system comprising:

• a processor;
• and
• a machine-readable storage medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
• receiving a plurality of natural language prompts for a first language model including receiving a first prompt that includes multiple chained utterances from a chat interface;
• storing the plurality of natural language prompts in a privacy protecting datastore in which the plurality of natural language prompts are inaccessible from outside of the privacy protecting datastore once stored in the privacy protecting datastore;
• prompting a third language model to generate a summary of the first prompt;
• storing the summary of the first prompt with the plurality of natural language prompts in the privacy protecting datastore;
• analyzing the plurality of natural language prompts stored in the privacy protecting datastore to generate embedding vectors representing the plurality of natural language prompts;
• storing the embedding vectors in the privacy protecting datastore;
• analyzing the embedding vectors using a clustering algorithm to generate clusters of related embedding vectors in the privacy protecting datastore;
• generating, for each cluster of at least a subset of the clusters of related embedding vectors, theme information summarizing the embedding vectors of the cluster;
• and
• storing the theme information associated with each cluster of the at least a subset of clusters in a second datastore outside of the privacy protecting datastore, the second datastore being accessible for querying data stored therein.

A method implemented in a data processing system for anonymizing prompts to a language model, the method comprising:

• receiving a plurality of natural language prompts for a first language model;
• storing the plurality of natural language prompts in a privacy protecting datastore in which the at least a subset natural language prompts are inaccessible from outside of the privacy protecting datastore once stored in the privacy protecting datastore;
• analyzing the plurality of natural language prompts stored in the privacy protecting datastore to generate embedding vectors representing the plurality of natural language prompts;
• storing the embedding vectors in the privacy protecting datastore;
• analyzing the embedding vectors using a clustering algorithm to generate clusters of related embedding vectors in the privacy protecting datastore;
• generating, for each cluster of at least a subset of the clusters of related embedding vectors, theme information summarizing the embedding vectors of the cluster by:
• fine-tuning a large language model (LLM) with differential privacy on each cluster of the at least a subset of clusters;
• prompting the LLM to generate a plurality of synthetic representative data points associated with each cluster;
• and
• prompting the LLM to generate the theme information for each cluster by summarizing the plurality of synthetic representative data points;
• and
• storing the theme information associated with each cluster of the at least a subset of clusters in a second datastore outside of the privacy protecting datastore, the second datastore being accessible for querying data stored therein.

A method implemented in a data processing system for anonymizing prompts to a language model, the method comprising:

• receiving a plurality of natural language prompts for a first language model including receiving a first prompt that includes multiple chained utterances from a chat interface;
• storing the plurality of natural language prompts in a privacy protecting datastore in which the plurality of natural language prompts are inaccessible from outside of the privacy protecting datastore once stored in the privacy protecting datastore;
• prompting a third language model to generate a summary of the first prompt;
• storing the summary of the first prompt with the plurality of natural language prompts in the privacy protecting datastore;
• analyzing the plurality of natural language prompts stored in the privacy protecting datastore to generate embedding vectors representing the plurality of natural language prompts;
• storing the embedding vectors in the privacy protecting datastore;
• analyzing the embedding vectors using a clustering algorithm to generate clusters of related embedding vectors in the privacy protecting datastore;
• generating, for each cluster of at least a subset of the clusters of related embedding vectors, theme information summarizing the embedding vectors of the cluster;
• and
• storing the theme information associated with each cluster of the at least a subset of clusters in a second datastore outside of the privacy protecting datastore, the second datastore being accessible for querying data stored therein.

A data processing system comprising:

• a processor;
• and
• a machine-readable storage medium storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:
• receiving natural language prompts for a first language model;
• processing the natural language prompts using a prompt anonymization pipeline configured to store the natural language prompts in a datastore that is inaccessible from outside of the prompt anonymization pipeline, the prompt anonymization pipeline being configured to anonymize the natural language prompts in the datastore by:
• generating embeddings for the natural language prompts providing a numerical representation of the natural language prompts;
• analyzing the embeddings using a clustering algorithm to identify clusters of related natural language prompts;
• analyzing at least a subset of the clusters of related natural language prompts using a third language model trained to summarize subject matter of the embeddings into a theme associated with the cluster;
• and
• storing the theme associated with each cluster of the at least the subset of the clusters in an anonymized prompt datastore;
• receiving a query for anonymized prompt information from a client device;
• executing the query on the anonymized prompt datastore to obtain query results;
• and
• providing the query results to the client device.