# US 2022/0036083 A1 · System and method for generation of process graphs from multi-media narratives

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036083 A1
- Document ID: US20220036083A1
- Status: Published application
- Kind: A1
- Application number: 16/942,485
- Filed: 2020-07-29
- Published: 2022-02-03
- Earliest claimed: 2020-07-29
- Assignees: Kpmg LLP
- Applicants: Kpmg LLP
- Inventors: Swaminathan Chandrasekaran; Shiwangi Singh; Shan-Chuan Li; Anand Sekhar; Qianhao Yu; Sathyanarayan Venkateswaran; Oliver Michael Baltay
- Web page: https://patentgrep.com/patent/US20220036083A1

## CPC

- G06F 40/30: Semantic analysis
- G06V 20/41: {Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items (segmenting video sequences G06V20/49)}
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06F 18/24: Classification techniques
- G06F 18/251: {of input or preprocessed data}
- G06F 18/29: {Graphical models, e.g. Bayesian networks}
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06Q 10/10: Office automation; Time management
- G06Q 30/02: Marketing; Price estimation or determination; Fundraising
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/803: {of input or preprocessed data}
- G06V 10/84: using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 40/174: {Facial expression recognition}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A system for characterizing content relating to a desired outcome is disclosed. The disclosed system can be configured to identify context included in content collected from various content sources, map the identified context into graph nodes and graph edges connecting the graph nodes, identify one or more features of the identified context and adjust at least one of: a graph node and a graph edge based on the identified one or more features, identify a graph incorporating the graph nodes, the graph edges, and at least one of an adjusted graph node and an adjusted graph edge, and provide a recommendation for at least one action for achieving the desired outcome based on the identified graph.

## Claims

### Claim 1

A system for characterizing content relating to a desired outcome, the system comprising:
at least one memory operable to store content collected from one or more content generators over a predetermined period of time; and
at least one processor communicatively coupled to the at least one memory, the processor being operable to:
identify context data included in the collected content;
map the identified context data into two or more graph nodes and one or more graph edges connecting the at least two graph nodes;
identify one or more features of the identified context data and adjust at least one of: a graph node and a graph edge based on the identified one or more features;
identify a graph incorporating the two or more graph nodes, the one or more graph edges, and at least one of an adjusted graph node and an adjusted graph edge; and
provide a recommendation for at least one action for achieving the desired outcome based on the identified graph.

### Claim 2 (depends on 1)

The system of claim 1, wherein the one of more content generators comprise at least one of audio content and video content obtained from an entity having information on the desired outcome.

### Claim 3 (depends on 2)

The system of claim 2, wherein the processor is operable to identify the context based on converting the audio content to text.

### Claim 4 (depends on 2)

The system of claim 2, wherein the processor is operable to identify the context by performing frame detection on the video content.

### Claim 5 (depends on 2)

The system of claim 2, wherein the processor is operable to identify at least one content generator based on one or more characteristics of the at least one content generator.

### Claim 6 (depends on 5)

The system of claim 5, wherein the one or more characteristics of the least one content generator comprises at least one of previous content history, previous review of generated content, and known qualifications of the at least one content generator.

### Claim 7 (depends on 1)

The system of claim 1, wherein the processor is operable to identify the context by translating the collected content to a native language.

### Claim 8 (depends on 1)

The system of claim 1, wherein the processor is configured to identify the one or more features by performing at least one of interference analysis, electrodermal emotions weighting, truthfulness analysis and weighting, visual posture analysis and weighting, and influence point detection and weighting.

### Claim 9 (depends on 1)

The system of claim 1, wherein the one or more features comprise at least one of tone and emotion of the identified context.

### Claim 10 (depends on 1)

The system of claim 1, wherein the processor is operable to identify the context based on one-shot learning.

### Claim 11 (depends on 1)

The system of claim 1, wherein the processor is operable to collect the content from the one or more content generators.

### Claim 12 (depends on 11)

The system of claim 11, wherein the processor is operable to collect the content over two or more periods of time.

### Claim 13

A system for generating a process flow graph from content data, comprising
one or more content generators for generating the content data,
a content collector for collecting the content data generated by the one or more content generators,
a domain specific knowledge unit for providing domain specific content data,
a content data pre-processing subsystem having a storage element for storing the content data and a processor for processing the content data and the domain specific content data based on the type of content data to generate processed content data and charactering the processed content data to generate an initial process graph having a plurality of nodes and a plurality of edges,
a database for storing the content data from the one or more content generators and the processed content data from the content data pre-processing subsystem, and
a processing and weighting subsystem for processing the initial process graph and weighting each of the plurality of nodes relative to each other to generate a final process graph.

### Claim 14 (depends on 13)

The system of claim 13, wherein the content data includes one or more of audio data and video data, and wherein the content data preprocessing subsystem includes one or more processors for:
converting the audio data to text data,
detecting frames in the video data,
decomposing the text data into selected components using a natural language processing technique,
identifying one or more selected components of the text data and then classifying the text data into at least one of a plurality of classifications using a classification technique, and
extracting nodes and edges from the content data and then generating the initial process graph therefrom.

### Claim 15 (depends on 14)

The system of claim 14, wherein the selected components of the text data comprise one or more of paragraph, sentences and phrases.

### Claim 16 (depends on 14)

The system of claim 14, further comprising a translation unit for translating the text data from one language to another language.

### Claim 17 (depends on 14)

The system of claim 14, wherein the processor is operable to identify at least one of the one or more content generators based on one or more characteristics of the one or more content generators.

### Claim 18 (depends on 17)

The system of claim 17, wherein the one or more characteristics of the at least one content generator comprises at least one of previous content history data, previous review of generated content data, and known qualifications of the at least one content generator.

### Claim 19 (depends on 13)

The system of claim 13, wherein the one or more content generators comprise one or more of an entity, a subject matter expert, and data associated with a biometric wearable device.

### Claim 20 (depends on 14)

The system of claim 14, wherein the content data includes audio data, video data, biometric data, physiological data, context data and electronic data

### Claim 21 (depends on 14)

The system of claim 14, wherein the classification technique includes one or more of a recurrent neural network (RNN) technique, bidirectional encoder technique, and long-short term memory (LSTM) technique.

### Claim 22 (depends on 13)

The system of claim 13, wherein the domain specific information of the domain knowledge unit includes one or more of domain specific entities, domain specific terms, domain specific actions, and domain specific glossary entries.

### Claim 23 (depends on 13)

The system of claim 13, wherein the processing and weighting subsystem includes a video posture analysis subsystem for processing biometric data associated with the content data, wherein the biometric data includes facial expression data of the content generator and posture data of the content generator, and wherein the video posture analysis subsystem comprises
a facial analysis unit for analyzing the facial expression data and identifying selected emotions associated with the facial expression data, and
a body language analysis unit for analyzing the posture data to identify selected types of body language therein,
wherein the facial expression data and the posture data can be used by the processing and weighting subsystem to determine the nodes in the final process graph.

### Claim 24 (depends on 23)

The system of claim 23, wherein the processing and weighting subsystem further comprises an emotion detection subsystem for analyzing the content data to identify and determine the emotion data associated with the content data, wherein the content data includes audio data, and wherein the emotion detection subsystem includes
an electrodermal tonal information unit for detecting tonal information in the content data associated with one or more of the content generators, and
a speech emotion recognition unit for detecting emotion in the audio data.

### Claim 25 (depends on 24)

The system of claim 24, wherein the tonal information includes one or more of confidence, calmness, happiness, anger, sadness, fearful, disgust, and surprise associated with the content generator.

### Claim 26 (depends on 24)

The system of claim 24, wherein the processing and weighting subsystem further includes a language analysis subsystem for analyzing the audio data portion of the content data, wherein the language analysis subsystem includes
a commonsense task inference analysis unit for generating task data from the content data, and
a truthfulness analysis and weighting unit for performing feature analysis on the content data.

### Claim 27 (depends on 26)

The system of claim 26, wherein the processing and weighting subsystem further comprises a graph edge-node generation unit for generating the final process graph from the initial process graph and then refining the graph nodes and edges of the initial process graph based on the data from one or more of the video posture analysis subsystem, the emotion detection subsystem, and the language analysis subsystem.

### Claim 28 (depends on 27)

The system of claim 27, further comprising
a multi-factorial context weighting unit for weighting the nodes and edges of the final process graph based on one or more types of addition context data,
an influence point detection unit for further weighting the nodes and edges of the final process graph by detecting one or more points of influence in the content data, and
a one-shot learning unit for classifying and weighing data associated with the graph nodes.

### Claim 29 (depends on 28)

The system of claim 28, wherein the one or more types of additional context data comprises attributes associated with one or more of the content generators

### Claim 30

A computer implemented method for generating a process flow graph from content data, comprising
providing one or more content generators for generating the content data,
collecting the content data generated by the one or more content generators with a content collector,
providing domain specific content data via a domain specific knowledge unit,
processing the content data and the domain specific content data with a content data pre-preprocessing subsystem based on the type of content data to generate processed content data and charactering the processed content data to generate an initial process graph having a plurality of nodes and a plurality of edges,
storing the content data from the one or more content generators and the processed content data from the content data pre-processing subsystem, and
processing the initial process graph and weighting each of the plurality of nodes relative to each other to generate a final process graph with a processing and weighting subsystem .

### Claim 31 (depends on 30)

The computer implemented method of claim 30, wherein the content data includes one or more of audio data and video data, and with the content data preprocessing subsystem:
converting the audio data to text data,
detecting frames in the video data,
decomposing the text data into selected components using a natural language processing technique,
identifying one or more selected components of the text data and then classifying the text data into at least one of a plurality of classifications using a classification technique, and
extracting nodes and edges from the content data and then generating the initial process graph therefrom.

### Claim 32 (depends on 31)

The computer implemented method of claim 31, wherein the selected components of the text data comprise one or more of paragraph, sentences and phrases. The computer implemented method of claim 31, further comprising translating the text data from one language to another language.

### Claim 34 (depends on 31)

The computer implemented method of claim 31, further comprising identifying at least one of the one or more content generators based on one or more characteristics of the one or more content generators.

### Claim 35 (depends on 34)

The computer implemented method of claim 34, wherein the one or more characteristics of the at least one content generator comprises at least one of previous content history data, previous review of generated content data, and known qualifications of the at least one content generator.

### Claim 36 (depends on 30)

The computer implemented method of claim 30, wherein the one or more content generators comprise one or more of an entity, a subject matter expert, and data associated with a biometric wearable device.

### Claim 37 (depends on 31)

The computer implemented method of claim 31, wherein the classification technique includes one or more of a recurrent neural network (RNN) technique, bidirectional encoder technique, and long-short term memory (LSTM) technique.

### Claim 38 (depends on 31)

The computer implemented method of claim 31, wherein the domain specific information of the domain knowledge unit includes one or more of domain specific entities, domain specific terms, domain specific actions, and domain specific glossary entries.

### Claim 39 (depends on 30)

The computer implemented method of claim 30, wherein the processing and weighting subsystem includes a video posture analysis subsystem for processing biometric data associated with the content data, wherein the biometric data includes facial expression data of the content generator and posture data of the content generator, and wherein the video posture analysis subsystem comprises a processor for:
analyzing the facial expression data and identifying selected emotions associated with the facial expression data with a facial analysis unit, and
analyzing the posture data to identify selected types of body language therein with a body language analysis unit,
wherein the facial expression data and the posture data can be used by the processing and weighting subsystem to determine the nodes in the final process graph.

### Claim 40 (depends on 39)

The computer implemented method of claim 39, wherein the processing and weighting subsystem further comprises an emotion detection subsystem for analyzing the content data to identify and determine the emotion data associated with the content data, wherein the content data includes audio data, and wherein the emotion detection subsystem includes a processor for:
detecting tonal information in the content data associated with one or more of the content generators with an electrodermal tonal information unit, and
detecting emotion in the audio data with a speech emotion recognition unit.

### Claim 41 (depends on 40)

The computer implemented method of claim 40, wherein the tonal information includes one or more of confidence, calmness, happiness, anger, sadness, fearful, disgust, and surprise associated with the content generator.

### Claim 42 (depends on 40)

The computer implemented method of claim 40, wherein the processing and weighting subsystem further includes a language analysis subsystem for analyzing the audio data portion of the content data, wherein the language analysis subsystem includes a processor for:
generating task data from the content data with a commonsense task inference analysis unit, and
performing feature analysis on the content data with a truthfulness analysis and weighting unit.

### Claim 43 (depends on 42)

The computer implemented method of claim 42, wherein the processing and weighting subsystem further comprises a processor for generating the final process graph from the initial process graph and then refining the graph nodes and edges of the initial process graph based on the data from one or more of the video posture analysis subsystem, the emotion detection subsystem, and the language analysis subsystem.

### Claim 44 (depends on 43)

The computer implemented method of claim 43, further comprising
weighting the nodes and edges of the final process graph based on one or more types of addition context data,
further weighting the nodes and edges of the final process graph by detecting one or more points of influence in the content data, and
classifying and weighing data associated with the graph nodes.
