# US 12,002,134 B2 · Automated flow chart generation and visualization system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,134 B2
- Document ID: US12002134B2
- Status: Granted
- Kind: B2
- Application number: 17/278,100
- Filed: 2021-01-08
- Published: 2023-10-26
- Granted: 2024-06-04
- Earliest claimed: 2021-01-08
- Assignees: PwC Product Sales LLC
- Applicants: PwC Product Sales LLC
- Inventors: Yizhuo Zhang; Jun Li; Weigang Li; Shawni Chen
- Web page: https://patentgrep.com/patent/US12002134B2

## CPC

- G06T 11/23: using straight lines or curves
- G06T 11/26: Drawing of charts or graphs
- G06T 11/60: Creating or editing images; Combining images with text
- G06V 10/82: using neural networks
- G06V 30/1444: {Selective acquisition, locating or processing of specific regions, e.g. highlighted text, fiducial marks or predetermined fields}
- G06V 30/1473: {Recognising objects as potential recognition candidates based on visual cues, e.g. shapes}
- G06V 30/412: Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
- G06V 30/414: Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system for generating digital flowcharts is provided. The system receives sketch image data comprising a plurality of shapes and text, and processes the sketch image data to generate flowchart data by applying a first model configured to generate shape data, applying a second model configured to generate text data, and generating linking data that associates shape data and text data. The system may generate and display a visualization of the flowchart data. The system may map the flowchart data to a region of a presentation slide and display a visualization of the flowchart data on the presentation slide.

## Claims

### Claim 1

A system for generating digital flowcharts, the system comprising:
a display;
one or more processors; and
memory storing computer program code executable by the one or more processors to cause the system to:
receive sketch image data comprising a plurality of shapes and text;
process the sketch image data to generate flowchart data, wherein processing the sketch image data comprises:
applying a first model configured to generate shape data based on the sketch image data, wherein applying the first model to generate the shape data comprises:
detecting the presence of a shape in the sketch image data;
generating classification data indicating a predicted class of the detected shape in the sketch image data; and
generating first location data indicating a determined location of the shape;
processing the shape data to generate the flowchart data, wherein generating the flowchart data comprises generating linking data that associates the shape data with one or more other objects in the flowchart data, wherein the one or more other objects comprise an arrow shape object and a text object;
wherein generating linking data that associates the shape data with the arrow shape object comprises:
calculating a minimum distance between the determined location of the shape and a location of a portion of the arrow shape object;
determining that the minimum distance is less than or equal to a predetermined threshold distance; and
in accordance with determining that the minimum distance is less than or equal to the predetermined threshold distance, generating the linking data that associates the shape data with the arrow shape object;
wherein generating linking data that associates the shape data with the text object comprises:
determining that a bounding box of the shape overlaps a bounding box of the text object;
in accordance with determining that the bounding box of the shape overlaps the bounding box of the text object, generating the linking data that associates the shape data with the text object;
wherein generating the flowchart data comprises, in accordance with generating the linking data that associates the shape data with the text object, modifying center coordinates of the text bounding box to match center coordinates of the shape bounding box; and
generate and display, on the display, a visualization of the flowchart data.

### Claim 2 (depends on 1)

The system of claim 1, wherein receiving the sketch image data comprises receiving a user input indicating selection of the sketch image data from among a plurality of sketches uploaded to the system.

### Claim 3 (depends on 1)

The system of claim 1, wherein processing the sketch image data further comprises one or more of:
cropping the sketch image data;
rotating the sketch image data; and
resizing the sketch image data.

### Claim 4 (depends on 1)

The system of claim 1, wherein the first model comprises one or more models selected from: a R-CNN model, a Faster-RCNN model, and a YOLO model.

### Claim 5 (depends on 1)

The system of claim 1, wherein predicting the class of the detected shape in the sketch image data comprises applying a classification algorithm to select the class from a predetermined set of classes.

### Claim 6 (depends on 5)

The system of claim 5, wherein the predetermined set of classes comprises two or more of arrow, line, rectangle, ellipse, diamond, and parallelogram.

### Claim 7 (depends on 1)

The system of claim 1, wherein the first location data comprises one or more of shape center coordinates, shape width, and shape height.

### Claim 8 (depends on 7)

The system of claim 7, wherein the first location data comprises a shape bounding box comprising four corner coordinates.

### Claim 9 (depends on 8)

The system of claim 8, wherein processing the shape data to generate the flowchart data further comprises:
defining a plurality of regions for a presentation page;
mapping the shape to one of the regions of the plurality of regions based on the center coordinates of the shape bounding box; and
generating modified first location data based on region location data associated with the one of the regions.

### Claim 10 (depends on 9)

The system of claim 9, wherein the one or more other objects are associated with second location data.

### Claim 11 (depends on 10)

The system of claim 10, wherein generating the linking data comprises generating data that associates the shape with the one or more other objects based on the modified first location data and the second location data.

### Claim 12 (depends on 10)

The system of claim 10, wherein the one or more other objects comprises an arrow and the second location data indicates a location of one or more of an endpoint of the arrow and a header of the arrow.

### Claim 13 (depends on 1)

The system of claim 1, wherein:
the one or more other objects comprises a text object;
processing the sketch image data further comprises:
applying a second model configured to generate text data based on the sketch image data, wherein applying the second model to generate the text data comprises:
detecting the presence of text in the sketch image data; and
generating third location data indicating a determined location of the text; and
generating the flowchart data is further based on processing the text data and generating the linking data such that the linking data associates the shape data with the text data.

### Claim 14 (depends on 13)

The system of claim 13, wherein the second model comprises one or more optical character recognition models selected from: a CNN model, a RNN model, and Attention Networks.

### Claim 15 (depends on 13)

The system of claim 13, wherein the third location data comprises a text bounding box comprising four corner coordinates.

### Claim 16 (depends on 13)

The system of claim 13, wherein the memory storing computer program code executable by the one or more processors is further configured to cause the system to generate and display, on the display:
a visualization of the generated text data, and
an editor interface configured to allow a user to modify the generated text.

### Claim 17 (depends on 1)

The system of claim 1, wherein displaying the visualization of the flowchart data comprises displaying the visualization of the flowchart data overlaid with the sketch image.

### Claim 18 (depends on 1)

The system of claim 1, wherein the memory storing computer program code executable by the one or more processors is further configured to cause the system to:
receive a first user input comprising an instruction to modify the flowchart data;
in response to receiving the first user input, modify the flowchart data; and
update display of the visualization of the flowchart based on the modified flowchart data.

### Claim 19 (depends on 1)

The system of claim 1, wherein the memory storing computer program code executable by the one or more processors is further configured to cause the system to interface with one or more APIs to generate one or more of PowerPoint slides and Google Slides based on the flowchart data.

### Claim 20

A method for generating digital flowcharts, the method performed at a system comprising a display, one or more processors, and memory storing computer program code executable by the one or more processors configured to cause the system to carry out the method, the method comprising:
receiving sketch image data comprising a plurality of shapes and text;
processing the sketch image data to generate flowchart data, wherein processing the sketch image data comprises:
applying a first model configured to generate shape data based on the sketch image data, wherein applying the first model to generate the shape data comprises:
detecting the presence of a shape in the sketch image data;
generating classification data indicating a predicted class of the detected shape in the sketch image data; and
generating first location data indicating a determined location of the shape; and
processing the shape data to generate the flowchart data, wherein generating the flowchart data comprises generating linking data that associates the shape data with one or more other objects in the flowchart data, wherein the one or more other objects comprise an arrow shape object and a text object;
wherein generating linking data that associates the shape data with the arrow shape object comprises:
calculating a minimum distance between the determined location of the shape and a location of a portion of the arrow shape object;
determining that the minimum distance is less than or equal to a predetermined threshold distance; and
in accordance with determining that the minimum distance is less than or equal to the predetermined threshold distance, generating the linking data that associates the shape data with the arrow shape object; and
wherein generating linking data that associates the shape data with the text object comprises:
determining that a bounding box of the shape overlaps a bounding box of the text object;
in accordance with determining that the bounding box of the shape overlaps the bounding box of the text object, generating the linking data that associates the shape data with the text object;
wherein generating the flowchart data comprises, in accordance with generating the linking data that associates the shape data with the text object, modifying center coordinates of the text bounding box to match center coordinates of the shape bounding box; and
generating and displaying, on the display, a visualization of the flowchart data.

### Claim 21

A non-transitory computer-readable storage medium for generating digital flowcharts, the non-transitory computer-readable storage medium storing computer program code configured to be executed by a system comprising a display and one or more processors to cause the system to:
receive sketch image data comprising a plurality of shapes and text;
process the sketch image data to generate flowchart data, wherein processing the sketch image data comprises:
applying a first model configured to generate shape data based on the sketch image data, wherein applying the first model to generate the shape data comprises:
detecting the presence of a shape in the sketch image data;
generating classification data indicating a predicted class of the detected shape in the sketch image data; and
generating first location data indicating a determined location of the shape;
processing the shape data to generate the flowchart data, wherein generating the flowchart data comprises generating linking data that associates the shape data with one or more other objects in the flowchart data, wherein the one or more other objects comprise an arrow shape object and a text object;
wherein generating linking data that associates the shape data with the arrow shape object comprises:
calculating a minimum distance between the determined location of the shape and a location of a portion of the arrow shape object;
determining that the minimum distance is less than or equal to a predetermined threshold distance; and
in accordance with determining that the minimum distance is less than or equal to the predetermined threshold distance, generating the linking data that associates the shape data with the arrow shape object;
wherein generating linking data that associates the shape data with the text object comprises:
determining that a bounding box of the shape overlaps a bounding box of the text object;
in accordance with determining that the bounding box of the shape overlaps the bounding box of the text object, generating the linking data that associates the shape data with the text object;
wherein generating the flowchart data comprises, in accordance with generating the linking data that associates the shape data with the text object, modifying center coordinates of the text bounding box to match center coordinates of the shape bounding box; and
generate and display, on the display, a visualization of the flowchart data.