# US 2018/0101512 A1 · Multi-Media Forms System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0101512 A1
- Document ID: US20180101512A1
- Status: Published application
- Kind: A1
- Application number: 15/721,084
- Filed: 2017-09-29
- Published: 2018-04-12
- Earliest claimed: 2016-10-07
- Assignees: Flowfinity Wireless, Inc.
- Inventors: Dmitry Mikhailov; Olga Sboychakova
- Web page: https://patentgrep.com/patent/US20180101512A1

## CPC

- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06F 16/29: Geographical information databases
- G06F 16/433: {using audio data}
- G06F 16/685: {using automatically derived transcript of audio data, e.g. lyrics (speech recognition G10L15/00)}
- G06F 16/84: Mapping; Conversion
- G06F 40/174: Form filling; Merging
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An inventory management or equipment inspection system that allows an audio or audio-video session to be recorded and converted to text, where the text is entered into semi-structured fields of a data form, while the audio track is being recorded. In other words, the system records an audio or audio-video session providing for concurrent collection of unstructured data stream, such as the audio track, and structured or semi-structured data, such as completion of text fields in the form. The system provides the ability to record the session, automatically convert the recorded voice to text, enter the text into data fields, and edit the text in the fields, while the audio or audio-video session is ongoing.

## Claims

### Claim 1

A method for recording multi-media data, comprising:
recording a data collection session including a data streaming session;
converting the data streaming session to a number of text entries while the data streaming session is in progress;
receiving the text entries into semi-structured data fields of a form while the data streaming session is in progress;
receiving edits to the text entries in the semi-structured data fields of the form while the data streaming session is in progress; and
storing the data streaming session together with the data form containing the edit text entries converted from the data streaming session.

### Claim 2 (depends on 1)

The method of claim 1, further comprising receiving automatic voice-to-text entries and manual text entries into the semi-structured data fields of the form while the data streaming session is in progress.

### Claim 3 (depends on 1)

The method of claim 1, further comprising displaying a location map and receiving data entries in association with the location map.

### Claim 4 (depends on 1)

The method of claim 1, further comprising displaying a schematic diagram and receiving data entries in association with the schematic diagram.

### Claim 5 (depends on 1)

The method of claim 1, further comprising displaying a diagram or illustration of a machine or mechanical part and receiving data entries in association with the a diagram or illustration of the machine or mechanical part.

### Claim 6 (depends on 1)

The method of claim 1, further comprising displaying a picture of a person, place or feature and receiving data entries in association with the picture of the person, place or feature.

### Claim 7 (depends on 1)

The method of claim 1, further comprising displaying a geo-map of a site of the data collection including indicia of structures to be inspected, an indication of a current location of location of a user, and displaying or recording a route traveled by the user during a data collection session.

### Claim 8 (depends on 1)

The method of claim 1, further comprising displaying a control panel and receiving user inputs controlling the data streaming session via the control panel.

### Claim 9 (depends on 1)

The method of claim 1, further comprising displaying a graphic image associated with a data collection session, and receiving and storing changes to the a graphic image during the data collection session.

### Claim 10 (depends on 1)

The method of claim 1, further comprising displaying or announcing navigation instructions to guide a user through the data collection session.

### Claim 11 (depends on 1)

The method of claim 1, further comprising displaying a split-screen user interface with data streaming controls in a first portion of the split-screen user interface, and a structured or semi-structured form in a second portion of the split-screen user interface.

### Claim 12 (depends on 1)

The method of claim 1, further comprising automatically advancing data collection from a first form field to a second form field in accordance with predefined conditions associated with the streaming data.

### Claim 13 (depends on 1)

The method of claim 1, further comprising playing back a stored data collection session and receiving edits to data entries of the session while playing back the stored data collection session.

### Claim 14

A non-transient data storage medium storing computer executable instructions for causing a computer-controlled device to perform steps, comprising:
recording a data collection session including a data streaming session;
converting the data streaming session to a number of text entries while the data streaming session is in progress;
receiving the text entries into semi-structured data fields of a form while the data streaming session is in progress;
receiving edits to the text entries in the semi-structured data fields of the form while the data streaming session is in progress; and
storing the data streaming session together with the data form containing the edit text entries converted from the data streaming session.

### Claim 15 (depends on 14)

The data storage medium of claim 14, further comprising receiving automatic voice-to-text entries and manual text entries into the semi-structured data fields of the form while the data streaming session is in progress.

### Claim 16 (depends on 14)

The data storage medium of claim 14, further comprising displaying a location map and receiving data entries in association with the location map.

### Claim 17 (depends on 14)

The data storage medium of claim 14, further comprising displaying a schematic diagram and receiving data entries in association with the schematic diagram.

### Claim 18 (depends on 14)

The data storage medium of claim 14, further comprising displaying a diagram or illustration of a machine or mechanical part and receiving data entries in association with the a diagram or illustration of the machine or mechanical part.

### Claim 19 (depends on 14)

The data storage medium of claim 14, further comprising displaying a picture of a person, place or feature and receiving data entries in association with the picture of the person, place or feature.

### Claim 20

A handheld portable computing device operative for:
recording a data collection session including a data streaming session;
converting the data streaming session to a number of text entries while the data streaming session is in progress;
receiving the text entries into semi-structured data fields of a form while the data streaming session is in progress;
receiving edits to the text entries in the semi-structured data fields of the form while the data streaming session is in progress; and
storing the data streaming session together with the data form containing the edit text entries converted from the data streaming session.
