# US 10,313,536 B2 · Information processing system, electronic apparatus, information processing apparatus, information processing method, electronic apparatus processing method and non-transitory computer readable medium to confirm difference in meeting contents using sound information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,313,536 B2
- Document ID: US10313536B2
- Status: Granted
- Kind: B2
- Application number: 15/613,932
- Filed: 2017-06-05
- Published: 2017-12-14
- Granted: 2019-06-04
- Earliest claimed: 2016-06-10
- Assignees: Konica Minolta, Inc.
- Applicants: Konica Minolta, Inc.
- Inventors: Yoshifumi Wagatsuma
- Web page: https://patentgrep.com/patent/US10313536B2

## CPC

- H04N 1/00188: {Printing, e.g. prints or reprints (H04N1/0019, H04N1/00196 take precedence)}
- B41J 3/44: Typewriters or selective printing mechanisms having dual functions or combined with, or coupled to, apparatus performing other functions
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/0485: Scrolling or panning
- G06F 3/1285: {Remote printer device, e.g. being remote from client or server}
- G06F 3/1292: {Mobile client, e.g. wireless printing}
- G11B 27/031: Electronic editing of digitised analogue information signals, e.g. audio or video signals
- G11B 27/102: {Programmed access in sequence to addressed parts of tracks of operating record carriers (access by moving the head G11B3/08, G11B5/54, G11B7/085, G11B21/022; by moving the record carrier G11B15/005, G11B17/005, by driving of both record carrier and head G11B15/1816)}
- G11B 27/34: Indicating arrangements {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)}
- H04N 1/00204: {with a digital computer or a digital computer system, e.g. an internet server (programmed control between transmitter and receiver or between image input and image output device H04N1/32561)}
- H04N 1/00267: {with a viewing or projecting apparatus, e.g. for reading image information from a film (H04N1/00262 takes precedence)}
- H04N 1/00307: {with a mobile telephone apparatus (constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00)}
- H04N 1/32112: {in a separate computer file, document page or paper sheet, e.g. a fax cover sheet (H04N1/32122 takes precedence)}
- H04N 1/32133: {on the same paper sheet, e.g. a facsimile page header}
- H04N 5/765: Interface circuits between an apparatus for recording and another apparatus
- H04N 2201/0036: Detecting or checking connection
- H04N 2201/0039: Connection via a network
- H04N 2201/0055: By radio
- H04N 2201/0094: Multifunctional device, i.e. a device capable of all of reading, reproducing, copying, facsimile transception, file transception
- H04N 2201/0096: Portable devices
- H04N 2201/3205: of identification information, e.g. name or ID code (H04N2201/3209 takes precedence)
- H04N 2201/3214: of a date
- H04N 2201/3215: of a time or duration
- H04N 2201/3226: of identification information or the like, e.g. ID code, index, title, part of an image, reduced-size image
- H04N 2201/3245: of image modifying data, e.g. handwritten addenda, highlights or augmented reality information
- H04N 2201/3253: Position information, e.g. geographical position at time of capture, GPS data
- H04N 2201/3264: of sound signals
- H04N 2201/3266: of text or character information, e.g. text accompanying an image (H04N2201/3226 takes precedence)
- H04N 2201/3269: of machine readable codes or marks, e.g. bar codes or glyphs
- H04N 2201/3271: Printing or stamping
- H04N 2201/3273: Display
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- B41J: TYPEWRITERS; SELECTIVE PRINTING MECHANISMS, i.e. MECHANISMS PRINTING OTHERWISE THAN FROM A FORME; CORRECTION OF TYPOGRAPHICAL ERRORS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

An information processing system that that confirms the meanings of difference information in meeting content with reference to the sound information is provided. Differences between meeting content contained in image information of a paper medium taken at each detection cycle and meeting content contained in master information are extracted. Surrounding sounds are recorded while the meeting is being held. The sound information of the recorded sounds is associated with the difference information of the extracted differences and also associated with the master information with the timing when the differences of the meeting content are detected.

## Claims

### Claim 1

An information processing system comprising:
an electronic imaging apparatus; and
a multi-function peripheral (MFP) apparatus that forms a print image on a paper medium and exchanges information with the electronic imaging apparatus, wherein
the electronic imaging apparatus comprises:
an imaging unit that takes image information of the paper medium during a repeating detection cycle;
an extracting unit that extracts a difference between a first meeting content contained in the taken image information and a second meeting content contained in master information saved in the MFP apparatus, wherein the second meeting content is same as the first meeting content at the time when the imaging unit starts taking the image information;
a sound recording unit that records a surrounding sound while a meeting is being held with respect to the second meeting content; and
a terminal side association processing unit that, in correspondence with the extraction of the difference, associates difference information of the difference with sound information of the recorded surrounding sound and with the timing of when the difference is detected,
the MFP apparatus forms the print image on the paper medium based on the second meeting content, and
the MFP apparatus comprises:
a main body side association processing unit that associates the master information with pair information that includes the difference information and the sound information, which are associated with each other by the terminal side association processing unit.

### Claim 2 (depends on 1)

The information processing system of claim 1 wherein the MFP apparatus further comprises:
a storage unit that stores the master information and attribute information of the master information;
an identification processing unit that generates identification image information from the attribute information stored in the storing unit; and
an image processing unit that forms the print image on the paper medium based on meeting material image information generated from the second meeting content contained in the master information and the identification image information generated by the identification processing unit, wherein
the image processing unit forms, as the print image, a meeting material image based on the meeting material image information and an identification image based on the identification image information, and
the identification image information includes storage destination information of the master information, which is an item of the attribute information.

### Claim 3 (depends on 2)

The information processing system of claim 2 wherein the MFP apparatus further comprises:
an image control unit that controls an interface image that is output with any one of the master information, the attribute information, the difference information, and the sound information, wherein
the interface image serves as an interface to output the sound information based on the difference information, and
the attribute information includes identification information for identifying the meeting content contained in the master information, host information about a host of the meeting, or output person information about an output person who commands formation of the print image on the paper medium in addition to the storage destination information of the master information.

### Claim 4 (depends on 3)

The information processing system of claim 3 wherein the electronic imaging apparatus further comprises:
an event management unit that manages the differences extracted by the extracting unit in units of events, wherein
the extracting unit extracts the sound information in correspondence with the event managed by the event management unit,
the event serves as an indicator showing a change in the taken image information in the progress of the meeting, starts when the difference is extracted, and terminates if no difference is extracted anew for a temporal waiting period, and
the terminal side association processing unit associates the sound information with the difference information for each event.

### Claim 5 (depends on 4)

The information processing system of claim 4 wherein
the terminal side association processing unit has a period for clipping the sound information, and
the period comprises a preceding period before starting the event and a succeeding period after terminating the event.

### Claim 6 (depends on 5)

The information processing system of claim 5 wherein
if a predetermined waiting time is not exceeded by an extraction interval at which the differences are extracted, the extracting unit extracts the differences based on the same master information.

### Claim 7 (depends on 6)

The information processing system of claim 6 wherein
if the position of the electronic imaging apparatus does not exceed a set movement range, the extracting unit extracts the difference based on the same master information.

### Claim 8 (depends on 7)

The information processing system of claim 7 wherein
the detection cycle, the temporal waiting period or the sound information clipping period can be changed.

### Claim 9 (depends on 8)

The information processing system of claim 8 wherein
the predetermined waiting time or the set movement range can be changed.

### Claim 10 (depends on 9)

The information processing system of claim 9 wherein
if the predetermined waiting time is not exceeded by the extraction interval, or if the position of the electronic imaging apparatus does not exceed the set movement range, the detection cycle, the temporal waiting period, the sound information clipping period, the predetermined waiting time or the set movement range can dynamically be changed within a change allowable range in accordance with the number of sheets contained in the paper medium on which the print image is formed.

### Claim 11 (depends on 10)

The information processing system of claim 10 wherein
if the difference disappears while the meeting is being held, the terminal side association processing unit dissolves the association between the difference information and the sound information.

### Claim 12 (depends on 11)

The information processing system of claim 11 wherein
if the predetermined waiting time is not exceeded by the extraction interval at which differences are extracted, or if the set movement range is not exceeded by the position of the electronic imaging apparatus and differences are extracted, the terminal side association processing unit sets the preceding period before starting the event to an extended preceding period.

### Claim 13 (depends on 12)

The information processing system of claim 12 further comprising:
a speech recognition unit that recognizes a speech, wherein
if a silent state is continued after the event occurs, the terminal side association processing unit starts extracting the sound information with the timing when the speech recognition unit recognizes a speech.

### Claim 14 (depends on 13)

The information processing system of claim 13 further comprising:
a character recognition unit that recognizes characters, wherein
when the character recognition unit detects start of writing characters to the paper medium, and the speech recognition unit detects a speech to be recognized, the terminal side association processing unit associates the sound information and characters written to the paper medium with each other in preference to the timing with which a difference is extracted in the detection cycle.

### Claim 15 (depends on 14)

The information processing system of claim 14 wherein
the pair information is associated with the master information for each output person information.

### Claim 16 (depends on 3)

The information processing system of claim 3 wherein
the interface image includes an image showing the progress state of the meeting, and
the image control unit associates the image showing the progress state of the meeting with the sound information and the difference information.

### Claim 17 (depends on 1)

The information processing system of claim 1 wherein
the electronic imaging apparatus further comprises a communication control unit that acquires the master information, which is used to extract differences with the extracting unit, based on the storage destination information of the master information,
the MFP apparatus further comprises an information determination unit that determines whether or not the pair information is transmitted,
when the communication control unit does not establish communication with the MFP apparatus, pair information is saved and when the communication is established, the communication control unit transmits the pair information to the MFP apparatus, and
when the information determination unit determines that the pair information is transmitted, the main body side association processing unit associates the pair information with the master information.

### Claim 18

An electronic imaging apparatus that exchanges information with a multi-functional peripheral (MFP) apparatus that forms a print image on a paper medium, comprising:
an imaging unit that takes image information of the paper medium during a repeating detection cycle;
an extracting unit that extracts a difference between a first meeting content contained in the taken image information and a second meeting content contained in master information saved in the MFP apparatus, wherein the second meeting content is same as the first meeting content at the time when the imaging unit starts taking the image information;
a sound recording unit that records a surrounding sound while a meeting is being held with respect to the second meeting content;
a terminal side association processing unit that, in correspondence with the extraction of the difference, associates difference information of the difference with sound information of the recorded surrounding sound and with the timing of when the difference is detected; and
a communication unit that transmits, to the MFP apparatus, pair information including the difference information and the sound information that are associated with each other by the terminal side association processing unit, and a command to associate the pair information with the master information, wherein
the MFP apparatus forms the print image on the paper medium based on the second meeting content.

### Claim 19 (depends on 18)

The electronic imaging apparatus of claim 18 wherein
the terminal side association processing unit associates the sound information with the difference information for each event, and
the event serves as an indicator showing a change in the taken image information in the progress of a meeting, starts when a difference is extracted, and terminates if no difference is extracted anew for a temporal waiting period.

### Claim 20

A multi-functional peripheral (MFP) apparatus that exchanges information with an electronic imaging apparatus, the MFP apparatus comprising:
a storage unit that stores master information and attribute information of the master information;
an identification processing unit that generates identification image information from the attribute information;
an image processing unit that forms a print image on a paper medium based on meeting material image information generated from a second meeting content contained in the master information and the identification image information; and
a main body side association processing unit that associates the master information and pair information with each other, wherein
the pair information includes:
sound information of recorded surrounding sounds; and
difference information of a difference between:
a first meeting content contained in taken image information of the paper medium, and
the second meeting content contained in the master information.

### Claim 21

An information processing method performed by an information processing system that includes an electronic imaging apparatus and a multi-functional peripheral (MFP) apparatus that forms a print image on a paper medium and exchanges information with the electronic imaging apparatus, the method comprising:
in the electronic imaging apparatus side:
taking image information of the paper medium during a repeating detection cycle by an imaging unit;
extracting a difference between a first meeting content contained in the taken image information and a second meeting content contained in master information saved in the MFP apparatus, wherein the second meeting content is same as the first meeting content at the time when the imaging unit starts taking the image information;
recording a surrounding sound while a meeting is being held with respect to the second meeting content; and
in correspondence with the extraction of the difference, associating difference information of the difference with sound information of the recorded surrounding sound and with the timing of when the difference is detected,
the MFP apparatus forms the print image on the paper medium based on the second meeting content, and
in the MFP apparatus side:
associating the master information with pair information including the difference information and the sound information that are associated with each other.

### Claim 22

An electronic imaging apparatus processing method performed by an electronic imaging apparatus that exchanges information with a multi-functional peripheral (MFP) apparatus that forms a print image on a paper medium, the method comprising:
taking image information of the paper medium during a repeating detection cycle by an imaging unit;
extracting a difference between a first meeting content contained in the taken image information and a second meeting content contained in master information saved in the MFP apparatus, wherein the second meeting content is same as the first meeting content at the time when the imaging unit starts taking the image information;
recording a surrounding sound while a meeting is being held with respect to the second meeting content;
in correspondence with the extraction of the difference, associating difference information of the difference with sound information of the recorded surrounding sound and with the timing of when the difference is detected; and
transmitting, to the MFP apparatus, pair information including the difference information and the sound information that are associated with each other, and a command to associate the pair information with the master information, wherein
the MFP apparatus forms the print image on the paper medium based on the second meeting content.

### Claim 23

A non-transitory computer readable medium including programmed instructions, wherein the instructions, when executed by a computer, cause the computer to control an electronic imaging apparatus that exchanges information with a multi-functional peripheral (MFP) apparatus that forms a print image on a paper medium, and causes the computer to function as:
an imaging unit that takes image information of the paper medium during a repeating detection cycle;
an extracting unit that extracts a difference between a first meeting content contained in the taken image information and a second meeting content contained in master information saved in the MFP apparatus, wherein the second meeting content is same as the first meeting content at the time when the imaging unit starts taking the image information;
a sound recording unit that records a surrounding sound while a meeting is being held with respect to the second meeting content;
a terminal side association processing unit that, in correspondence with the extraction of the difference, associates difference information of the difference with sound information of the recorded surrounding sound and with the timing of when the difference is detected; and
a communication unit that transmits, to the MFP apparatus, pair information including the difference information and the sound information that are associated with each other, and a command to associate the pair information with the master information, wherein
the MFP apparatus forms the print image on the paper medium based on the second meeting content.

### Claim 24

An information processing method performed by a multi-functional peripheral (MFP) apparatus that exchanges information with an electronic imaging apparatus, the method comprising:
storing master information and attribute information of the master information;
generating identification image information from the attribute information; and
forming a print image on a paper medium based on meeting material image information generated from a second meeting content contained in the master information and the identification image information; and
associating the master information and pair information with each other, wherein
the pair information includes:
sound information of recorded surrounding sounds; and
difference information of a difference between:
a first meeting content contained in taken image information of the paper medium, and
the second meeting content.

### Claim 25

A non-transitory computer readable medium including programmed instructions, wherein the instructions, when executed by a computer, cause the computer to control a multi-functional peripheral (MFP) apparatus that exchanges information with an electronic imaging apparatus, and causes the computer to function as:
a storage unit that stores master information and attribute information of the master information;
an identification processing unit that generates identification image information from the attribute information;
an image processing unit that forms a print image on a paper medium based on meeting material image information generated from a second meeting content contained in the master information and the identification image information; and
a main body side association processing unit that associates the master information and pair information with each other, wherein
the pair information includes:
sound information of recorded surrounding sounds; and
difference information of a difference between:
a first meeting content contained in taken image information of the paper medium, and
a second meeting content.