# US 2015/0106828 A1 · Method and apparatus for identifying point of interest and inserting accompanying information into a multimedia signal

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0106828 A1
- Document ID: US20150106828A1
- Status: Published application
- Kind: A1
- Application number: 14/055,021
- Filed: 2013-10-16
- Published: 2015-04-16
- Earliest claimed: 2013-10-16
- Inventors: Gal Gubesi; Elran Ahronee; Gal Sont
- Web page: https://patentgrep.com/patent/US20150106828A1

## CPC

- H04N 21/23424: {involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement}
- G06Q 30/0269: {based on user profile or attribute}
- H04H 60/33: Arrangements for monitoring the users' behaviour or opinions
- H04H 60/40: for identifying broadcast time
- H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/26241: {involving the time of distribution, e.g. the best time of the day for inserting an advertisement or airing a children programme}
- H04N 21/439: Processing of audio elementary streams
- H04N 21/44218: {Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme (methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45)}
- H04N 21/458: Scheduling content for creating a personalised stream, e.g. by combining a locally stored advertisement with an incoming stream; Updating operations, e.g. for OS modules {; time-related management operations (arrangements for replacing or switching information during the broadcast or during the distribution H04H20/10)}
- H04N 21/4725: using interactive regions of the image, e.g. hot spots {(details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94)}
- H04N 21/4728: for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
- H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)}
- H04N 21/8583: {by creating hot-spots}
- H04H 20/106: {Receiver-side switching}
- H04H 60/37: for identifying segments of broadcast information, e.g. scenes or extracting programme ID
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N)

## Abstract

An apparatus arranged to insert accompanying information into a multimedia signal, the apparatus constituted of: a multimedia receiver; a point of interest (POI) identifier arranged to identify a POI in the received multimedia signal responsive to content of the received multimedia signal; an inserter arranged to insert accompanying information into the received multimedia signal responsive to an output of the POI identifier; an output device arranged to transmit the output of the inserter to a display device, the identification, insertion and output being contemporaneous with the receiving; a server; a user gesture signal receiver arranged to receive a signal responsive to a user gesture; and a communication device, wherein the inserted accompanying information encourages a user to perform a gesture with a device in communication with the user gesture signal receiver, and wherein the communication device is arranged to transmit the received signal responsive to the user gesture.

## Claims

### Claim 1

An apparatus arranged to insert accompanying information into a multimedia signal, the apparatus comprising:
a multimedia receiver arranged to receive the multimedia signal;
a point of interest identifier in communication with the received multimedia signal and arranged to identify a point of interest in the received multimedia signal responsive to content of the received multimedia signal;
an inserter arranged to insert accompanying information into the received multimedia signal responsive to an output of said point of interest identifier;
an output device arranged to transmit the output of said inserter to a display device, said identification, insertion and output being contemporaneous with said receiving;
a server;
a user gesture signal receiver arranged to receive a signal responsive to a user gesture; and
a communication device in communication with said server and said user gesture signal receiver,
wherein said inserted accompanying information encourages a user to perform a gesture with at least one device in communication with said user gesture signal receiver, and wherein said communication device is arranged to transmit said received signal responsive to the user gesture to said server.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein said insertion of said accompanying information by said inserter replaces the received multimedia signal.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein said insertion of said accompanying information by said inserter results in a combined signal of said accompanying information and the received multimedia signal output to the display device by said output device.

### Claim 4 (depends on 1)

The apparatus of claim 1, further comprising a communication device arranged to receive information regarding said point of interest, said point of interest identifier responsive to said received information.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein said inserter is further arranged to alternately pass said received multimedia signal.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the content of the received multimedia signal comprises a flag associated with the point of interest, and wherein said point of interest identifier is arranged to identify the flag, thereby identifying the point of interest in the received multimedia signal.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein said point of interest identifier is responsive to subtitle information accompanying the received multimedia signal.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein said point of interest identifier comprises an audio converter responsive to an audio portion of the received multimedia signal, said audio converter arranged to output a digital signal indicative of at least one identified audio portion, wherein said identification of the point of interest in the received multimedia signal is responsive to the identified audio portion.

### Claim 9 (depends on 1)

The apparatus of claim 1, wherein said point of interest identifier is further arranged to identify an image of the received multimedia signal, wherein said identification of the point of interest in the received multimedia signal is responsive to said identified image.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein said identified point of interest in the received multimedia signal represents a predetermined temporal point in the received multimedia signal.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein said identified point of interest in the received multimedia signal represents a predetermined point in time.

### Claim 12 (depends on 1)

The apparatus of claim 1, wherein said server comprises an indexed plurality of accompanying information.

### Claim 13 (depends on 1)

The apparatus of claim 1, further comprising a database in communication with said server, said database comprising:
an identifier associated with each point of interest;
an identifier of the particular multimedia signal content associated with each point of interest; and
a classifier of each point of interest.

### Claim 14 (depends on 1)

The apparatus of claim 1, wherein said communication device is further arranged to perform communication between said server and said point of interest identifier, wherein said multimedia receiver, said point of interest identifier, said inserter, said output device and said communication device are collocated with the display device, and said server is remote from said display device, wherein said communication device is arranged to receive information from said server regarding said point of interest, said point of interest identifier responsive to said received information.

### Claim 15 (depends on 14)

The apparatus according to claim 14, further comprising a user device in communication with said user gesture signal receiver, wherein said user device is arranged responsive to a user gesture to transmit the signal responsive to the user gesture to be received by said user gesture signal receiver.

### Claim 16 (depends on 14)

The apparatus according to claim 14, further comprising:
a program identifier arranged to obtain identifying information regarding said received multimedia signal; and
an information retriever in communication with said program identifier, said server via said communication device, and said inserter, said information retriever arranged to retrieve from said server via said communication device information associated with the obtained identifying information,
wherein said inserter is further arranged to insert the retrieved information into the received multimedia signal.

### Claim 17 (depends on 11)

The apparatus according to claim 11, further comprising a mobile device in communication with the user gesture signal receiver, wherein said mobile device is arranged responsive to a user gesture to communicate with the server responsive to the accompanying information.

### Claim 18

A method of inserting accompanying information into a multimedia signal, the method comprising:
receiving the multimedia signal;
identifying a point of interest in the received multimedia signal responsive to content of the received multimedia signal;
inserting accompanying information into said received multimedia signal responsive to said identified point of interest;
transmitting said received multimedia signal comprising said inserted accompanying information to a display device, said identifying, inserting and transmitting being contemporaneous with said receiving;
providing a server;
providing a user gesture signal receiver; and
providing a communication device in communication with said provided server and said provided user gesture signal receiver,
wherein said accompanying information encourages a user to perform a gesture with a user device in communication with said provided user gesture signal receiver, said user device transmitting a signal to said user gesture signal receiver indicative of said user gesture, said user gesture signal receiver transmitting information to said server via said provided communication device regarding said transmitted signal.

### Claim 19 (depends on 18)

The method of claim 18, wherein said insertion of said accompanying information replaces said received multimedia signal.

### Claim 20 (depends on 18)

The method of claim 18, wherein said insertion of said accompanying information results in a combined signal of said accompanying information and said received multimedia signal.

### Claim 21 (depends on 18)

The method of claim 18, further comprising receiving information regarding the point of interest, said identifying the point of interest responsive to said received information.

### Claim 22 (depends on 18)

The method of claim 18, further comprising, in the event that no point of interest is identified, passing said received multimedia signal.

### Claim 23 (depends on 18)

The method of claim 18, wherein the content of the received multimedia signal comprises a flag associated with the point of interest, the method further comprising identifying the flag, thereby identifying the point of interest in said received multimedia signal.

### Claim 24 (depends on 18)

The method of claim 18, wherein said identifying the point of interest is responsive to subtitle information accompanying said received multimedia signal.

### Claim 25 (depends on 18)

The method of claim 18, further comprising:
identifying an audio portion of said received multimedia signal; and
outputting a digital signal indicative of said identified at least one identified audio portion,
wherein said identifying the point of interest is responsive to the identified audio portion.

### Claim 26 (depends on 18)

The method of claim 18, further comprising identifying an image of the received multimedia signal,
wherein said identifying the point of interest is responsive to said identified image.

### Claim 27 (depends on 18)

The method of claim 18, wherein said identified point of interest in said received multimedia signal represents a predetermined temporal point in said received multimedia signal.

### Claim 28 (depends on 18)

The method of claim 18, wherein said identified point of interest in said received multimedia signal represents a predetermined point in time.

### Claim 29 (depends on 18)

The method of claim 18, wherein said provided server comprises a plurality of indexed accompanying information.

### Claim 30 (depends on 18)

The method of claim 18, further comprising providing a database in communication with said provided server, said provided database comprising:
an identifier associated with each point of interest;
an identifier of the particular multimedia signal content associated with each point of interest; and
a classifier of each point of interest.

### Claim 31 (depends on 30)

The method of claim 30, further comprising:
obtaining identifying information regarding said received multimedia signal;
retrieving from said provided server information associated with said obtained identifying information; and
inserting said retrieved information into said received multimedia signal.

### Claim 32 (depends on 31)

The method of claim 31, further comprising:
providing a user device in communication with said provided server;
receiving a user gesture at a mobile device; and
responsive to said received user gesture, transmitting a message responsive to said inserted accompanying information to said provided server.

### Claim 33 (depends on 18)

The method of claim 18, further comprising:
obtaining identifying information regarding said received multimedia signal; and
retrieving from said provided server via said provided communication device information associated with the obtained identifying information; and
inserting the retrieved information into the received multimedia signal.
