# US 2014/0196082 A1 · Comment information generating apparatus and comment information generating method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2014/0196082 A1
- Document ID: US20140196082A1
- Status: Published application
- Kind: A1
- Application number: 14/204,469
- Filed: 2014-03-11
- Published: 2014-07-10
- Earliest claimed: 2012-07-17
- Assignees: Panasonic Corporation
- Applicants: Panasonic Corporation
- Inventors: Ayako MARUYAMA; Kunio NOBORI; Koichi HOTTA; Shuhei SASAKURA
- Web page: https://patentgrep.com/patent/US20140196082A1

## CPC

- H04N 21/8133: {specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme}
- G06F 16/00: Information retrieval; Database structures therefor; File system structures therefor
- G06F 16/7837: {using objects detected or recognised in the video content}
- G06F 16/7867: {using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings}
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06T 11/00: Two-dimensional [2D] image generation
- G09G 5/00: Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators (image data processing or generation, in general G06T)
- G09G 5/38: with means for controlling the display position
- G11B 27/00: Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B 27/034: on discs (G11B27/036, G11B27/038 take precedence)
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/235: Processing of additional data, e.g. scrambling of additional data or processing content descriptors
- H04N 21/2743: Video hosting of uploaded data from client
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/4788: communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)}
- H04N 21/488: Data services, e.g. news ticker {(systems specially adapted for using meteorological information in broadcast systems H04H60/71)}
- H04N 21/4888: {for displaying teletext characters}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- G06T 2207/30241: Trajectory
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

A comment information generating apparatus includes: a video obtaining unit which receives a video; a comment input receiving unit which receives position information of an object in the video and a comment that is displayed with the object and is caused to track the object starting from a predetermined timing; a target time determining unit which determines, to be a target value indicating a time length, a target time during which the comment is displayed, based on the comment; an object motion determining unit which determines, for the object, a motion trajectory having a time length substantially equal to the target time and along which the comment is caused to track the object; a comment information generating unit which generates comment information including the comment and the motion trajectory of the object; and an output unit which outputs the comment information.

## Claims

### Claim 1

A comment information generating apparatus comprising:
a video obtaining unit configured to receive a video;
a comment input receiving unit configured to receive inputs of position information of an object in the video received by the video obtaining unit and a comment that is displayed with the object and is caused to track the object starting from a predetermined timing;
a target time determining unit configured to determine a target time during which the comment is displayed, based on the comment received by the comment input receiving unit, the target time being a target value indicating a time length;
an object motion determining unit configured to determine, for the object shown by the position information, a motion trajectory having a time length that is substantially equal to the target time and along which the comment is displayed with the object and is caused to track the object;
a comment information generating unit configured to generate comment information to be output including the comment and the motion trajectory of the object determined by the object motion determining unit; and
an output unit configured to output the comment information generated by the comment information generating unit.

### Claim 2 (depends on 1)

The comment information generating apparatus according to claim 1,
wherein the target time determining unit is configured to calculate the target time which lengthens as the comment received by the comment input receiving unit lengthens, and
the output unit is configured to output the motion trajectory having the time length which lengthens as the comment input to the comment input receiving unit lengthens.

### Claim 3 (depends on 2)

The comment information generating apparatus according to claim 2,
wherein the target time determining unit is configured to calculate, as the target time, a value obtained by multiplying a unit display time which is a predetermined display time per character and the number of characters in the comment received by the comment input receiving unit, and
the output unit is configured to output the motion trajectory having, as the time length, the value obtained by multiplying the unit display time and the number of characters, when the comment is input to the comment input receiving unit.

### Claim 4 (depends on 2)

The comment information generating apparatus according to claim 2,
wherein the target time determining unit is further configured to calculate, as the target time, a predetermined visual recognition time required to visually recognize the characters, when the calculated target time is shorter than the predetermined visual recognition time, and
the output unit is configured to output, as the motion trajectory, a motion trajectory which has a time length larger than or equal to the visual recognition time even if the comment is shorter than the visual recognition time, by outputting the motion trajectory which corresponds to a longer one of the visual recognition time and a value obtained by multiplying a unit display time which is a predetermined display time per character and the number of characters in the comment when the comment is input to the comment input receiving unit.

### Claim 5 (depends on 1)

The comment information generating apparatus according to claim 1,
wherein the output unit is configured to output a plurality of the motion trajectories which are different from each other for a plurality of the comments received by the comment input receiving unit, when the plurality of the comments have different numbers of characters and are provided at a same position in a same frame.

### Claim 6 (depends on 1)

The comment information generating apparatus according to claim 1,
wherein the object motion determining unit is configured to calculate a plurality of the motion trajectories for the object which is in the video and shown by the position information received by the comment input receiving unit, using a plurality of motion estimation methods or motion estimation parameters, and determine, for the object, the motion trajectory which is selected as having a time length closest to the target time from among the plurality of the motion trajectories calculated for the object.

### Claim 7 (depends on 6)

The comment information generating apparatus according to claim 6,
wherein the object motion determining unit is configured to calculate a plurality of the motion trajectories for the object which is in the video and shown by the position information received by the comment input receiving unit, using, as the plurality of motion estimation parameters, one kind of (i) allowance parameters having different values which affect trackability of the object, (ii) search window regions having different sizes, and (iii) feature values having different values, and determine, for the object, the motion trajectory which is selected as having a time length closest to the target time from among the plurality of the motion trajectories calculated for the object.

### Claim 8 (depends on 6)

The comment information generating apparatus according to claim 6,
wherein, in a case of occurrence of a “motion estimation impossible state” in which the motion trajectory closest to the target time cannot be determined for the object using the plurality of motion estimation methods or motion estimation parameters, the object motion determining unit is further configured to determine whether the “motion estimation impossible state” is caused by occlusion or a scene change, and switch between object motion determining methods based on a result of the determination,

### Claim 9 (depends on 8)

The comment information generating apparatus according to claim 8,
wherein when determining that the “motion estimation impossible state” is caused by the occlusion, the object motion determining unit is configured to determine the motion trajectory having the time length closest to the target time for the object by extrapolating a motion trajectory of the object from a frame at which the occlusion occurred to a following frame, based on a motion trajectory of the object to the frame at which the occlusion occurred.

### Claim 10 (depends on 8)

The comment information generating apparatus according to claim 8,
wherein when determining that the “motion estimation impossible state” is caused by the scene change, the object motion determining unit is configured to determine, for the object, a motion trajectory to a frame at which the scene change occurred as the motion trajectory to be output.

### Claim 11 (depends on 8)

The comment information generating apparatus according to claim 8,
wherein the object motion determining unit is configured to determine that the “motion estimation impossible state” is caused by the scene change, when a difference between luminance histograms of frames composing the video is larger than or equal to a predetermined threshold value, and that the “motion estimation impossible state” is caused by the occlusion, when a difference between the luminance histograms of the frames composing the video is smaller than the predetermined threshold value.

### Claim 12 (depends on 6)

The comment information generating apparatus according to claim 6,
wherein when a motion trajectory among the plurality of the motion trajectories of the object calculated using the plurality of motion estimation methods or motion estimation parameters is shorter than the time length corresponding to the target time by a predetermined time or more, the object motion determining unit is configured to determine, as the motion trajectory, a motion trajectory which is closest to the target time by connecting, to a forward end of a motion trajectory of the object, a motion trajectory which is estimated between (i) the frame having the position information and the input comment received by the comment input receiving unit and including a position which is shown by the position information received by the comment input receiving unit and (ii) a frame positioned backward on a time axis.

### Claim 13 (depends on 6)

The comment information generating apparatus according to claim 6,
wherein when a motion trajectory among the plurality of the motion trajectories of the object calculated using the plurality of motion estimation methods or motion estimation parameters is shorter than the time length corresponding to the target time by a predetermined time or more, the object motion determining unit is configured to determine a motion trajectory having a time length closest to the target time as the motion trajectory of the object shown by the position information received by the comment input receiving unit from among the plurality of the motion trajectories calculated for the object starting at positions within a predetermined distance range from positions shown by the position information of the object received by the comment input receiving unit.

### Claim 14 (depends on 6)

The comment information generating apparatus according to claim 6,
wherein when a motion trajectory among the plurality of the motion on trajectories of the object calculated using the plurality of motion estimation methods or motion estimation parameters is shorter than the time length corresponding to the target time by a predetermined time or more, the object motion determining unit is configured to determine a motion trajectory having a time length closest to the target time as the motion trajectory of the object shown by the position information received by the comment input receiving unit from among the plurality of the motion trajectories calculated for the object based on positions within a user-specified range including positions shown by the position information of the object received by the comment input receiving unit.

### Claim 15 (depends on 6)

The comment information generating apparatus according to claim 6,
wherein when a motion trajectory among the plurality of the motion trajectories of the object calculated using the plurality of motion estimation methods or motion estimation parameters is shorter than the time length corresponding to the target time by a predetermined time or more, the object motion determining unit is configured to divide the object into regions, and determine a motion trajectory having a time length closest to the target time as the motion trajectory of the object, the determined motion trajectory ending in one of the regions.

### Claim 16 (depends on 1)

The comment information generating apparatus according to claim 1,
wherein the object motion determining unit is further configured to determine, as the motion trajectory, a motion trajectory having a time length closest to the target time, modify the determined motion trajectory for the object to be a virtual motion trajectory from a position at which a comment received by the comment input receiving unit is provided, based on a relative position relationship between the position at which the comment is provided and a center of gravity of the object shown by the position information received by the comment input receiving unit, and output the modified motion trajectory.

### Claim 17 (depends on 1)

The comment information generating apparatus according to claim 1, further comprising
a target time adjusting unit configured to adjust the target time by lengthening the target time with an increase in a motion velocity of the object, based on the motion trajectory of the object determined by the object motion determining unit,
wherein the object motion determining unit is further configured to determine, for the object, a new motion trajectory having a time length corresponding to the target time adjusted by the target time adjusting unit, the new motion trajectory being for displaying the comment with the object shown by the position information and causing the comment to track the object.

### Claim 18

A comment information generating method comprising,
receiving a video as an input;
receiving inputs of position information of an object in the video received in the receiving and a comment that is displayed with the object and is caused to track the object;
determining a target time during which the comment is displayed, based on the comment received in the receiving, the target time being a target value indicating a time length;
determining, for the object shown by the position information, a motion trajectory having the time length that is substantially equal to the target time and along which the comment is displayed with the object and is caused to track the object;
generating comment information to be output including the comment and the motion trajectory of the object determined in the determining of the motion trajectory; and
outputting the comment information generated in the generating.

### Claim 19 (depends on 18)

A non-transitory computer-readable recording medium storing thereon a program causing a computer to execute the comment information generating method according to claim 18.