# US 11,451,858 B2 · Method and system of processing information flow and method of displaying comment information

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,451,858 B2
- Document ID: US11451858B2
- Status: Granted
- Kind: B2
- Application number: 17/018,137
- Filed: 2020-09-11
- Published: 2021-03-18
- Granted: 2022-09-20
- Earliest claimed: 2019-09-12
- Assignees: Shanghai Bilibili Technology Co., Ltd.
- Applicants: Shanghai Bilibili Technology Co., Ltd.
- Inventors: Zhiwei Chen; Longwen Gao; Yingxuan Shi; Chaoran Li; Qi Tang; Jianqiang Ding; Bo Wang; Tianxiao Ye; Xiaolong Yan; Dada Liao; Xiang Zhu
- Web page: https://patentgrep.com/patent/US11451858B2

## CPC

- H04N 21/2743: Video hosting of uploaded data from client
- G06T 7/11: Region-based segmentation
- G06T 11/60: Creating or editing images; Combining images with text
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 21/2187: Live feed
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/4302: {Content synchronisation processes, e.g. decoder synchronisation}
- H04N 21/43072: {of multiple content streams on the same device}
- H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window}
- H04N 21/4622: {Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet (web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859)}
- H04N 21/4788: communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)}
- H04N 21/8133: {specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme}
- H04N 21/8547: involving timestamps for synchronizing content
- G06T 2207/10016: Video; Image sequence
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure describes techniques of processing information flow and synchronously displaying image s, mask frames and comment information. According to the present disclosure, a system processes a first information stream uploaded by a stream pushing terminal to obtain a frame image and mask frame data, the mask frame data corresponding to the frame image and identifying a timestamp offset indicative of a time difference between the mask frame data and a playing timestamp of the frame image. The system synthesizes the frame image and the mask frame data to generate a second information stream; and output the second information stream to a client for synchronously displaying the image frame, a corresponding mask frame, and the comment information.

## Claims

### Claim 1

A method of processing information flow, comprising:
receiving, from a first client computing device, a first data unit among a plurality of first data units comprised in a first information stream, wherein the first data unit comprises image data and meta-information;
obtaining an image and a first timestamp corresponding to the image by decoding the first data unit;
identifying a main area of the image, wherein the main area of the image comprises a person area, an animal area, a landscape area, an architectural area, an artwork area, and a text area in the image;
generating mask frame data based at least in part on the identified main area of the image, wherein the mask frame data comprise a timestamp offset associated with the first timestamp; and
generating a second information stream by synthesizing the image and the mask frame data corresponding to the image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining the timestamp offset of the mask frame data based on the first timestamp and the meta-information.

### Claim 3 (depends on 1)

The method of claim 1, wherein the meta-information comprises a second timestamp, the second timestamp indicating a time of the image to be played on a second client computing device.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
encoding the image and the mask frame data corresponding to the image to obtain second data information; and
generating a second data unit combining the second data information with the meta-information.

### Claim 5 (depends on 4)

The method of claim 4, further comprising:
combining a plurality of second data units to generate the second information stream; and
sending the second information stream to a second client computing device.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
identifying the main area of the image using an image segmentation algorithm.

### Claim 7

A method of synchronously displaying images frames, mask frames and comment information, comprising:
receiving a second data unit a second information stream and comment information, wherein the second information stream is generated by synthesizing at least one image and mask frame data corresponding to the at least one image, the second information stream comprises a plurality of second data units, and the second data unit comprises meta-information and second data information;
obtaining an image and mask frame data corresponding to the image by decoding the second data information, wherein the mask frame data comprise a mask frame generated based on a main area of the image, the mask frame data further comprise a timestamp offset indicative of a time difference between a timestamp of the mask frame and a playing timestamp of the image, and the main area of the image comprises a person area, an animal area, a landscape area, an architectural area, an artwork area, and a text area in the image; and
synchronously displaying the image, the mask frame corresponding to the image, and the comment information based on the meta-information and the timestamp offset.

### Claim 8 (depends on 7)

The method of claim 7, wherein the meta-information comprises a timestamp, wherein the timestamp indicates a time of the image to be displayed on a client computing device.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
displaying the comment information in areas other than an area where the mask frame is displayed.

### Claim 10 (depends on 9)

The method of claim 9, wherein the mask frame is a transparent channel.

### Claim 11

A system of processing information flow, comprising:
at least one processor; and
at least one memory communicatively coupled to the at least one processor and storing instructions that upon execution by the at least one processor cause the system to:
receive, from a first client computing device, a first data unit among a plurality of first data units comprised in a first information stream, wherein the first data unit comprises image data and meta-information;
obtain an image and a first timestamp corresponding to the image by decoding the first data unit;
identify a main area the image, wherein the main area of the image comprises a person area, an animal area, a landscape area, an architectural area, an artwork area, and a text area in the image;
generate mask frame data based at least in part on the identified main area of the image, wherein the mask frame data comprise a timestamp offset associated with the first timestamp; and
generate a second information stream by synthesizing the image and the mask frame data corresponding to the image.

### Claim 12 (depends on 11)

The system of claim 11, the at least one memory further storing instructions that upon execution by the at least one processor cause the system to:
determine the timestamp offset of the mask frame data based on the first timestamp and the meta-information.

### Claim 13 (depends on 11)

The system of claim 11, wherein the meta-information comprises a second timestamp, the second timestamp indicating a time of the image to be played on a second client computing device.

### Claim 14 (depends on 11)

The system of claim 11, the at least one memory further storing instructions that upon execution by the at least one processor cause the system to:
encode the image and the mask frame data corresponding to the image to obtain second data information; and
generate a second data unit by combining the second data information with the meta-information.

### Claim 15 (depends on 14)

The system of claim 14, the at one memory further storing instructions that upon execution by the at least one processor cause the system to:
combine a plurality of second data units to generate the second information stream; and
send the second information stream to a second client computing device.

### Claim 16 (depends on 11)

The system of claim 11, the at least one memory further storing instructions that upon execution by the at least one processor cause the system to:
identify the main area of the image using an image segmentation algorithm.

### Claim 17

A system of synchronously displaying images frames, corresponding mask frames and comment information, comprising:
at least one processor; and
at one memory communicatively coupled to the at least one processor and storing instructions that upon execution by the at least one processor cause the system to:
receive a second data unit in a second information stream and comment information, wherein the second information stream is generated by synthesizing at least one image and mask frame data corresponding to the at least one image, the second information stream comprises a plurality of second data units, and the second data unit comprises meta-information and second data information;
obtain an image and mask frame data corresponding to the image by decoding the second data information, wherein the mask frame data comprise a mask frame generated based on a main area of the image, the mask frame data further comprise a timestamp offset indicative of a time difference between a timestamp of the mask frame and a playing timestamp of the image, and the main area of the image comprises a person area, an animal area, a landscape area, an architectural area, an artwork area, and a text area in the image; and
synchronously displaying the image, the mask frame corresponding to the image, and the comment information based on the meta-information and the timestamp offset.

### Claim 18 (depends on 17)

The system of claim 17, wherein the meta-information comprises a timestamp, wherein the timestamp indicates a time of the image to be displayed on a client computing device.

### Claim 19 (depends on 17)

The system claim 17, the at least one memory further storing instructions that upon execution by the at one processor cause the system to:
display the comment information in areas other than an area where the mask frame displayed.

### Claim 20 (depends on 19)

The system of claim 19, wherein the mask frame is a transparent channel.