# US 11,442,606 B2 · User interface interaction method and system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,606 B2 - Document ID: US11442606B2 - Status: Granted - Kind: B2 - Application number: 17/180,026 - Filed: 2021-02-19 - Published: 2021-08-26 - Granted: 2022-09-13 - Earliest claimed: 2020-02-25 - Assignees: Shanghai Bilibili Technology Co., Ltd. - Applicants: Shanghai Bilibili Technology Co., Ltd. - Inventors: Yingxuan Shi; Lulu Yu; Chaoran Li; Zhiwei Chen; Longwen Gao - Web page: https://patentgrep.com/patent/US11442606B2 ## CPC - G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range - G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence) - G06V 20/40: in video content (extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44) - H04N 21/2408: {Monitoring of the upstream path of the transmission network, e.g. client requests (monitoring data switching networks utilization H04L43/0876; scheduling or organising the servicing of application requests H04L67/60)} - H04N 21/2668: Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles {(information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063)} - H04N 21/4312: {involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations} - H04N 21/44222: {Analytics of user selections, e.g. selection of programmes or purchase activity (monitoring of user selections in data processing systems G06F11/34; arrangements for monitoring the user's behaviour or opinions in broadcast systems H04H60/33)} - H04N 21/4756: {for rating content, e.g. scoring a recommended movie} - H04N 21/4788: communicating with other users, e.g. chatting {(arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104)} - H04N 21/4884: {for displaying subtitles} - H04N 21/6582: {Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number (arrangements where receivers interact with the broadcast H04H20/38)} - H04N 21/8583: {by creating hot-spots} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The present disclosure describes techniques for implementing user interface interaction. The disclosed techniques comprise playing a video on an interface; monitoring user input performed on the interface; determining a target frame based at least in part on the user input; determining a location where the user input is performed on the interface; determining whether the location of the user input is in a predetermined area of the target frame, wherein the predetermined area is associated with at least one object in the target frame; and implementing an operation associated with the at least one object in response to determining that the location of the user input is in the predetermined area of the target frame. ## Claims ### Claim 1 A method, comprising: playing a video on an interface, wherein the video comprises at least one frame; monitoring user input performed on the interface; determining a target frame among the at least one frame based at least in part on the user input; determining a location where the user input is performed on the interface; determining whether the location of the user input is in a predetermined area of the target frame, wherein the predetermined area is associated with at least one object in the target frame, wherein the at least one object has different types of object corresponding to different types of operation on the video, and wherein the at least one object comprises different function elements by which a user performs the different types of operation; and implementing a corresponding operation on the video in response to determining that the location of the user input is in the predetermined area associated with the at least one object in the target frame, wherein the corresponding operation is among the different types of operation. ### Claim 2 (depends on 1) The method of claim 1, further comprising: obtaining information associated with each of the at least one frame of the video, wherein the information associated with each frame comprises information indicating a type of at least one object in each frame, location information associated with the at least one object in each frame, and timestamp information associated with each frame. ### Claim 3 (depends on 2) The method of claim 2, wherein information associated with a same type of objects in consecutive frames is stored in a single array when the objects with the same type are respectively located at adjacent positions in the consecutive frames, wherein the single array comprises information indicating the same type of the objects, average location information of the objects in the consecutive frames, timestamp information of a first frame among the consecutive frames, and timestamp information of a last frame among the consecutive frames. ### Claim 4 (depends on 1) The method of claim 1, further comprising: obtaining information associated with the at least one object in the target frame by performing object recognition on the target frame using an object recognition model. ### Claim 5 (depends on 1) The method of claim 1, further comprising: determining the target frame based on a play time of the video and a time when the user input is performed on the interface. ### Claim 6 (depends on 1) The method of claim 1, wherein the predetermined area of the target frame is associated with a location of the at least one object in the target frame. ### Claim 7 (depends on 1) The method of claim 1, further comprising: determining a type of the at least one object in the target frame; and implementing the operation associated with the at least one object based on the type of the at least one object. ### Claim 8 A system, comprising: at least one processor; and at least one memory communicatively coupled to the at least one processor and storing instructions that upon execution by the at least one processor cause the system to perform operations, the operations comprising: playing a video on an interface, wherein the video comprises at least one frame; monitoring user input performed on the interface; determining a target frame among the at least one frame based at least in part on the user input; determining a location where the user input is performed on the interface; determining whether the location of the user input is in a predetermined area of the target frame, wherein the predetermined area is associated with at least one object in the target frame, wherein the at least one object has different types of object corresponding to different types of operation on the video, and wherein the at least one object comprises different function elements by which a user performs the different types of operation; and implementing a corresponding operation on the video n response to determining that the location of the user input is in the predetermined area associated with the at least one object in the target frame, wherein the corresponding operation is among the different types of operation. ### Claim 9 (depends on 8) The system of claim 8, the operations further comprising: obtaining information associated with each of the at least one frame of the video, wherein the information associated with each frame comprises information indicating a type of at least one object in each frame, location information associated with the at least one object in each frame, and timestamp information associated with each frame. ### Claim 10 (depends on 9) The system of claim 9, wherein information associated with a same type of objects in consecutive frames is stored in a single array when the objects with the same type are respectively located at adjacent positions in the consecutive frames, wherein the single array comprises information indicating the same type of the objects, average location information of the objects in the consecutive frames, timestamp information of a first frame among the consecutive frames, and timestamp information of a last frame among the consecutive frames. ### Claim 11 (depends on 8) The system of claim 8, the operations further comprising: obtaining information associated with the at least one object in the target frame by performing object recognition on the target frame using an object recognition model. ### Claim 12 (depends on 8) The system of claim 8, the operations further comprising: determining the target frame based on a play time of the video and a time when the user input is performed on the interface. ### Claim 13 (depends on 8) The system of claim 8, wherein the predetermined area of the target frame is associated with a location of the at least one object in the target frame. ### Claim 14 (depends on 8) The system of claim 8, the operations further comprising: determining a type of the at least one object in the target frame; and implementing the operation associated with the at least one object based on the type of the at least one object. ### Claim 15 A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations comprising: playing a video on an interface, wherein the video comprises at least one frame; monitoring user input performed on the interface; determining a target frame among the at least one frame based at least in part on the user input; determining a location where the user input is performed on the interface; determining whether the location of the user input is in a predetermined area of the target frame, wherein the predetermined area is associated with at least one object in the target frame, wherein the at least one object has different types of object corresponding to different types of operation on the video, and wherein the at least one object comprises different function elements by which a user performs the different types of operation; and implementing a corresponding operation on the video in response to determining that the location of the user input is in the predetermined area associated with the at least one object in the target frame, wherein the corresponding operation is among the different types of operation. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium of claim 15, the operations further comprising: obtaining information associated with each of the at least one frame of the video, wherein the information associated with each frame comprises information indicating a type of at least one object in each frame, location information associated with the at least one object in each frame, and timestamp information associated with each frame. ### Claim 17 (depends on 16) The non-transitory computer-readable storage medium of claim 16, wherein information associated with a same type of objects in consecutive frames is stored in a single array when the objects with the same type are respectively located at adjacent positions in the consecutive frames, wherein the single array comprises information indicating the same type of the objects, average location information of the objects in the consecutive frames, timestamp information of a first frame among the consecutive frames, and timestamp information of a last frame among the consecutive frames. ### Claim 18 (depends on 15) The non-transitory computer-readable storage medium of claim 15, the operations further comprising: obtaining information associated with the at least one object in the target frame by performing object recognition on the target frame using an object recognition model. ### Claim 19 (depends on 15) The non-transitory computer-readable storage medium of claim 15, the operations further comprising: determining the target frame based on a play time of the video and a time when the user input is performed on the interface. ### Claim 20 (depends on 15) The non-transitory computer-readable storage medium of claim 15, the operations further comprising: determining a type of the at least one object in the target frame; and implementing the operation associated with the at least one object based on the type of the at least one object.