# US 11,115,706 B2 · Method, client, and terminal device for screen recording > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,115,706 B2 - Document ID: US11115706B2 - Status: Granted - Kind: B2 - Application number: 16/717,861 - Filed: 2019-12-17 - Published: 2020-12-03 - Granted: 2021-09-07 - Earliest claimed: 2019-04-24 - Assignees: Wangsu Science & Technology Co., Ltd. - Applicants: Wangsu Science & Technology Co., Ltd. - Inventors: Yanpeng Chen - Web page: https://patentgrep.com/patent/US11115706B2 ## CPC - H04N 21/2343: involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)} - H04N 21/4334: {Recording operations (recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27)} - H04N 21/236: Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream {(multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00)} - H04N 21/2387: Stream processing in response to a playback request from an end-user, e.g. for trick-play - H04N 21/4147: PVR [Personal Video Recorder] (H04N5/76 takes precedence {; arrangements for broadcast with accumulation-type receivers H04H20/40}) - H04N 21/4307: {Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen} - H04N 21/47217: {for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks (specific graphical features in visual interfaces H04N21/4312)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method, client, and terminal device for screen recording are provided, and the method includes: enabling a screen recording data synthesizing module (401) when a screen recording instruction is received (101); inputting an encoded audio data of a player into the screen recording data synthesizing module (401) for superposition, to obtain a merged audio data (102); and inputting a video data of the player into the screen recording data synthesizing module (401), and merging the video data with the merged audio data, to obtain a screen recording data (103). In the present disclosure, all the audio and video data in a live broadcasting scenario can be completely recorded to ensure the integrity of the live broadcasting scenario. ## Claims ### Claim 1 A method for screen recording, performed on a client, comprising: obtaining an audio play parameter of each audio play unit of a plurality of audio play units in a player when the player is initialized; receiving a screen recording instruction; for each respective audio play unit of the plurality of audio play units of the player, collecting respective audio data of the respective audio play unit based on the audio play parameter of the respective audio play unit and periodically encoding the respective audio data based on a preset audio unified encoding parameter to obtain respective encoded audio data in response to the screen recording instruction; superposing encoded audio data of the plurality of audio play units to obtain merged audio data; and merging video data of the player with the merged audio data to obtain screen recording data; wherein the method further comprises: determining an on-going screen recording state, and performing a corresponding processing based on the screen recording state, wherein the screen recording state at least includes a waiting timeout state; wherein after an operation of audio collecting and encoding performed on all audio data of any one of the plurality of audio play units is completed, the operation of audio collecting and encoding enters a waiting state; wherein determining the on-going screen recording state comprises: starting timing when the operation of audio collecting and encoding enters a waiting state; and determining that the screen recording state is the waiting timeout state if no operation of audio collecting and encoding performed on any other audio play unit enters the waiting state within a preset waiting time range. ### Claim 2 (depends on 1) The method according to claim 1, further comprising: managing audio collecting and encoding when the screen recording instruction is received, to simultaneously collect audio data of the plurality of audio play units; and managing the audio collecting and encoding when an end instruction is received, to simultaneously stop collecting the audio data of the plurality of the audio play units. ### Claim 3 (depends on 1) The method according to claim 1, wherein the screen recording state further includes a normal state, a cancel state, and an end state. ### Claim 4 (depends on 3) The method according to claim 3, wherein the screen recording state is determined as the cancel state in a case that an end operation is triggered when a minimum duration requirement for a screen recording duration is not satisfied after screen recording starts, or, a cancel operation is triggered when the screen recording starts and does not end normally or abnormally. ### Claim 5 (depends on 4) The method according to claim 4, wherein when the screen recording state is determined as the cancel state, performing the corresponding processing based on the screen recording state comprises: stopping on-going operations of audio collecting and encoding, video collecting and encoding and screen recording data synthesizing when receiving a screen recoding cancel instruction, and feeding back information of exception log to the client. ### Claim 6 (depends on 3) The method according to claim 3, wherein the end state includes a normal ending and an abnormal ending; and when the end state is the abnormal ending, performing a corresponding processing based on the screen recording state comprises: stopping on-going operations of audio collecting and encoding and video collecting and encoding, superposing encoded audio data to obtain merged audio data, and merging the video data with the merged audio data to obtain screen recording data. ### Claim 7 (depends on 1) The method according to claim 1, further comprising: collecting video data of the player based on a preset video collecting parameter when the screen recording instruction is received. ### Claim 8 (depends on 1) The method according to claim 1, wherein when the screen recording state is determined as the waiting timeout state, performing the corresponding processing based on the screen recording state comprises: stopping on-going operations of audio collecting and encoding, video collecting and encoding and screen recording data synthesizing, and feeding back information of exception log to the client. ### Claim 9 A terminal device for screen recording, comprising: at least one processor; and memory communicatively coupled to the at least one processor; wherein the memory stores instructions which, when executed by the at least one processor, cause the at least one processor to: obtain an audio play parameter of each audio play unit of a plurality of audio play units in a player when the player is initialized; receive a screen recording instruction; for each respective audio play unit of the plurality of audio play units of the player, collect respective audio data of the respective audio play unit based on the audio play parameter of the respective audio play unit and periodically encode the respective audio data based on a preset audio unified encoding parameter to obtain respective encoded audio data in response to the screen recording instruction; superpose encoded audio data of the plurality of audio play units to obtain merged audio data; and merge video data of the player with the merged audio data to obtain screen recording data; wherein the at least one processor is further configured to: determine an on-going screen recording state, and perform a corresponding processing based on the screen recording state, wherein the screen recording state at least includes a waiting timeout state; wherein after an operation of audio collecting and encoding performed on all audio data of any one of the plurality of audio play units is completed, the operation of audio collecting and encoding enters a waiting state; wherein the at least one processor is configured to determine the on-going screen recording state by: starting timing when the operation of audio collecting and encoding enters a waiting state; and determining that the screen recording state is the waiting timeout state if no operation of audio collecting and encoding performed on any other audio play unit enters the waiting state within a preset waiting time range. ### Claim 10 (depends on 9) The terminal device according to claim 9, wherein the at least one processor is further configured to: manage audio collecting and encoding when the screen recording instruction is received, to simultaneously collect the audio data of the plurality of audio play units; and manage the audio collecting and encoding when an end instruction is received, to simultaneously stop collecting the audio data of the plurality of the audio play units. ### Claim 11 (depends on 9) The terminal device according to claim 9, wherein the screen recording state further includes a normal state, a cancel state, and an end state. ### Claim 12 (depends on 11) The terminal device according to claim 11, wherein the screen recording state is determined as the cancel state in a case that an end operation is triggered when a minimum duration requirement for a screen recording duration is not satisfied after screen recording starts, or, a cancel operation is triggered when the screen recording starts and does not end normally or abnormally. ### Claim 13 (depends on 12) The terminal device according to claim 12, wherein when the screen recording state is determined as the cancel state, the at least one processor is further configured to: stop on-going operations of audio collecting and encoding, video collecting and encoding and screen recording data synthesizing when receiving a screen recoding cancel instruction, and feed back information of exception log to the client. ### Claim 14 (depends on 11) The terminal device according to claim 11, wherein the end state includes a normal ending and an abnormal ending; and when the end state is the abnormal ending, the at least one processor is further configured to: stop on-going operations of audio collecting and encoding and video collecting and encoding, superpose encoded audio data to obtain merged audio data, and merge the video data with the merged audio data to obtain the screen recording data. ### Claim 15 (depends on 9) The terminal device according to claim 9, wherein the at least one processor is further configured to: collect video data of the player based on a preset video collecting parameter when the screen recording instruction is received. ### Claim 16 (depends on 9) The terminal device according to claim 9, wherein when the screen recording state is determined as the waiting timeout state, the at least one processor is further configured to: stop on-going operations of audio collecting and encoding, video collecting and encoding and screen recording data synthesizing, and feed back information of exception log to the client.