# US 12,217,462 B2 · Image processing method and apparatus, electronic device, and storage medium for reducing time delay caused by decoding an image sequence and improving the smoothness of image processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,462 B2 - Document ID: US12217462B2 - Status: Granted - Kind: B2 - Application number: 17/728,810 - Filed: 2022-04-25 - Published: 2022-08-11 - Granted: 2025-02-04 - Earliest claimed: 2020-07-01 - Assignees: Tencent Technology (Shenzhen) Company Limited - Applicants: Tencent Technology (Shenzhen) Company Limited - Inventors: Xiao Li - Web page: https://patentgrep.com/patent/US12217462B2 ## CPC - G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/156: Availability of hardware or computational resources, e.g. encoding based on power-saving criteria - H04N 19/172: the region being a picture, frame or field - H04N 19/20: using video object coding - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract This application discloses an image processing method performed by an electronic device. The method includes: receiving padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence; inputting the image frame set to a cache queue of a decoder, and determining an image frame currently located at the first position of the cache queue as a target image frame; inserting padding frames between the target image frame and a next image frame subsequent the target image frame based on padding frame information, and performing the step of inserting padding frames between the target image frame and a next image frame subsequent the target image frame based on the padding frame information, until all image frames in the image frame set are decoded; and processing the decoded image frames and displaying a processing result. The solution reduces time delay caused by decoding an image sequence. ## Claims ### Claim 1 An image processing method performed at an electronic device, the method comprising: receiving padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence, the padding frame information being generated by the server according to the image frame set; inputting the image frame set frame by frame to a cache queue of a decoder, and determining an image frame currently located at a first position of the cache queue as a target image frame; inserting padding frames between the target image frame and a next image frame subsequent the target image frame based on the padding frame information, so that the remaining frames other than the target image frame in the cache queue are the padding frames; using the next image frame subsequent the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performing the operation of inserting padding frames between the new target image frame and a next image frame subsequent the new target image frame based on the padding frame information, until all image frames in the image frame set are decoded; and processing the decoded image frames and displaying a processing result of the decoded image frames. ### Claim 2 (depends on 1) The method according to claim 1, wherein the inserting padding frames between the target image frame and a next image frame subsequent the target image frame based on the padding frame information comprises: extracting a padding frame set generated by the server according to the image frame set from the padding frame information; selecting a padding frame corresponding to the target image frame in the padding frame set to obtain a target padding frame; and inserting a preset quantity of target padding frames between the target image frame and the next image frame subsequent the target image frame. ### Claim 3 (depends on 2) The method according to claim 2, wherein the inserting a preset quantity of target padding frames between the target image frame and the next image frame subsequent the target image frame comprises: detecting a maximum cache quantity corresponding to the cache queue; generating a padding quantity according to the target image frame and the maximum cache quantity; and inserting target padding frames corresponding to the padding quantity between the target image frame and the next image frame subsequent the target image frame. ### Claim 4 (depends on 3) The method according to claim 3, wherein after the inserting target padding frames corresponding to the padding quantity between the target image frame and the next image frame subsequent the target image frame, the method further comprises: extracting a frame sequence number of the target image frame; and assigning frame sequence numbers of the padding frames in the cache queue based on the frame sequence number and corresponding positions of the padding frames in the cache queue. ### Claim 5 (depends on 4) The method according to claim 4, wherein the assigning frame sequence numbers of the padding frames in the cache queue based on the frame sequence number and corresponding positions of the padding frames in the cache queue comprises: extracting an offset corresponding to the padding frames in the cache queue from the padding frame information; and assigning the frame sequence numbers of the padding frames in the cache queue based on the frame sequence number, the corresponding positions of the padding frames in the cache queue, and the offset corresponding to the padding frames in the cache queue. ### Claim 6 (depends on 1) The method according to claim 1, wherein the using the next image frame subsequent the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performing the operation of inserting padding frames between the new target image frame and a next image frame subsequent the new target image frame based on padding frame information, until all image frames in the image frame set are decoded comprises: using the new target image frame to obtain a currently processed object in response to detecting that the target image frame is decoded; detecting a type of the currently processed object; inserting the padding frames between the currently processed object and a next image frame of the currently processed object based on the padding frame information by using a frame sequence number of the currently processed object as a benchmark, when a detection result indicates that the currently processed object is a key frame; adjusting a frame sequence number of the currently processed object according to a frame sequence number corresponding to a previous frame of the currently processed object in response to detecting that the currently processed object is not the key frame, and inserting the padding frames between the currently processed object and the next image frame of the currently processed object based on the padding frame information; and performing the operation of using the new target image frame to obtain a currently processed object until all the image frames are decoded. ### Claim 7 (depends on 1) The method according to claim 1, wherein the inputting the image frame set frame by frame to a cache queue of a decoder, and determining an image frame currently located at the first position of the cache queue as a target image frame comprises: obtaining a key frame of a currently rendered to-be-processed image sequence to obtain a target key frame; determining a next image frame of the target key frame in the image frame set as a target image frame; and storing the target image frame at the first position of the cache queue. ### Claim 8 An electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of being run on the processor, the processor, when executing the program, causing the electronic device to perform an image processing method including: receiving padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence, the padding frame information being generated by the server according to the image frame set; inputting the image frame set frame by frame to a cache queue of a decoder, and determining an image frame currently located at a first position of the cache queue as a target image frame; inserting padding frames between the target image frame and a next image frame subsequent the target image frame based on the padding frame information, so that the remaining frames other than the target image frame in the cache queue are the padding frames; using the next image frame subsequent the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performing the operation of inserting padding frames between the new target image frame and a next image frame subsequent the new target image frame based on the padding frame information, until all image frames in the image frame set are decoded; and processing the decoded image frames and displaying a processing result of the decoded image frames. ### Claim 9 (depends on 8) The electronic device according to claim 8, wherein the inserting padding frames between the target image frame and a next image frame subsequent the target image frame based on the padding frame information comprises: extracting a padding frame set generated by the server according to the image frame set from the padding frame information; selecting a padding frame corresponding to the target image frame in the padding frame set to obtain a target padding frame; and inserting a preset quantity of target padding frames between the target image frame and the next image frame subsequent the target image frame. ### Claim 10 (depends on 9) The electronic device according to claim 9, wherein the inserting a preset quantity of target padding frames between the target image frame and the next image frame subsequent the target image frame comprises: detecting a maximum cache quantity corresponding to the cache queue; generating a padding quantity according to the target image frame and the maximum cache quantity; and inserting target padding frames corresponding to the padding quantity between the target image frame and the next image frame subsequent the target image frame. ### Claim 11 (depends on 10) The electronic device according to claim 10, wherein after the inserting target padding frames corresponding to the padding quantity between the target image frame and the next image frame subsequent the target image frame, the method further comprises: extracting a frame sequence number of the target image frame; and assigning frame sequence numbers of the padding frames in the cache queue based on the frame sequence number and corresponding positions of the padding frames in the cache queue. ### Claim 12 (depends on 11) The electronic device according to claim 11, wherein the assigning frame sequence numbers of the padding frames in the cache queue based on the frame sequence number and corresponding positions of the padding frames in the cache queue comprises: extracting an offset corresponding to the padding frames in the cache queue from the padding frame information; and assigning the frame sequence numbers of the padding frames in the cache queue based on the frame sequence number, the corresponding positions of the padding frames in the cache queue, and the offset corresponding to the padding frames in the cache queue. ### Claim 13 (depends on 8) The electronic device according to claim 8, wherein the using the next image frame subsequent the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performing the operation of inserting padding frames between the new target image frame and a next image frame subsequent the new target image frame based on padding frame information, until all image frames in the image frame set are decoded comprises: using the new target image frame to obtain a currently processed object in response to detecting that the target image frame is decoded; detecting a type of the currently processed object; inserting the padding frames between the currently processed object and a next image frame of the currently processed object based on the padding frame information by using a frame sequence number of the currently processed object as a benchmark, when a detection result indicates that the currently processed object is a key frame; adjusting a frame sequence number of the currently processed object according to a frame sequence number corresponding to a previous frame of the currently processed object in response to detecting that the currently processed object is not the key frame, and inserting the padding frames between the currently processed object and the next image frame of the currently processed object based on the padding frame information; and performing the operation of using the new target image frame to obtain a currently processed object until all the image frames are decoded. ### Claim 14 (depends on 8) The electronic device according to claim 8, wherein the inputting the image frame set frame by frame to a cache queue of a decoder, and determining an image frame currently located at the first position of the cache queue as a target image frame comprises: obtaining a key frame of a currently rendered to-be-processed image sequence to obtain a target key frame; determining a next image frame of the target key frame in the image frame set as a target image frame; and storing the target image frame at the first position of the cache queue. ### Claim 15 A non-transitory computer-readable storage medium, storing a computer program, the computer program, when executed by a processor of an electronic device, causing the electronic device to perform an image processing method including: receiving padding frame information transmitted by a server and an image frame set of a to-be-processed image sequence, the padding frame information being generated by the server according to the image frame set; inputting the image frame set frame by frame to a cache queue of a decoder, and determining an image frame currently located at a first position of the cache queue as a target image frame; inserting padding frames between the target image frame and a next image frame subsequent the target image frame based on the padding frame information, so that the remaining frames other than the target image frame in the cache queue are the padding frames; using the next image frame subsequent the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performing the operation of inserting padding frames between the new target image frame and a next image frame subsequent the new target image frame based on the padding frame information, until all image frames in the image frame set are decoded; and processing the decoded image frames and displaying a processing result of the decoded image frames. ### Claim 16 (depends on 15) The non-transitory computer-readable storage medium according to claim 15, wherein the inserting padding frames between the target image frame and a next image frame subsequent the target image frame based on the padding frame information comprises: extracting a padding frame set generated by the server according to the image frame set from the padding frame information; selecting a padding frame corresponding to the target image frame in the padding frame set to obtain a target padding frame; and inserting a preset quantity of target padding frames between the target image frame and the next image frame subsequent the target image frame. ### Claim 17 (depends on 16) The non-transitory computer-readable storage medium according to claim 16, wherein the inserting a preset quantity of target padding frames between the target image frame and the next image frame subsequent the target image frame comprises: detecting a maximum cache quantity corresponding to the cache queue; generating a padding quantity according to the target image frame and the maximum cache quantity; and inserting target padding frames corresponding to the padding quantity between the target image frame and the next image frame subsequent the target image frame. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium according to claim 17, wherein after the inserting target padding frames corresponding to the padding quantity between the target image frame and the next image frame subsequent the target image frame, the method further comprises: extracting a frame sequence number of the target image frame; and assigning frame sequence numbers of the padding frames in the cache queue based on the frame sequence number and corresponding positions of the padding frames in the cache queue. ### Claim 19 (depends on 15) The non-transitory computer-readable storage medium according to claim 15, wherein the using the next image frame subsequent the target image frame as a new target image frame in response to detecting that the target image frame is decoded, and performing the operation of inserting padding frames between the new target image frame and a next image frame subsequent the new target image frame based on padding frame information, until all image frames in the image frame set are decoded comprises: using the new target image frame to obtain a currently processed object in response to detecting that the target image frame is decoded; detecting a type of the currently processed object; inserting the padding frames between the currently processed object and a next image frame of the currently processed object based on the padding frame information by using a frame sequence number of the currently processed object as a benchmark, when a detection result indicates that the currently processed object is a key frame; adjusting a frame sequence number of the currently processed object according to a frame sequence number corresponding to a previous frame of the currently processed object in response to detecting that the currently processed object is not the key frame, and inserting the padding frames between the currently processed object and the next image frame of the currently processed object based on the padding frame information; and performing the operation of using the new target image frame to obtain a currently processed object until all the image frames are decoded. ### Claim 20 (depends on 15) The non-transitory computer-readable storage medium according to claim 15, wherein the inputting the image frame set frame by frame to a cache queue of a decoder, and determining an image frame currently located at the first position of the cache queue as a target image frame comprises: obtaining a key frame of a currently rendered to-be-processed image sequence to obtain a target key frame; determining a next image frame of the target key frame in the image frame set as a target image frame; and storing the target image frame at the first position of the cache queue.