# US 10,771,798 B2 · Multi-stream image processing apparatus and method of the same > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,771,798 B2 - Document ID: US10771798B2 - Status: Granted - Kind: B2 - Application number: 16/511,668 - Filed: 2019-07-15 - Published: 2020-07-02 - Granted: 2020-09-08 - Earliest claimed: 2018-12-26 - Assignees: Xiamen Sigmastar Technology Ltd - Applicants: Xiamen Sigmastar Technology Ltd. - Inventors: Ming-Yong Sun; Hao Wang - Web page: https://patentgrep.com/patent/US10771798B2 ## CPC - G06T 1/60: Memory management - H04N 19/40: using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence) - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The present disclosure discloses a multi-stream image processing method that includes steps outlined below. Image streams that include a main image stream and at least one sub image stream are generated by a former stage circuit, wherein a resolution of the main image stream is higher than the resolution of the sub image stream. Within an image frame processing time period, a N-th sub image frame is stored in a current sub image storage block in a memory module and a N-th main image frame is stored in a current main image storage block in the memory module by the former stage circuit. Within a first sub period of the image frame processing time period, a N−1-th sub image frame is read from a former sub image storage block in the memory module by a latter stage circuit to perform processing. Within a second sub period of the image frame processing time period, the N-th main image frame is read by the latter stage circuit to perform processing. ## Claims ### Claim 1 A multi-stream image processing method, comprising: generating a plurality of image streams by a former stage circuit according to a same image source, wherein the image streams comprise a main image stream and at least one sub image stream, and a resolution of the main image stream is higher than a resolution of the sub image stream, and the main image stream comprises a plurality of main image frames and the sub image stream comprises a plurality of sub image frames; processing the main image stream and the sub image stream for a plurality of image frame processing time periods, each comprising a first sub period and a second sub period behind the first sub period, to generate a processed main image stream and at least one processed sub image stream, wherein for an N-th image frame processing time period: storing an N-th sub image frame of the sub image stream by the former stage circuit in at least one current sub image storage block of a memory module during the N-th image frame processing time period; storing an N-th main image frame of the main image stream by the former stage circuit in a main image storage block of the memory module during the N-th image frame processing time period; reading and processing an N−1-th sub image of the sub image stream stored in a previous sub image storage block of the memory module by a latter stage circuit during the first sub period of the N-th image frame processing time period, to generate an N−1-th processed sub image frame in the processed sub image stream; and reading and processing the N-th main image frame stored in the main image storage block by the latter stage circuit during the second sub period of the N-th image frame processing time period, to generate an N-th processed main image frame in the processed main image stream. ### Claim 2 (depends on 1) The multi-stream image processing method of claim 1, wherein a capacity of the main image storage block is larger than or equals to a size of the N-th main image frame. ### Claim 3 (depends on 1) The multi-stream image processing method of claim 1, further comprising: retrieving frame processing information of the former stage circuit and the latter stage circuit by a synchronization circuit; determining whether a storing row number of the N-th main image frame that the former stage circuit currently stores exceeds a reading row number of the N-th main image frame that the latter stage circuit currently reads according to the frame processing information by the synchronization circuit; when the storing row number exceeds the reading row number, allowing the latter stage circuit to keep reading the N-th main image frame and process the N-th main image frame; and when the storing row number does not exceed the reading row number, stopping the latter stage circuit from reading the N-th main image frame. ### Claim 4 (depends on 3) The multi-stream image processing method of claim 3, wherein a capacity of the main image storage block is smaller than a size of the N-th main image frame, and the main image storage block operates as a ring buffer, the multi-stream image processing method further comprises: determining whether a difference between the storing row number and the reading row number exceeds the capacity of the main image storage block according to the frame processing information by the synchronization circuit; when the difference does not exceed the capacity of the main image storage block, allowing the former stage circuit to keep storing the N-th main image frame; and when the difference exceeds the capacity of the main image storage block, stopping the former stage circuit from storing the N-th main image frame. ### Claim 5 (depends on 1) The multi-stream image processing method of claim 1, wherein a synchronization signal time period is presented between the image frame processing time period and a former image frame processing time period, the former stage circuit and the latter stage circuit do not perform data processing during the synchronization signal time period. ### Claim 6 (depends on 1) The multi-stream image processing method of claim 1, wherein a capacity of the main image storage block is smaller than a size of the N-th main image frame, and the main image storage block operates as a ring buffer, a synchronization signal time period is presented between the image frame processing time period and a former image frame processing time period, the multi-stream image processing method further comprises: reading an N−1-th main image frame stored in the main image storage block by the latter stage circuit and processing the N−1-th main image frame by the latter stage circuit during a third sub period, wherein the third sub period is during the synchronization signal time period and during a section of the image frame processing time period that is before the first sub period; retrieving frame processing information of the former stage circuit and the latter stage circuit by a synchronization circuit; calculating a difference between a size of the N−1-th main image frame and a reading row number of the N−1-th main image frame that the latter stage circuit currently reads according to the frame processing information by the synchronization circuit; determining whether a sum of the difference and a storing row number of the N-th main image frame that the former stage circuit currently stores exceeds the capacity of the main image storage block according to the frame processing information by the synchronization circuit; when the sum does not exceed the capacity of the main image storage block, allowing the former stage circuit to keep storing the N-th main image frame; and when the sum exceeds the capacity of the main image storage block, stopping the former stage circuit from storing the N-th main image frame. ### Claim 7 (depends on 1) The multi-stream image processing method of claim 1, wherein the former stage circuit is an image signal processor and the latter stage circuit is an encoder. ### Claim 8 A multi-stream image processing apparatus, comprising: a memory module; a former stage circuit; a latter stage circuit; and a processing circuit electrically coupled to the memory module, the former stage circuit and the latter stage circuit, and configured to execute a plurality of software or firmware executable commands to execute a multi-stream image processing method, wherein the multi-stream image processing method comprises: generating a plurality of image streams by the former stage circuit according to a same image source, wherein the image streams comprise a main image stream and at least one sub image stream, and a resolution of the main image stream is higher than a resolution of the sub image stream, and the main image stream comprises a plurality of main image frames and the sub image stream comprises a plurality of sub image frames; processing the main image stream and the sub image stream for a plurality of image frame processing time periods, each comprising a first sub period and a second sub period behind the first sub period, to generate a processed main image stream and at least one processed sub image stream, wherein for an N-th image frame processing time period: storing an N-th sub image frame of the sub image stream by the former stage circuit in at least one current sub image storage block of a memory module during the N-th image frame processing time period; storing an N-th main image frame of the main image stream by the former stage circuit in a main image storage block of the memory module during the N-th image frame processing time period; reading and processing an N−1-th sub image frame of the sub image stream stored in a previous sub image storage block of the memory module by the latter stage circuit during the first sub period of the N-th image frame processing time period, to generate an N−1-th processed sub image frame in the processed sub image stream; and reading and processing the N-th main image frame stored in the main image storage block by the latter stage circuit during the second sub period of the N-th image frame processing time period, to generate an N-th processed main image frame in the processed main image stream. ### Claim 9 (depends on 8) The multi-stream image processing apparatus of claim 8, wherein a capacity of the main image storage block is larger than or equals to a size of the N-th main image frame. ### Claim 10 (depends on 8) The multi-stream image processing apparatus of claim 8, further comprising a synchronization circuit electrically coupled to the processing circuit, the former stage circuit and the latter stage circuit, the multi-stream image processing method further comprises: retrieving frame processing information of the former stage circuit and the latter stage circuit by a synchronization circuit; determining whether a storing row number of the N-th main image frame that the former stage circuit currently stores exceeds a reading row number of the N-th main image frame that the latter stage circuit currently reads according to the frame processing information by the synchronization circuit; when the storing row number exceeds the reading row number, allowing the latter stage circuit to keep reading the N-th main image frame and process the N-th main image frame; and when the storing row number does not exceeds the reading row number, stopping the latter stage circuit from reading the N-th main image frame. ### Claim 11 (depends on 10) The multi-stream image processing apparatus of claim 10, wherein a capacity of the main image storage block is smaller than a size of the N-th main image frame, and the main image storage block operates as a ring buffer, the multi-stream image processing method further comprises: determining whether a difference between the storing row number and the reading row number exceeds the capacity of the main image storage block according to the frame processing information by the synchronization circuit; when the difference does not exceed the capacity of the main image storage block, allowing the former stage circuit to keep storing the N-th main image frame; and when the difference exceeds the capacity of the main image storage block, stopping the former stage circuit from storing the N-th main image frame. ### Claim 12 (depends on 8) The multi-stream image processing apparatus of claim 8, wherein a synchronization signal time period is presented between the image frame processing time period and a former image frame processing time period, the former stage circuit and the latter stage circuit do not perform data processing during the synchronization signal time period. ### Claim 13 (depends on 8) The multi-stream image processing apparatus of claim 8, further comprising a synchronization circuit electrically coupled to the processing circuit, the former stage circuit and the latter stage circuit, wherein a capacity of the main image storage block is smaller than a size of the N-th main image frame, and the main image storage block operates as a ring buffer, a synchronization signal time period is presented between the image frame processing time period and a former image frame processing time period, the multi-stream image processing method further comprises: reading an N−1-th main image frame stored in the main image storage block by the latter stage circuit and processing the N−1-th main image frame by the latter stage circuit during a third sub period, wherein the third sub period is during the synchronization signal time period and during a section of the image frame processing time period that is before the first sub period; retrieving frame processing information of the former stage circuit and the latter stage circuit by a synchronization circuit; calculating a difference between a size of the N−1-th main image frame and a reading row number of the N−1-th main image frame that the latter stage circuit currently reads according to the frame processing information by the synchronization circuit; determining whether a sum of the difference and a storing row number of the N-th main image frame that the former stage circuit currently stores exceeds the capacity of the main image storage block according to the frame processing information by the synchronization circuit; when the sum does not exceed the capacity of the main image storage block, allowing the former stage circuit to keep storing the N-th main image frame; and when the sum exceeds the capacity of the main image storage block, stopping the former stage circuit from storing the N-th main image frame. ### Claim 14 (depends on 8) The multi-stream image processing apparatus of claim 8, wherein the former stage circuit is an image signal processor and the latter stage circuit is an encoder.