# US 11,445,227 B2 · Method and system for automatic real-time frame segmentation of high resolution video streams into constituent features and modifications of features in each frame to simultaneously create multiple different linear views from same video source > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,227 B2 - Document ID: US11445227B2 - Status: Granted - Kind: B2 - Application number: 16/973,739 - Filed: 2019-02-06 - Published: 2021-08-19 - Granted: 2022-09-13 - Earliest claimed: 2018-06-12 - Applicants: Ela Kliots Shapira; Abraham Zvi Yehiel Elbaum - Inventors: Ela Kliots Shapira; Abraham Zvi Yehiel Elbaum - Web page: https://patentgrep.com/patent/US11445227B2 ## CPC - H04N 21/23439: {for generating different versions} - H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)} - G11B 27/036: Insert-editing - H04N 9/64: Circuits for processing colour signals (H04N9/77 takes precedence; camera processing pipelines for processing colour signals H04N23/84) - H04N 11/24: High-definition television systems - H04N 21/23412: {for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects} - H04N 21/2343: involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)} - H04N 21/236: Assembling of a multiplex stream, e.g. transport stream, by combining a video stream with other content or additional data, e.g. inserting a URL [Uniform Resource Locator] into a video stream, multiplexing software data into a video stream; Remultiplexing of multiplex streams; Insertion of stuffing bits into the multiplex stream, e.g. to obtain a constant bit-rate; Assembling of a packetised elementary stream {(multiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00)} - H04N 21/44: Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs - H04N 21/4621: {Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen (arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10)} - H04N 21/4854: {for modifying image parameters, e.g. image brightness, contrast} - H04N 21/8146: {involving graphical data, e.g. 3D object, 2D graphics} - H04N 21/845: Structuring of content, e.g. decomposing content into time segments - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) ## Abstract A method, a programmed computer system, for example, a network-based hardware device, and a machine readable medium containing a software program for modifying a high definition video data stream in real time, with no visible delays, to add content on a frame by frame basis, thus simultaneously compositing multiple different customized linear views, for purposes such as creating and broadcasting targeted advertising in real time. The method employs conventional video processing technology in novel and inventive ways to achieve the desired objective by passing data selected by the program back and forth between a GPU and a CPU of a computer. The method is also usable with data streams having lower than high definition where real time processing is desired and yields better results than conventional methods. In such applications, all processing may be done by the CPU of a sufficiently powerful general-purpose computer. ## Claims ### Claim 1 A method of automatically modifying high resolution video data in real time using a computer including a GPU, the method comprising: receiving a video data stream by the computer, wherein the video data stream includes multiple frames; extracting color data for the individual pixels of the frames of the video data stream to create color layers; cascading the color layers into a unified color map of features in the frame; assembling instructions to modify pixels of the frame at a level of the unified color map of features from stored data according to predefined instructions, wherein the instructions at least include additional graphical data to be composited with original graphical data of the video data stream; creating multiple modified sets of video data streams according to the instructions, output resolution requirements and desired file formats, while original source of the video data stream remains intact, wherein the modified sets of video data are different from each other and the creating includes: merging the color layers and the additional graphical data into single-layer color frames with uniform resolution, wherein each single-layer color frame is based on a discrete set of the instructions, rendering each single-layer color frame based on the output resolution requirements to create at least one final frame, encoding, based on the resolution, each final frame to at least one codec to create video components, and multiplexing the video components into the desired file formats; and making the modified set of video data streams available for access by one or more intended recipients. ### Claim 2 (depends on 1) The method according to claim 1, wherein the video stream is provided by any source and in any native format, including a digital camera, a recorded video file, a streaming video file obtained over a network, and a commercial video program obtained directly off air. ### Claim 3 (depends on 2) The method according to claim 2 further including decoding the received video data stream from its native format to an internal processing format employed by the computer. ### Claim 4 (depends on 1) The method according to claim 1, wherein the instructions include modifications of color of the original graphical data of the video data stream. ### Claim 5 (depends on 1) The method according to claim 1, wherein the additional graphical data includes other video data from any source and in any native format, and has been decoded, from its native format to an internal processing format employed by the computer by which the method is being performed. ### Claim 6 (depends on 1) The method according to claim 1, wherein the additional graphical data includes images, definitions of 3D models to be rendered by the GPU, and definitions of text to be rendered by the GPU. ### Claim 7 (depends on 1) The method according to claim 1, wherein the instructions transmitted to the GPU include instructions for adding graphical elements. ### Claim 8 (depends on 1) The method according to claim 1, wherein the instructions include storyboard script created prior to processing of the video data stream and read during the processing, and/or commands sent from a user during the processing. ### Claim 9 (depends on 1) The method according to claim 1, wherein the instructions are generated automatically by a computer algorithm external to the computer performing the method. ### Claim 10 (depends on 1) The method according to claim 1, wherein part of frame or a whole frame, as color layers, is transferred from the GPU to a CPU of the computer for processing by computer vision algorithms and/or a business-intelligence algorithm. ### Claim 11 (depends on 10) The method according to claim 10, wherein the color layers which will be transferred to the CPU for subsequent processing undergo processing by the GPU prior to transfer, to prepare them for external logic application. ### Claim 12 (depends on 10) The method according to claim 10, wherein results of the processing are used to automatically generate the instructions. ### Claim 13 (depends on 1) The method according claim 1, further including executing a self-calibrating routine by accessing succeeding video frames to reduce sensitivity to detection of irrelevant changes in the video data. ### Claim 14 (depends on 1) The method according to claim 1, wherein the instructions for the color layers, including additional graphical elements, are transmitted to the GPU based on stored data for each intended recipient of each modified video stream. ### Claim 15 (depends on 14) The method according to claim 14, wherein the instructions for the color layers include data provided by external systems for each intended recipient of each modified video stream. ### Claim 16 (depends on 1) The method according to claim 1, wherein each frame is modified without introducing visible delays. ### Claim 17 (depends on 1) The method according to claim 1, wherein the multiple modified video data streams are created simultaneously and made available to multiple users. ### Claim 18 (depends on 17) The method according to claim 17, wherein modified video data streams which are based on the original graphical data of the video data stream are synchronized with respect to the video's timeline. ### Claim 19 (depends on 1) The method according to claim 1, wherein the video data stream is characterized by lower than high definition, and the steps of claim 1 are performed by a CPU and active memory of the computer without utilization of the GPU. ### Claim 20 A computer system for automatically modifying high resolution video data in real time, comprising: one or more central processing units and/or one or more graphics processing units; one or more network or other communication interface units; a shared memory including mass storage capability; wherein the memory contains a program that configures the computer system to: receive a video data stream, wherein the video data stream includes multiple frames; extract color data for individual pixels of the frames of the video data stream to create color layers; cascade the color layers into a unified color map of features in the frame; assemble instructions to modify pixels of the frame at a level of the unified color map of features from stored data according to predefined instructions, wherein the instructions include additional graphical data to be composited with original graphical data of the video data stream; create multiple modified sets of video data streams according to the instructions, output resolution requirements and desired file formats, while original source of the video data stream remains intact, wherein the modified sets of video data are different from each other and the creation includes: merging the color layers and the additional graphical data of the video data stream into single-layer color frames with uniform resolution, wherein each single-layer color frame is based on a discrete set of the instructions, rendering each single-layer color frame based on the output resolution requirements to create at least one final frame, encoding, based on the resolution, each final frame to at least one codec to create video components, and multiplexing the video components into the desired file formats; and make the modified sets of video data streams available for access by one or more intended recipients.