# US 11,910,121 B2 · Converting dual-context video data to full color video > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,910,121 B2 - Document ID: US11910121B2 - Status: Granted - Kind: B2 - Application number: 17/158,209 - Filed: 2021-01-26 - Published: 2022-07-28 - Granted: 2024-02-20 - Earliest claimed: 2021-01-26 - Assignees: ZF Friedrichshafen AG - Applicants: Zf Friedrichshafen AG - Inventors: Anthony Louis Rivas; Matthew W. Warmuth; Christopher Guirlanda - Web page: https://patentgrep.com/patent/US11910121B2 ## CPC - H04N 5/265: Mixing - G06T 7/269: using gradient-based methods - H04N 7/013: {the incoming video signal comprising different parts having originally different frame rate, e.g. video and graphics} - H04N 9/43: Conversion of monochrome picture signals to colour picture signals for colour picture display - H04N 9/77: Circuits for processing the brightness signal and the chrominance signal relative to each other, e.g. adjusting the phase of the brightness signal relative to the colour signal, correcting differential gain or differential phase (circuits for matrixing H04N9/67) - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference - H04N 19/186: the unit being a colour or a chrominance component - H04N 19/537: Motion estimation other than block-based - H04N 19/587: involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence - H04N 23/10: for generating image signals from different wavelengths - H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio - H04N 25/40: Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled - H04N 25/445: by skipping some contiguous pixels within the read portion of the array - G06T 2207/10024: Color image - G06T 2207/20081: Training; Learning - G06T 2207/20221: Image fusion; Image merging - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Systems and methods are provided for generating color video from a dual-context camera. A dual-context camera provides a first series of video frames encoded in accordance with a full color model and a second series of video frames encoded in accordance with an underdetermined color model. The two series of video frames are interleaved as to form a series of pairs of frames, each comprising a color video frame and an underdetermined video frame. An image merger generates a composite image for each pair of frames in the series of frames. The composite image includes a set of brightness values from the underdetermined video frame and a set of chrominance values from the color video frame. A color video source replaces the underdetermined image in each of the series of pairs of frames with the composite image generated for the pair of frames to provide a color video stream. ## Claims ### Claim 1 A system comprising: a dual-context camera that provides a first series of video frames encoded in accordance with a full color model and a second series of video frames encoded in accordance with an underdetermined color model, the first series of video frames and the second series of video frames being interleaved as to form a series of pairs of frames, each comprising a color video frame and an underdetermined video frame; an image merger that generates a composite image for each pair of frames in the series of frames, the composite image comprising a set of brightness values from the underdetermined video frame and a set of chrominance values from the color video frame; and a color video source that replaces the underdetermined video frame in each of the series of pairs of frames with the composite image generated for the pair of frames to provide a color video stream, such that the color video stream comprises the first series of video frames and the composite image generated for each pair of frames of the series of pairs of frames. ### Claim 2 (depends on 1) The system of claim 1, wherein each of the first series of video frames is encoded in a luma-chroma color model with each of a plurality of pixels in the color video frame represented as a luminosity value and two chrominance values, and the image merger generating the composite image such that each pixel of a plurality of pixels in the composite image is represented as the brightness value from a corresponding pixel in the underdetermined video frame associated with the composite image and the two chrominance values associated with a corresponding pixel in the color video frame associated with the composite image. ### Claim 3 (depends on 1) The system of claim 1, wherein the system further comprises a motion detector that uses an optical flow algorithm that detects motion between the color video frame and the underdetermined video frame in a given pair of frames and selects the set of chrominance values for the composite image from the color video frame according to the detected motion. ### Claim 4 (depends on 1) The system of claim 1, wherein each of the first series of video frames is encoded in a trichromatic color model with each of a plurality of pixels in the color video frame represented as three chrominance values, the system further comprising a color model transformer that generates, from the color video frame, a color image encoded in a luma-chroma color model with each of a plurality of pixels in the color video frame represented as a luminosity value and two chrominance values. ### Claim 5 (depends on 4) The system of claim 4, the image merger generating the composite image such that each pixel of a plurality of pixels in the composite image is represented as the brightness value from a corresponding pixel in the underdetermined video frame associated with the composite image and the two chrominance values associated with a corresponding pixel in the color video frame associated with the composite image. ### Claim 6 (depends on 1) The system of claim 1, wherein the underdetermined video frame has a resolution less than that of the composite image, and a given pixel in the underdetermined video frame provides a brightness value for a set of multiple pixels in the composite image. ### Claim 7 (depends on 1) The system of claim 1, wherein the color video frame has a resolution greater than that of the composite image, and a plural set of pixels in the color video frame is used to provide a chrominance value for a pixel in the composite image. ### Claim 8 (depends on 1) The system of claim 1, the system further comprising a machine learning model that receives the first series of color video frames and the second series of video frames from the dual-context camera and a video display that receives the color video stream. ### Claim 9 (depends on 1) The system of claim 1, wherein a dual-context camera comprises a Red-Yellow-Yellow-Cyan color filter, such that the first series of video frames are encoded in accordance with a Red-Yellow-Cyan color model. ### Claim 10 A method comprising: providing a first series of video frames encoded in accordance with a full color model and a second series of video frames encoded in accordance with an underdetermined color model, the first series of video frames and the second series of video frames being interleaved as to form a series of pairs of frames, with each pair of frames in the series of frames comprising a color video frame and an underdetermined video frame; generating a composite image for each pair of frames in the series of frames comprising a set of brightness values from the underdetermined video frame and a set of chrominance values from the color video frame; and replacing the underdetermined video frame in each of the series of pairs of frames with the composite image generated for the pair of frames to provide a color video stream, such that the color video stream comprises the first series of video frames and the composite image generated for each pair of frames of the series of pairs of frames. ### Claim 11 (depends on 10) The method of claim 10, further comprising: applying, for each pair of frames of the series of pairs of frames, an optical flow algorithm that detects motion between the color video frame and the underdetermined video frame in the pair of frames; and selecting the set of chrominance values from the color video frame for the composite image according to the detected motion. ### Claim 12 (depends on 10) The method of claim 10, wherein each of the first series of video frames is encoded in a trichromatic color model with each of a plurality of pixels in the color video frame represented as three chrominance values, the method further comprising: generating, from the color video frame, a color image encoded in a luma-chroma color model with each of a plurality of pixels in the color video frame represented as a luminosity value and two chrominance values; wherein generating a composite image for each pair of frames comprises generating the composite image such that each pixel of a plurality of pixels in the composite image is represented as the brightness value from a corresponding pixel in the underdetermined video frame associated with the composite image and the two chrominance values associated with a corresponding pixel in the color video frame associated with the composite image. ### Claim 13 (depends on 10) The method of claim 10, wherein the underdetermined video frame has a resolution less than that of the composite image, and a given pixel in the underdetermined video frame provides a brightness value for a set of multiple pixels in the composite image. ### Claim 14 (depends on 10) The method of claim 10, wherein each of the first series of video frames is encoded in a luma-chroma color model with each of a plurality of pixels in the color video frame represented as a luminosity value and two chrominance values, and generating a composite image for each pair of frames comprises generating the composite image such that each pixel of a plurality of pixels in the composite image is represented as the brightness value from a corresponding pixel in the underdetermined video frame associated with the composite image and the two chrominance values associated with a corresponding pixel in the color video frame associated with the composite image. ### Claim 15 (depends on 10) The method of claim 10, wherein the second series of video frames are provided as grayscale video frames. ### Claim 16 A system comprising: a dual-context camera that provides a series of color video frames and a series of grayscale video frames, the series of color video frames and the series of grayscale video frames being interleaved as to form a series of pairs of frames, each comprising a color video frame and a grayscale video frame; an image merger that generates a composite image for each pair of frames in the series of frames, the composite image comprising a set of brightness values from the grayscale video frame and a set of chrominance values from the color video frame; and a color video source that replaces the grayscale video frame in each of the series of pairs of frames with the composite image generated for the pair of frames to provide a color video stream, such that the color video stream comprises the first series of color video frames and the composite image generated for each pair of frames of the series of pairs of frames. ### Claim 17 (depends on 16) The system of claim 16, wherein each of the series of color video frames is encoded in a trichromatic color model with each of a plurality of pixels in the color video frame represented as a three chrominance values, the system further comprising a color model transformer that generates, from the color video frame, a color image encoded in a luma-chroma color model with each of a plurality of pixels in the color video frame represented as a luminosity value and two chrominance values, the image merger generating the composite image such that each pixel of a plurality of pixels in the composite image is represented as the brightness value from a corresponding pixel in the grayscale video frame associated with the composite image and the two chrominance values associated with a corresponding pixel in the color video frame associated with the composite image. ### Claim 18 (depends on 16) The system of claim 16, wherein each of the series of color video frames is encoded in a luma-chroma color model with each of a plurality of pixels in the color video frame represented as a luminosity value and two chrominance values, and the image merger generating the composite image such that each pixel of a plurality of pixels in the composite image is represented as the brightness value from a corresponding pixel in the grayscale video frame associated with the composite image and the two chrominance values associated with a corresponding pixel in the color video frame associated with the composite image. ### Claim 19 (depends on 16) The system of claim 16, the system further comprising a machine learning model that receives the series of color video frames and the series of grayscale video frames from the dual-context camera and a video display that receives the color video stream. ### Claim 20 (depends on 16) The system of claim 16, wherein the system further comprises a motion detector that uses an optical flow algorithm that detects motion between the color video frame and the grayscale video frame in a given pair of frames and selects the set of chrominance values for the composite image from the color video frame according to the detected motion.