# US 12,120,336 B2 · Embedding frame masks in a video stream > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,120,336 B2 - Document ID: US12120336B2 - Status: Granted - Kind: B2 - Application number: 18/107,841 - Filed: 2023-02-09 - Published: 2023-06-15 - Granted: 2024-10-15 - Earliest claimed: 2021-06-23 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Henrik Valdemar Turbell; David Yuheng Zhao; Zhuangfei Wu - Web page: https://patentgrep.com/patent/US12120336B2 ## CPC - H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/17: the unit being an image region, e.g. an object - H04N 19/172: the region being a picture, frame or field - H04N 19/186: the unit being a colour or a chrominance component - H04N 19/85: using pre-processing or post-processing specially adapted for video compression - H04N 5/202: Gamma control (circuits for controlling camera response irrespective of the scene brightness H04N23/82) - H04N 9/75: Chroma key - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Techniques are described for efficiently embedding frame masks in a video stream. In some solutions, a computer implemented method includes operations for encoding a frame of video data comprising an array of pixels to generate an encoded video frame, and transmitting the encoded video frame to a video decoder. The array of pixels include foreground pixels and background pixels. The foreground pixels have respective first chroma component values which are bounded within a first chroma component range. Encoding the frame of video data can include converting the first chroma component values of the foreground pixels to second chroma component values by applying a mapping function. The second chroma component values are bounded within a second chroma component range. The second chroma component range is shifted or compressed from the first chroma component range. ## Claims ### Claim 1 A computer-implemented method comprising: encoding a frame of video data comprising an array of pixels to generate an encoded video frame, wherein the array of pixels comprise foreground pixels and background pixels; and transmitting the encoded video frame to a video decoder; wherein the foreground pixels have respective first chroma component values, wherein the first chroma component values are bounded within a first chroma component range; wherein encoding the frame of video data comprises converting the first chroma component values of the foreground pixels to second chroma component values by applying a mapping function, wherein the second chroma component values are bounded within a second chroma component range, and wherein the second chroma component range is shifted or compressed from the first chroma component range, wherein both the first chroma component range and the second chroma component range are subranges of a full range of a selected chroma component in a color space. ### Claim 2 (depends on 1) The method of claim 1, wherein converting the first chroma component values to second chroma component values comprises: performing both chroma component range shifting and chroma component range compression by linearly mapping a first chroma component value within the first chroma component range to a corresponding second chroma component value within the second chroma component range. ### Claim 3 (depends on 1) The method of claim 1, wherein converting the first chroma component values to second chroma component values comprises: performing chroma component range compression by mapping the first chroma component values within the first chroma component range to the second chroma component values within the second chroma component range, wherein the second chroma component range is smaller than the first chroma component range. ### Claim 4 (depends on 1) The method of claim 1, wherein converting the first chroma component values to second chroma component values comprises: performing chroma component range shifting by mapping the first chroma component values within the first chroma component range to the second chroma component values within the second chroma component range, wherein a lower limit of the second chroma component range is larger than a lower limit of the first chroma component range, and an upper limit of the second chroma component range is larger than an upper limit of the first chroma component range. ### Claim 5 (depends on 4) The method of claim 4, wherein encoding the frame of video data further comprises setting chroma component values of the background pixels to a background chroma component value that is smaller than the lower limit of the second chroma component range. ### Claim 6 (depends on 1) The method of claim 1, wherein converting the first chroma component values to second chroma component values comprises: performing chroma component range shifting by mapping the first chroma component values within the first chroma component range to the second chroma component values within the second chroma component range, wherein a lower limit of the second chroma component range is smaller than a lower limit of the first chroma component range, and an upper limit of the second chroma component range is smaller than an upper limit of the first chroma component range. ### Claim 7 (depends on 6) The method of claim 6, wherein encoding the frame of video data further comprises setting chroma component values of the background pixels to a background chroma component value that is larger than the upper limit of the second chroma component range. ### Claim 8 (depends on 1) The method of claim 1, wherein shifting or compressing from the first chroma component range to the second chroma component range is performed in two dimensions corresponding to two different color components of a YUV color space. ### Claim 9 (depends on 1) The method of claim 1, wherein the encoded video frame is usable by the video decoder to: decode the encoded video frame to obtain decoded chroma component values corresponding to the array of pixels; extract a foreground mask based on the decoded chroma component values, wherein the foreground mask identifies a subset of the array of pixels to be in a foreground of the frame; and convert the decoded chroma component values of the subset of the array of pixels identified by the foreground mask to new chroma component values by applying an inverse mapping function to the decoded chroma component values of the subset of the array of pixels identified by the foreground mask, wherein the inverse mapping function reverses the mapping function, and wherein the new chroma component values are bounded within the first chroma component range. ### Claim 10 A computing device comprising: memory; one or more hardware processors coupled to the memory; and one or more non-transitory computer-readable storage media storing instructions that, when loaded into the memory, cause the one or more hardware processors to perform operations comprising: encoding a frame of video data comprising an array of pixels to generate an encoded video frame, wherein the array of pixels comprise foreground pixels and background pixels; and transmitting the encoded video frame to a video decoder; wherein the foreground pixels have respective first chroma component values, wherein the first chroma component values are bounded within a first chroma component range; wherein encoding the frame of video data comprises converting the first chroma component values of the foreground pixels to second chroma component values by applying a mapping function, wherein the second chroma component values are bounded within a second chroma component range, and wherein the second chroma component range is shifted or compressed from the first chroma component range, wherein both the first chroma component range and the second chroma component range are subranges of a full range of a selected chroma component in a color space. ### Claim 11 (depends on 10) The computing device of claim 10, wherein converting the first chroma component values to second chroma component values comprises: performing both chroma component range shifting and chroma component range compression by linearly mapping a first chroma component value within the first chroma component range to a corresponding second chroma component value within the second chroma component range. ### Claim 12 (depends on 10) The computing device of claim 10, wherein converting the first chroma component values to second chroma component values comprises: performing chroma component range compression by mapping the first chroma component values within the first chroma component range to the second chroma component values within the second chroma component range, wherein the second chroma component range is smaller than the first chroma component range. ### Claim 13 (depends on 10) The computing device of claim 10, wherein converting the first chroma component values to second chroma component values comprises: performing chroma component range shifting by mapping the first chroma component values within the first chroma component range to the second chroma component values within the second chroma component range, wherein a lower limit of the second chroma component range is larger than a lower limit of the first chroma component range, and an upper limit of the second chroma component range is larger than an upper limit of the first chroma component range. ### Claim 14 (depends on 13) The computing device of claim 13, wherein encoding the frame of video data further comprises setting chroma component values of the background pixels to a background chroma component value that is smaller than the lower limit of the second chroma component range. ### Claim 15 (depends on 10) The computing device of claim 10, wherein converting the first chroma component values to second chroma component values comprises: performing chroma component range shifting by mapping the first chroma component values within the first chroma component range to the second chroma component values within the second chroma component range, wherein a lower limit of the second chroma component range is smaller than a lower limit of the first chroma component range, and an upper limit of the second chroma component range is smaller than an upper limit of the first chroma component range. ### Claim 16 (depends on 15) The computing device of claim 15, wherein encoding the frame of video data further comprises setting chroma component values of the background pixels to a background chroma component value that is larger than the upper limit of the second chroma component range. ### Claim 17 (depends on 10) The computing device of claim 10, wherein shifting or compressing from the first chroma component range to the second chroma component range is performed in two dimensions corresponding to two different color components of a YUV color space. ### Claim 18 (depends on 10) The computing device of claim 10, wherein the encoded video frame is usable by the video decoder to: decode the encoded video frame to obtain decoded chroma component values corresponding to the array of pixels; extract a foreground mask based on the decoded chroma component values, wherein the foreground mask identifies a subset of the array of pixels to be in a foreground of the frame; and convert the decoded chroma component values of the subset of the array of pixels identified by the foreground mask to new chroma component values by applying an inverse mapping function to the decoded chroma component values of the subset of the array of pixels identified by the foreground mask, wherein the inverse mapping function reverses the mapping function, and wherein the new chroma component values are bounded within the first chroma component range. ### Claim 19 One or more non-transitory computer-readable media having encoded thereon computer-executable instructions causing one or more processors to perform a method, comprising: encoding a frame of video data comprising an array of pixels to generate an encoded video frame, wherein the array of pixels comprise foreground pixels and background pixels; and transmitting the encoded video frame to a video decoder; wherein the foreground pixels have respective first chroma component values, wherein the first chroma component values are bounded within a first chroma component range; wherein encoding the frame of video data comprises converting the first chroma component values of the foreground pixels to second chroma component values by applying a mapping function, wherein the second chroma component values are bounded within a second chroma component range, and wherein the second chroma component range is shifted or compressed from the first chroma component range, wherein both the first chroma component range and the second chroma component range are subranges of a full range of a selected chroma component in a color space. ### Claim 20 (depends on 19) The one or more non-transitory computer-readable media of claim 19, wherein shifting or compressing from the first chroma component range to the second chroma component range is performed in two dimensions corresponding to two different color components of a YUV color space.