# US 10,503,458 B2 · System, method and computer program product for generating remote views in a virtual mobile device platform using efficient macroblock comparison during display encoding, including efficient detection of unchanged macroblocks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,458 B2 - Document ID: US10503458B2 - Status: Granted - Kind: B2 - Application number: 15/661,971 - Filed: 2017-07-27 - Published: 2018-02-01 - Granted: 2019-12-10 - Earliest claimed: 2016-07-28 - Assignees: Intelligent Waves LLC - Applicants: Intelligent Waves LLC - Inventors: Robert David Lee; Hemal Yogesh Gandhi - Web page: https://patentgrep.com/patent/US10503458B2 ## CPC - G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay} - G06F 9/452: {Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131)} - G06F 9/45558: {Hypervisor-specific management and integration aspects} - G06F 21/53: by executing in a restricted environment, e.g. sandbox or secure virtual machine - G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining - G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) - H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] - H04L 67/565: Conversion or adaptation of application format or content (adding application control or application functional data H04L67/561) - H04N 19/103: Selection of coding mode or of prediction mode - H04N 19/14: Coding unit complexity, e.g. amount of activity or edge presence estimation (H04N19/146 takes precedence) - H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/186: the unit being a colour or a chrominance component - H04N 19/27: involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC] - H04N 19/423: characterised by memory arrangements (H04N19/433 takes precedence) - H04N 19/503: involving temporal prediction (adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109) - H04N 19/517: by encoding - H04N 19/547: Motion estimation performed in a transform domain - H04N 19/61: in combination with predictive coding - H04N 19/625: using discrete cosine transform [DCT] - G06F 2221/032: Protect output to user by software means - G06F 2221/2149: Restricted operating environment - G09G 5/12: Synchronisation between the display unit and other units, e.g. other display units, video-disc players - G09G 2340/02: Handling of images in compressed format, e.g. JPEG, MPEG - G09G 2340/06: Colour space transformation - G09G 2350/00: Solving problems of bandwidth in display systems - G09G 2358/00: Arrangements for display data security - G09G 2360/121: using a cache memory - G09G 2360/18: Use of a frame buffer in a display terminal, inclusive of the display panel - G09G 2370/022: Centralised management of display operation, e.g. in a server instead of locally - G09G 2370/027: Arrangements and methods specific for the display of internet documents - H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference - H04N 19/172: the region being a picture, frame or field - H04W 88/02: Terminal devices - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00) - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract Embodiments disclosed herein provide systems, methods and computer readable media for generating remote views in a virtual mobile device platform. A virtual mobile device platform may be coupled to a physical mobile device over a network and generate frames of data for generating views on the physical device. These frames can be generated using an efficient display encoding pipeline on the virtual mobile device platform. Such efficiencies may include, for example, the synchronization of various processes or operations, the governing of various processing rates, the elimination of duplicative or redundant processing, the application of different encoding schemes, the efficient detection of duplicative or redundant data or the combination of certain operations. ## Claims ### Claim 1 A system for a virtual mobile device platform with efficient frame processing, comprising: a virtual mobile device platform coupled to a physical mobile device over a network, the virtual mobile device platform including a processor executing instructions on a non-transitory computer readable medium for implementing: a virtual machine executing a virtual mobile device associated with the physical mobile device communicating with the virtual mobile device platform over the network, the virtual mobile device including a guest operating system (OS) and one or more applications executing on the guest OS, the guest OS generating a frame of display data from an application executing on the guest OS; and a video encoder including an input/output (I/O) thread for generating a converted frame by performing color space conversion on the frame of display data generated by the guest OS and a display encoder for generating an encoded frame by encoding the converted frame generated by the I/O thread, wherein the I/O thread includes a zero motion vector (ZMV) detector to identify, for each macroblock of the frame, whether the macroblock is a ZMV macroblock by comparing the frame to a previous frame, wherein the processor further executes the instructions for: sending, by the virtual mobile device platform, the encoded frame to the physical mobile device. ### Claim 2 (depends on 1) The system of claim 1, wherein the frame is stored in a first buffer and the previous frame is stored in a second buffer and the comparison of the frame to the previous frame is integrated with a copy of the frame from the first buffer to the second buffer. ### Claim 3 (depends on 2) The system of claim 2, wherein each macroblock of the frame is copied from the first buffer to the second buffer if the macroblock is not identified as a ZMV macroblock. ### Claim 4 (depends on 1) The system of claim 1, wherein comparing the frame to the previous frame comprises comparing each macroblock of the frame to a corresponding macroblock of the previous frame, and during the comparing of the frame to the previous frame a first scanline for a row of macroblocks of the frame is compared to a corresponding first scanline for a corresponding row of macroblocks of the previous frame in an X direction before a second scanline for the row of macroblocks of the frame is compared to a corresponding second scanline for the corresponding row of macroblocks of the previous frame. ### Claim 5 (depends on 4) The system of claim 4, wherein if the comparison fails for the first scanline for a particular macroblock in the row of macroblocks, the comparison of the second scanline is skipped for the particular macroblock when comparing the second scanlines for the rows of macroblocks. ### Claim 6 (depends on 1) The system of claim 1, wherein the identification of whether each macroblock is a ZMV macroblock is stored in metadata associated with the frame. ### Claim 7 A method for efficient frame processing in a virtual mobile device platform, comprising: at a virtual mobile device platform coupled to a physical mobile device over a network, executing a virtual mobile device on a virtual machine, the virtual mobile device associated with the physical mobile device communicating with the virtual mobile device platform over the network, the virtual mobile device including a guest operating system (OS) and one or more applications executing on the guest OS, the guest OS generating a frame of display data from an application executing on the guest OS; generating a converted frame by an input/output (I/O) thread of a video encoder by performing color space conversion on the frame of display data generated by the guest OS; generating an encoded frame at a display encoder of the video encoder by encoding the converted frame generated by the I/O thread, wherein the I/O thread includes a zero motion vector (ZMV) detector to identify, for each macroblock of the frame, whether the macroblock is a ZMV macroblock by comparing the frame to a previous frame; and sending, by the virtual mobile device platform, the encoded frame to the physical mobile device. ### Claim 8 (depends on 7) The method of claim 7, wherein the frame is stored in a first buffer and the previous frame is stored in a second buffer and the comparison of the frame to the previous frame is integrated with a copy of the frame from the first buffer to the second buffer. ### Claim 9 (depends on 8) The method of claim 8, wherein each macroblock of the frame is copied from the first buffer to the second buffer if the macroblock is not identified as a ZMV macroblock. ### Claim 10 (depends on 7) The method of claim 7, wherein comparing the frame to the previous frame comprises comparing each macroblock of the frame to a corresponding macroblock of the previous frame, and during the comparing of the frame to the previous frame a first scanline for a row of macroblocks of the frame is compared to a corresponding first scanline for a corresponding row of macroblocks of the previous frame in an X direction before a second scanline for the row of macroblocks of the frame is compared to a corresponding second scanline for the corresponding row of macroblocks of the previous frame. ### Claim 11 (depends on 10) The method of claim 10, wherein if the comparison fails for the first scanline for a particular macroblock in the row of macroblocks, the comparison of the second scanline is skipped for the particular macroblock when comparing the second scanlines for the rows of macroblocks. ### Claim 12 (depends on 7) The method of claim 7, wherein the identification of whether each macroblock is a ZMV macroblock is stored in metadata associated with the frame. ### Claim 13 A non-transitory computer readable media comprising instructions for efficient frame processing in a virtual mobile device platform, including instructions for: at a virtual mobile device platform coupled to a physical mobile device over a network, executing a virtual mobile device on a virtual machine, the virtual mobile device associated with the physical mobile device communicating with the virtual mobile device platform over the network, the virtual mobile device including a guest operating system (OS) and one or more applications executing on the guest OS, the guest OS generating a frame of display data from an application executing on the guest OS; generating a converted frame by an input/output (I/O) thread of a video encoder by performing color space conversion on the frame of display data generated by the guest OS; generating an encoded frame at a display encoder of the video encoder by encoding the converted frame generated by the I/O thread, wherein the I/O thread includes a zero motion vector (ZMV) detector to identify, for each macroblock of the frame, whether the macroblock is a ZMV macroblock by comparing the frame to a previous frame; and sending, by the virtual mobile device platform, the encoded frame to the physical mobile device. ### Claim 14 (depends on 13) The computer readable media of claim 13, wherein the frame is stored in a first buffer and the previous frame is stored in a second buffer and the comparison of the frame to the previous frame is integrated with a copy of the frame from the first buffer to the second buffer. ### Claim 15 (depends on 14) The computer readable media of claim 14, wherein each macroblock of the frame is copied from the first buffer to the second buffer if the macroblock is not identified as a ZMV macroblock. ### Claim 16 (depends on 13) The computer readable media of claim 13, wherein comparing the frame to the previous frame comprises comparing each macroblock of the frame to a corresponding macroblock of the previous frame, and during the comparing of the frame to the previous frame a first scanline for a row of macroblocks of the frame is compared to a corresponding first scanline for a corresponding row of macroblocks of the previous frame in an X direction before a second scanline for the row of macroblocks of the frame is compared to a corresponding second scanline for the corresponding row of macroblocks of the previous frame. ### Claim 17 (depends on 16) The computer readable media of claim 16, wherein if the comparison fails for the first scanline for a particular macroblock in the row of macroblocks, the comparison of the second scanline is skipped for the particular macroblock when comparing the second scanlines for the rows of macroblocks. ### Claim 18 (depends on 13) The computer readable media of claim 13, wherein the identification of whether each macroblock is a ZMV macroblock is stored in metadata associated with the frame.