# US 2020/0401363 A1 · System, method and computer program product for generating remote views in a virtual mobile device platform using efficient color space conversion and frame encoding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401363 A1
- Document ID: US20200401363A1
- Status: Published application
- Kind: A1
- Application number: 17/008,489
- Filed: 2020-08-31
- Published: 2020-12-24
- Earliest claimed: 2016-07-28
- Assignees: Intelligent Waves LLC
- Applicants: Intelligent Waves LLC
- Inventors: Robert David LEE; Hermal Yogesh Gandhi; Akshay P. Joshi
- Web page: https://patentgrep.com/patent/US20200401363A1

## CPC

- G06F 3/1454: {involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay}
- G06F 9/452: {Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131)}
- G06F 9/45558: {Hypervisor-specific management and integration aspects}
- G06F 21/53: by executing in a restricted environment, e.g. sandbox or secure virtual machine
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00)
- H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
- H04L 67/565: Conversion or adaptation of application format or content (adding application control or application functional data H04L67/561)
- H04N 19/103: Selection of coding mode or of prediction mode
- H04N 19/14: Coding unit complexity, e.g. amount of activity or edge presence estimation (H04N19/146 takes precedence)
- H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/186: the unit being a colour or a chrominance component
- H04N 19/27: involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]
- H04N 19/423: characterised by memory arrangements (H04N19/433 takes precedence)
- H04N 19/503: involving temporal prediction (adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109)
- H04N 19/517: by encoding
- H04N 19/547: Motion estimation performed in a transform domain
- H04N 19/61: in combination with predictive coding
- H04N 19/625: using discrete cosine transform [DCT]
- G06F 2221/032: Protect output to user by software means
- G06F 2221/2149: Restricted operating environment
- G09G 5/12: Synchronisation between the display unit and other units, e.g. other display units, video-disc players
- G09G 2340/02: Handling of images in compressed format, e.g. JPEG, MPEG
- G09G 2340/06: Colour space transformation
- G09G 2350/00: Solving problems of bandwidth in display systems
- G09G 2358/00: Arrangements for display data security
- G09G 2360/121: using a cache memory
- G09G 2360/18: Use of a frame buffer in a display terminal, inclusive of the display panel
- G09G 2370/022: Centralised management of display operation, e.g. in a server instead of locally
- G09G 2370/027: Arrangements and methods specific for the display of internet documents
- H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference
- H04N 19/172: the region being a picture, frame or field
- H04W 88/02: Terminal devices
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

Embodiments disclosed herein provide systems, methods and computer readable media for generating remote views in a virtual mobile device platform. A virtual mobile device platform may be coupled to a physical mobile device over a network and generate frames of data for generating views on the physical device. These frames can be generated using an efficient display encoding pipeline on the virtual mobile device platform. Such efficiencies may include, for example, the synchronization of various processes or operations, the governing of various processing rates, the elimination of duplicative or redundant processing, the application of different encoding schemes, the efficient detection of duplicative or redundant data or the combination of certain operations.

## Claims

### Claim 1

A system for a virtual mobile device platform with efficient frame processing, comprising:
a virtual mobile device platform coupled to a physical mobile device over a network, the virtual mobile device platform including a processor executing instructions on a non-transitory computer readable medium for implementing:
a virtual machine executing a virtual mobile device associated with a physical mobile device communicating with the virtual mobile device platform over the network, the virtual mobile device including a guest operating system (OS) and one or more applications executing on the guest OS, the guest OS generating a frame of display data from an application executing on the guest operating system;
a video encoder including an input/output (I/O) thread for generating a converted frame by performing color space conversion on the frame of display data generated by the guest OS and a display encoder for generating an encoded frame by encoding the converted frame generated by the I/O thread, wherein the I/O thread includes a zero motion vector (ZMV) detector to identify, for each macroblock of the frame, whether the macroblock is a ZMV macroblock, and performing color space conversion based on the identification of the macroblock; and
sending, by the virtual mobile device platform, the encoded frame to the physical mobile device.

### Claim 2 (depends on 1)

The system of claim 1, wherein performing color space conversion of the frame based on the identification of the macroblock comprises performing color space conversion only on macroblocks of the frame not identified as ZMV macroblocks.

### Claim 3 (depends on 1)

The system of claim 1, wherein the converted frame may be encoded based on the identification of macroblocks of the frame, and the encoding of the converted frame comprises only encoding macroblocks of the converted frame not identified as ZMV macroblocks.

### Claim 4 (depends on 1)

The system of claim 1, wherein the identification of whether each macroblock is a ZMV macroblock is stored in metadata associated with the frame.

### Claim 5

A system for a virtual mobile device platform with efficient frame processing, comprising:
a virtual mobile device platform coupled to a physical mobile device over a network, the virtual mobile device platform including a processor executing instructions on a non-transitory computer readable medium for implementing:
a virtual machine executing a virtual mobile device associated with a physical mobile device communicating with the virtual mobile device platform over the network, the virtual mobile device including a guest operating system (OS) and one or more applications executing on the guest OS, the guest OS generating a frame of display data from an application executing on the guest operating system;
a video encoder including an input/output (I/O) thread for generating a converted frame by performing color space conversion on the frame of display data generated by the guest OS and a display encoder for generating an encoded frame by encoding the converted frame generated by the I/O thread, wherein the I/O thread includes a content type detector to identify, for each macroblock of the frame, whether the macroblock is a high-frequency macroblock or a low-frequency macroblock, and each macroblock is encoded by the display encoder based on the identification of the macroblock; and
sending, by the virtual mobile device platform, the encoded frame to the physical mobile device.

### Claim 6 (depends on 5)

The system of claim 5, wherein the identification of each macroblock as a high-frequency macroblock or a low-frequency macroblock is stored in metadata associated with the frame.

### Claim 7 (depends on 6)

The system of claim 6, wherein the display encoder includes a high-frequency encoder and a standard encoder, and encoding the converted frame comprises encoding macroblocks of the converted frame identified as high-frequency macroblocks with the high-frequency encoder and encoding macroblocks of the converted frame identified as low-frequency macroblocks with the standard encoder.

### Claim 8 (depends on 7)

The system of claim 7, wherein the high-frequency encoder is a zip encoder and the low-frequency encoder is a H.264 encoder.

### Claim 9 (depends on 5)

The system of claim 5, wherein identifying a macroblock as a high-frequency macroblock or a low-frequency macroblock comprises determining a spectral difference between an upper half of the macroblock and a lower half of the macroblock and identifying the macroblock as a high-frequency macroblock or a low-frequency macroblock-based on the spectral difference.

### Claim 10 (depends on 9)

The system of claim 9, wherein determining the spectral difference comprises performing a zig-zag scan to determine a Discrete Cosine Transform (DCT) coefficient array representing a spectral energy of the macroblock.

### Claim 11 (depends on 10)

The system of claim 10, wherein determining the spectral difference comprises determining a low frequency energy from the DCT coefficient array based on low frequency coefficients of the DCT coefficient array and a high frequency energy based on high frequency coefficients of the DCT coefficient array, and comparing the low frequency energy with the high frequency energy to determine the spectral difference.

### Claim 12

A method for efficient frame processing in a virtual mobile device platform, comprising:
at a virtual mobile device platform coupled to a physical mobile device over a network,
executing a virtual mobile device on a virtual machine, the virtual mobile device associated with a physical mobile device communicating with the virtual mobile device platform over the network, the virtual mobile device including a guest operating system (OS) and one or more applications executing on the guest OS, the guest OS generating a frame of display data from an application executing on the guest OS;
generating a converted frame by an input/output (I/O) thread of a video encoder by performing color space conversion on the frame of display data generated by the guest OS;
generating an encoded frame at a display encoder of the video encoder by encoding the converted frame generated by the I/O thread, wherein the I/O thread includes a zero motion vector (ZMV) detector to identify, for each macroblock of the frame, whether the macroblock is a ZMV macroblock, and performing color space conversion based on the identification of the macroblock; and
sending, by the virtual mobile device platform, the encoded frame to the physical mobile device.

### Claim 13 (depends on 12)

The method of claim 12, wherein performing color space conversion of the frame based on the identification of the macroblock comprises performing color space conversion only on macroblocks of the frame not identified as ZMV macroblocks.

### Claim 14 (depends on 12)

The method of claim 12, wherein the converted frame may be encoded based on the identification of macroblocks of the frame, and the encoding of the converted frame comprises only encoding macroblocks of the converted frame not identified as ZMV macroblocks.

### Claim 15 (depends on 12)

The method of claim 12, wherein the identification of whether each macroblock is a ZMV macroblock is stored in metadata associated with the frame.
