# US 11,310,513 B2 · Apparatus and method for enhancing motion estimation based on user input > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,310,513 B2 - Document ID: US11310513B2 - Status: Granted - Kind: B2 - Application number: 16/893,187 - Filed: 2020-06-04 - Published: 2020-09-24 - Granted: 2022-04-19 - Earliest claimed: 2012-12-19 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Helix He - Web page: https://patentgrep.com/patent/US11310513B2 ## CPC - H04N 19/162: User input - H04N 19/56: Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search - H04N 19/61: in combination with predictive coding - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract A method and apparatus for enhancing motion estimation based on user input are provided. The motion estimation apparatus used for video encoding comprises a receiver operable to receive a user based input and an input analysis module operable to analyzed the user based input. The apparatus also comprises an encoder that is operable to compute displacement coordinates from the analyzed user based input for a current block in a target frame of a video stream and operable to determine a search area in a reference frame to search for a best match for the current block using the displacement coordinates. The encoder can also comprise a block match module operable to find a best match block for the current block in the search area of the reference frame using a block matching procedure. ## Claims ### Claim 1 A method comprising: determining, based at least in part on input data representative of a user-based input, an input value; computing, based at least in part on the input value, displacement coordinates for a current block within a target frame of a video stream with respect to a reference frame of the video stream; based at least in part on the displacement coordinates, executing a search within the reference frame for a best match block corresponding to the current block within the target frame; and determining a prediction error based at least in part on computing a difference between the current block within the target frame and the best match block within the reference frame identified based at least in part on the search. ### Claim 2 (depends on 1) The method of claim 1, wherein the search begins at an initial search location determined using the displacement coordinates. ### Claim 3 (depends on 1) The method of claim 1, wherein the executing the search includes using a block matching procedure. ### Claim 4 (depends on 3) The method of claim 3, wherein the block matching procedure includes at least one of: full search, cross search, spiral search, three step search, four step search, orthogonal search, hierarchical search, or diamond search. ### Claim 5 (depends on 1) The method of claim 1, further comprising: calculating a motion vector for the current block using a first position of the current block and a second position of the best match block; and encoding the current block based at least in part on the motion vector and the prediction error to generate an encoded current block. ### Claim 6 (depends on 1) The method of claim 1, further comprising: encoding the current block based at least in part on the prediction error to generate an encoded current block; and transmitting the encoded current block to a remote computer for decoding to recover the current block. ### Claim 7 (depends on 1) The method of claim 1, further comprising converting the input value to a device agnostic input value based at least in part on a type of input device used to generate the user-based input, wherein the computing the displacement coordinates is based at least in part on the device agnostic input value. ### Claim 8 (depends on 1) The method of claim 1, further comprising receiving the input data comprises over a communication interface from a remote computer. ### Claim 9 (depends on 1) The method of claim 1, further comprising generating the video stream using a virtual graphics processing unit (vGPU). ### Claim 10 A system comprising: a receiver to receive input data representative of a user-based input to an input device; an input analyzer to determine an input value corresponding to the user-based input; and an encoder to: compute, based at least in part on the input value, displacement coordinates for a current block within a target frame of a video stream with respect to a reference frame of the video stream; based at least in part on the displacement coordinates, execute a search within the reference frame for a best match block corresponding to the current block within the target frame; determine a prediction error based at least in part on computing a difference between the current block within the target frame and the best match block within the reference frame; and encode the current block based at least in part on the prediction error. ### Claim 11 (depends on 10) The system of claim 10, wherein the search begins at an initial search location determined using the displacement coordinates. ### Claim 12 (depends on 10) The system of claim 10, wherein the executing the search includes using a block matching procedure. ### Claim 13 (depends on 10) The system of claim 10, wherein the encoder is further to: calculate a motion vector for the current block using a first position of the current block and a second position of the best match block, wherein the encoding the current block is further based at least in part on the motion vector. ### Claim 14 (depends on 10) The system of claim 10, further comprising a transmitter to transmit the current block after the encoding to a remote computer for decoding thereof. ### Claim 15 (depends on 10) The system of claim 10, wherein the encoder is implemented using a virtual graphics processing unit (vGPU). ### Claim 16 (depends on 10) The system of claim 10, wherein the input analyzer is further to convert the input value to a device agnostic input value based at least in part on a type of the input device. ### Claim 17 A method comprising: receiving, from a remote client device, input data representative of a user-based input to an input device communicatively coupled to the remote client device; determining an input value based at least in part on input data; computing, based at least in part on the input value, displacement coordinates for a current block within a target frame of a video stream with respect to a reference frame of the video stream; based at least in part on the displacement coordinates, executing a search within the reference frame for a best match block corresponding to the current block within the target frame; encoding the current block to generate an encoded current block by, at least in part, determining a prediction error corresponding to a difference between the current block within the target frame and the best match block within the reference frame; and transmitting data representative of at least the encoded current block to the remote client device. ### Claim 18 (depends on 17) The method of claim 17, further comprising: calculating a motion vector for the current block using a first position of the current block and a second position of the best match block, wherein the encoding the current block is further based at least in part on the motion vector. ### Claim 19 (depends on 17) The method of claim 17, wherein the search begins at an initial search location determined using the displacement coordinates. ### Claim 20 (depends on 17) The method of claim 17, further comprising converting the input value to a device agnostic input value based at least in part on a type of input device used to generate the user-based input, wherein the computing the displacement coordinates is based at least in part on the device agnostic input value.