# US 2022/0046321 A1 · Pipelined Video Interface for Remote Controlled Aerial Vehicle with Camera

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0046321 A1
- Document ID: US20220046321A1
- Status: Published application
- Kind: A1
- Application number: 17/404,764
- Filed: 2021-08-17
- Published: 2022-02-10
- Earliest claimed: 2015-06-30
- Assignees: GoPro, Inc.
- Applicants: GoPro, Inc.
- Inventors: Joseph Anthony Enke; Anupam Joshi
- Web page: https://patentgrep.com/patent/US20220046321A1

## CPC

- H04N 21/440281: {by altering the temporal resolution, e.g. by frame skipping (television signal recording using magnetic recording on tape for reproducing at a rate different from the recording rate H04N5/783)}
- B64U 10/14: with four distinct rotor axes, e.g. quadcopters
- B64U 20/87: Mounting of imaging devices, e.g. mounting of gimbals
- B64U 30/291: Detachable rotors or rotor supports
- B64U 30/293: Foldable or collapsible rotors or rotor supports
- B64U 60/40: foldable or retractable
- H04L 65/762: {at the source (reformatting of additional data in video distribution servers H04N21/2355)}
- H04N 7/185: {from a mobile camera, e.g. for remote control}
- H04N 21/43637: {involving a wireless protocol, e.g. Bluetooth®, RF or wireless LAN [IEEE 802.11] (arrangements for wireless networking or broadcasting of information in indoor or near-field type systems H04B10/114)}
- H04N 23/54: Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
- H04N 23/60: Control of cameras or camera modules
- H04N 23/661: Transmitting camera control signals through networks, e.g. control via the Internet
- H04N 23/80: Camera processing pipelines; Components thereof
- B64U 60/10: specially adapted for use on water
- B64U 2101/30: for imaging, photography or videography
- B64U 2201/20: Remote controls
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- B64U: UNMANNED AERIAL VEHICLES [UAV]; EQUIPMENT THEREFOR
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Disclosed is a system and method for reducing the total latency for transferring a frame from the low latency camera system mounted on an aerial vehicle to the display of the remote controller. The method includes reducing the latency through each of the modules of the system, i.e. through a camera module, an encoder module, a wireless interface transmission, wireless interface receiver module, a decoder module and a display module. To reduce the latency across the modules, methods such as overclocking the image processor, pipelining the frame, squashing the processed frame, using a fast hardware encoder that can perform slice based encoding, tuning the wireless medium using queue sizing, queue flushing, bitrate feedback, physical medium rate feedback, dynamic encoder parameter tuning and wireless radio parameter adjustment, using a fast hardware decoder that can perform slice based decoding and overclocking the display module are used.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A method, comprising:
overclocking a pipeline of an image processor of a low latency camera system;
forwarding a processed frame from a first pipeline stage to a second pipeline stage of the image processor;
responsive to detecting an interface associated with the low latency camera system, squashing image data of the processed frame, wherein the squashing comprises aligning an external interface in the pipeline to when data is ready from the pipeline to avoid buffering, wherein a clock of the interface is distinct from a clock of the image processor;
encoding the processed frame; and
transmitting the encoded frame.

### Claim 22 (depends on 21)

The method of claim 21, wherein the processed frame is processed via one or more image processing operations including at least one of sharpening, compression, color conversion, performing a Bayer transformation, demosaicing, noise reduction, image stabilization, or rolling shutter artifact reduction.

### Claim 23 (depends on 22)

The method of claim 22, wherein the one or more image processing operations are performed on one or more unprocessed frames in each pipeline stage of the image processor.

### Claim 24 (depends on 23)

The method of claim 23, wherein the overclocking drops a frame of the one or more unprocessed frames to increase an operating frequency of the image processor.

### Claim 25 (depends on 21)

The method of claim 21, wherein squashing the image data of the processed frame reduces latency by one or more frames.

### Claim 26 (depends on 21)

The method of claim 21, wherein squashing the image data of the processed frame reduces a size of the image data without losing image information.

### Claim 27 (depends on 21)

The method of claim 21, comprising:
storing the processed frame in a memory of the low latency camera system.

### Claim 28 (depends on 21)

The method of claim 21, comprising:
preprocessing the processed frame via one or more preprocessing operations including at least one of scaling, resizing, or cropping.

### Claim 29 (depends on 28)

The method of claim 28, wherein encoding the processed frame comprises:
performing one or more image encoding operations on the preprocessed frame, the one or more image encoding operations including slice based encoding that encodes input frames when a slice of the processed frame is available to reduce the latency associated with an input buffer.

### Claim 30

A non-transitory computer readable storage medium configured to store instructions, the instructions when executed by a processor cause the processor to:
overclock a pipeline of an image processor of a low latency camera system;
process a frame via one or more processing operations;
forward the processed frame from a first pipeline stage to a second pipeline stage of the image processor;
responsive to detecting an interface associated with the low latency camera system, squash image data of the processed frame, wherein the squashing comprises aligning an external interface in the pipeline to when data is ready from the pipeline, wherein a clock of the interface is distinct from a clock of the image processor;
encode the processed frame; and
transmit the encoded frame.

### Claim 31 (depends on 30)

The non-transitory computer readable storage medium of claim 30, wherein the one or more processing operations includes at least one of sharpening, compression, color conversion, performing a Bayer transformation, demosaicing, noise reduction, image stabilization, or rolling shutter artifact reduction.

### Claim 32 (depends on 31)

The non-transitory computer readable storage medium of claim 31, wherein the one or more processing operations are performed on one or more unprocessed frames in each pipeline stage of the image processor.

### Claim 33 (depends on 32)

The non-transitory computer readable storage medium of claim 32, wherein the overclocking drops a frame of the one or more unprocessed frames to increase an operating frequency of the image processor.

### Claim 34 (depends on 30)

The non-transitory computer readable storage medium of claim 30, wherein the image data of the processed frame is squashed to reduce latency by one or more frames.

### Claim 35 (depends on 30)

The non-transitory computer readable storage medium of claim 30, wherein the image data of the processed frame is squashed to reduce a size of the image data without losing image information.

### Claim 36 (depends on 30)

The non-transitory computer readable storage medium of claim 30, the instructions when executed by the processor causing the processor to:
store the processed frame in a memory of the low latency camera system.

### Claim 37 (depends on 30)

The non-transitory computer readable storage medium of claim 30, wherein the overclocking increases idle time of the image processor.

### Claim 38 (depends on 30)

The non-transitory computer readable storage medium of claim 30, the instructions when executed by the processor causing the processor to:
preprocess the processed frame via one or more preprocessing operations including at least one of scaling, resizing, or cropping.

### Claim 39 (depends on 38)

The non-transitory computer readable storage medium of claim 38, wherein to encode the processed frame by an encoder comprises to:
perform one or more image encoding operations on the preprocessed frame, the one or more image encoding operations including slice based encoding that encodes input frames when a slice of the processed frame is available to reduce the latency associated with an input buffer.

### Claim 40

A low latency camera system, comprising:
a memory; and
a processor configured to execute instructions stored on the memory to:
overclock an image processor pipeline,
forward a frame from a first pipeline stage to a second pipeline stage,
detect an interface associated with the low latency camera system, and
squash image data of the frame, wherein the squashing comprises aligning an external interface in the image processor pipeline to when data is ready from the image processor pipeline; and
an encoder configured to encode the frame prior to transmission.
