# US 2023/0065931 A1 · Projecting images captured using fisheye lenses for feature detection in autonomous machine applications

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0065931 A1
- Document ID: US20230065931A1
- Status: Published application
- Kind: A1
- Application number: 18/053,512
- Filed: 2022-11-08
- Published: 2023-03-02
- Earliest claimed: 2020-04-06
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Karsten Patzwaldt
- Web page: https://patentgrep.com/patent/US20230065931A1

## CPC

- G06T 7/73: using feature-based methods
- G01C 21/265: {constructional aspects of navigation devices, e.g. housings, mountings, displays (G01C21/3688 takes precedence)}
- G05D 1/0221: {involving a learning process}
- G05D 1/0246: {using a video camera in combination with image processing means}
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06V 10/147: Details of sensors, e.g. sensor lenses (fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19)
- G06V 10/24: Aligning, centring, orientation detection or correction of the image
- G06V 10/40: Extraction of image or video features
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 10/94: Hardware or software architectures specially adapted for image or video understanding
- G06V 20/56: exterior to a vehicle by using sensors mounted on the vehicle
- H04N 5/2628: {Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation}
- H04N 23/698: for achieving an enlarged field of view, e.g. panoramic image capture
- G06N 3/045: Combinations of networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T 2207/30261: Obstacle
- G06V 10/247: {by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01C: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY (measuring liquid level G01F; radio navigation, determining distance or velocity by use of propagation effects, e.g. Doppler effects, propagation time, of radio waves, analogous arrangements using other waves G01S)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

In various examples, live perception from wide-view sensors may be leveraged to detect features in an environment of a vehicle. Sensor data generated by the sensors may be adjusted to represent a virtual field of view different from an actual field of view of the sensor, and the sensor data—with or without virtual adjustment—may be applied to a stereographic projection algorithm to generate a projected image. The projected image may then be applied to a machine learning model—such as a deep neural network (DNN)—to detect and/or classify features or objects represented therein. In some examples, the machine learning model may be pre-trained on training sensor data generated by a sensor having a field of view less than the wide-view sensor such that the virtual adjustment and/or projection algorithm may update the sensor data to be suitable for accurate processing by the pre-trained machine learning model.

## Claims

### Claim 1

A method comprising:
virtually rotating, based at least on one or more orientations of one or more image sensors, one or more fields of view corresponding to the one or more image sensors to generate one or more images having one or more adjusted fields of view;
projecting the one or more images onto one or more planes to generate one or more projected images;
computing, using one or more neural networks and based at least on at least one projected image of the one or more projected images, one or more outputs; and
performing one or more operations using one or more machines based at least on the one or more outputs.

### Claim 2 (depends on 1)

The method of claim 1, wherein the projecting is performed using one or more of a stereographic projection algorithm or a gnomonic projection algorithm.

### Claim 3 (depends on 1)

The method of claim 1, wherein the virtually rotating aligns one or more locations corresponding to the one or more image sensors with one or more identified regions in the one or more fields of view.

### Claim 4 (depends on 3)

The method of claim 3, wherein the one or more locations include a virtual center corresponding to the one or more image sensors, and at least one identified region of the one or more identified regions corresponds to a horizon in the one or more fields of view.

### Claim 5 (depends on 1)

The method of claim 1, wherein the virtually rotating includes virtually rotating the one or more fields of view by one or more angles determined based at least on one or more mounting angles of the one or more image sensors.

### Claim 6 (depends on 1)

The method of claim 1, wherein the virtually rotating includes virtually rotating the one or more fields of view along at least one or more vertical directions by one or more predetermined degrees.

### Claim 7 (depends on 1)

The method of claim 1, wherein the virtually rotating includes virtually rotating one or more virtual spheres associated with the one or more fields of view by one or more degrees.

### Claim 8 (depends on 1)

The method of claim 1, wherein the one or more neural networks are trained using training image data representative of images generated using one or more second image sensors having one or more second fields of view that are less than the one or more fields of view.

### Claim 9 (depends on 1)

The method of claim 1, wherein the one or more machines include a vehicle, and the one or more operations correspond to navigating the vehicle along a driving surface.

### Claim 10

A system comprising:
one or more processing devices to:
generate updated image data corresponding to one or more first fields of view, at least, by virtually rotating one or more second fields of view corresponding to one or more image sensors based at least on one or more orientations of the one or more image sensors;
generate projected image data representative of one or more projected images based at least on applying the updated image data to one or more image projection algorithms;
determine, using one or more machine learning models (MLMs) and based at least on the projected image data, output data; and
perform one or more operations using one or more machines based at least on the output data.

### Claim 11 (depends on 10)

The system of claim 10, wherein the one or more image projection algorithms include one or more of a stereographic projection algorithm or a gnomonic projection algorithm.

### Claim 12 (depends on 10)

The system of claim 10, wherein the virtually, rotating includes virtually rotating the one or more second fields of view to substantially align one or more locations corresponding to the one or more image sensors with one or more regions in the one or more second fields of view.

### Claim 13 (depends on 10)

The system of claim 10, wherein the virtually rotating includes virtually rotating the one or more second fields of view by one or more angles that correspond to one or more mounting angles of the one or more image sensors.

### Claim 14 (depends on 10)

The system of claim 10, wherein the virtually rotating includes virtually rotating the one or more second fields of view along at least one or more vertical directions by one or more degrees.

### Claim 15 (depends on 10)

The system of claim 10, wherein the system is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing ray-tracing operations;
a system for performing deep learning operations;
a system implemented using a robot;
a system for presenting at least one of virtual reality content or augmented reality content; or
a system implemented at least partially using cloud computing resources.

### Claim 16

A processor comprising:
one or more circuits to perform one or more operations using one or more machines based at least on one or more outputs, the one or more outputs computed based at least on one or more machine learning models (MLMs) processing data representative of at least one projected image generated, at least, by virtually rotating one or more fields of view corresponding to one or more image sensors toward a horizon.

### Claim 17 (depends on 16)

The processor of claim 16, wherein the at least one projected image is further generated, at least, by using one or more of a stereographic projection algorithm or a gnomonic projection algorithm.

### Claim 18 (depends on 16)

The processor of claim 16, wherein the virtually rotating includes virtually rotating the one or more fields of view to substantially align one or more locations corresponding to the one or more image sensors with one or more regions in the one or more fields of view, the one or more regions including, at least, the horizon.

### Claim 19 (depends on 16)

The processor of claim 16, wherein the virtually rotating includes virtually rotating the one or more fields of view by one or more angles that correspond to one or more mounting angles of the one or more image sensors.

### Claim 20 (depends on 16)

The processor of claim 16, wherein the processor is comprised in at least one of:
a control system for an autonomous or semi-autonomous machine;
a perception system for an autonomous or semi-autonomous machine;
a system for performing simulation operations;
a system for performing ray-tracing operations;
a system for performing deep learning operations;
a system implemented using a robot;
a system for presenting at least one of virtual reality content or augmented reality content; or
a system implemented at least partially using cloud computing resources.
