# US 2025/0157172 A1 · Systems and methods for equivarience in three-dimensional (3d) transformations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0157172 A1
- Document ID: US20250157172A1
- Status: Published application
- Kind: A1
- Application number: 18/652,569
- Filed: 2024-05-01
- Published: 2025-05-15
- Earliest claimed: 2023-11-09
- Assignees: Toyota Research Institute, Inc.; Toyota Jidosha Kabushiki Kaisha
- Applicants: Toyota Research Institute, Inc.
- Inventors: YINSHUANG XU; VITOR GUIZILINI; SERGEY ZAKHAROV; RARES A. AMBRUS; ADRIEN D. GAIDON; DIAN CHEN; KATHERINE Y. LIU
- Web page: https://patentgrep.com/patent/US20250157172A1

## CPC

- G06V 10/242: {by image rotation, e.g. by 90 degrees}
- G06V 10/46: Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features (colour feature extraction G06V10/56)
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and methods for enhanced computer vision capabilities, particularly including 3D transformation equivariance, which may be applicable to autonomous vehicle operation are described. A vehicle may be equipped with an 3D transformation equivariance architecture for performing equivariance of image data in the 3D space for image analysis and computer vision functions. The 3D Transformation Equivariance system and method can be configured to replace Fourier positional embedding with spherical harmonics, ensuring equivariance to 3D rotations for the input embedding. Furthermore, the 3D Transformation Equivariance system can be designed with varying architectures that utilize equivariant self-attention and cross-attention modules that are tailer to the spherical harmonics embedding within the general architecture.

## Claims

### Claim 1

A vehicle, comprising:
a processor device receiving image data of one or more objects having rotations in a three-dimensional (3D) space, wherein the 3D space is associated with a surrounding environment for the vehicle; and
a controller device performing equivariance of the image data in the 3D space from the processor device for image analysis and computer vision functions, and performing one or more autonomous operations in response to the image analysis and computer vision functions based on the equivariance.

### Claim 2 (depends on 1)

The vehicle of claim 1, wherein the processor device comprises a 3D transformation equivariance component.

### Claim 3 (depends on 2)

The vehicle of claim 2, wherein the 3D transformation equivariance component comprises spherical harmonics.

### Claim 4 (depends on 3)

The vehicle of claim 3, wherein the spherical harmonics provide equivariance to rotations in the 3D space for input embeddings.

### Claim 5 (depends on 4)

The vehicle of claim 4, wherein the equivariance of the image data comprises equivariance to simultaneous spatial transformations of input and output.

### Claim 6 (depends on 3)

The vehicle of claim 3, wherein the 3D transformation equivariance component performs positional encoding with the spherical harmonics for input ray vectors and the translation of cameras associated with the image data.

### Claim 7 (depends on 2)

The vehicle of claim 2, wherein the image data comprises image embeddings and camera embeddings.

### Claim 8 (depends on 7)

The vehicle of claim 7, wherein the image data is captured by one or more vehicle cameras at the multiple viewpoints associated with rotations in the 3D space.

### Claim 9 (depends on 1)

The vehicle of claim 1, wherein the processor device comprises a computer vision component performing one or more computer visual capabilities for the one or more autonomous operations.

### Claim 10 (depends on 9)

The vehicle of claim 9, wherein the one or more computer visual capabilities comprise object detection.

### Claim 11 (depends on 1)

The vehicle of claim 1, wherein the vehicle comprises an autonomous vehicle.

### Claim 12

A system, comprising:
an equivariant cross-attention module encoding one or more image embeddings and one or more camera embeddings and outputting encoded information, wherein the encoding comprises positional encoding with spherical harmonics for input ray vectors and translation of cameras associated with the one or more image embeddings and the one or more camera embeddings.

### Claim 13 (depends on 12)

The system of claim 12, further comprising:
an equivariant self-attention with Fourier Transform module applying features associated with the one or more image embeddings and the one or more camera embeddings as Fourier coefficients to obtain spherical features, and applying self-attention to the spherical features followed by a Fourier Transform to retrieve equivariant features.

### Claim 14 (depends on 13)

The system of claim 13, further comprising:
an equivariant cross-attention module applying equivariant cross-attention after the self-attention to obtain invariant features for equivariance predictions.

### Claim 15 (depends on 14)

The system of claim 14, further comprising:
an equivariant self-attention without Fourier Transform module obtaining equivariant features without sampling associated with a Fourier Transform.

### Claim 16 (depends on 15)

The system of claim 15, wherein further comprising:
an invariant cross-attention after self-attention module applying invariant cross-attention after the self-attention enabling decoding of high frequency information to obtain invariant features for equivariance predictions.
