US 12,738,176 B2Granted

Systems and methods for training a scene simulator using real and simulated agent data

AssigneesToyota Research Institute, Inc., Toyota Jidosha Kabushiki Kaisha, Massachusetts Institute of Technology

InventorsTsun-Hsuan Wang, Alexander Amini, Wilko Schwarting, Igor Gilitschenski, Sertac Karaman, Daniela Rus

Life of this patent

Estimating the term…

Loading patent…

Abstract

System, methods, and other embodiments described herein relate to training a scene simulator for rendering 2D scenes using data from real and simulated agents. In one embodiment, a method includes acquiring trajectories and three-dimensional (3D) views for multiple agents from observations of real vehicles. The method also includes generating a 3D scene having the multiple agents using the 3D views and information from simulated agents. The method also includes training a scene simulator to render scene projections using the 3D scene. The method also includes outputting a 2D scene having simulated observations for a driving scene using the scene simulator.

Claims20

Dependent claims

A learning system, comprising:

a processor;
and
a memory storing instructions that, when executed by the processor, cause the processor to:
acquire trajectories and three-dimensional (3D) views for multiple agents from observations of real vehicles;
generate a 3D scene having the multiple agents by casting ambient light within the 3D views to the real vehicles associated with an infinite distance with the multiple agents and information from simulated agents, wherein the ambient light is associated with an average color of the 3D scene;
train a scene simulator with a policy to render scene projections using the 3D scene with a foreground mask, and a reinforcement model computes the policy and increases a reward for a driving task associated with an ego vehicle within a driving scene that is using the policy;
and
output a 2D scene having simulated observations for the driving scene using the scene simulator.

5 more elements

A non-transitory computer-readable medium comprising:

instructions that when executed by a processor cause the processor to:
acquire trajectories and three-dimensional (3D) views for multiple agents from observations of real vehicles;
generate a 3D scene having the multiple agents by casting ambient light within the 3D views to the real vehicles associated with an infinite distance with the multiple agents and information from simulated agents, wherein the ambient light is associated with an average color of the 3D scene;
train a scene simulator with a policy to render scene projections using the 3D scene with a foreground mask, and a reinforcement model computes the policy and increases a reward for a driving task associated with an ego vehicle within a driving scene that is using the policy;
and
output a 2D scene having simulated observations for the driving scene using the scene simulator.

3 more elements

Dependencies are inferred from the claim text.