# US 11,299,170 B2 · Passenger support device, passenger support method and program storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,299,170 B2
- Document ID: US11299170B2
- Status: Granted
- Kind: B2
- Application number: 16/266,443
- Filed: 2019-02-04
- Published: 2019-08-08
- Granted: 2022-04-12
- Earliest claimed: 2018-02-06
- Assignees: Toyota Jidosha Kabushiki Kaisha
- Applicants: Toyota Jidosha Kabushiki Kaisha
- Inventors: Akira Sasaki; Eisuke Ando; Misako Hayashima; Saki Kitoh; Masahiro Yamashita
- Web page: https://patentgrep.com/patent/US11299170B2

## CPC

- B60W 50/0098: {Details of control systems ensuring comfort, safety or stability not otherwise provided for}
- B25J 11/00: Manipulators not otherwise provided for
- B25J 11/008: {Manipulators for service tasks}
- B60K 35/10: Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
- B60K 35/22: Display screens
- B60K 35/26: using acoustic output
- B60K 35/28: characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
- B60K 35/29: Instruments characterised by the way in which information is handled, e.g. showing information on plural displays or prioritising information according to driving conditions
- B60K 35/53: Movable instruments, e.g. slidable
- B60W 50/10: Interpretation of driver requests or demands
- G05D 1/00: Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots (drive control systems specially adapted for autonomous road vehicles B60W60/00)
- B60K 2360/161: Explanation of functions, e.g. instructions
- B60K 2360/175: Autonomous driving
- B60K 2360/1868: according to driving situations
- B60W 2710/30: Auxiliary equipments
- B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})
- B60K: ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

There is provided a passenger support device including (i) an informing portion that sets behavior items to be performed by a passenger by a time a vehicle arrives at a destination and sets an order of performing the behavior items, the informing portion informing the passenger of the behavior items to be performed based on the order, (ii) an action recognition portion that recognizes an action of the passenger performing the behavior items, and (iii) an action support portion that supports the action of the passenger performing the behavior items, based on each of the behavior items and the action of the passenger recognized by the action recognition portion.

## Claims

### Claim 1

A passenger support device comprising:
a camera;
a processor; and
memory storing instructions that, when executed by the processor, cause the processor to:
set behavior items to be performed by a passenger by a time a vehicle arrives at a destination, set an order of performing the behavior items, and inform the passenger of the behavior items to be performed based on the order;
determine that the behavior items cannot be performed by the time the vehicle arrives at the destination;
request that the passenger confirm a behavior item can be omitted;
remove the confirmed behavior item from the behavior items to be performed;
analyze, using artificial intelligence, actions of the passenger performing the behavior items from images captured by the camera;
predict, using the artificial intelligence and an analysis result, an action of the passenger performing the behavior items;
support the predicted action of the passenger performing the behavior items, based on each of the behavior items;
store a history of behavior items performed, a time required to perform the behavior items, and a frequency of performing the behavior items; and
select the behavior items to be performed in order of the frequency such that performing the behavior items is finished by the time the vehicle arrives at the destination.

### Claim 2 (depends on 1)

The passenger support device according to claim 1, wherein the processor indicates a behavior item to be performed next after performing each of the behavior items is finished.

### Claim 3 (depends on 1)

The passenger support device according to claim 1, wherein the processor indicates each of the behavior items by display on a display medium and by sound output from a sound output portion.

### Claim 4 (depends on 1)

The passenger support device according to claim 1, wherein the processor further: receives input indicating addition, removal, and a change in the order of performing the behavior items, from the passenger; and sets the behavior items to be performed and the order of performing the behavior items based on the received input.

### Claim 5 (depends on 4)

The passenger support device according to claim 4, wherein the processor further acquires a required time for the vehicle to reach the destination, wherein the processor sets the behavior items to be performed such that the behavior items are finished by the required time.

### Claim 6 (depends on 4)

The passenger support device according to claim 4, wherein the processor receives the input indicating addition, the removal, and the order of performing the behavior items on an interactive basis.

### Claim 7 (depends on 1)

The passenger support device according to claim 1, further comprising a robot arm that supports the action of the passenger performing one of the behavior items.

### Claim 8 (depends on 7)

The passenger support device according to claim 7, wherein the robot arm is activated in accordance with the behavior item and the recognized action of the passenger.

### Claim 9 (depends on 1)

The passenger support device according to claim 1, wherein the processor further: receives input indicating a change in the order of performing the behavior items, without addition or removal, from the passenger; and change the order of performing the behavior items based on the received input.

### Claim 10

A passenger support method, comprising:
setting behavior items to be performed by a passenger, by a time a vehicle arrives at a destination and setting an order of performing the behavior items;
informing the passenger of the behavior items to be performed based on the order;
determining that the behavior items cannot be performed by the time the vehicle arrives at the destination;
requesting that the passenger confirm a behavior item can be omitted;
removing the confirmed behavior item from the behavior items to be performed;
analyzing, using artificial intelligence, actions of the passenger performing the behavior items from images captured by a camera;
predicting, using artificial intelligence and an analysis result, an action of the passenger performing the behavior items;
supporting the predicted action of the passenger performing the behavior items, based on each of the behavior items;
storing a history of behavior items performed, a time required to perform the behavior items, and a frequency of performing the behavior items; and
selecting the behavior items to be performed in order of the frequency such that performing the behavior items is finished by the time the vehicle arrives at the destination.

### Claim 11 (depends on 10)

The method according to claim 10, further comprising: receiving input indicating a change in the order of performing the behavior items, without addition or removal, from the passenger; and changing the order of performing the behavior items based on the received input.

### Claim 12

A non-transitory storage medium that stores a program causing a computer to execute a passenger support processing, the processing comprising:
setting behavior items to be performed by a passenger, by a time a vehicle arrives at a destination and setting an order of performing the behavior items;
informing the passenger of the behavior items to be performed based on the order;
determining that the behavior items cannot be performed by the time the vehicle arrives at the destination;
requesting that the passenger confirm a behavior item can be omitted;
removing the confirmed behavior item from the behavior items to be performed;
analyzing, using artificial intelligence, actions of the passenger performing the behavior items from images captured by a camera; predicting, using the artificial intelligence and an analysis result, an action of the passenger performing the behavior items;
supporting the predicted action of the passenger performing the behavior items, based on each of the behavior items;
storing a history of behavior items performed, a time required to perform the behavior items, and a frequency of performing the behavior items; and
selecting the behavior items to be performed in order of the frequency such that performing the behavior items is finished by the time the vehicle arrives at the destination.

### Claim 13 (depends on 12)

The non-transitory storage medium according to claim 12, wherein the processing further comprises: receiving input indicating a change in the order of performing the behavior items, without addition or removal, from the passenger; and changing the order of performing the behavior items based on the received input.