# US 11,993,271 B2 · Information processing apparatus, non-transitory storage medium, and information processing method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,993,271 B2 - Document ID: US11993271B2 - Status: Granted - Kind: B2 - Application number: 17/658,452 - Filed: 2022-04-08 - Published: 2022-10-20 - Granted: 2024-05-28 - Earliest claimed: 2021-04-14 - Assignees: Toyota Jidosha Kabushiki Kaisha - Applicants: Toyota Jidosha Kabushiki Kaisha - Inventors: Toyokazu Nakashima; Ryuichi Kamaga; Mitsuhiro Miura; Yasuhiro Baba; Tomokazu Maya; Ryosuke Kobayashi; Genshi Kuno - Web page: https://patentgrep.com/patent/US11993271B2 ## CPC - B60W 40/09: Driving style or behaviour - G06V 20/597: {Recognising the driver's state or behaviour, e.g. attention or drowsiness} - G06V 40/176: {Dynamic expression} - B60W 2420/403: Image sensing, e.g. optical camera - B60W 2540/21: Voice - B60W 2540/22: Psychological state; Stress level or workload - B60W: CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract An information processing apparatus communicable with a vehicle includes a controller. The controller is configured to store behavior information of a user in the vehicle in association with a date and time, acquire an image via an imaging unit of the vehicle when a difference between behavior information at a specific time and behavior information at a time later than the specific time is detected, an store the image in association with the date and time and the behavior information. ## Claims ### Claim 1 An information processing apparatus communicable with a vehicle, the information processing apparatus comprising: at least one processor programmed to: store behavior information of a user in the vehicle in association with a date and time, acquire an image of surroundings of the vehicle via an imaging unit of the vehicle when a difference between behavior information at a specific time and behavior information at a time later than the specific time is detected, and store the image in association with the date and time and the behavior information, wherein the behavior information includes driving information of the vehicle, the driving information includes a destination, a frequency of visits to a specific destination, and a moving route to a specific destination, and the at least one processor is further configured to: determine that the difference between the destination at the specific time and the destination at the time later than the specific time is detected, the destination at the specific time being a destination that is arrived at after the vehicle has travelled thereto, acquire the image of the surroundings of the vehicle when the at least one processor determines that the difference between the destination at the specific time and the destination at the time later than the specific time is detected, determine that the difference between the destination at the specific time and the destination at the time later than the specific time is not detected, the destination at the specific time being the destination that is arrived at after the vehicle has travelled thereto, not acquire the image of the surroundings of the vehicle when the at least one processor determines that the difference between the destination at the specific time and the destination at the time later than the specific time is not detected, determine that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is detected, the frequency of visits to the specific destination at the specific time being a frequency at which the vehicle has travelled to the specific destination that is a destination that is arrived at after the vehicle has travelled thereto, acquire the image of the surroundings of the vehicle when the at least one processor determines that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is detected, determine that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is not detected, the frequency of visits to the specific destination at the specific time being a frequency at which the vehicle has travelled to the specific destination that is the destination that is arrived at after the vehicle has travelled thereto, not acquire the image of the surroundings of the vehicle when the at least one processor determines that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is not detected, determine that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is detected, the moving route to the specific destination at the specific time being a moving route to the specific destination that is a destination that is arrived at after the vehicle has travelled thereto, acquire the image of the surroundings of the vehicle when the at least one processor determines that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is detected, determine that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is not detected, the moving route to the specific destination at the specific time being the moving route to the specific destination that is the destination that is arrived at after the vehicle has travelled thereto, and not acquire the image of the surroundings of the vehicle when the at least one processor determines that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is not detected. ### Claim 2 (depends on 1) The information processing apparatus according to claim 1, wherein the behavior information includes motion information of the user. ### Claim 3 (depends on 2) The information processing apparatus according to claim 2, wherein the motion information includes information about a motion showing an emotion of the user. ### Claim 4 (depends on 1) The information processing apparatus according to claim 1, wherein the at least one processor is further programmed to acquire a voice of the user via a voice input unit of the vehicle when the difference is detected. ### Claim 5 (depends on 1) The information processing apparatus according to claim 1, wherein the at least one processor is further programmed to output an image before the difference and an image after the difference to an output unit of the vehicle when the difference is detected. ### Claim 6 (depends on 1) A vehicle comprising the information processing apparatus according to claim 1. ### Claim 7 A non-transitory storage medium storing instructions that are executable by one or more processors and that cause the one or more processors to perform functions comprising: storing behavior information of a user in a vehicle in association with a date and time; acquiring an image of surroundings of the vehicle via an imaging unit of the vehicle when a difference between behavior information at a specific time and behavior information at a time later than the specific time is detected; and storing the image in association with the date and time and the behavior information, wherein the behavior information includes driving information of the vehicle, the driving information includes a destination, a frequency of visits to a specific destination, and a moving route to a specific destination, and the functions further comprise: determining that the difference between the destination at the specific time and the destination at the time later than the specific time is detected, the destination at the specific time being a destination that is arrived at after the vehicle has travelled thereto, acquiring the image of the surroundings of the vehicle when it is determined that the difference between the destination at the specific time and the destination at the time later than the specific time is detected, determining that the difference between the destination at the specific time and the destination at the time later than the specific time is not detected, the destination at the specific time being the destination that is arrived at after the vehicle has travelled thereto, not acquiring the image of the surroundings of the vehicle when it is determined that the difference between the destination at the specific time and the destination at the time later than the specific time is not detected, determining that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is detected, the frequency of visits to the specific destination at the specific time being a frequency at which the vehicle has travelled to the specific destination that is a destination that is arrived at after the vehicle has travelled thereto, acquiring the image of the surroundings of the vehicle when it is determined that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is detected, determining that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is not detected, the frequency of visits to the specific destination at the specific time being a frequency at which the vehicle has travelled to the specific destination that is the destination that is arrived at after the vehicle has travelled thereto, not acquiring the image of the surroundings of the vehicle when it is determined that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is not detected, determining that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is detected, the moving route to the specific destination at the specific time being a moving route to the specific destination that is a destination that is arrived at after the vehicle has travelled thereto, acquiring the image of the surroundings of the vehicle when it is determined that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is detected, determining that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is not detected, the moving route to the specific destination at the specific time being the moving route to the specific destination that is the destination that is arrived at after the vehicle has travelled thereto, and not acquiring the image of the surroundings of the vehicle when it is determined that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is not detected. ### Claim 8 (depends on 7) The non-transitory storage medium according to claim 7, wherein the behavior information includes motion information of the user. ### Claim 9 (depends on 8) The non-transitory storage medium according to claim 8, wherein the motion information includes information about a motion showing an emotion of the user. ### Claim 10 (depends on 7) The non-transitory storage medium according to claim 7, wherein the functions further include acquiring a voice of the user via a voice input unit of the vehicle when the difference is detected. ### Claim 11 An information processing method executed by an information processing apparatus communicable with a vehicle, the information processing method comprising: storing behavior information of a user in the vehicle in association with a date and time; acquiring an image of surroundings of the vehicle via an imaging unit of the vehicle when a difference between behavior information at a specific time and behavior information at a time later than the specific time is detected; and storing the image in association with the date and time and the behavior information, wherein the behavior information includes driving information of the vehicle, the driving information includes a destination, a frequency of visits to a specific destination, and a moving route to a specific destination, and the method further comprises: determining that the difference between the destination at the specific time and the destination at the time later than the specific time is detected, the destination at the specific time being a destination that is arrived at after the vehicle has travelled thereto, acquiring the image of the surroundings of the vehicle when it is determined that the difference between the destination at the specific time and the destination at the time later than the specific time is detected, determining that the difference between the destination at the specific time and the destination at the time later than the specific time is not detected, the destination at the specific time being the destination that is arrived at after the vehicle has travelled thereto, not acquiring the image of the surroundings of the vehicle when it is determined that the difference between the destination at the specific time and the destination at the time later than the specific time is not detected, determining that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is detected, the frequency of visits to the specific destination at the specific time being a frequency at which the vehicle has travelled to the specific destination that is a destination that is arrived at after the vehicle has travelled thereto, acquiring the image of the surroundings of the vehicle when it is determined that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is detected, determining that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is not detected, the frequency of visits to the specific destination at the specific time being a frequency at which the vehicle has travelled to the specific destination that is the destination that is arrived at after the vehicle has travelled thereto, not acquiring the image of the surroundings of the vehicle when it is determined that the difference between the frequency of visits to the specific destination at the specific time and the frequency of visits to the specific destination at the time later than the specific time is not detected, determining that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is detected, the moving route to the specific destination at the specific time being a moving route to the specific destination that is a destination that is arrived at after the vehicle has travelled thereto, acquiring the image of the surroundings of the vehicle when it is determined that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is detected, determining that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is not detected, the moving route to the specific destination at the specific time being the moving route to the specific destination that is the destination that is arrived at after the vehicle has travelled thereto, and not acquiring the image of the surroundings of the vehicle when it is determined that the difference between the moving route to the specific destination at the specific time and the moving route to the specific destination at the time later than the specific time is not detected. ### Claim 12 (depends on 11) The information processing method according to claim 11, wherein the behavior information includes motion information of the user. ### Claim 13 (depends on 12) The information processing method according to claim 12, wherein the motion information includes information about a motion showing an emotion of the user. ### Claim 14 (depends on 11) The information processing method according to claim 11, further comprising acquiring a voice of the user via a voice input unit of the vehicle when the difference is detected.