# US 2025/0094731 A1 · Geometrically grounded large language models for zero-shot human activity forecasting in human-aware task planning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094731 A1
- Document ID: US20250094731A1
- Status: Published application
- Kind: A1
- Application number: 18/614,200
- Filed: 2024-03-22
- Published: 2025-03-20
- Earliest claimed: 2023-09-14
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co, Ltd.
- Inventors: Moritz Alexander GRAULE; Ibrahim Volkan ISLER
- Web page: https://patentgrep.com/patent/US20250094731A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G05D 1/225: operated by off-board computers
- G05D 1/249: from positioning sensors located off-board the vehicle, e.g. from cameras
- G05D 1/645: {Vehicle-induced nuisance abatement, e.g. minimising noise or visual impact}
- G05D 2101/10: using artificial intelligence [AI] techniques
- G05D 2105/10: for cleaning, vacuuming or polishing
- G05D 2107/40: Indoor domestic environment
- G05D 2109/10: Land vehicles
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G05D: SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES

## Abstract

A method includes receiving data from one or more sensors, detecting, from the received data, one or more activities of a person in a space, converting the detected one or more activities into a natural language narration; extracting one or more items from a map corresponding to the space; determining a relevancy score for each of the extracted one or more items based on an output of a language model that receives as input a combination of the narration with a binding sequence; correlating the extracted one or more items with one or more locations on the map based on the determined relevancy for each of the extracted one or more items; and outputting a control signal for controlling a movement of one more devices based on the correlation of the extracted one or more items with the one or more locations on the map.

## Claims

### Claim 1

A method performed by at least one processor, the method comprising:
receiving data from one or more sensors corresponding to a first time period;
detecting, from the received data, one or more activities of a person in a space;
converting the detected one or more activities into a natural language narration;
combining the natural language narration with sequence text that corresponds to an action that is sequential to the detected one or more activities;
extracting one or more items from a map corresponding to the space;
determining a relevancy score for each of the extracted one or more items based on an output of a language model that receives as input the combination of the narration with the sequence text;
correlating the extracted one or more items with one or more locations on the map based on the determined relevancy for each of the extracted one or more items; and
outputting a control signal for controlling a movement of one more devices based on the correlation of the extracted one or more items with the one or more locations on the map.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the correlating the extracted one or more items with the one or more locations on the map further comprises:
aggregating the relevancy score of each of the one or more items belonging to a same partition in the map;
wherein a first partition in the map having a higher aggregated relevancy score than a second partition in the map represents a higher likelihood that the person will be located in the first partition than the second partition during the second time period.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the outputting the control signal for controlling the movement of the one or more devices is based on the aggregated relevancy score of each partition in the map.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the correlating the extracted one or more items with the one or more locations on the map further comprises:
determining, for each path from the person's current location to the one or more locations on the map, a likelihood that a respective path will be traversed.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the map includes, for each extracted item, a semantic label, a position, and a bounding box.

### Claim 6 (depends on 1)

The method according to claim 1, wherein each of the extracted one or more items are located at a distance from the person that is less than or equal to a distance threshold.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the language model is a large language model (LLM) trained on a text corpus.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the one or more devices comprises an autonomous robot.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the outputting the control signal for controlling the movement of the one or more devices further comprises:
causing the autonomous robot to avoid the person during the second time period.

### Claim 10 (depends on 8)

The method according to claim 8, wherein the outputting the control signal for controlling the movement of the one or more devices further comprises:
causing the autonomous robot to assist the person during the second time period.

### Claim 11

An apparatus comprising:
a memory;
processing circuitry coupled to the memory, the processing circuitry configured to:
receive data from one or more sensors corresponding to a first time period,
detect, from the received data, one or more activities of a person in a space,
convert the detected one or more activities into a natural language narration;
combine the natural language narration with sequence text that corresponds to an action that is sequential to the detected one or more activities,
extract one or more items from a map corresponding to the space,
determine a relevancy score for each of the extracted one or more items based on an output of a language model that receives as input the combination of the narration with the sequence text,
correlate the extracted one or more items with one or more locations on the map based on the determined relevancy for each of the extracted one or more items, and
a control signal for controlling a movement of one more devices based on the correlation of the extracted one or more items with the one or more locations on the map.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein the correlation of the extracted one or more items with the one or more locations on the map comprises the processing circuitry further configured to:
aggregate the relevancy score of each of the one or more items belonging to a same partition in the map,
wherein a first partition in the map having a higher aggregated relevancy score than a second partition in the map represents a higher likelihood that the person will be located in the first partition than the second partition during the second time period.

### Claim 13 (depends on 12)

The apparatus according to claim 12, wherein the control of the operation of the one or more devices is based on the aggregated relevancy score of each partition in the map.

### Claim 14 (depends on 11)

The apparatus according to claim 11, wherein the correlation of the extracted one or more items with the one or more locations on the map comprises the processing circuitry further configured to:
determine, for each path from the person's current location to the one or more locations on the map, a likelihood that the respective path will be traversed.

### Claim 15 (depends on 11)

The apparatus according to claim 11, wherein the map includes, for each extracted item, a semantic label, a position, and a bounding box.

### Claim 16 (depends on 11)

The apparatus according to claim 11, wherein each of the extracted one or more items are located at a distance from the person that is less than or equal to a distance threshold.

### Claim 17 (depends on 11)

The apparatus according to claim 11, wherein the language model is a large language model (LLM) trained on a text corpus.

### Claim 18 (depends on 11)

The apparatus according to claim 11, wherein the one or more devices comprises an autonomous robot.

### Claim 19 (depends on 18)

The apparatus according to claim 18, wherein the processing circuitry is further configured to output the control signal to control an operation of the autonomous robot to avoid the person during the second time period.

### Claim 20

A non-transitory computer readable medium having instructions stored therein, which when executed by a processor cause the processor to execute a method comprising:
receiving data from one or more sensors corresponding to a first time period,
detecting, from the received data, one or more activities of a person in a space,
converting the detected one or more activities into a natural language narration;
combining the natural language narration with sequence text that corresponds to an action that is sequential to the detected one or more activities;
extracting one or more items from a map corresponding to the space;
determining a relevancy score for each of the extracted one or more items based on an output of a language model that receives as input the combination of the narration with the sequence text;
correlating the extracted one or more items with one or more locations on the map based on the determined relevancy for each of the extracted one or more items; and
outputting a control signal for controlling a movement of one more devices based on the correlation of the extracted one or more items with the one or more locations on the map.
