# US 2024/0256794 A1 · Systems, devices, and methods for operating a robotic system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0256794 A1
- Document ID: US20240256794A1
- Status: Published application
- Kind: A1
- Application number: 18/425,577
- Filed: 2024-01-29
- Published: 2024-08-01
- Earliest claimed: 2023-01-30
- Assignees: Sanctuary Cognitive Systems Corporation
- Applicants: Sanctuary Cognitive Systems Corporation
- Inventors: Suzanne Gildert
- Web page: https://patentgrep.com/patent/US20240256794A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- B25J 9/1602: {characterised by the control system, structure, architecture}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1653: {parameters identification, estimation, stiffness, accuracy, error analysis}
- B25J 9/1658: {characterised by programming language}
- B25J 9/1661: {characterised by task planning, object-oriented languages}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems}
- B25J 9/1679: {characterised by the tasks executed}
- B25J 9/1697: {Vision controlled systems}
- B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means}
- B25J 13/003: {by means of an audio-responsive input (audible safety signals B25J19/061)}
- G06F 40/205: Parsing
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})

## Abstract

A robotic system includes a robot, an object recognition subsystem, an interface to a large language model (LLM), and a system controller. The robot operates in an environment that includes a first and a second object. In an example method of operation of the robotic system, the object recognition subsystem assigns a first label to the first object. The interface sends a query, including the first label, to the LLM. The interface receives a response from the LLM, the response in reply to the query and including a second label. The object recognition subsystem assigns the second label to the second object. In some implementations, the object recognition subsystem includes sensors and a sensor data processor. The sensors scan the environment to generate sensor data, and the sensor data processor detects the presence of the first and the second object based at least in part on the sensor data.

## Claims

### Claim 1

A robotic system comprising:
a robot operable in an environment, the environment comprising a plurality of objects, the plurality of objects including a first object and a second object;
an object recognition subsystem communicatively coupled to the robot; and
an interface to a large language model (LLM) communicatively coupled to the object recognition system, wherein the object recognition subsystem comprises at least one processor and at least one non-transitory processor-readable storage medium communicatively coupled to the at least one processor, the at least one non-transitory processor-readable storage medium storing processor-executable instructions and/or data that, when executed by the at least one processor, cause the robotic system to perform a method for recognizing objects in the environment, the method which includes:
assigning a first label to the first object;
sending, by the interface, a query to the LLM, the query comprising the first label;
receiving, by the interface, a response from the LLM, the response in reply to the query, the response comprising a second label; and
assigning the second label to the second object.

### Claim 2 (depends on 1)

The robotic system of claim 1, the object recognition subsystem comprising a plurality of sensors and a sensor data processor, the method further comprising:
scanning the environment, by the plurality of sensors, to generate sensor data;
detecting, by the sensor data processor, the presence of the first object and the second object, wherein the detecting, by the sensor data processor, the presence of the first object and the second object is based at least in part on the sensor data.

### Claim 3 (depends on 2)

The robotic system of claim 2, wherein the assigning, by the object recognition subsystem, a first label to the first object includes:
identifying the first object based at least in part on the sensor data; and
assigning a natural language label to the first object.

### Claim 4 (depends on 3)

The robotic system of claim 3, wherein the sending, by the interface, a query to the LLM includes formulating a natural language statement, the natural language statement comprising the natural language label assigned to the first object.

### Claim 5 (depends on 3)

The robotic system of claim 3, the method further comprising determining a degree of confidence in the identifying of the first object exceeds a determined confidence threshold, wherein the determining a degree of confidence in the identifying of the first object exceeds a determined confidence threshold includes determining a probability.

### Claim 6 (depends on 2)

The robotic system of claim 2, wherein the scanning the environment, by the plurality of sensors, to generate sensor data includes generating at least one of image data, video data, audio data, or haptic data.

### Claim 7 (depends on 2)

The robotic system of claim 2, wherein the detecting, by the sensor data processor, the presence of the first object and the second object includes detecting, by the sensor data processor, the presence of the first object and the second object in real time.

### Claim 8 (depends on 2)

The robotic system of claim 2, the method further comprising assigning, by the object recognition subsystem, a third label to the second object, wherein the assigning, by the object recognition subsystem, a third label to the second object includes:
identifying the second object based at least in part on the sensor data; and
determining a degree of confidence in the identifying of the second object fails to exceed a determined confidence threshold.

### Claim 9 (depends on 8)

The robotic system of claim 8, wherein the assigning, by the object recognition subsystem, the second label to the second object includes updating the degree of confidence in the identifying of the second object.

### Claim 10 (depends on 1)

The robotic system of claim 1, wherein the sending, by the interface, a query to the LLM includes formulating a natural language statement, the natural language statement comprising the first label.

### Claim 11 (depends on 10)

The robotic system of claim 10, wherein the formulating a natural language statement includes structuring the natural language statement to cause the response from the LLM to follow a defined structure.

### Claim 12 (depends on 11)

The robotic system of claim 11, wherein the receiving, by the interface, a response from the LLM includes:
receiving a natural language statement, the natural language statement comprising a natural language label; and
parsing the natural language statement to extract the natural language label.

### Claim 13 (depends on 12)

The robotic system of claim 12, wherein the assigning, by the object recognition subsystem, a second label to the second object includes assigning the natural language label to the second object.

### Claim 14 (depends on 1)

The robotic system of claim 1, wherein the assigning, by the object recognition subsystem, a first label to the first object includes:
identifying the first object; and
assigning a natural language label to the first object.

### Claim 15 (depends on 14)

The robotic system of claim 14, the method further comprising: determining a degree of confidence in the identifying of the first object exceeds a determined confidence threshold, wherein the determining a degree of confidence in the identifying of the first object exceeds a determined confidence threshold includes determining a probability.

### Claim 16 (depends on 14)

The robotic system of claim 14, wherein the sending, by the interface, a query to the LLM includes formulating a natural language statement, the natural language statement comprising the natural language label.

### Claim 17 (depends on 16)

The robotic system of claim 16, wherein the formulating a natural language statement includes structuring the natural language statement to cause the response from the LLM to follow a defined structure.

### Claim 18 (depends on 1)

The robotic system of claim 1, wherein the receiving, by the interface, a response from the LLM includes:
receiving a natural language statement, the natural language statement comprising a natural language label; and
parsing the natural language statement to extract the natural language label.

### Claim 19 (depends on 1)

The robotic system of claim 1, the method further comprising assigning, by the object recognition subsystem, a third label to the second object, wherein the assigning, by the object recognition subsystem, a second label to the second object includes comparing the second label with the third label.

### Claim 20 (depends on 19)

The robotic system of claim 19, wherein the assigning, by the object recognition subsystem, a second label to the second object further includes updating a degree of confidence.
