# US 12,762,351 B2 · Systems, devices, and methods for operating a robotic system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,762,351 B2
- Document ID: US12762351B2
- Status: Granted
- Kind: B2
- Application number: 18/425,557
- Filed: 2024-01-29
- Published: 2024-08-01
- Granted: 2026-10-13
- Earliest claimed: 2023-01-30
- Assignees: Sanctuary Cognitive Systems Corporation
- Applicants: Sanctuary Cognitive Systems Corporation
- Inventors: Suzanne Gildert
- Web page: https://patentgrep.com/patent/US12762351B2

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- B25J 9/1602: {characterised by the control system, structure, architecture}
- B25J 9/163: {learning, adaptive, model based, rule based expert control}
- B25J 9/1653: {parameters identification, estimation, stiffness, accuracy, error analysis}
- B25J 9/1658: {characterised by programming language}
- B25J 9/1661: {characterised by task planning, object-oriented languages}
- B25J 9/1664: {characterised by motion, path, trajectory planning}
- B25J 9/1671: {characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems}
- B25J 9/1679: {characterised by the tasks executed}
- B25J 9/1697: {Vision controlled systems}
- B25J 11/0005: {Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means}
- B25J 13/003: {by means of an audio-responsive input (audible safety signals B25J19/061)}
- G06F 40/205: Parsing
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50})

## Abstract

A robotic system includes a robot, an object recognition subsystem, an interface to a large language model (LLM), and a system controller. The robot operates in an environment that includes a first and a second object. In an example method of operation of the robotic system, the object recognition subsystem assigns a first label to the first object. The interface sends a query, including the first label, to the LLM. The interface receives a response from the LLM, the response in reply to the query and including a second label. The object recognition subsystem assigns the second label to the second object. In some implementations, the object recognition subsystem includes sensors and a sensor data processor. The sensors scan the environment to generate sensor data, and the sensor data processor detects the presence of the first and the second object based at least in part on the sensor data.

## Claims

### Claim 1

A method of autonomously recognizing objects in an environment, the method performed by a robotic system comprising a robot, an object recognition subsystem, and an interface to a large language model (LLM), and the environment comprising a first object and at least a second object, the method comprising:
identifying, by the object recognition subsystem, the first object;
determining, by the object recognition subsystem, that a degree of confidence in identifying the first object exceeds a threshold;
assigning, by the object recognition subsystem, a first label to the first object;
identifying, by the object recognition subsystem, the second object;
determining, by the object recognition subsystem, that a degree of confidence in identifying the second object fails to exceed the threshold;
sending, by the interface, a query to the LLM, the query comprising the first label and a request to provide a list of labels for the second object, the list of labels ranked according to an estimate of the degree of confidence that each label matches the second object;
receiving, by the interface, a response from the LLM, the response in reply to the query, the response comprising the list of labels for the second object;
matching, by the object recognition subsystem, the second object to a second label in the list of labels in the response from the LLM; and
assigning, by the object recognition subsystem, the second label to the second object.

### Claim 2 (depends on 1)

The method of claim 1, the object recognition subsystem comprising a plurality of sensors and a sensor data processor, the method further comprising:
scanning the environment, by the plurality of sensors, to generate sensor data; and
detecting, by the sensor data processor, the presence of the first object and the second object, wherein the detecting, by the sensor data processor, the presence of the first object and the second object is based at least in part on the sensor data.

### Claim 3 (depends on 2)

The method of claim 2, wherein the assigning, by the object recognition subsystem, a first label to the first object includes:
identifying the first object based at least in part on the sensor data; and
assigning a natural language label to the first object.

### Claim 4 (depends on 3)

The method of claim 3, wherein the sending, by the interface, a query to the LLM includes formulating a natural language statement, the natural language statement comprising the natural language label assigned to the first object and the request to provide the list of labels for the second object.

### Claim 5 (depends on 3)

The method of claim 3, wherein the determining a degree of confidence in the identifying of the first object exceeds a determined confidence threshold includes determining a probability.

### Claim 6 (depends on 2)

The method of claim 2, wherein the scanning the environment, by the plurality of sensors, to generate sensor data includes generating at least one of image data, video data, audio data, or haptic data.

### Claim 7 (depends on 2)

The method of claim 2, wherein the detecting, by the sensor data processor, the presence of the first object and the second object includes detecting, by the sensor data processor, the presence of the first object and the second object in real time.

### Claim 8 (depends on 2)

The method of claim 2, wherein the robot includes haptic sensors, and wherein:
detecting, by the sensor data processor, the presence of the first object and the second object includes establishing, by the haptic sensors, respective haptic profiles of the first object and the second object;
identifying, by the object recognition system, the first object includes identifying, by the object recognition system, the first object based at least in part on the haptic profile of the first object; and
identifying, by the object recognition system, the second object includes identifying, by the object recognition system, the second object based at least in part on the haptic profile of the second object.

### Claim 9 (depends on 1)

The method of claim 1, wherein the assigning, by the object recognition subsystem, the second label to the second object includes updating the degree of confidence in the identifying of the second object.

### Claim 10 (depends on 1)

The method of claim 1, wherein the sending, by the interface, a query to the LLM includes formulating a natural language statement, the natural language statement comprising the first label and the request to provide the list of labels for the second object.

### Claim 11 (depends on 10)

The method of claim 10, wherein the formulating a natural language statement includes structuring the natural language statement to cause the response from the LLM to follow a defined structure.

### Claim 12 (depends on 1)

The method of claim 1, wherein the receiving, by the interface, a response from the LLM includes:
receiving a natural language statement, the natural language statement comprising the list of labels for the second object; and
parsing the natural language statement to extract the list of labels for the second object.

### Claim 13 (depends on 1)

The method of claim 1 wherein matching, by the object recognition subsystem, the second object to a second label in the list of labels in the response from the LLM includes increasing the degree of confidence in identifying the second object to exceed the threshold.