# US 12,223,949 B2 · Semantic rearrangement of unknown objects from natural language commands > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,223,949 B2 - Document ID: US12223949B2 - Status: Granted - Kind: B2 - Application number: 17/930,349 - Filed: 2022-09-07 - Published: 2023-03-09 - Granted: 2025-02-11 - Earliest claimed: 2021-09-07 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Christopher Jason Paxton; Weiyu Liu; Tucker Ryer Hermans; Dieter Fox - Web page: https://patentgrep.com/patent/US12223949B2 ## CPC - B25J 11/008: {Manipulators for service tasks} - G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning} - B25J 13/003: {by means of an audio-responsive input (audible safety signals B25J19/061)} - G06F 40/30: Semantic analysis - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 40/216: using statistical methods - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 2015/223: {Execution procedure of a spoken command} - B25J: MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES ({manipulators specially adapted for use in surgery A61B34/70; manipulators used in cleaning hollow articles B08B9/04}; manipulators associated with rolling mills B21B39/20; manipulators associated with forging machines B21J13/10; {manipulators associated with picking-up and placing mechanisms B23P19/007}; means for holding wheels or parts thereof B60B30/00; {vehicles with ground-engaging propulsion means, e.g. walking members B62D57/02, B62D57/032; devices for picking-up and depositing articles or materials between conveyors B65G47/90, B65G47/91; manipulators with gripping or holding means for transferring packages B65H67/065}; cranes B66C; {manipulators used in the protection or supervision of pipe-line installations F17D5/00; walking equipment adapted for nuclear steam-generators F22B37/006}; manipulators specially adapted for, or associated with, nuclear reactors G21C; {apparatus used for handling wafers during manufacture or treatment of semiconductor H10P72/50}) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A robotic system is provided for performing rearrangement tasks guided by a natural language instruction. The system can include a number of neural networks used to determine a selected rearrangement of the objects in accordance with the natural language instruction. A target object predictor network processes a point cloud of the scene and the natural language instruction to identify a set of query objects that are to-be-rearranged. A language conditioned prior network processes the point cloud, natural language instruction, and the set of query objects to sample a distribution of rearrangements to generate a number of sets of pose offsets for the set of query objects. A discriminator network then processes the samples to generate scores for the samples. The samples may be refined until a score for at least one of the sample generated by the discriminator network is above a threshold value. ## Claims ### Claim 1 A method, comprising: receiving a point cloud and a natural language instruction; generating, via a first network, an instance mask based on the natural language instruction, wherein the instance mask identifies a set of query objects in the point cloud; generating, via a second network, samples of pose offsets for the set of query objects in accordance with the natural language instruction; and identifying, via a discriminator network, a selected rearrangement of the set of query objects in accordance with scores. ### Claim 2 (depends on 1) The method of claim 1, wherein the point cloud represents a scene including a plurality of objects, and wherein the set of query objects is a subset of the objects in the scene that are identified to be moved in order to satisfy the natural language instruction. ### Claim 3 (depends on 1) The method of claim 1, wherein at least one of the first network or the second network is configured to generate embeddings for the point cloud and the natural language instruction used as inputs to the first network or the second network. ### Claim 4 (depends on 1) The method of claim 1, wherein at least one of the first network or the second network is a transformer-based neural network comprising an encoder and a decoder, and wherein each of the encoder and the decoder comprises a plurality of layers. ### Claim 5 (depends on 4) The method of claim 4, wherein a layer of the encoder includes a first sub-layer that includes a multi-head attention mechanism and a second sub-layer that includes a fully-connected feed-forward layer. ### Claim 6 (depends on 5) The method of claim 5, wherein a layer of the decoder includes a first sub-layer that includes a masked multi-head attention mechanism, a second sub-layer that includes a multi-head attention mechanism, and a third sub-layer that includes a fully-connected feed-forward layer. ### Claim 7 (depends on 1) The method of claim 1, wherein each sample comprises a set of pose offsets corresponding to the set of query objects, and wherein each pose offset comprises a parameterization of a six degree of freedom (6-DOF) transformation. ### Claim 8 (depends on 1) The method of claim 1, wherein identifying the selected rearrangement comprises: comparing each of the scores to a threshold value; and selecting a particular sample having a score that is greater than the threshold value as the selected rearrangement, or responsive to determining that no samples have a score greater than the threshold value, refining the samples using a cross-entropy method. ### Claim 9 (depends on 1) The method of claim 1, further comprising: rearranging a number of objects using a robot based on the selected rearrangement. ### Claim 10 (depends on 1) The method of claim 1, further comprising training the discriminator network using a set of scenes and a set of randomly perturbed scenes as positive and negative samples, respectively. ### Claim 11 A system for rearranging objects in a scene, the system comprising: a memory storing a point cloud and a natural language instruction; and at least one processor configured to: generate, via a first network, an instance mask based on the natural language instruction, wherein the instance mask identifies a set of query objects in the point cloud; generate, via a second network, samples of pose offsets for the set of query objects in accordance with the natural language instruction; and identify, via a discriminator network, a selected rearrangement of the set of query objects in accordance with scores. ### Claim 12 (depends on 11) The system of claim 11, wherein the point cloud represents a scene including a plurality of objects, and wherein the set of query objects is a subset of the objects in the scene that are identified to be moved in order to satisfy the natural language instruction. ### Claim 13 (depends on 11) The system of claim 11, wherein at least one of the first network or the second network is configured to generate embeddings for the point cloud and the natural language instruction used as inputs to the first network or the second network. ### Claim 14 (depends on 11) The system of claim 11, wherein at least one of the first network or the second network is a transformer-based neural network comprising an encoder and a decoder, and wherein each of the encoder and the decoder comprises a plurality of layers. ### Claim 15 (depends on 14) The system of claim 14, wherein a layer of the encoder includes a first sub-layer that includes a multi-head attention mechanism and a second sub-layer that includes a fully-connected feed-forward layer. ### Claim 16 (depends on 15) The system of claim 15, wherein a layer of the decoder includes a first sub-layer that includes a masked multi-head attention mechanism, a second sub-layer that includes a multi-head attention mechanism, and a third sub-layer that includes a fully-connected feed-forward layer. ### Claim 17 (depends on 11) The system of claim 11, wherein each sample comprises a set of pose offsets corresponding to the set of query objects, and wherein each pose offset comprises a parameterization of a six degree of freedom (6-DOF) transformation. ### Claim 18 (depends on 11) The system of claim 11, wherein identifying the selected rearrangement comprises: comparing each of the scores to a threshold value; and selecting a particular sample having a score that is greater than the threshold value as the selected rearrangement, or responsive to determining that no samples have a score greater than the threshold value, refining the samples using a cross-entropy method. ### Claim 19 (depends on 11) The system of claim 11, further comprising a robot that rearranges a number of objects based on the selected rearrangement. ### Claim 20 A non-transitory computer readable medium storing instructions that, responsive to being executed by at least one processor, cause a system to: receive a point cloud and a natural language instruction; generate, via a first network, an instance mask based on the natural language instruction, wherein the instance mask identifies a set of query objects in the point cloud; generate, via a second network, samples of pose offsets for the set of query objects in accordance with the natural language instruction; and identify, via a discriminator network, a selected rearrangement of the set of query objects in accordance with scores.