# US 12,112,446 B1 · Object correction using scene graphs > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,446 B1 - Document ID: US12112446B1 - Status: Granted - Kind: B1 - Application number: 18/237,480 - Filed: 2023-08-24 - Granted: 2024-10-08 - Earliest claimed: 2021-02-24 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Angela Blechschmidt; Daniel Ulbricht; Alexander S. Polichroniadis - Web page: https://patentgrep.com/patent/US12112446B1 ## CPC - G06T 17/005: {Tree description, e.g. octree, quadtree} - G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts - G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80) - G06T 17/00: Three-dimensional [3D] modelling for computer graphics - G06V 10/82: using neural networks - G06V 20/20: in augmented reality scenes - G06V 20/64: Three-dimensional [3D] objects - G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations - G06T 2207/10028: Range image; Depth image; 3D point clouds - G06T 2210/61: Scene description - G06T 2215/16: Using real world measurements to influence rendering - G06T 2219/2004: Aligning objects, relative positioning of parts - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Various implementations disclosed herein include devices, systems, and methods that uses object relationships represented in the scene graph to adjust the position of objects. For example, an example process may include obtaining a three-dimensional (3D) representation of a physical environment that was generated based on sensor data obtained during a scanning process, detecting positions of a set of objects in the physical environment based on the 3D representation, generating a scene graph for the 3D representation of the physical environment based on the detected positions of the set of objects, wherein the scene graph represents the set of objects and relationships between the objects, and determining a refined 3D representation of the physical environment by refining the position of at least one object in the set of objects based on the scene graph and an alignment rule associated with a relationship in the scene graph. ## Claims ### Claim 1 A method comprising: at an electronic device having a processor: obtaining a three-dimensional (3D) representation of a physical environment that was generated based on sensor data obtained during a scanning process; detecting positions of a set of objects in the physical environment based on the 3D representation; generating an object relationship model for the 3D representation of the physical environment based on the detected positions of the set of objects and the sensor data, wherein the object relationship model represents the set of objects and relationships between the objects; and determining a refined 3D representation of the physical environment by refining the position of at least one object in the set of objects based on the object relationship model and an alignment rule associated with a relationship in the object relationship model. ### Claim 2 (depends on 1) The method of claim 1, wherein refining the position of the at least one object is based on an angular position of the at least one object with respect to a wall structure. ### Claim 3 (depends on 1) The method of claim 1, wherein refining the position of the at least one object is based on a position of the at least one object with respect to a different object. ### Claim 4 (depends on 1) The method of claim 1, wherein the set of objects is a first set of objects, the method further comprising: detecting positions of a second set of objects in the physical environment based on the 3D representation; updating the object relationship model for the 3D representation of the physical environment based on the detected positions of the second set of objects; and determining, based on the updated object relationship model, that the second set of objects is a reflection of at least a portion of the first set of objects. ### Claim 5 (depends on 1) The method of claim 1, wherein the 3D representation is associated with 3D semantic data that includes a 3D point cloud that includes semantic labels associated with at least a portion of 3D points within the 3D point cloud. ### Claim 6 (depends on 5) The method of claim 5, wherein the semantic labels identify walls, wall attributes, objects, and classifications of the objects of the physical environment. ### Claim 7 (depends on 5) The method of claim 5, wherein a neural network is trained to: refine the accuracy of identified alignments of the set of objects based on the positions and the 3D semantic data associated with each respective object in the set of objects; and output the refined 3D representation of the physical environment based on the refined accuracy of the identified alignments of the set of objects. ### Claim 8 (depends on 1) The method of claim 1, wherein the 3D representation is an extended reality (XR) experience that is presented to a user. ### Claim 9 (depends on 1) The method of claim 1, wherein the sensor data comprises depth data and light intensity image data obtained during the scanning process. ### Claim 10 (depends on 1) The method of claim 1, wherein the electronic device is a head-mounted device (HMD). ### Claim 11 A device comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: obtaining a three-dimensional (3D) representation of a physical environment that was generated based on sensor data obtained during a scanning process; detecting positions of a set of objects in the physical environment based on the 3D representation; generating an object relationship model for the 3D representation of the physical environment based on the detected positions of the set of objects and the sensor data, wherein the object relationship model represents the set of objects and relationships between the objects; and determining a refined 3D representation of the physical environment by refining the position of at least one object in the set of objects based on the object relationship model and an alignment rule associated with a relationship in the object relationship model. ### Claim 12 (depends on 11) The device of claim 11, wherein refining the position of the at least one object is based on an angular position of the at least one object with respect to a wall structure. ### Claim 13 (depends on 11) The device of claim 11, wherein refining the position of the at least one object is based on a position of the at least one object with respect to a different object. ### Claim 14 (depends on 11) The device of claim 11, wherein the set of objects is a first set of objects, and wherein the program instructions, when executed on the one or more processors, further cause the one or more processors to perform operations comprising: detecting positions of a second set of objects in the physical environment based on the 3D representation; updating the object relationship model for the 3D representation of the physical environment based on the detected positions of the second set of objects; and determining, based on the updated object relationship model, that the second set of objects is a reflection of at least a portion of the first set of objects. ### Claim 15 (depends on 11) The device of claim 11, wherein the 3D representation is associated with 3D semantic data that includes a 3D point cloud that includes semantic labels associated with at least a portion of 3D points within the 3D point cloud. ### Claim 16 (depends on 15) The device of claim 15, wherein the semantic labels identify walls, wall attributes, objects, and classifications of the objects of the physical environment. ### Claim 17 (depends on 15) The device of claim 15, wherein a neural network is trained to: refine the accuracy of identified alignments of the set of objects based on the positions and the 3D semantic data associated with each respective object in the set of objects; and output the refined 3D representation of the physical environment based on the refined accuracy of the identified alignments of the set of objects. ### Claim 18 (depends on 11) The device of claim 11, wherein the device is a head-mounted device (HMD). ### Claim 19 (depends on 11) The device of claim 11, wherein the sensor data comprises depth data and light intensity image data obtained during the scanning process. ### Claim 20 A non-transitory computer-readable storage medium, storing program instructions executable by one or more processors of an electronic device to perform operations comprising: obtaining a three-dimensional (3D) representation of a physical environment that was generated based on sensor data obtained during a scanning process; detecting positions of a set of objects in the physical environment based on the 3D representation; generating an object relationship model for the 3D representation of the physical environment based on the detected positions of the set of objects and the sensor data, wherein the object relationship model represents the set of objects and relationships between the objects; and determining a refined 3D representation of the physical environment by refining the position of at least one object in the set of objects based on the object relationship model and an alignment rule associated with a relationship in the object relationship model.