# US 11,029,805 B2 · Real-time preview of connectable objects in a physically-modeled virtual space > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,029,805 B2 - Document ID: US11029805B2 - Status: Granted - Kind: B2 - Application number: 16/508,197 - Filed: 2019-07-10 - Published: 2021-01-14 - Granted: 2021-06-08 - Earliest claimed: 2019-07-10 - Assignees: Magic Leap, Inc. - Applicants: Magic Leap, Inc. - Inventors: Edmund Graves Brown, IV; Javier Antonio Busto; Jeffrey A. Scott; Jeremy Vanhoozer - Web page: https://patentgrep.com/patent/US11029805B2 ## CPC - G06F 3/04815: Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object - G02B 27/0172: {characterised by optical features} - G06F 1/163: {Wearable computers, e.g. on a belt} - G06F 1/1686: {the I/O peripheral being an integrated camera} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)} - G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)} - G06F 3/0304: {Detection arrangements using opto-electronic means (constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042)} - G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour - G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units} - G06F 3/147: using display panels - G06V 20/20: in augmented reality scenes - G02B 27/0093: {with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking} - G02B 2027/0138: {comprising image capture systems, e.g. camera} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Virtual reality (VR), augmented reality (AR), and/or mixed reality (MR) systems may enable one or more users to connect two or more connectable objects together. These connectable objects may be real objects from the user's environment, virtual objects, or a combination thereof. A preview system may be included as a part of the VR, AR, and/or MR systems that provide a preview of the connection between the connectable objects prior to the user(s) connecting the connectable objects. The preview may include a representation of the connectable objects in a connected state along with an indication of whether the connected state is valid or invalid. The preview system may continuously physically model the connectable objects while simultaneously displaying a preview of the connection process to the user of the VR, AR, or MR system. ## Claims ### Claim 1 An augmented reality (AR) system, comprising: a user input device configured to receive user input in an environment of a user of the AR system; a display configured to present virtual content to the user of the AR system; non-transitory computer storage configured to store: a list of two or more connectable virtual objects including spatial information for each of the connectable objects, wherein each connectable virtual object has at least one node, wherein a node defines a point in space with respect to the corresponding connectable virtual object at which the connectable virtual object can be connected to another connectable virtual object, and a list of one or more nodes for each of the connectable virtual objects; and a hardware processor in communication with the non-transitory computer storage, the hardware processor programmed to: receive user input from the user input device indicative of a manipulation of at least one of the connectable virtual objects; identify a first of the at least one of the connectable virtual objects as an active object based on the user input; identify at least one of the connectable virtual objects as a target object; determine that a node of the active object and a node of the target object are compatible; and generate a preview of the active object connected to the target object. ### Claim 2 (depends on 1) The system of claim 1, wherein the preview includes a visual indication of whether the connection between the active object and the target object is valid. ### Claim 3 (depends on 1) The system of claim 1, wherein the spatial information includes at least one of a position, a rotation, or a scale for each of the connectable objects. ### Claim 4 (depends on 1) The system of claim 1, wherein the hardware processor is further programmed to: update the spatial information for each of the connectable virtual objects based on a physics simulation. ### Claim 5 (depends on 1) The system of claim 1, wherein the hardware processor is further programmed to: generate a list of active objects based on the user input, and generate a list of active nodes comprising available nodes from the nodes of the active objects. ### Claim 6 (depends on 5) The system of claim 5, wherein the hardware processor is further programmed to: generate a list of target nodes comprising available nodes from the nodes of connectable virtual objects excluding the active objects, generate a list of node pairs comprising valid pairs of nodes between the list of active nodes and the list of target nodes, and select the target object based on the list of node pairs. ### Claim 7 (depends on 6) The system of claim 6, wherein to select the candidate connectable object, the hardware processor is further programmed to: calculate a connection score for each of the node pairs using a connection score heuristic, select the node pair having a highest or lowest connection score as an active node pair, and select the target object sharing a node in the active node pair. ### Claim 8 (depends on 7) The system of claim 7, wherein the connection score heuristic comprises a measurement of a distance between the nodes for each of the node pairs. ### Claim 9 (depends on 7) The system of claim 7, wherein to generate the preview, the hardware processor is further programmed to: reposition and rotate the preview to be connected to the target object via the active node pair. ### Claim 10 (depends on 7) The system of claim 7, wherein the hardware processor is further programmed to: determine whether the nodes of the active node pair are compatible using a compatibility heuristic, and generate a visual indication of whether the connection between the active object and the candidate connectable object is valid based on the determination of whether the nodes of the active node pair are compatible. ### Claim 11 (depends on 7) The system of claim 7, wherein the hardware processor is further programmed to: determine whether the preview overlaps another connectable object or a non-connectable object, and generate a visual indication of whether the preview overlaps the other connectable object or the non-connectable object. ### Claim 12 (depends on 1) The system of claim 1, wherein the list of connectable objects includes at least one virtual connectable object and at least one virtual representation of a real-world connectable object. ### Claim 13 (depends on 1) The system of claim 1, wherein the list of connectable objects includes only virtual connectable objects. ### Claim 14 (depends on 1) The system of claim 1, wherein the hardware processor is further programmed to: determine that the nodes of the active object are not within a threshold distance of any nodes of the other connectable objects, and hide the preview from being displayed in response to determining that the nodes of the active object are not within the threshold distance of any of the nodes of the other connectable objects. ### Claim 15 (depends on 1) The system of claim 1, wherein the user input device comprises a plurality of user input devices, and wherein the hardware processor is further programmed to: receive input from a plurality of users via the user input devices, generate a list of active objects based on the input received from the plurality of user input devices, and for each of the active objects, generate a preview the corresponding active object connected to a corresponding target object. ### Claim 16 (depends on 1) The system of claim 1, wherein the user input device comprises an outward facing camera configured to detect gestures of the user. ### Claim 17 A method comprising: under control of a preview system for connectable virtual objects comprising computing hardware: receiving user input from a user input device indicative of a manipulation of at least one of a plurality of connectable virtual objects from a list of connectable virtual objects, the list of connectable virtual objects including spatial information for each of the connectable virtual objects and a list of one or more nodes for each of the connectable virtual objects, wherein each connectable virtual object has at least one node, and wherein a node defines a point in space with respect to the corresponding connectable virtual object at which the connectable virtual object can be connected to another connectable virtual object; identifying a first of the at least one of the connectable virtual objects as an active object based on the user input; identifying at least one of the connectable virtual objects as a target object; determining that a node of the active object and a node of the target object are compatible; and generate a preview of the active object connected to the target object.