# US 2019/0318150 A1 · System and method for semantic or visual collaborative augmented reality > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0318150 A1 - Document ID: US20190318150A1 - Status: Published application - Kind: A1 - Application number: 15/955,586 - Filed: 2018-04-17 - Published: 2019-10-17 - Earliest claimed: 2018-04-17 - Assignees: Fuji Xerox Co., Ltd. - Applicants: Fuji Xerox Co., Ltd. - Inventors: David A. SHAMMA; Laurent DENOUE; Matthew L. COOPER - Web page: https://patentgrep.com/patent/US20190318150A1 ## CPC - G06V 20/653: {by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces} - G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)} - G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves} - G06F 3/02: Input arrangements using manually operated switches, e.g. using keyboards or dials - G06F 3/0346: with detection of the device orientation or free movement in a three-dimensional [3D] space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors - G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)} - G06V 20/20: in augmented reality scenes - H04M 1/72427: for supporting games or graphical animations - H04M 2250/52: including functional features of a camera - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08) ## Abstract An augmented reality (AR) environment synchronization method, includes, for each of a plurality of devices associated with respective users in the AR environment, receiving plane information, and generating object/unique identifier information; across the devices, coordinating the plane information; performing context-aware matching of the object/unique identifier information across the devices to generate a match between respective objects sensed by the devices; and providing synchronization control to the devices, to permit an annotation of the matched object to be locked to a landmark and the plane of one of the devices with respect to others of the devices. ## Claims ### Claim 1 An augmented reality (AR) environment synchronization method, comprising: for each of a plurality of devices associated with respective users in the AR environment, receiving plane information, and generating object/unique identifier information; across the devices, coordinating the plane information; performing context-aware matching of the object/unique identifier information across the devices to generate a match between respective objects sensed by the devices; and providing synchronization control to the devices, to permit an annotation of the matched object to be locked to a landmark and the plane of one of the devices with respect to others of the devices. ### Claim 2 (depends on 1) The method of claim 1, wherein instructions are provided to the devices to grant rights or access to perform prescribed annotations. ### Claim 3 (depends on 1) The method of claim 1, further comprising restricting or permitting annotation to certain users or objects, and in certain contexts or situations, either manually or automatically, based on a user input. ### Claim 4 (depends on 1) The method of claim 1, wherein the receiving plane information comprising receiving information on the users associated with the devices, or ambient sensed information from the devices. ### Claim 5 (depends on 1) The method of claim 1, wherein the devices comprise AR camera devices, and information received from the devices comprises one or more of geolocation, ambient conditions, user profile, user social graph or public information, where permitted by the user. ### Claim 6 (depends on 1) The method of claim 1, wherein the matching comprises a context-aware operation that uses visual and semantic indicia to perform the matching, and the objects are similar but different in at least some aspects between ones of the devices. ### Claim 7 (depends on 1) The method of claim 1, wherein the synchronization control comprising locking the annotation to the landmark and the plane, such that a movement of the object by one of the users permits the annotation to move therewith. ### Claim 8 A non-transitory computer-readable medium including instructions for augmented reality (AR) environment synchronization, the instructions comprising: for each of a plurality of devices associated with respective users in the AR environment, receiving plane information, and generating object/unique identifier information; across the devices, coordinating the plane information; performing context-aware matching of the object/unique identifier information across the devices to generate a match between respective objects sensed by the devices; and providing synchronization control to the devices, to permit an annotation of the matched object to be locked to a landmark and the plane of one of the devices with respect to others of the devices. ### Claim 9 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein additional instructions are provided to the devices to grant rights or access to perform prescribed annotations. ### Claim 10 (depends on 8) The non-transitory computer-readable medium of claim 8, further comprising restricting or permitting annotation to certain users or objects, and in certain contexts or situations, either manually or automatically, based on a user input. ### Claim 11 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the receiving plane information comprising receiving information on the users associated with the devices, or ambient sensed information from the devices. ### Claim 12 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the devices comprise AR camera devices, and information received from the devices comprises one or more of geolocation, ambient conditions, user profile, user social graph or public information, where permitted by the user. ### Claim 13 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the matching comprises a context-aware operation that uses visual and semantic indicia to perform the matching, and the objects are similar but different in at least some aspects between ones of the devices. ### Claim 14 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the synchronization control comprising locking the annotation to the landmark and the plane, such that a movement of the object by one of the users permits the annotation to move therewith. ### Claim 15 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein one or more of the respective users is provided with control over the context-aware matching, and the context-aware matching is based on at least one of visual, semantic, category, functional, and context features in response to at least one of visual, geolocation, temperature, humidity, and light exposure inputs. ### Claim 16 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the annotation properties in the AR environment is controlled to permit or restrict propagation of the annotation through the matched objects, based on a user indication of a status of the matched objects. ### Claim 17 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein the annotation properties of the matched objects in the AR environment are controlled based on a characteristic of one or more of the respective users. ### Claim 18 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein the characteristic comprises at least one of metadata, social media information, preferences in a user profile, and historical behavior associated with the one or more of the respective users. ### Claim 19 (depends on 8) The non-transitory computer-readable medium of claim 8, wherein computer vision and AR are applied to determine context, and perform the matching, based on at least one of geolocation, demographic, metadata, behavioral, historical, social graph, role of the respective users, and human or non-human status of the respective users.