# US 12,182,848 B2 · Method, system, and computer-readable medium for determining positioning of items on a surface > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,182,848 B2 - Document ID: US12182848B2 - Status: Granted - Kind: B2 - Application number: 18/167,206 - Filed: 2023-02-10 - Published: 2023-06-22 - Granted: 2024-12-31 - Earliest claimed: 2019-04-05 - Assignees: Shopify Inc. - Applicants: Shopify Inc. - Inventors: Daniel Beauchamp - Web page: https://patentgrep.com/patent/US12182848B2 ## CPC - G06Q 30/0631: {Recommending goods or services} - G06Q 30/0627: {by specifying product or service characteristics, e.g. product dimensions} - G06Q 30/0641: {utilising user interfaces specially adapted for shopping} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract Computer-implemented methods and systems including receiving feature data identifying a physical surface determined using augmented reality software and an imaging device operated by a computing device in a physical environment comprising the physical surface; determining a positioning of at least one surface item upon the physical surface based on surface dimensions determined from the feature data; and causing an augmented reality interface to be displayed by the augmented reality software on the computing device, the augmented reality interface comprising an illustrative layout of the physical surface generated using the feature data and showing the determined positioning of the at least one surface item upon the physical surface in the illustrative layout while imaging the physical surface in the physical environment. ## Claims ### Claim 1 A computer-implemented method comprising: receiving feature data identifying a physical surface determined using feature points recognized by augmented reality software coupled to an imaging device operated by a computing device while the computing device is capturing the feature data from a physical environment in which the physical surface is located; determining how to position multiple surface items to fit together and scale with dimensions of the physical surface based on three-dimensional data of the surface items and two-dimensional surface dimensions of the physical surface determined from the feature points identified in the feature data obtained while in the physical environment comprising the physical surface; and causing an augmented reality interface to be displayed by the augmented reality software on the computing device, the augmented reality interface comprising an illustrative layout of the physical surface generated using the feature data and showing the multiple surface items and a determined positioning of the surface items in three-dimensions upon the two-dimensional physical surface in the illustrative layout according to how the multiple surface items fit together and scale with the dimensions of the physical surface, while imaging the physical surface in the physical environment. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising: selecting, with a recommendation engine, a first surface item and a second surface item of the multiple surface items to be positioned upon the physical surface and visualized by the computing device using the augmented reality software, the selecting based at least in part on the determined surface dimensions of the physical surface and the three-dimensional data of each of the first surface item and the second surface item. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein the positioning of the first surface item and the second surface item upon the physical surface is determined based on the determined surface dimensions, the three-dimensional data of the first surface item and the second surface item, and one or more rules defining a rule-based spacing for the first surface item and the second surface item, wherein the rule-based spacing determines how the first surface item and the second surface item fit together and scale with the dimensions of the physical surface. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, further comprising the recommendation engine selecting at least one of the first surface item and the second surface item based on a product selection criterion received from the computing device. ### Claim 5 (depends on 2) The computer-implemented method of claim 2, further comprising the recommendation engine selecting at least one of the first surface item and the second surface item based on a customer profile. ### Claim 6 (depends on 2) The computer-implemented method of claim 2, further comprising the recommendation engine selecting at least one of the first surface item and the second surface item based on past purchase history of a customer. ### Claim 7 (depends on 2) The computer-implemented method of claim 2, further comprising the recommendation engine selecting at least one of the first surface item and the second surface item based on product availability. ### Claim 8 (depends on 2) The computer-implemented method of claim 2, further comprising the recommendation engine selecting at least one of the first surface item and the second surface item based on a merchant profile. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the feature data comprises three-dimensional data for the multiple surface items and the physical surface. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, further comprising determining, based on the surface data, a dimensional layout of the physical surface for the illustrative layout, wherein determining how to position the multiple surface items comprises using the dimensional layout of the physical surface and how quantities of the multiple surface items fit on the physical surface. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein determining how to position the multiple surface items comprises using layout rules for objects on surfaces and how the at least one surface item fits on the physical surface. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein determining how to position the multiple surface items comprises using a machine learning algorithm based on layout characteristics for objects on surfaces and how the multiple surface items fit on the physical surface. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, further comprising determining at least one of a size or scale for the multiple surface items in determining how the multiple surface items fit and scale with the surface dimensions. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, wherein the illustrative layout of the physical surface comprises determined quantities of each of a first surface item and a second surface item positioned on the physical surface. ### Claim 15 A system comprising: a processor; at least one communications interface coupled to the processor, the at least one communications interface configured to communicate with a computing device; and at least one memory, the at least one memory comprising processor executable instructions that, when executed by the at least one processor, causes the system to: receive feature data identifying a physical surface determined using feature points recognized by augmented reality software coupled to an imaging device operated by a computing device while the computing device is capturing the feature data from a physical environment in which the physical surface is located; determine how to position multiple surface items to fit together and scale with dimensions of the physical surface based on three-dimensional data of the surface items and two-dimensional surface dimensions of the physical surface determined from the feature points identified in the feature data obtained while in the physical environment comprising the physical surface; and cause an augmented reality interface to be displayed by the augmented reality software on the computing device, the augmented reality interface comprising an illustrative layout of the physical surface generated using the feature data and showing the multiple surface items and a determined positioning of the surface items in three-dimensions upon the two-dimensional physical surface in the illustrative layout according to how the multiple surface items fit together and scale with the dimensions of the physical surface, while imaging the physical surface in the physical environment. ### Claim 16 (depends on 15) The system of claim 15, further comprising instructions that, when executed by the at least one processor, cause the system to: select, with a recommendation engine, a first surface item and a second surface item of the multiple surface items to be positioned upon the physical surface and visualized by the computing device using the augmented reality software, the selecting based at least in part on the determined surface dimensions of the physical surface and the three-dimensional data of each of the first surface item and the second surface item. ### Claim 17 (depends on 16) The system of claim 16, wherein the positioning of the first surface item and the second surface item upon the physical surface is determined based on the determined surface dimensions, the three-dimensional data of the first surface item and the second surface item, and one or more rules defining a rule-based spacing for the first surface item and the second surface item, wherein the rule-based spacing determines how the first surface item and the second surface item fit together and scale with the dimensions of the physical surface. ### Claim 18 (depends on 16) The system of claim 16, further comprising instructions that, when executed by the at least one processor, cause the recommendation engine to select at least one of the first surface item and the second surface item based on a product selection criterion received from the computing device. ### Claim 19 (depends on 16) The system of claim 16, further comprising instructions that, when executed by the at least one processor, cause the recommendation engine to select at least one of the first surface item and the second surface item based on a customer profile. ### Claim 20 (depends on 16) The system of claim 16, further comprising instructions that, when executed by the at least one processor, cause the recommendation engine to select at least one of the first surface item and the second surface item based on product availability. ### Claim 21 (depends on 16) The system of claim 16, further comprising instructions that, when executed by the at least one processor, cause the recommendation engine to select at least one of the first surface item and the second surface item based on a merchant profile. ### Claim 22 (depends on 15) The system of claim 15, wherein the feature data comprises three-dimensional data for the multiple surface items and the physical surface. ### Claim 23 (depends on 15) The system of claim 15, further comprising instructions that, when executed by the at least one processor, cause the system to determine, based on the surface data, a dimensional layout of the physical surface for the illustrative layout, wherein determining how to position of the multiple surface items comprises using the dimensional layout of the physical surface and how quantities of the multiple surface items fit on the physical surface. ### Claim 24 (depends on 15) The system of claim 15, wherein the illustrative layout of the physical surface comprises determined quantities of each of a first surface item and a second surface item positioned on the physical surface. ### Claim 25 A non-transitory computer-readable medium comprising processor executable instructions that, when executed by a processor, cause the processor to: receive feature data identifying a physical surface determined using feature points recognized by augmented reality software coupled to an imaging device operated by a computing device while the computing device is capturing the feature data from a physical environment in which the physical surface is located; determine how to position multiple surface items to fit together and scale with dimensions of the physical surface based on three-dimensional data of the surface items and two-dimensional surface dimensions of the physical surface determined from the feature points identified in the feature data obtained while in the physical environment comprising the physical surface; and cause an augmented reality interface to be displayed by the augmented reality software on the computing device, the augmented reality interface comprising an illustrative layout of the physical surface generated using the feature data and showing the multiple surface items and a determined positioning of the surface items in three-dimensions upon the two-dimensional physical surface in the illustrative layout according to how the multiple surface items fit together and scale with the dimensions of the physical surface, while imaging the physical surface in the physical environment.