# US 2014/0156459 A1 · In-store product detection system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0156459 A1 - Document ID: US20140156459A1 - Status: Published application - Kind: A1 - Application number: 13/693,126 - Filed: 2012-12-04 - Published: 2014-06-05 - Earliest claimed: 2012-12-04 - Assignees: eBay Inc. - Inventors: Matthew Scott Zises - Web page: https://patentgrep.com/patent/US20140156459A1 ## CPC - G06Q 30/0643: {graphically representing goods, e.g. 3D product representation} - G06Q 30/0623: {by investigating goods or services} - G06Q 30/0641: {utilising user interfaces specially adapted for shopping} - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract A method and system create, on a portable electronic device, a real-time graphical view of an in-store product display that pinpoints the location of one or more target products on the product display. The graphical view is generated by capturing merchandise display data, such as a video feed of the in-store product display, and processing the data to detect the one or more target products based on predefined product recognition criteria uniquely associated with the respective target products. ## Claims ### Claim 1 A method comprising: receiving merchandise display data captured by a portable electronic device at a store, the merchandise display data indicating physical presence of multiple products on a product display at the store; in an automated operation using one or more processors, detecting a target product among the multiple products on the in-store product display based on the merchandise display data; displaying a graphical view of at least a part of the in-store product display on a screen of the portable electronic device; and providing as a part of the graphical view a location indicator that indicates an on-screen position of the target product. ### Claim 2 (depends on 1) The method of claim 1, wherein the detecting of the target product comprises processing the merchandise display data onboard the portable electronic device to identify one or more product recognition criteria for the target product in the merchandise display data. ### Claim 3 (depends on 2) The method of claim 2, further comprising capturing image information by use of an onboard camera of the portable electronic device, the image information representing an image of at least part of the in-store product display, to provide the merchandise display data. ### Claim 4 (depends on 3) The method of claim 3, wherein the detecting of the target product comprises processing the image information to recognize a visual indicator that is unique to the target product. ### Claim 5 (depends on 4) The method of claim 4, wherein the visual indicator comprises one or more trade dress elements of a packaging of the target product. ### Claim 6 (depends on 4) The method of claim 4, wherein the visual indicator comprises a visual product code carried on the target product. ### Claim 7 (depends on 3) The method of claim 3, wherein capturing the image information comprises capturing video content at the portable electronic device, the displaying of the graphical view comprising displaying a video image based on the captured video content, the video image showing at least part of the in-store product display substantially in real time together with the location indicator, to continuously indicate the on-screen position of the target product in the video image. ### Claim 8 (depends on 3) The method of claim 3, wherein the receiving of the merchandise display data comprises receiving respective wireless identifier signals from the multiple products, the identifying of the product recognition criteria comprising identifying a particular one of the identifier signals as being a unique identifier signal of the target product. ### Claim 9 (depends on 8) The method of claim 8, further comprising receiving at the portable electronic device the one or more product recognition criteria that define one or more unique product attributes of the target product. ### Claim 10 (depends on 9) The method of claim 9, further comprising transmitting to a store computer system a search query for the target product, the product recognition criteria being received by the portable electronic device from the store computer system responsive to the search query. ### Claim 11 (depends on 10) The method of claim 10, wherein the search query comprises a search image, the method further comprising capturing the search image by use of the onboard camera of the portable electronic device. ### Claim 12 (depends on 11) The method of claim 11, wherein the search query comprises a visual product code unique to the target product, the method further comprising scanning an instance of the visual product code by use of the onboard camera of the portable electronic device. ### Claim 13 (depends on 2) The method of claim 2, wherein the one or more product recognition criteria is for a shopping list comprising a plurality of target products, the processing of the merchandise display data being to identify any of the plurality of target products, the method further comprising providing as part of the graphical view a respective location indicator for each of the plurality of target products that are identified. ### Claim 14 (depends on 13) The method of claim 13, further comprising receiving the one or more product recognition criteria for the shopping list from a home pantry system of a user of the portable electronic device. ### Claim 15 (depends on 14) The method of claim 14, further comprising: monitoring stock levels of products in the home pantry system; determining that a stock level for a particular product in the home pantry system is below a threshold level; responsive to the determination, automatically adding the particular product to the shopping list; and communicating the shopping list to the portable electronic device from the home pantry system by transmitting the one or more product recognition criteria for the respective target products forming part of the shopping list. ### Claim 16 A method comprising automatically recognizing a product of interest in video content captured by a mobile electronic device in a store, and displaying an augmented reality view of the video content substantially in real time on a screen of the mobile electronic device, the product of interest being highlighted in real time in the augmented reality view. ### Claim 17 A system comprising: a receiver module to receive receiving merchandise display data captured by a portable electronic device at a store, the merchandise display data indicating physical presence of multiple products on a product display at the store; a product locator module to detect a target product among the multiple products on the in-store product display based on the merchandise display data; a display module to display a graphical view of at least a part of the in-store product display on a screen of the portable electronic device; and an indicator module to provide in the graphical view a location indicator that indicates an on-screen position of the target product. ### Claim 18 (depends on 17) The system of claim 17, wherein the product locator module is provided by the portable electronic device, the locator module being configured to process the merchandise display data to identify one or more product recognition criteria for the target product in the merchandise display data. ### Claim 19 (depends on 18) The system of claim 18, wherein the product locator module is configured to process image information captured with an onboard camera of the portable electronic device, the image information comprising the merchandise display data and comprising one or more images of the in-store product display. ### Claim 20 (depends on 19) The system of claim 19, wherein the product locator module is configured to detect the target product by processing the image information to recognize a visual indicator that is unique to the target product. ### Claim 21 (depends on 20) The system of claim 20, wherein the product locator module is configured to recognize one or more trade dress elements of a packaging of the target product. ### Claim 22 (depends on 20) The system of claim 20, wherein the product locator module is configured to recognize a visual product code carried on the target product. ### Claim 23 (depends on 19) The system of claim 19, wherein the image information comprises video content captured by the onboard camera, the display module being configured such that the graphical view comprises a substantially real-time video image of the in-store product display, the indicator module being configured to dynamically adapt an on-screen position of the location indicator to coincide with the on-screen position of the target product. ### Claim 24 A portable electronic device that comprises a product detection module configured to: generate an augmented reality display of multiple consumer products based on image data captured by a camera of the device; detect a product of interest among the multiple consumer products; and dynamically pinpoint the product of interest in the augmented reality display. ### Claim 25 A non-transitory machine-readable storage medium storing instructions for a machine to: receive receiving merchandise display data captured by a portable electronic device at a store, the merchandise display data indicating physical presence of multiple products on a product display at the store; detect a target product among the multiple products on the in-store product display based on the merchandise display data; display a graphical view of at least a part of the in-store product display on a screen of the portable electronic device; and provide as a part of the graphical view a location indicator that indicates an on-screen position of the target product.