# US 2019/0318408 A1 · Concurrent visualization tool using augmented reality to improve decision making

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318408 A1
- Document ID: US20190318408A1
- Status: Published application
- Kind: A1
- Application number: 15/950,935
- Filed: 2018-04-11
- Published: 2019-10-17
- Earliest claimed: 2018-04-11
- Assignees: Wells Fargo Bank, N.A.
- Applicants: Wells Fargo Bank, N.A.
- Inventors: Waverly W. Chao; Haritha Devarajan; Thomas W. Gross; Kristine Ing Kushner; Muhammad Farukh Munir; Pamela M. Murphy
- Web page: https://patentgrep.com/patent/US20190318408A1

## CPC

- G06Q 30/0631: {Recommending goods or services}
- G02B 27/017: {Head mounted}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- H04W 4/025: {using location based information parameters}
- H04W 4/35: for the management of goods or merchandise
- H04W 4/80: Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

Examples described herein relate to apparatuses and methods of providing real-time notifications on affordability and advisability of purchasing goods or services using augmented reality. The method includes receiving, from AR device associated with a user, visual data representing a view through a viewing area of the AR device. The visual data comprises geolocation data indicating a geolocation of the AR device. The method includes identifying an image of an object included in the visual data. The method includes determining, based on the visual data, a product identifier associated with the object. The method includes determining an attempt by the user to purchase the object. The method includes generating, in response to determining the attempt, recommendation data based on the product identifier and an account associated with the user. The method includes sending the recommendation data to the AR device to cause the AR device to display the recommendation data to the user.

## Claims

### Claim 1

A method of providing real-time notifications on affordability and advisability of purchasing goods or services using augmented reality (AR), the method comprising:
receiving, by a recommendation server and from an AR device associated with a user, visual data representing a view through a viewing area of the AR device, the visual data comprises geolocation data indicating a geolocation of the AR device;
identifying, by the recommendation server, an image of an object included in the visual data;
determining, by the recommendation server and based on the visual data, a product identifier associated with the object;
determining, by the recommendation server, an attempt by the user to purchase the object;
generating, by the recommendation server and in response to determining the attempt, recommendation data based on the product identifier and an account associated with the user; and
sending, by the recommendation server, the recommendation data to the AR device to cause the AR device to display the recommendation data to the user.

### Claim 2 (depends on 1)

The method of claim 1, wherein identifying the object comprises:
determining the object occupies a predetermined amount of a predetermined region of the viewing area of the AR device.

### Claim 3 (depends on 1)

The method of claim 1, wherein identifying the object comprises:
determining the object occupies a foreground of the viewing area of the AR device.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining the product identifier comprises:
extracting, by the recommendation server, the image of the object from the visual data;
comparing, by the recommendation server, the extracted image and a plurality of images to identify an image of the plurality of images matching the extracted image, wherein each image of the plurality of images comprises a known object; and
retrieving, by the recommendation server and from a database, a product identifier associated with the image of the plurality of images.

### Claim 5 (depends on 1)

The method of claim 1, wherein determining the attempt by the user to purchase the object comprises:
extracting, by the recommendation server, the geolocation data from the visual data;
comparing, by the recommendation server, the extracted geolocation data and historical geolocation data associated with the account of the user to determine a number of matches; and
determining, by the recommendation server, the number of matches exceeds a predetermined threshold.

### Claim 6 (depends on 1)

The method of claim 1, wherein determining the attempt by the user to purchase the object comprises:
comparing, by the recommendation server, the product identifier associated with the object and contextual data associated with the account of the user; wherein the contextual data extracted from at least one of a social media account, a text message, and a transcript of a voice call.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the recommendation data comprises:
discovering, by the recommendation server and based on the account of the user, a predetermined financial milestone associated with the user; and
comparing, by the recommendation server, the predetermined financial milestone, purchase data associated with a purchase of the object, and the account of the user to determine an affordability value, wherein the affordability value indicates whether the purchase of the object prevents the user from achieving the predetermined financial milestone; and
generating, by the recommendation server, the recommendation data based on the affordability value.

### Claim 8 (depends on 7)

The method of claim 7, wherein generating the recommendation data comprises:
selecting, by the recommendation server based on the affordability value, a cross-product associated with the product identifier; and
generating, by the recommendation server, the recommendation data based on the plurality of cross-products, the product identifier, and the account associated with the user;
wherein the recommendation data causes the AR device to display the cross-product to the user.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the recommendation data comprises:
discovering, by the recommendation server and based on the account of the user, a predetermined financial milestone associated with the user;
retrieving, by the recommendation server and from a database, visual content associated with the user;
generating, by the recommendation server and based on the visual content, a digital video simulating the user interacting with an object associated with the predetermined financial milestone; and
generating, by the recommendation server, the recommendation data based on the digital video, the product identifier, and the account associated with the user; wherein the recommendation data causes the AR device to display the digital video to the user.

### Claim 10 (depends on 1)

The method of claim 1, wherein the recommendation data causes the AR device to display the recommendation data in a first region of the viewing area of the AR device separate from a second region of the viewing area of the AR device occupied by the object.

### Claim 11 (depends on 1)

The method of claim 1, wherein the recommendation data causes the AR device to display an indicator deterring the user from purchasing the object.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
receiving, by the recommendation server and from the AR device, a request to share the recommendation data with a second AR device associated with a second user; and
sending, by the recommendation server, the recommendation data to the second AR device to cause the second AR device to display the recommendation data to the second user.

### Claim 13

A system for providing real-time notifications on affordability and advisability of purchasing goods or services using augmented reality (AR), the system comprising a recommendation server configured to:
receive from an AR device associated with a user, visual data representing a view through a viewing area of the AR device, the visual data comprises geolocation data indicating a geolocation of the AR device;
identify an image of an object included in the visual data;
determine, based on the visual data, a product identifier associated with the object;
determine an attempt by the user to purchase the object;
generate, in response to determining the attempt, recommendation data based on the product identifier and an account associated with the user; and
send the recommendation data to the AR device to cause the AR device to display the recommendation data to the user.

### Claim 14 (depends on 13)

The system of claim 13, wherein the recommendation server further configured to at least one of:
determine the object occupies a predetermined amount of a predetermined region of the viewing area of the AR device; and
determine the object occupies a foreground of the viewing area of the AR device.

### Claim 15 (depends on 13)

The system of claim 13, wherein the recommendation server further configured to:
extract the image of the object from the visual data;
compare the extracted image and a plurality of images to identify an image of the plurality of images matching the extracted image, wherein each image of the plurality of images comprises a known object; and
retrieve, from a database, a product identifier associated with the image of the plurality of images.

### Claim 16 (depends on 13)

The system of claim 13, wherein the recommendation server further configured to:
extract the geolocation data from the visual data;
compare the extracted geolocation data and historical geolocation data associated with the account of the user to determine a number of matches; and
determine the number of matches exceeds a predetermined threshold.

### Claim 17 (depends on 13)

The system of claim 13, wherein the recommendation server further configured to:
compare, by the recommendation server, the product identifier associated with the object and contextual data associated with the account of the user; wherein the contextual data extracted from at least one of a social media account, a text message, and a transcript of a voice call.

### Claim 18 (depends on 13)

The system of claim 13, wherein the recommendation server further configured to:
discover, based on the account of the user, a predetermined financial milestone associated with the user;
compare the financial milestone, purchase data associated with a purchase of the object, and the account of the user to determine an affordability value, wherein the affordability value indicates whether the purchase of the object prevents the user from achieving the financial milestone;
select, based on the affordability value, a cross-product associated with the product identifier; and
generate the recommendation data based on the plurality of cross-products, the product identifier, and the account associated with the user, wherein the recommendation data causes the AR device to display the cross-product and wherein the recommendation data causes the AR device to display an indicator deterring the user from purchasing the object.

### Claim 19 (depends on 13)

The system of claim 13, wherein the recommendation server further configured to:
discover, based on the account of the user, a financial milestone predetermined by the user;
retrieve, from a database, visual content associated with the user;
generate, based on the visual content, a digital video simulating the user interacting with an object associated with the financial milestone; and
generate the recommendation data based on the digital video, the product identifier, and the account associated with the user; wherein the recommendation data causes the AR device to display the digital video to the user.

### Claim 20

A method for providing real-time notifications on affordability and advisability of purchasing goods or services using augmented reality, the method comprising:
detecting, by AR device associated with a user, an observation of an object by a user through a viewing area of the AR device;
capturing, by the AR device and in response to the detection of the observation, visual data representing a view through the viewing area of the AR device;
sending, by the AR device, the visual data to a recommendation server, the visual data comprises geolocation data indicating the geolocation of the AR device;
receiving, by the AR device, recommendation data from the recommendation server, the recommendation data indicating whether a purchase of the object prevents the user from achieving a predetermined financial milestone associated with the user;
displaying, by the AR device and in the viewing area of the AR device, the recommendation data to the user;
detecting, by the AR device and in the viewing area of the AR device, an overlap of the recommendation data and the object; and
moving, by the AR device, the recommendation data to a region of the viewing area of the AR device to avoid the overlap.