# US 2026/0087783 A1 · Generating training data based on gaze captured at a source location for training a replacement model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0087783 A1
- Document ID: US20260087783A1
- Status: Published application
- Kind: A1
- Application number: 18/891,284
- Filed: 2024-09-20
- Published: 2026-03-26
- Earliest claimed: 2024-09-20
- Assignees: Maplebear Inc.
- Applicants: Maplebear Inc.
- Inventors: Sonal Jain; Julia Singer; Helen Kuo; Karuna Ahuja
- Web page: https://patentgrep.com/patent/US20260087783A1

## CPC

- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06Q 30/0631: {Recommending goods or services}
- G06V 40/197: {Matching; Classification}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

An online system receives information captured by a gaze tracking device describing a gaze point of a user and video data captured within a source location, detects a location associated with a first item that matches the gaze point based on the received information, and determines the first item is not available at the source location based on the video data. The system receives a signal indicating the user collected a second item from the source location, determines the second item is a replacement for the first item, and generates a new training example indicating the second item is an acceptable replacement for the first item for the user. The system trains a machine-learning model to generate a score indicating whether a candidate item is an acceptable replacement for a target item for a user, in which the model is trained using training data that includes the new training example.

## Claims

### Claim 1

A method, performed at a computer system comprising a processor and a computer-readable medium, comprising:
receiving, at an online system, information captured by a gaze tracking device describing a gaze point of a user and video data captured within a source location;
detecting, within the source location, an item location associated with a first item that matches the gaze point of the user based at least in part on the received information;
determining that the first item is not available at the source location based at least in part on the video data;
receiving a signal indicating that the user collected a second item from the source location;
determining that the second item is a replacement for the first item;
generating a new training example for a training dataset, wherein the new training example indicates the second item is an acceptable replacement for the first item for the user;
training a machine-learning model to generate a score indicating whether a candidate item is an acceptable replacement for a target item for a particular user of the online system, wherein the machine-learning model is trained using the training dataset that includes the new training example; and
storing parameters of the trained machine-learning model on a non-transitory computer-readable medium.

### Claim 2 (depends on 1)

The method of claim 1, wherein detecting, within the source location, the item location associated with the first item that matches the gaze point of the user based at least in part on the received information comprises:
detecting, within the source location, the item location associated with the first item that matches the gaze point of the user based on a layout of the source location, wherein the layout of the source location describes a set of item locations within the source location associated with each item of a plurality of items included among an inventory of the source location.

### Claim 3 (depends on 2)

The method of claim 2, wherein detecting, within the source location, the item location associated with the first item that matches the gaze point of the user based on the layout of the source location comprises:
comparing a portion of the video data that matches the gaze point of the user with the layout of the source location; and
detecting, within the source location, the item location associated with the first item that matches the gaze point of the user based at least in part on the comparing.

### Claim 4 (depends on 1)

The method of claim 1, wherein detecting, within the source location, the item location associated with the first item that matches the gaze point of the user based at least in part on the received information comprises:
applying one or more computer vision algorithms to a portion of the video data that matches the gaze point of the user to detect the item location associated with the first item that matches the gaze point of the user.

### Claim 5 (depends on 1)

The method of claim 1, wherein determining that the first item is not available at the source location based at least in part on the video data comprises:
accessing an image of the first item;
applying one or more computer vision algorithms to the video data to detect one or more objects depicted in the video data;
determining whether the one or more objects depicted in the video data match the image of the first item; and
responsive to determining that the one or more objects depicted in the video data do not match the image of the first item, determining that the first item is not available at the source location.

### Claim 6 (depends on 1)

The method of claim 1, wherein generating the new training example for the training dataset comprises:
including, in the new training example, a set of user data for the user, wherein the set of user data for the user comprises a set of preferences of the user.

### Claim 7 (depends on 6)

The method of claim 6, wherein training the machine-learning model to generate the score indicating whether a candidate item is an acceptable replacement for a target item for a particular user of the online system comprises:
receiving item data for a plurality of items included among one or more inventories of one or more source locations;
receiving user data for a plurality of users of the online system;
receiving, for each pair of an item and an additional item included among the plurality of items, a label indicating whether the item is an acceptable replacement for the additional item for a set of users of the online system; and
training the machine-learning model based at least in part on the item data, the user data, and the label for each pair of an item and an additional item included among the plurality of items.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining that the second item is a replacement for the first item is based at least in part on one or more of: a measure of similarity between the first item and the second item, a hierarchical taxonomy into which the first item and the second item are organized, a proximity between the item location associated with the first item and a location at which the second item was collected, an amount of time elapsed since a time that the gaze point of the user matched the item location associated with the first item and a time that the user collected the second item, or the score indicating whether the second item is an acceptable replacement for the first item.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
receiving a request from a client device associated with a picker to recommend an acceptable replacement for the first item for an additional user of the online system, wherein the request includes information describing the source location;
retrieving a set of item data for the first item and for each candidate item of a set of candidate items included among an inventory of the source location;
retrieving a set of user data for the additional user;
accessing the machine-learning model trained to generate the score indicating whether a candidate item is an acceptable replacement for a target item for a particular user of the online system;
for each candidate item of the set of candidate items, applying the machine-learning model to generate the score indicating whether a corresponding candidate item is an acceptable replacement for the first item for the additional user of the online system based at least in part on the set of user data for the additional user and the set of item data for the first item and the corresponding candidate item;
ranking the set of candidate items based at least in part on the score indicating whether each candidate item is an acceptable replacement for the first item for the additional user;
selecting, from the set of candidate items, a replacement for the first item for the additional user based at least in part on the ranking; and
storing parameters of the trained machine-learning model on a non-transitory computer-readable medium.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
sending information describing the selected replacement for the first item for the additional user to the client device associated with the picker.

### Claim 11

A computer program product comprising a non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps comprising:
receiving, at an online system, information captured by a gaze tracking device describing a gaze point of a user and video data captured within a source location;
detecting, within the source location, an item location associated with a first item that matches the gaze point of the user based at least in part on the received information;
determining that the first item is not available at the source location based at least in part on the video data;
receiving a signal indicating that the user collected a second item from the source location;
determining that the second item is a replacement for the first item;
generating a new training example for a training dataset, wherein the new training example indicates the second item is an acceptable replacement for the first item for the user; and
training a machine-learning model to generate a score indicating whether a candidate item is an acceptable replacement for a target item for a particular user of the online system, wherein the machine-learning model is trained using the training dataset that includes the new training example.

### Claim 12 (depends on 11)

The computer program product of claim 11, wherein detecting, within the source location, the item location associated with the first item that matches the gaze point of the user based at least in part on the received information comprises:
detecting, within the source location, the item location associated with the first item that matches the gaze point of the user based on a layout of the source location, wherein the layout of the source location describes a set of item locations within the source location associated with each item of a plurality of items included among an inventory of the source location.

### Claim 13 (depends on 12)

The computer program product of claim 12, wherein detecting, within the source location, the item location associated with the first item that matches the gaze point of the user based on the layout of the source location comprises:
comparing a portion of the video data that matches the gaze point of the user with the layout of the source location; and
detecting, within the source location, the item location associated with the first item that matches the gaze point of the user based at least in part on the comparing.

### Claim 14 (depends on 11)

The computer program product of claim 11, wherein detecting, within the source location, the item location associated with the first item that matches the gaze point of the user based at least in part on the received information comprises:
applying one or more computer vision algorithms to a portion of the video data that matches the gaze point of the user to detect the item location associated with the first item that matches the gaze point of the user.

### Claim 15 (depends on 11)

The computer program product of claim 11, wherein determining that the first item is not available at the source location based at least in part on the video data comprises:
accessing an image of the first item;
applying one or more computer vision algorithms to the video data to detect one or more objects depicted in the video data;
determining whether the one or more objects depicted in the video data match the image of the first item; and
responsive to determining that the one or more objects depicted in the video data do not match the image of the first item, determining that the first item is not available at the source location.

### Claim 16 (depends on 11)

The computer program product of claim 11, wherein generating the new training example for the training dataset comprises:
including, in the new training example, a set of user data for the user, wherein the set of user data for the user comprises a set of preferences of the user.

### Claim 17 (depends on 16)

The computer program product of claim 16, wherein training the machine-learning model to generate the score indicating whether a candidate item is an acceptable replacement for a target item for a particular user of the online system comprises:
receiving item data for a plurality of items included among one or more inventories of one or more source locations;
receiving user data for a plurality of users of the online system;
receiving, for each pair of an item and an additional item included among the plurality of items, a label indicating whether the item is an acceptable replacement for the additional item for a set of users of the online system; and
training the machine-learning model based at least in part on the item data, the user data, and the label for each pair of an item and an additional item included among the plurality of items.

### Claim 18 (depends on 11)

The computer program product of claim 11, wherein determining that the second item is a replacement for the first item is based at least in part on one or more of: a measure of similarity between the first item and the second item, a hierarchical taxonomy into which the first item and the second item are organized, a proximity between the item location associated with the first item and a location at which the second item was collected, an amount of time elapsed since a time that the gaze point of the user matched the item location associated with the first item and a time that the user collected the second item, or the score indicating whether the second item is an acceptable replacement for the first item.

### Claim 19 (depends on 11)

The computer program product of claim 11, wherein the computer-readable storage medium further has instructions encoded thereon that, when executed by the processor, cause the processor to perform steps comprising:
receiving a request from a client device associated with a picker to recommend an acceptable replacement for the first item for an additional user of the online system, wherein the request includes information describing the source location;
retrieving a set of item data for the first item and for each candidate item of a set of candidate items included among an inventory of the source location;
retrieving a set of user data for the additional user;
accessing the machine-learning model trained to generate the score indicating whether a candidate item is an acceptable replacement for a target item for a particular user of the online system;
for each candidate item of the set of candidate items, applying the machine-learning model to generate the score indicating whether a corresponding candidate item is an acceptable replacement for the first item for the additional user of the online system based at least in part on the set of user data for the additional user and the set of item data for the first item and the corresponding candidate item;
ranking the set of candidate items based at least in part on the score indicating whether each candidate item is an acceptable replacement for the first item for the additional user;
selecting, from the set of candidate items, a replacement for the first item for the additional user based at least in part on the ranking; and
sending information describing the selected replacement for the first item for the additional user to the client device associated with the picker.

### Claim 20

A computer system comprising:
a processor; and
a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, perform actions comprising:
receiving, at an online system, information captured by a gaze tracking device describing a gaze point of a user and video data captured within a source location;
detecting, within the source location, an item location associated with a first item that matches the gaze point of the user based at least in part on the received information;
determining that the first item is not available at the source location based at least in part on the video data;
receiving a signal indicating that the user collected a second item from the source location;
determining that the second item is a replacement for the first item;
generating a new training example for a training dataset, wherein the new training example indicates the second item is an acceptable replacement for the first item for the user;
training a machine-learning model to generate a score indicating whether a candidate item is an acceptable replacement for a target item for a particular user of the online system, wherein the machine-learning model is trained using the training dataset that includes the new training example; and
storing parameters of the trained machine-learning model on a non-transitory computer-readable medium.
