# US 2017/0372048 A1 · Image Based CAPTCHA Challenges

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0372048 A1
- Document ID: US20170372048A1
- Status: Published application
- Kind: A1
- Application number: 15/700,395
- Filed: 2017-09-11
- Published: 2017-12-28
- Earliest claimed: 2015-12-03
- Assignees: Google Inc.
- Applicants: Google Inc.
- Inventors: Wei Liu; Vinay Damodar Shet; Ying Liu; Aaron Malenfant; Haidong Shao; Hongshu Liao; Jiexing Gu; Edison Tan
- Web page: https://patentgrep.com/patent/US20170372048A1

## CPC

- G06F 21/316: {by observing the pattern of computer usage, e.g. typical user behaviour}
- G06F 18/40: Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
- G06F 21/36: by graphic or iconic representation
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- H04L 63/12: {Applying verification of the received information (cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32)}
- G06F 2221/2133: Verifying human interaction, e.g., Captcha
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Systems and methods of determining image characteristics are provided. More particularly, a first image having an unknown characteristic can be obtained. The first image can be provided to a plurality of user devices in a verification challenge. The verification challenge can include one or more instructions to be presented to a user of each user device. The instructions being determined based at least in part on the first image. User responses can be received, and an unknown characteristic of the first image can be determined based at least in part on the received responses. Subsequent to determining the unknown characteristic of the first image, one or more machine learning models can be trained based at least in part on the determined characteristic.

## Claims

### Claim 1

A computer-implemented method of determining image characteristics, the method comprising:
obtaining, by the one or more computing devices, one or more first images having an unknown characteristic;
providing, by the one or more computing devices, the one or more first images to a plurality of user devices in a verification challenge, the verification challenge comprising one or more instructions to be presented to a user of each user device, the instructions being determined based at least in part on the one or more first images;
receiving, by the one or more computing devices, a response to the verification challenge from at least a subset of the plurality of user devices;
determining, by the one or more computing devices, the unknown characteristic of the one or more first images based at least in part on the received responses; and
subsequent to determining, by the one or more computing devices, the unknown characteristic of the one or more first images, training, by the one or more computing devices, one or more machine learning models based at least in part on the determined characteristic.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising receiving, by one or more computing devices, a request to engage in a verification process from a user computing device, and wherein the one or more first images are obtained responsive to receiving the request.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
obtaining, by the one or more computing devices, a plurality of second images having known characteristics; and
providing, by the one or more computing devices, at least a subset of the plurality of second images along with the one or more first images in the verification challenge.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the unknown characteristic of the one or more first images is determined to be unknown based at least in part on the one or more machine learning models, the one or more machine learning models configured to receive one or more images as an input and to determine one or more characteristics of the one or more images as an output.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the one or more characteristics of the one or more images relate to one or more items or objects depicted in the one or more images.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, wherein the one or more machine learning models are further configured to provide a certainty probability indicative of a degree of certainty that the output of the model is correct.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the unknown characteristic of the one or more first images is determined to be unknown when the certainty probability associated with the one or more first images is below a certainty threshold.

### Claim 8 (depends on 4)

The computer-implemented method of claim 4, wherein the known characteristics of the plurality of second images are determined based at least in part on the one or more machine learning models.

### Claim 9 (depends on 3)

The computer-implemented method of claim 3, wherein the one or more instructions associated with the verification challenge prompts the user to interact with one or more of the first or second images that meet a criterion.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the one or more instructions associated with the verification challenge prompts the user to interact with at least a portion of the one or more first images that meets a criterion.

### Claim 11 (depends on 10)

The computer-implemented method of claim 10, wherein the one or more instructions associated with the verification challenge prompt the user to select a portion of at least one of the one or more first images.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the verification challenge is determined based at least in part on the unknown characteristic of the one or more first images.

### Claim 13 (depends on 3)

The computer-implemented method of claim 3, further comprising providing, by the one or more computing devices, a third image in the verification challenge, the third image depicting an item or object, and wherein the one or more instructions associated with the verification challenge prompt the user to select one or more of the one or more first images or the at least a subset of second images that depict an item or object that from the same classification as the item or object depicted in the third image.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, further comprising determining, by the one or more computing devices, whether to verify each of the plurality of user devices based at least in part on the responses to the verification challenge.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, wherein determining, by the one or more computing devices, the unknown characteristic of the one or more first images comprises inferring, by the one or more computing devices, the unknown characteristic of the one or more first images when a threshold number of user responses are indicative of a characteristic of the one or more first images.

### Claim 16

A computing system, comprising:
one or more processors; and
one or more memory devices, the one or more memory devices storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations, the operations comprising:
obtaining a first image having an unknown characteristic;
providing the first image to a plurality of user devices in a verification challenge, the verification challenge providing one or more instructions to the user based at least in part on the first image;
receiving a response to the verification challenge from at least a subset of the plurality of user devices;
determining the unknown characteristic of the first image based at least in part on the received responses; and
subsequent to determining the unknown characteristic of the first image, training one or more machine learning models based at least in part on the determined characteristic.

### Claim 17 (depends on 16)

The computing system of claim 16, further comprising:
obtaining a plurality of second images having known characteristics; and
providing at least a subset of the plurality of second images along with the first image in the verification challenge.

### Claim 18 (depends on 17)

The computing system of claim 17, wherein determining the unknown characteristic of the first image comprises inferring the unknown characteristic of the first image when a threshold number of user responses are indicative of a characteristic of the first image.

### Claim 19

One or more tangible, non-transitory computer-readable media storing computer-readable instructions that when executed by one or more processors cause the one or more processors to perform operations, the operations comprising:
obtaining a first image having an unknown characteristic;
providing the first image to a plurality of user devices in a verification challenge, the verification challenge providing one or more instructions to the user based at least in part on the first image;
receiving a response to the verification challenge from at least a subset of the plurality of user devices;
determining the unknown characteristic of the first image based at least in part on the received responses; and
subsequent to determining the unknown characteristic of the first image, training one or more machine learning models based at least in part on the determined characteristic.

### Claim 20 (depends on 19)

The one or more tangible, non-transitory computer-readable media of claim 19 wherein determining the unknown characteristic of the first image comprises inferring the unknown characteristic of the first image when a threshold number of user responses are indicative of a characteristic of the first image.
