# US 2021/0176519 A1 · System and method for in-video product placement and in-video purchasing capability using augmented reality with automatic continuous user authentication

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0176519 A1
- Document ID: US20210176519A1
- Status: Published application
- Kind: A1
- Application number: 17/111,394
- Filed: 2020-12-03
- Published: 2021-06-10
- Earliest claimed: 2019-06-06
- Assignees: Augmented And Segmented Media Interface Corp.
- Applicants: Augmented and Segmented Media Interface Corp.
- Inventors: Sambhab Thapaliya
- Web page: https://patentgrep.com/patent/US20210176519A1

## CPC

- H04N 21/4316: {for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window}
- H04N 21/812: {involving advertisement data (advertising per se G06Q30/02)}
- H04M 1/72454: according to context-related or environment-related conditions
- H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)}
- H04N 21/42222: {Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device}
- H04N 21/441: Acquiring end-user identification {, e.g. using personal code sent by the remote control or by inserting a card}
- H04N 21/4524: {involving the geographical location of the client (retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00)}
- H04N 21/4662: {characterized by learning algorithms}
- H04N 21/4666: {using neural networks, e.g. processing the feedback provided by the user}
- H04N 21/4667: {Processing of monitored end-user data, e.g. trend analysis based on the log file of viewer selections}
- H04N 21/47815: {Electronic shopping (payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- H04M 1/72442: for playing music files
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/23424: {involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement}
- H04N 21/252: {Processing of multiple end-users' preferences to derive collaborative data}
- H04N 21/2542: {for selling goods, e.g. TV shopping (payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12)}
- H04N 21/26603: {for automatically generating descriptors from content, e.g. when it is not made available by its provider, using content analysis techniques}
- H04N 21/47202: {for requesting content on demand, e.g. video on demand}
- H04N 21/4725: using interactive regions of the image, e.g. hot spots {(details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94)}
- H04N 21/6582: {Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number (arrangements where receivers interact with the broadcast H04H20/38)}
- H04N 21/8583: {by creating hot-spots}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Techniques are provided by which the digital delivery of a viewer-requested video along with the best chosen advertisement for the viewer is improved. These techniques may be particularly suited for the short video industry. An innovative video analytics mechanism and user-behavioral analytics mechanism are provided, with which the best match of an exact product on the video for the particular viewer is advertised on the video, while the viewer is viewing the video. Further, techniques are provided that enable the viewer to purchase the product while still in the video, not having to leave the video or the site to complete the purchase. Further, techniques are provided by which the users and their smartphones are automatically continuously being authenticated.

## Claims

### Claim 1

A system, comprising:
a smartphone, wherein the smartphone comprises: an accelerometer sensor, a gyroscope sensor, and a GPS sensor, wherein each sensor collects raw data from manipulations of the smartphone for purposes of user authentication;
a rest filter model configured to receive the raw data and configured to process the raw data using a motion detection module to identify valid data and invalid data therefrom, wherein valid data is data reflecting motion of the smartphone and invalid data is data reflecting the smartphone at rest;
a user authentication convolutional neural network (“CNN”) model configured to receive the identified valid and invalid data and generate a prediction that the manipulations of the smartphone are from a legitimate user or from an intruder; and
a server configured to train and store the user authentication CNN model and further configured to download the trained user authentication CNN model to the smartphone on a predetermined time basis.

### Claim 2 (depends on 1)

The system of claim 1, wherein the system is configured to process a testing phase and a training phase, and when the user authentication CNN model is trained, the smartphone is configured to perform continuous user authentication.

### Claim 3 (depends on 1)

The system of claim 1, wherein raw data reflects typical activities including typing, scrolling, calling, chatting, walking, holding, and location, and the user authentication CNN model determines one or more patterns of typical behavior for a user, based on the raw data reflects typical activities.

### Claim 4 (depends on 1)

The system of claim 1, wherein said system is a component of a network-accessible platform that enables in-video product purchasing and wherein the system is configured to provide automatic user authentication is response to a request for purchasing an in-video product.

### Claim 5 (depends on 1)

The system of claim 1, wherein the user authentication CNN model is trained on an on-going basis.

### Claim 6 (depends on 1)

The system of claim 1, wherein the smartphone is configured to collect the raw data in local storage up to 5 mb and when over 5 mb, the smartphone automatically uploads the raw data to the server for training the user authentication CNN model and the smartphone is configured to send the GPS location data to the server along with the raw data.

### Claim 7 (depends on 1)

The system of claim 1, wherein the sensors are configured to, at testing phase, continuously accumulate data at 50 Hz.

### Claim 8 (depends on 1)

The system of claim 1, wherein the smartphone further comprises a feature extraction component that is configured to extract features from the data and feed such extracted features to the rest filter model and wherein the rest filter model is further configured to receive the extracted features and use the features for identifying the invalid data and further configured to send the valid data to the user authentication CNN model.

### Claim 9 (depends on 8)

The system of claim 8, wherein the features comprise:
absolute energy;
area under the curve;
interquartile range;
kurtosis;
maximum value;
minimum value;
mean;
median absolute deviation;
standard deviation;
maximum frequency;
entropy;
negative turning points;
ECDF (empirical cumulative distribution function) Percentile along the time axis;
median absolute difference;
spectral distance;
wavelet energy;
wavelet variance; and
power spectrum density bandwidth.

### Claim 10 (depends on 1)

The system of claim 1, wherein the user authentication CNN model is configured to classify a user as legitimate or intruder when the model consistently outputs the same class (legitimate or intruder) thrice and the classified output is considered to be valid when the prediction probability is above 0.8.

### Claim 11 (depends on 1)

The system of claim 1 is further configured to use GPS to track a legitimate user by using a predetermined circle of radius, with center as the user's recorded, during training, GPS location, that created, wherein the radius is the threshold that is considered and wherein during testing, if the current GPS location of the user's lies within a circle, the identification of the location is determined valid and no penalty is given to its predicted probability, else the application assigns a penalty of 0.85, wherein enforcing the penalty improves the robustness of the innovative user authentication CNN model because, to overcome the penalty the user is required to have a prediction output greater than a predetermined value.

### Claim 12 (depends on 1)

The system of claim 1, wherein the smartphone is configured to push data to the server every 5 mb at a time and wherein the server is configured so that once the uploaded data achieves an accuracy rate that is flattening out, the data uploading stops.

### Claim 13 (depends on 1)

The system of claim 1, wherein the smartphone is configured to collect data when the smartphone detects motion and at every 12 seconds.

### Claim 14 (depends on 1)

The system of claim 1, wherein the user authentication CNN model runs in a 90 second time interval.

### Claim 15 (depends on 1)

The system of claim 1, wherein the user authentication CNN model is configured to constantly learn from user data by being configured to determine that once a measurement of confidence is above a par value of 80%, then the model is considered confident is marked trained.

### Claim 16 (depends on 1)

The system of claim 1, wherein when a user is watching a plurality of videos and they like an object on a particular video, the user is enabled to click and buy the object based on the user authentication CNN model producing a output that indicates that the user is legitimate.

### Claim 17 (depends on 1)

The system of claim 1, wherein the system is further configured to run the user authentication CNN model again if the user is classified as an intruder to confirm and if the user is classified as an intruder more than three times, then another test is performed in server and the user is marked illegitimate.

### Claim 18 (depends on 1)

The system of claim 1, wherein when there is no moment in the gyroscope or accelerometer sensors for more than 10 seconds, the smartphone is marked as idle and the data are not further processed.

### Claim 19

A method, comprising:
collecting raw data from manipulations of a smartphone for purposes of user authentication, wherein the smartphone comprises: an accelerometer sensor, a gyroscope sensor, and a GPS sensor, for collecting the data;
receiving, at a rest filter model, the raw data and processing the raw data using a motion detection module to identify valid data and invalid data therefrom, wherein valid data is data reflecting motion of the smartphone and invalid data is data reflecting the smartphone at rest;
receiving, by a user authentication convolutional neural network (“CNN”) model, the identified valid and invalid data and generating therefrom a prediction that the manipulations of the smartphone are from a legitimate user or from an intruder; and
training and storing, by a server, the user authentication CNN model and downloading the trained user authentication CNN model to the smartphone on a predetermined time basis.

### Claim 20

A non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, performs the steps of:
collecting raw data from manipulations of a smartphone for purposes of user authentication, wherein the smartphone comprises: an accelerometer sensor, a gyroscope sensor, and a GPS sensor, for collecting the data;
receiving, at a rest filter model, the raw data and processing the raw data using a motion detection module to identify valid data and invalid data therefrom, wherein valid data is data reflecting motion of the smartphone and invalid data is data reflecting the smartphone at rest;
receiving, by a user authentication convolutional neural network (“CNN”) model, the identified valid and invalid data and generating therefrom a prediction that the manipulations of the smartphone are from a legitimate user or from an intruder; and
training and storing, by a server, the user authentication CNN model and downloading the trained user authentication CNN model to the smartphone on a predetermined time basis.
