# US 11,029,834 B2 · Utilizing biometric feedback to allow users to scroll content into a viewable display area

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,029,834 B2
- Document ID: US11029834B2
- Status: Granted
- Kind: B2
- Application number: 15/848,677
- Filed: 2017-12-20
- Published: 2019-06-20
- Granted: 2021-06-08
- Earliest claimed: 2017-12-20
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: James E. Bostick; John M. Ganci, Jr.; Martin G. Keen; Sarbajit K. Rakshit
- Web page: https://patentgrep.com/patent/US11029834B2

## CPC

- G06F 3/0485: Scrolling or panning
- G06F 3/013: {Eye tracking input arrangements (G06F3/015 takes precedence)}
- G06F 3/015: {Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection}
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06N 20/00: Machine learning
- G06V 40/174: {Facial expression recognition}
- G06V 40/20: Movements or behaviour, e.g. gesture recognition (recognition of facial expressions G06V40/16)
- A61F 4/00: Methods or devices enabling patients or disabled persons to operate an apparatus or a device not forming part of the body
- G06F 3/012: {Head tracking input arrangements}
- G06F 2203/011: Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
- G06V 40/15: {Biometric patterns based on physiological signals, e.g. heartbeat, blood flow}
- G10L 25/63: for estimating an emotional state
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- A61F: FILTERS IMPLANTABLE INTO BLOOD VESSELS; PROSTHESES; DEVICES PROVIDING PATENCY TO, OR PREVENTING COLLAPSING OF, TUBULAR STRUCTURES OF THE BODY, e.g. STENTS; ORTHOPAEDIC, NURSING OR CONTRACEPTIVE DEVICES; FOMENTATION; TREATMENT OR PROTECTION OF EYES OR EARS; BANDAGES, DRESSINGS OR ABSORBENT PADS; FIRST-AID KITS (dental prosthetics A61C)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Systems and methods are provided for displaying visual content on an electronic device to a user. In some aspects, a viewable display area on a display of the device is determined, the viewable display area representing the portion of the display that the user is capable of seeing. Biometric data representing a state of the user is obtained from biometric device(s). Each state is mapped to one or more predefined control functions, the control functions determining the scrolling direction and scrolling speed of text through the viewable display area of the electronic device. A control function is selected based on a current state of the user, and the selected control function is used to control scrolling of visual content in real time or near real time. The system comprises a machine learning system to provide advanced adaptive capabilities.

## Claims

### Claim 1

A method of providing visual content displayed on an electronic device to a motor-impaired user, the method comprising:
determining, for the motor-impaired user, a viewable display area on a display of the electronic device, wherein the viewable display area is a fixed area corresponding to a stationary location on the display and wherein the viewable display area is smaller in size than the display;
determining a baseline state of the motor-impaired user;
analyzing the visual content to divide into multiple sections comprising key objects;
dividing the visual content into the multiple sections;
determining a content navigation path to scroll the multiple sections of the visual content based on a relevance or prominence of each key object of the key objects;
automatically scrolling the visual content through the viewable display area according to the determined content navigation path;
obtaining biometric data including motor-impaired user data from the motor-impaired user, wherein the biometric data is used by a cognitive control system having a processor to determine a state of the motor-impaired user selected from a plurality of states, the plurality of states corresponding to a stress or interest level of the motor-impaired user, and wherein the biometric data obtained from the motor-impaired user is selected from a group consisting of a heart rate, a tone of voice, and a rate of respiration;
mapping, by the cognitive control system having the processor, each state of the plurality of states to one or more control functions;
selecting a control function from the one or more control functions based on the determined state of the motor-impaired user; and
adjusting, via the selected control function, the scrolling of the visual content outside the viewable display area through the viewable display area on the electronic device to return the determined state of the motor-impaired user to the baseline state, wherein additional biometric data is obtained and processed to determine that the determined state of the motor-impaired user has returned to the baseline state, and wherein the additional biometric data obtained from the motor-impaired user is selected from the group consisting of the heart rate, the tone of voice, and the rate of respiration.

### Claim 2 (depends on 1)

The method of claim 1, wherein the viewable display area is determined by calculating a user distance and a viewing angle of the motor-impaired user relative to the display.

### Claim 3 (depends on 1)

The method of claim 1, further comprising continuously modifying the one or more control functions to control the scrolling of the visual content based upon real-time or near real-time biometric data obtained from the motor-impaired user.

### Claim 4 (depends on 1)

The method of claim 1, wherein the biometric data is obtained from one or more wearable biometric devices.

### Claim 5 (depends on 1)

The method of claim 1, wherein the biometric data includes data corresponding to an emotional, psychological, or a physiological state of the motor-impaired user, and wherein a machine learning system maps the biometric data to the determined state of the motor-impaired user.

### Claim 6 (depends on 1)

The method of claim 1, wherein the one or more control functions are scroll up, scroll left, scroll down, scroll right, provide confirmation, increase scrolling speed, decrease scrolling speed, or pause.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
receiving a voice instruction from the motor-impaired user; and
moving the visual content into the viewable display area based on the voice instruction.

### Claim 8

A system for providing visual content displayed on an electronic device to a motor-impaired user, the system comprising at least one processor configured to:
determine, for the motor-impaired user, a viewable display area on a display of the electronic device,
wherein the viewable display area is a fixed area corresponding to a stationary location on the display and wherein the viewable display area is smaller in size than the display;
determine a baseline state of the motor-impaired user;
analyze the visual content to divide into multiple sections comprising key objects;
divide the visual content into the multiple sections;
determine a content navigation path to scroll the multiple sections of the visual content based on a relevance or prominence of each key object of the key objects;
automatically scroll the visual content through the viewable display area according to the determined content navigation path;
obtain biometric data including motor-impaired user data from the motor-impaired user, wherein the biometric data is used by a cognitive control system via the at least one processor to determine a state of the motor-impaired user selected from a plurality of states, the plurality of states corresponding to a stress or interest level of the motor-impaired user, and wherein the biometric data obtained from the motor-impaired user is selected from a group consisting of a heart rate, a tone of voice, and a rate of respiration;
map, by the cognitive control system via the at least one processor, each state of the plurality of states to one or more control functions;
select a control function from the one or more control functions based on the determined state of the motor-impaired user; and
adjust, via the selected control function, the scrolling of the visual content outside the viewable display area through the viewable display area on the electronic device to return the determined state of the motor-impaired user to the baseline state, wherein additional biometric data is obtained and processed to determine that the determined state of the motor-impaired user has returned to the baseline state, and wherein the additional biometric data obtained from the motor-impaired user is selected from the group consisting of the heart rate, the tone of voice, and the rate of respiration.

### Claim 9 (depends on 8)

The system of claim 8, wherein the viewable display area is determined by calculating a user distance and a viewing angle of the motor-impaired user relative to the display.

### Claim 10 (depends on 8)

The system of claim 8, wherein the at least one processor is further configured to continuously modify the one or more control functions to control the scrolling of the visual content based upon real-time or near real-time biometric data obtained from the motor-impaired user.

### Claim 11 (depends on 8)

The system of claim 8, wherein the biometric data is obtained from one or more wearable biometric devices.

### Claim 12 (depends on 8)

The system of claim 8, wherein the biometric data includes data corresponding to an emotional, psychological, or a physiological state of the motor-impaired user, and wherein a machine learning system maps the biometric data to the determined state of the motor-impaired user.

### Claim 13 (depends on 8)

The system of claim 8, wherein the at least one processor is further configured to:
receive a voice instruction from the motor-impaired user; and
move the visual content into the viewable display area based on the voice instruction.

### Claim 14

A computer program product for providing visual content displayed on an electronic device to a motor-impaired user, the computer program product comprising one or more computer readable storage media collectively having program instructions embodied therewith, the program instructions executable by a processor to:
determine, for the motor-impaired user, a viewable display area on a display of the electronic device, wherein the viewable display area is a fixed area corresponding to a stationary location on the display and wherein the viewable display area is smaller in size than the display;
determine a baseline state of the motor-impaired user;
analyze the visual content to divide into multiple sections comprising key objects;
divide the visual content into the multiple sections;
determine a content navigation path to scroll the multiple sections of the visual content based on a relevance or prominence of each key object of the key objects;
automatically scroll the visual content through the viewable display area according to the determined content navigation path;
obtain biometric data including motor-impaired user data from the motor-impaired user, wherein the biometric data is used by a cognitive control system via the processor to determine a state of the motor-impaired user selected from a plurality of states, the plurality of states corresponding to a stress or interest level of the motor-impaired user, and wherein the biometric data obtained from the motor-impaired user is selected from a group consisting of a heart rate, a tone of voice, and a rate of respiration;
map each state of the plurality of states to one or more control functions;
select a control function from the one or more control functions based on the determined state of the motor-impaired user; and
adjust, via the selected control function, the scrolling of the visual content outside the viewable display area through the viewable display area on the electronic device to return the determined state of the motor-impaired user to the baseline state, wherein additional biometric data is obtained and processed to determine that the determined state of the motor-impaired user has returned to the baseline state, and wherein the additional biometric data obtained from the motor-impaired user is selected from the group consisting of the heart rate, the tone of voice, and the rate of respiration.

### Claim 15 (depends on 14)

The computer program product of claim 14, wherein the viewable display area is determined by calculating a user distance and a viewing angle of the motor-impaired user relative to the display.

### Claim 16 (depends on 14)

The computer program product of claim 14, wherein the program instructions are executable to continuously modify the one or more control functions to control the scrolling of the visual content based upon real-time or near real-time biometric data obtained from the motor-impaired user.

### Claim 17 (depends on 14)

The computer program product of claim 14, wherein the biometric data includes data corresponding to an emotional, psychological, or a physiological state of the motor-impaired user, and wherein a machine learning system maps the biometric data to the determined state of the motor-impaired user.

### Claim 18 (depends on 14)

The computer program product of claim 14, wherein the program instructions are executable to: receive a voice instruction from the motor-impaired user; and
move the visual content into the viewable display area based on the voice instruction.

### Claim 19 (depends on 14)

The computer program product of claim 14, wherein the biometric data is obtained from one or more wearable biometric devices.