# US 11,907,739 B1 · Annotating screen content in a mobile environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,907,739 B1
- Document ID: US11907739B1
- Status: Granted
- Kind: B1
- Application number: 18/106,218
- Filed: 2023-02-06
- Granted: 2024-02-20
- Earliest claimed: 2014-07-11
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Matthew Sharifi; David Petrou
- Web page: https://patentgrep.com/patent/US11907739B1

## CPC

- G06F 9/451: Execution arrangements for user interfaces
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04845: for image manipulation, e.g. dragging, rotation, expansion or change of colour
- G06F 16/156: {Query results presentation}
- G06F 16/2228: {Indexing structures}
- G06F 16/24578: {using ranking}
- G06F 16/583: using metadata automatically derived from the content
- G06F 16/5854: {using shape and object relationship}
- G06F 16/5866: {using information manually generated, e.g. tags, keywords, comments, manually generated location and time information}
- G06F 16/957: Browsing optimisation, e.g. caching or content distillation
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 40/134: Hyperlinking
- G06F 40/169: Annotation, e.g. comment data or footnotes
- G06F 40/295: Named entity recognition
- G06Q 10/1097: {using calendar-based scheduling for task assignment}
- G06V 30/416: Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
- H04L 67/06: specially adapted for file transfer, e.g. file transfer protocol [FTP]
- H04W 4/18: Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

Systems and methods are provided for sharing a screen from a mobile device. For example, a method includes receiving, at a second mobile device, an image of a screen captured from a first mobile device and determining whether to trigger an automated action. The method may also include displaying, responsive to not triggering the automated action, annotation data generated for the image with the image on a display of the second mobile device, the annotation data including at least one visual cue corresponding to content in the image relevant to a user of the second mobile device. The method may further include, responsive to triggering the automated action, determining that a mobile application associated with the image is installed on the second mobile device and replaying user input actions received with the image on the second mobile device starting from a reference screen associated with the mobile application.

## Claims

### Claim 1

A method implemented by one or more processors, the method comprising:
identifying an area of actionable content on a current screen of a display of a client device, the current screen of the display of the client device being associated with a software application that is accessible at the client device;
in response to identifying the area of actionable content, generating a selectable visual cue for the area of actionable content, the selectable visual cue for the area of actionable content being associated with multiple candidate actions for the area of actionable content;
graphically annotating the area of actionable content on the current screen of the display of the client device with the selectable visual cue;
receiving, from a user of the client device, a user selection of the selectable visual cue;
in response to receiving the user selection of the selectable visual cue:
causing an indication of the multiple candidate action to be provided for presentation to the user via the display of the client device;
in response to receiving a further user selection, from the user of the client device, of a first candidate action, of the multiple candidate actions:
causing a first screen that is associated with the first candidate action to overlay the current screen of the display of the client device without leaving a context of the current application;
in response to receiving a further user selection, from the user of the client device, of a second candidate action, of the multiple candidate actions:
causing a second screen that is associated with the second candidate action to overlay the current screen of the display of the client device without leaving a context of the current application,
wherein the second candidate action differs from the first candidate action, and
wherein the second screen that is associated with the second candidate action differs from the first screen that is associated with the first candidate action; and
in response to receiving a further user selection, from the user of the client device, of a third candidate action, of the multiple candidate actions:
causing the client device to launch an additional software application associated with the third candidate action and that is also accessible at the client device,
wherein the third candidate action differs from both the first candidate action and the second candidate action, and
wherein the additional software application associated with the third candidate action is in addition to the software application.

### Claim 2 (depends on 1)

The method of claim 1, wherein graphically annotating the area of actionable content on the current screen of the display of the client device with the selectable visual cue comprises one or more of:
underlining textual content that is associated with the area of actionable content;
highlighting textual content that is associated with the area of actionable content;
circling textual content that is associated with the area of actionable content; or
outlining textual content that is associated with the area of actionable content.

### Claim 3 (depends on 2)

The method of claim 2, wherein receiving the user selection of the selectable visual cue comprises:
receiving, from the user of the client device, touch input directed to the textual content that is associated with the area of actionable content via the display of the client device.

### Claim 4 (depends on 3)

The method of claim 3, wherein the textual content that is associated with the area of actionable content comprises one or more of: a telephone number, an email address, a street address, a time, or a date.

### Claim 5 (depends on 3)

The method of claim 3, wherein causing the indication of the multiple candidate action to be provided for presentation to the user via the display of the client device comprises:
causing the indication of the multiple candidate action to overlay the current screen of the display of the client device.

### Claim 6 (depends on 5)

The method of claim 5, wherein the first candidate action is associated with a first portion of the display, wherein the second candidate action is associated with a second portion of the display, and wherein the second portion of the display that is associated with the second candidate action differs from the first portion of the display that is associated with the first candidate action.

### Claim 7 (depends on 6)

The method of claim 6, wherein receiving the further user selection of the first candidate action comprises:
receiving, from the user of the client device, further touch input directed to the first portion of the display that is associated with the first candidate action.

### Claim 8 (depends on 6)

The method of claim 6, wherein receiving the further user selection of the second candidate action comprises:
receiving, from the user of the client device, further touch input directed to the second portion of the display that is associated with the second candidate action.

### Claim 9 (depends on 1)

The method of claim 1, wherein generating the selectable visual cue for the area of actionable content comprises:
generating annotation data for the area of actionable content, wherein the annotation data includes the selectable visual cue.

### Claim 10

A client device, the client device comprising:
at least one processor; and
memory storing instructions that, when executed by the at least one processor, cause the at least one processor to:
identify an area of actionable content on a current screen of a display of the client device, the current screen of the display of the client device being associated with a software application that is accessible at the client device;
in response to identifying the area of actionable content, generate a selectable visual cue for the area of actionable content, the selectable visual cue for the area of actionable content being associated with multiple candidate actions for the area of actionable content;
graphically annotate the area of actionable content on the current screen of the display of the client device with the selectable visual cue;
receive, from a user of the client device, a user selection of the selectable visual cue;
in response to receiving the user selection of the selectable visual cue:
cause an indication of the multiple candidate action to be provided for presentation to the user via the display of the client device;
in response to receiving a further user selection, from the user of the client device, of a first candidate action, of the multiple candidate actions:
cause a first screen that is associated with the first candidate action to overlay the current screen of the display of the client device without leaving a context of the current application;
in response to receiving a further user selection, from the user of the client device, of a second candidate action, of the multiple candidate actions:
cause a second screen that is associated with the second candidate action to overlay the current screen of the display of the client device without leaving a context of the current application,
wherein the second candidate action differs from the first candidate action, and
wherein the second screen that is associated with the second candidate action differs from the first screen that is associated with the first candidate action; and
in response to receiving a further user selection, from the user of the client device, of a third candidate action, of the multiple candidate actions:
cause the client device to launch an additional software application associated with the third candidate action and that is also accessible at the client device,
wherein the third candidate action differs from both the first candidate action and the second candidate action, and
wherein the additional software application associated with the third candidate action is in addition to the software application.

### Claim 11 (depends on 10)

The client device of claim 10, wherein the instructions to graphically annotate demarcate the area of actionable content on the current screen of the display of the client device with the selectable visual cue comprise instructions to one or more of:
underline textual content that is associated with the area of actionable content;
highlight textual content that is associated with the area of actionable content;
circle textual content that is associated with the area of actionable content; or
outline textual content that is associated with the area of actionable content.

### Claim 12 (depends on 11)

The client device of claim 11, wherein the instructions to receive the user selection of the selectable visual cue comprise instructions to:
receive, from the user of the client device, touch input directed to the textual content that is associated with the area of actionable content via the display of the client device.

### Claim 13 (depends on 12)

The client device of claim 12, wherein the textual content that is associated with the area of actionable content comprises one or more of: a telephone number, an email address, a street address, a time, or a date.

### Claim 14 (depends on 12)

The client device of claim 12, wherein the instructions to cause the indication of the multiple candidate action to be provided for presentation to the user via the display of the client device comprise instructions to:
cause the indication of the multiple candidate action to overlay the current screen of the display of the client device.

### Claim 15 (depends on 14)

The client device of claim 14, wherein the first candidate action is associated with a first portion of the display, wherein the second candidate action is associated with a second portion of the display, and wherein the second portion of the display that is associated with the second candidate action differs from the first portion of the display that is associated with the first candidate action.

### Claim 16 (depends on 15)

The client device of claim 15, wherein the instructions to receive the further user selection of the first candidate action comprise instructions to:
receive, from the user of the client device, further touch input directed to the first portion of the display that is associated with the first candidate action.

### Claim 17 (depends on 15)

The client device of claim 15, wherein the instructions to receive the further user selection of the second candidate action comprise instructions to:
receive, from the user of the client device, further touch input directed to the second portion of the display that is associated with the second candidate action.

### Claim 18

A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations, the operations comprising:
identifying an area of actionable content on a current screen of a display of a client device, the current screen of the display of the client device being associated with a software application that is accessible at the client device;
in response to identifying the area of actionable content, generating a selectable visual cue for the area of actionable content, the selectable visual cue for the area of actionable content being associated with multiple candidate actions for the area of actionable content;
graphically annotating the area of actionable content on the current screen of the display of the client device with the selectable visual cue;
receiving, from a user of the client device, a user selection of the selectable visual cue;
in response to receiving the user selection of the selectable visual cue:
causing an indication of the multiple candidate action to be provided for presentation to the user via the display of the client device;
in response to receiving a further user selection, from the user of the client device, of a first candidate action, of the multiple candidate actions:
causing a first screen that is associated with the first candidate action to overlay the current screen of the display of the client device without leaving a context of the current application;
in response to receiving a further user selection, from the user of the client device, of a second candidate action, of the multiple candidate actions:
causing a second screen that is associated with the second candidate action to overlay the current screen of the display of the client device without leaving a context of the current application,
wherein the second candidate action differs from the first candidate action, and
wherein the second screen that is associated with the second candidate action differs from the first screen that is associated with the first candidate action; and
in response to receiving a further user selection, from the user of the client device, of a third candidate action, of the multiple candidate actions:
causing the client device to launch an additional software application associated with the third candidate action and that is also accessible at the client device,
wherein the third candidate action differs from both the first candidate action and the second candidate action, and
wherein the additional software application associated with the third candidate action is in addition to the software application.