# US 2023/0053425 A1 · Computer Device and Method for Facilitating an Interactive Conversational Session with a Digital Conversational Character in an Augmented Environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0053425 A1
- Document ID: US20230053425A1
- Status: Published application
- Kind: A1
- Application number: 17/982,240
- Filed: 2022-11-07
- Published: 2023-02-23
- Earliest claimed: 2019-04-03
- Assignees: HIA Technologies, Inc.
- Applicants: HIA Technologies, Inc.
- Inventors: Vacit Arat; Richard Cardran; Rick King
- Web page: https://patentgrep.com/patent/US20230053425A1

## CPC

- G06F 8/38: for implementing user interfaces
- G06F 8/35: model driven
- G06F 9/453: {Help systems}
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/73: using feature-based methods
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 20/20: in augmented reality scenes
- G06V 20/64: Three-dimensional [3D] objects
- G06V 30/153: {using recognition of characters or words}
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G16H 10/20: for electronic clinical trials or questionnaires
- G16H 80/00: ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
- G06N 3/08: Learning methods
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Disclosed herein is a software technology for facilitating an interactive conversational session between a user and a digital conversational character. For instance, in one aspect, the disclosed process may involve two primary phases: (1) an authoring phase that involves a first user accessing a content authoring tool to create a given type of visual conversation application that facilitates interactions between a second user and a digital conversational character in an interactive conversational session, and (2) a rendering phase that involves the second user accessing the created visual conversation application to interact with the digital conversational character in an interactive conversational session. In one implementation, accessing the created visual conversation application may involve detecting an object and identifying information associated with the detected object. The digital conversational character involved in the interactive conversational session may be superimposed onto a real-world environment.

## Claims

### Claim 1

A computing device comprising:
a network interface;
at least one processor;
a non-transitory computer-readable medium; and
program instructions stored on the non-transitory computer-readable medium that are executable by the at least one processor such that the computing device is configured to:
detect, via one or more sensors of the computing device, an object in a real-world environment;
determine information associated with the detected object;
present a view of a given type of visual conversation application that enables a user to interact with a digital conversational character in an interactive conversational session, wherein the view comprises the digital conversational character superimposed onto the real-world environment;
determine a given interaction mode from a plurality of interaction modes for the interactive conversational session between the user and the digital conversational character, wherein the plurality of interaction modes comprises a personal mode and a non-personal mode, and wherein, if some of the information associated with the detected object matches information associated with an account of the user for the given type of visual conversation application, determine that the given interaction mode should be in the personal mode; and
facilitate the interactive conversational session between the user and the digital conversational character based on the determined information associated with the detected object and the given interaction mode.

### Claim 2 (depends on 1)

The computing device of claim 1, wherein the one or more sensors comprises a camera, and wherein detecting the object in the real-world environment comprises detecting the object via the camera.

### Claim 3 (depends on 1)

The computing device of claim 1, wherein the one or more sensors comprises a microphone, and wherein detecting the object in the real-world environment comprises detecting sound signals associated with the object via the microphone.

### Claim 4 (depends on 3)

The computing device of claim 3, wherein the object comprises a Quick Response (QR) code, and wherein detecting the object in the real-world environment comprises detecting the QR code via the camera.

### Claim 5 (depends on 1)

The computing device of claim 1, wherein the one or more sensors comprises a Global Positioning System (GPS), and wherein determining information associated with the detected object comprises determining a location of the computing device that is near the detected object via the GPS.

### Claim 6 (depends on 1)

The computing device of claim 1, wherein the object in the real-world environment comprises a two-dimensional object or a three-dimensional object.

### Claim 7 (depends on 1)

The computing device of claim 1, wherein the computing device further comprises program instructions stored in the non-transitory computer-readable medium that are executable by the at least one processor such that the computing device is configured to:
cause the computing device to present a view of the given type of visual conversation application that enables the user to detect the object, wherein the view of the given type of visual conversation application that enables the user to detect the object comprises an indication on how to position the object.

### Claim 8 (depends on 1)

The computing device of claim 1, wherein the given type of visual conversation application is a web application.

### Claim 9 (depends on 1)

The computing device of claim 1, wherein determining information associated with the detected object comprises:
sending data identifying the detected object from the computing device to a computing system; and
receiving the information associated with the detected object from the computing system.

### Claim 10 (depends on 1)

The computing device of claim 1, wherein the computing device further comprises program instructions stored in the non-transitory computer-readable medium that are executable by the at least one processor such that the computing device is configured to:
determine the position and orientation of the computing device.

### Claim 11 (depends on 1)

The computing device of claim 1, wherein the detected object in the real-world environment comprises a given product that includes a product packaging, and wherein determining information associated with the detected object comprises extracting product information on the product packaging of the given product.

### Claim 12 (depends on 1)

The computing device of claim 1, wherein the detected object in the real-world environment comprises a given product, and wherein the view comprises the digital conversational character superimposed onto the given product in the real-world environment.

### Claim 13 (depends on 1)

The computing device of claim 1, wherein the detected object in the real-world environment comprises a given product, and wherein the view comprises the digital conversational character superimposed onto a given location in proximity to the given product in the real-world environment.

### Claim 14 (depends on 1)

The computing device of claim 1, wherein the detected object in the real-world environment comprises a given product, and wherein the digital conversational character comprises a digital representation of an image on the given product.

### Claim 15 (depends on 1)

The computing device of claim 1, wherein the computing device further comprises program instructions stored in the non-transitory computer-readable medium that are executable by the at least one processor such that the computing device is configured to:
before causing the computing device to present the view of the given type of visual conversation application, determine the given type of visual conversation application that should be presented based on the determined information associated with the detected object.

### Claim 16 (depends on 1)

The computing device of claim 1, wherein facilitating the interactive conversational session between the user and the digital conversational character comprises:
receiving a given question regarding the detected object;
determining a response for the digital conversational character based on the given interaction mode and information associated with the detected object; and
causing the digital conversational character to utter the determined response to the given question.

### Claim 17 (depends on 1)

The computing device of claim 1, wherein facilitating the interactive conversational session between the user and the digital conversational character comprises:
determining a given question that the digital conversational character should ask the user based on a location of the user;
based on the determination, causing the digital conversational character to utter the given question.

### Claim 18 (depends on 1)

The computing device of claim 1, wherein the digital conversational character comprises a digital representation of a human, an animal, an object, or a robot.

### Claim 19

A method performed by a computing device, the method comprising:
detecting, via one or more sensors of the computing device, an object in a real-world environment;
determining information associated with the detected object;
presenting a view of a given type of visual conversation application that enables a user to interact with a digital conversational character in an interactive conversational session, wherein the view comprises the digital conversational character superimposed onto the real-world environment;
determining a given interaction mode from a plurality of interaction modes for the interactive conversational session between the user and the digital conversational character, wherein the plurality of interaction modes comprises a personal mode and a non-personal mode, and wherein, if some of the information associated with the detected object matches information associated with an account of the user for the given type of visual conversation application, determine that the given interaction mode should be in the personal mode; and
facilitating the interactive conversational session between the user and the digital conversational character based on the determined information associated with the detected object and the given interaction mode.

### Claim 20

A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is provisioned with program instructions that are executable to cause a computing device to:
detect, via one or more sensors of the computing device, an object in a real-world environment;
determine information associated with the detected object;
present a view of a given type of visual conversation application that enables a user to interact with a digital conversational character in an interactive conversational session, wherein the view comprises the digital conversational character superimposed onto the real-world environment;
determine a given interaction mode from a plurality of interaction modes for the interactive conversational session between the user and the digital conversational character, wherein the plurality of interaction modes comprises a personal mode and a non-personal mode, and wherein, if some of the information associated with the detected object matches information associated with an account of the user for the given type of visual conversation application, determine that the given interaction mode should be in the personal mode; and
facilitate the interactive conversational session between the user and the digital conversational character based on the determined information associated with the detected object and the given interaction mode.
