# US 2021/0334077 A1 · Coding training method using augmented reality

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334077 A1
- Document ID: US20210334077A1
- Status: Published application
- Kind: A1
- Application number: 16/616,448
- Filed: 2019-05-14
- Published: 2021-10-28
- Earliest claimed: 2018-06-20
- Assignees: Coding & Play Inc.
- Applicants: Coding & Play Inc.
- Inventors: Kyung Hee RYOO
- Web page: https://patentgrep.com/patent/US20210334077A1

## CPC

- G06F 8/34: Graphical or visual programming
- G09B 5/02: with visual presentation of the material to be studied, e.g. using film strip
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G09B 19/0053: {Computers, e.g. programming}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Provided is a coding training method using augmented reality as a method performed by a computing device including a camera, the coding training method including: a first step of obtaining coding training content; a second step of selecting one or more points included in the coding training content; a third step of obtaining an image photographed by the camera; a fourth step of recognizing the photographed image and matching one or more objects included in the photographed image with the one or more points; a fifth step of displaying an image and the matched one or more points; a sixth step of displaying a virtual robot on the photographed image; a seventh step of displaying one or more instruction blocks each including one or more instructions for controlling a movement of the virtual robot; an eighth step of arranging the one or more instruction blocks at the one or more points based on user input for the one or more instruction blocks displayed; a ninth step of controlling the virtual robot in order in which the one or more instruction blocks are arranged, moving the virtual robot along the one or more points, and performing instructions corresponding to the instruction blocks arranged at each point; and a tenth step of displaying the movement of the virtual robot according the control result.

## Claims

### Claim 1

A coding training method using augmented reality as a method performed by a computing device including a camera, the coding training method comprising:
a first step of obtaining coding training content;
a second step of selecting one or more points included in the coding training content;
a third step of obtaining an image photographed by the camera;
a fourth step of recognizing the photographed image and matching one or more objects included in the photographed image with the one or more points;
a fifth step of displaying an image and the matched one or more points;
a sixth step of displaying a virtual robot on the photographed image;
a seventh step of displaying one or more instruction blocks each including one or more instructions for controlling a movement of the virtual robot;
an eighth step of arranging the one or more instruction blocks at the one or more points based on user input for the one or more instruction blocks displayed;
a ninth step of controlling the virtual robot in order in which the one or more instruction blocks are arranged, moving the virtual robot along the one or more points, and performing instructions corresponding to the instruction blocks arranged at each point; and
a tenth step of displaying the movement of the virtual robot according the control result.

### Claim 2 (depends on 1)

The coding training method using augmented reality of claim 1, wherein the first step includes:
an eleventh step of obtaining a mission included in the coding training content; and
a twelfth step of displaying the mission, and
the ninth step includes:
a thirteenth step of determining whether or not the mission is achieved according to the control result; and
a fourteenth step of displaying feedback according to whether or not the mission is achieved.

### Claim 3 (depends on 2)

The coding training method using augmented reality of claim 2, wherein the thirteenth step includes:
a fifteenth step of determining the interaction between the one or more object and the virtual robot; and
a sixteenth step of determination whether or not the mission is achieved at each of the one or more points based on the interaction, and
the fourteenth step includes:
a seventeenth step of displaying the feedback according to whether or not the mission is achieved at each of the one or more points.

### Claim 4 (depends on 3)

The coding training method using augmented reality of claim 3, wherein the fifteenth step includes:
an eighteenth step of determining one or more contact points between the one or more object and the virtual robot;
a nineteenth step of determining portions of the one or more objects and the virtual robot corresponding to each of the one or more contact points; and
a twentieth step of determining the interaction based on positions and movement of the one or more contact points and the portions corresponding to each of the one or more contact points.

### Claim 5 (depends on 4)

The coding training method using augmented reality of claim 4, wherein the twentieth step include:
a twenty-first step of dividing a first contact point included in the one or more contact point into a second contact point and a third contact point;
a twenty-second step of tracking movement of the second contact point and the third contact point;
a twenty-third step of determining the portions of the one or more objects and the virtual robot corresponding to the second contact point and the third contact point, respectively, when a distance between the second contact point and the third contact point is greater than a predetermined reference value; and
a twenty-fourth step of determining the interaction based on the positions and the movement of the second contact point and the third contact point and the portions corresponding to the second contact point and the third contact point, respectively.

### Claim 6 (depends on 2)

The coding training method using augmented reality of claim 2, wherein the eleventh step includes:
a twenty-fifth step of determining a type of the one or more objects;
a twenty-sixth step of generating a moving route of the virtual robot based on the positions and the type of the one or more objects included in the photographed image;
a twenty-seventh step of obtaining a first mission which causes the virtual robot to move along the moving route; and
a twenty-eighth step of obtaining a second mission which causes the virtual robot to perform at least one of the one or more objects with a predetermined interaction.

### Claim 7 (depends on 6)

The coding training method using augmented reality of claim 6, wherein the eleventh step includes:
a twenty-ninth of generating and displaying one or more virtual objects on the moving route;
a thirtieth step of determining a type of the one or more virtual objects;
a thirtieth-first step of updating the moving route based on the position and type of the one or more virtual objects;
a thirtieth-two of obtaining a third mission which causes the virtual robot to move along the updated moving route; and
a thirtieth-third step of obtaining a fourth mission which causes the virtual robot to perform a predetermined interaction with at least one of the one or more virtual objects.

### Claim 8 (depends on 1)

The coding training method using augmented reality of claim 1, wherein the eighth step further includes:
a thirtieth-fourth step of arranging at least a part of the one or more command blocks in the virtual robot, and
the ninth step further includes:
a thirtieth-fifth step of performing command corresponding to the command block arranged on the virtual robot;
a thirtieth-sixth step of performing command corresponding to the command block arranged at the reached point when the virtual robot reaches the one or more points;
a thirtieth-seventh of stopping an operation of the virtual robot and displaying the feedback when the command corresponding to the command block arranged on the virtual robot collide with the command corresponding to the command block arranged at the reached point.
