# US 11,439,907 B2 · Audio feedback that varies based on direction of input stroke > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,439,907 B2 - Document ID: US11439907B2 - Status: Granted - Kind: B2 - Application number: 16/899,889 - Filed: 2020-06-12 - Published: 2020-12-17 - Granted: 2022-09-13 - Earliest claimed: 2019-06-14 - Assignees: Nintendo Co., Ltd. - Applicants: Nintendo Co., Ltd. - Inventors: Masami Yone; Yuichi Ozaki - Web page: https://patentgrep.com/patent/US11439907B2 ## CPC - A63F 13/54: involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall - A63F 13/2145: the surface being also a display device, e.g. touch screens - A63F 13/426: involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun - G06F 1/1626: {with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]} - G06F 1/1669: {Detachable keyboards} - G06F 3/041: Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means - G06F 3/04883: for inputting data by handwriting, e.g. gesture or text - G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path} - G06T 11/23: using straight lines or curves - A63F 13/24: Constructional details thereof, e.g. game controllers with detachable joystick handles - A63F 2300/1043: being characterized by constructional details - A63F 2300/1075: using a touch screen - A63F 2300/6081: generating an output signal, e.g. under timing constraints, for spatialization - G09B 19/025: {with electrically operated apparatus or devices} - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G09B: EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS (devices for psychotechnics or for testing reaction times A61B5/16; games, sports, amusements A63; projectors, projector screens G03B) ## Abstract A storage medium storing therein a program, an information processing apparatus, an information processing method, a game system and a mobile device are disclosed. For example, disclosed is a non-transitory storage medium storing therein a program that is readable by a computer of an information processing apparatus, the program being configured to cause the computer to: acquire input coordinates of input performed based on an operation performed by a user; generate drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing; and output the sound information of the virtual writing sound to a speaker. ## Claims ### Claim 1 A non-transitory storage medium storing therein a program that is readable by a computer of an information processing apparatus, the program being configured to cause the computer to: acquire input coordinates of input performed based on an operation by a user, the input coordinates are set in a coordinate plane of an orthogonal coordinate system; generate drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing, where the virtual writing sound is changed according to the movement direction with respect to a reference line, the direction of the reference line is inclined with respect to a coordinate axis of the coordinate plane; and output the sound information of the virtual writing sound to a speaker. ### Claim 2 (depends on 1) The storage medium according to claim 1, wherein the program is further configured to cause the computer to generate the sound information where a volume of the virtual writing sound is changed according to the movement direction with respect to the reference line. ### Claim 3 (depends on 2) The storage medium according to claim 2, wherein the program is further configured to cause the computer to generate the sound information where when the movement direction is close to being perpendicular to the reference line, the volume is greater compared to a case where the movement direction is close to being parallel to the reference line. ### Claim 4 (depends on 1) The storage medium according to claim 1, wherein the program is further configured to cause the computer to generate the sound information where a pitch of the virtual writing sound is changed according to the movement direction with respect to the reference line. ### Claim 5 (depends on 4) The storage medium according to claim 4, wherein the program is further configured to cause the computer to generate the sound information where when the movement direction is close to being perpendicular to the reference line, the pitch is higher compared to a case where the movement direction is close to being parallel to the reference line. ### Claim 6 (depends on 1) The storage medium according to claim 1, wherein the program is further configured to cause the computer to generate the sound information where a tone of the virtual writing sound is changed according to the movement direction with respect to the reference line. ### Claim 7 (depends on 6) The storage medium according to claim 6, wherein the program is further configured to cause the computer to generate the sound information where the tone is changed depending on whether an angle formed by the movement direction with respect to the direction of the reference line is greater than or equal to 0 degrees and less than 180 degrees or greater than or equal to 180 degrees and less than 360 degrees. ### Claim 8 (depends on 1) The storage medium according to claim 1, wherein the reference line is inclined by about 45 degrees with respect to the coordinate axis. ### Claim 9 (depends on 1) The storage medium according to claim 1, wherein the program is further configured to cause the computer to determine an input velocity based on a change in the input coordinates and change the sound information of the virtual writing sound according to the determined input velocity. ### Claim 10 (depends on 9) The storage medium according to claim 9, wherein the program is further configured to cause the computer to generate the sound information where the change in the sound information of the virtual writing sound is greater when the determined input velocity is faster. ### Claim 11 (depends on 9) The storage medium according to claim 9, wherein the program is further configured to cause the computer to generate the sound information of the virtual writing sound where virtual writing sound has a higher pitch when the determined input velocity is faster. ### Claim 12 An information processing apparatus comprising: at least one processor; at least one storage medium; a program stored in the storage medium; wherein the program is configured to cause the at least one processor to: acquire input coordinates of input performed based on an operation by a user, the input coordinates are set in a coordinate plane of an orthogonal coordinate system; generate drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing, where the virtual writing sound is changed according to the movement direction with respect to a reference line, the direction of the reference line is inclined with respect to a coordinate axis of the coordinate plane ; and output the sound information of the virtual writing sound to a speaker. ### Claim 13 An information processing method comprising: acquiring input coordinates of input performed based on an operation of a user, the input coordinates are set in a coordinate plane of an orthogonal coordinate system; generating drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determining a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; and generating sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing, where the virtual writing sound is changed according to the movement direction with respect to a reference line, the direction of the reference line is inclined with respect to a coordinate axis of the coordinate plane; and outputting the sound information of the virtual writing sound to a speaker. ### Claim 14 A game system comprising: at least one processor; at least one storage medium; a program stored in the storage medium; wherein the program is configured to cause the at least one processor to: acquire input coordinates of input performed based on an operation by a user, the input coordinates are set in a coordinate plane of an orthogonal coordinate system; generate drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing, where the virtual writing sound is changed according to the movement direction with respect to a reference line, the direction of the reference line is inclined with respect to a coordinate axis of the coordinate plane; and output the sound information of the virtual writing sound to a speaker. ### Claim 15 A mobile device comprising: a display comprising a touch panel; at least one speaker; and a processor configured to: acquire input coordinates of input performed on the touch panel, the input coordinates are set in a coordinate plane of an orthogonal coordinate system; generate drawing information for displaying a path corresponding to the input coordinates in the display; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing, where the virtual writing sound is changed according to the movement direction with respect to a reference line, the direction of the reference line is inclined with respect to a coordinate axis of the coordinate plane; and output the sound information of the virtual writing sound to the at least one speaker. ### Claim 16 A non-transitory storage medium storing therein a program that is readable by a computer of an information processing apparatus, the program being configured to cause the computer to: acquire input coordinates of input performed based on an operation by a user; generate drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing, where the virtual writing sound is changed according to the movement direction with respect to a reference line, where a direction of the reference line is changed depending on whether the user is set as being right-handed or is set as being left-handed; and output the sound information of the virtual writing sound to a speaker. ### Claim 17 A non-transitory storage medium storing therein a program that is readable by a computer of an information processing apparatus, the program being configured to cause the computer to: acquire input coordinates of input performed based on an operation by a user; generate drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing; and output the sound information of the virtual writing sound to a speaker, wherein the sound information includes a base sound that does not change according to the movement direction. ### Claim 18 An information processing apparatus comprising: at least one processor; at least one storage medium; a program stored in the storage medium; wherein the program is configured to cause the at least one processor to: acquire input coordinates of input performed based on an operation by a user; generate drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing, where the virtual writing sound is changed according to the movement direction with respect to a reference line, where a direction of the reference line is changed depending on whether the user is set as being right-handed or is set as being left-handed; and output the sound information of the virtual writing sound to a speaker. ### Claim 19 An information processing apparatus comprising: at least one processor; at least one storage medium; a program stored in the storage medium; wherein the program is configured to cause the at least one processor to: acquire input coordinates of input performed based on an operation by a user; generate drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing, and output the sound information of the virtual writing sound to a speaker, wherein the sound information includes a base sound that does not change according to the movement direction. ### Claim 20 An information processing method comprising: acquiring input coordinates of input performed based on an operation of a user; generating drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determining a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generating sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing, where the virtual writing sound is changed according to the movement direction with respect to a reference line, where a direction of the reference line is changed depending on whether the user is set as being right-handed or is set as being left-handed; and outputting the sound information of the virtual writing sound to a speaker. ### Claim 21 An information processing method comprising: acquiring input coordinates of input performed based on an operation of a user; generating drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determining a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generating sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing; and outputting the sound information of the virtual writing sound to a speaker, wherein the sound information includes a base sound that does not change according to the movement direction. ### Claim 22 A game system comprising: at least one processor; at least one storage medium; a program stored in the storage medium; wherein the program is configured to cause the at least one processor to: acquire input coordinates of input performed based on an operation by a user; generate drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing, where the virtual writing sound is changed according to the movement direction with respect to a reference line, where a direction of the reference line is changed depending on whether the user is set as being right-handed or is set as being left-handed; and output the sound information of the virtual writing sound to a speaker. ### Claim 23 A game system comprising: at least one processor; at least one storage medium; a program stored in the storage medium; wherein the program is configured to cause the at least one processor to: acquire input coordinates of input performed based on an operation by a user; generate drawing information for displaying a path corresponding to the input coordinates on a display apparatus; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing; and output the sound information of the virtual writing sound to a speaker, wherein the sound information includes a base sound that does not change according to the movement direction. ### Claim 24 A mobile device comprising: a display comprising a touch panel; at least one speaker; and a processor configured to: acquire input coordinates of input performed on the touch panel; generate drawing information for displaying a path corresponding to the input coordinates in the display; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing, where the virtual writing sound is changed according to the movement direction with respect to a reference line, where a direction of the reference line is changed depending on whether the user is set as being right-handed or is set as being left-handed; and output the sound information of the virtual writing sound to the at least one speaker. ### Claim 25 A mobile device comprising: a display comprising a touch panel; at least one speaker; and a processor configured to: acquire input coordinates of input performed on the touch panel; generate drawing information for displaying a path corresponding to the input coordinates in the display; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing; and output the sound information of the virtual writing sound to the at least one speaker, wherein the sound information includes a base sound that does not change according to the movement direction. ### Claim 26 A mobile device comprising: a display comprising a touch panel; at least one speaker; and a processor configured to: acquire input coordinates of input performed on the touch panel; generate drawing information for displaying a path corresponding to the input coordinates in the display; determine a movement direction of the input based on a change in the input coordinates, which are acquired continuously while the input is performed; generate sound information of a virtual writing sound that is changed based on the movement direction in response to the input coordinates changing; and output the sound information of the virtual writing sound to the at least one speaker, wherein the processor is configured to generate a first audio data and a third audio data and combine the first audio data and the third audio data to generate the sound information of the virtual writing sound when the determined direction is a first direction; and wherein the processor is configured to generate a second audio data and the third audio data and combine the second audio data and the third audio data to generate the sound information of the virtual writing sound when the determined direction is a second direction, the first audio data and the second audio data being different.