# US 10,870,056 B2 · Emoji-based communications derived from facial features during game play > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,870,056 B2 - Document ID: US10870056B2 - Status: Granted - Kind: B2 - Application number: 15/801,203 - Filed: 2017-11-01 - Published: 2019-05-02 - Granted: 2020-12-22 - Earliest claimed: 2017-11-01 - Assignees: Sony Interactive Entertainment Inc. - Applicants: Sony Interactive Entertainment Inc. - Inventors: Michael Taylor; Glenn Black - Web page: https://patentgrep.com/patent/US10870056B2 ## CPC - A63F 13/655: by importing photos, e.g. of the player - A63F 13/213: comprising photodetecting means, e.g. cameras, photodiodes or infrared cells (A63F13/219 takes precedence) - A63F 13/87: Communicating with other players during game play, e.g. by e-mail or chat - G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships} - G06V 40/174: {Facial expression recognition} - H04L 51/10: Multimedia information - H04L 51/52: for supporting social networking services - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Techniques for emoji-based communications derived from facial features during game play provide a communication channel accessible by players associated with gameplay hosted by the online game platform and capture an image of a user using a controller that includes a camera. The techniques further determine facial features of the user based on the image, generate an emoji based on the facial features of the user, and transmit the emoji to the players associated with the online game platform over the communication channel. ## Claims ### Claim 1 A method for emoji-based communications derived from facial features, the method comprising: establishing a vector-space based on a plurality of facial expressions, each facial expression associated with a group of position vectors mapped based on facial features that each correspond to a dimension in the vector-space, and wherein establishing the vector-space includes establishing threshold conditions between different facial expressions; storing a plurality of emojis in a lookup table in memory, each emoji corresponding to a different one of the facial expressions; capturing an image of a user with a camera during gameplay, the image captured when a gameplay status meets a predetermined gameplay milestone; extracting facial features of the user from the image as captured when the gameplay status met the predetermined gameplay milestone; identifying a facial expression associated with the extracted facial features by comparing a query vector that maps the extracted facial features to a plurality of position vectors associated with the different facial expressions, wherein the facial expression is identified based on proximity of a mapped position vector to the query vector and the threshold conditions; identifying one of the stored emojis in the lookup table as corresponding to the identified facial expression; modifying the identified emoji based on the extracted facial features of the user; and transmitting the emoji to at least one peer of the user over a communication channel, the at least one peer associated with the gameplay when the gameplay status met the predetermined gameplay milestone. ### Claim 2 (depends on 1) The method of claim 1, wherein capturing another image of the user is based on one or more of an image capture command or a scheduled time. ### Claim 3 (depends on 1) The method of claim 1, wherein the captured image is among multiple images captured of the user, and further comprising: determining a change in the facial features of the user based on the multiple images; and selecting one of the multiple images based on the determined change, wherein the facial features are extracted from the selected image. ### Claim 4 (depends on 1) The method of claim 1, wherein extracting the facial features of the user further comprises determining landmarks in the image based on a facial recognition algorithm. ### Claim 5 (depends on 1) The method of claim 1, wherein the identified emoji is a model emoji, and wherein modifying the identified emoji further comprises adjusting portions of the model emoji based on the extracted facial features to form a modified model emoji. ### Claim 6 (depends on 1) The method of claim 1, wherein identifying the emoji corresponding to the identified facial expression further comprises comparing the extracted facial features to one or more facial features mapped to the stored plurality of emoji. ### Claim 7 (depends on 1) The method of claim 1, wherein identifying the emoji corresponding to the identified facial expression further comprises determining an emotion associated with the extracted facial features, wherein the identified emoji corresponds to the determined emotion. ### Claim 8 (depends on 1) The method of claim 1, further comprising: mapping a plurality of images to a plurality of position vectors in the vector-space based on the facial features extracted from each image; and assigning one or more position vectors to a group that represents a same emotion based on proximity in the vector-space. ### Claim 9 (depends on 8) The method of claim 8, wherein the vector-space includes a plurality of groups that each correspond to a different emotion. ### Claim 10 (depends on 1) The method of claim 1, wherein the transmitted emoji includes one or more of an ideogram, a smiley, a pictograph, or an emoticon. ### Claim 11 (depends on 1) The method of claim 1, further comprising generating a model emoji specific to the user based on the extracted features. ### Claim 12 (depends on 11) The method of claim 11, wherein generating the model emoji further comprises mapping one or more of the extracted features to a corresponding portion of the model emoji. ### Claim 13 A system for emoji-based communications derived from facial features, the system comprising: memory that stores: a vector-space established based on a plurality of facial expressions, each facial expression associated with a group of position vectors mapped based on facial features that each correspond to a dimension in the vector-space, and wherein the vector-space includes threshold conditions between different facial expressions, and a plurality of emojis in a lookup table, each emoji corresponding to a different one of the facial expressions; a camera that captures an image of a user during gameplay, the image captured when a gameplay status meets a predetermined gameplay milestone; a processor that executes instructions stored in the memory, wherein execution of the instructions by the processor: extracts facial features of the user from the image as captured when the gameplay status met the predetermined gameplay milestone; identifies a facial expression associated with the extracted facial features by comparing a query vector that maps the extracted facial features to a plurality of position vectors associated with the different facial expressions, wherein the identified facial expression has is identified based on proximity of a mapped position vector to the query vector and the threshold conditions; identifies one of the stored emojis in the lookup table as corresponding to the identified facial expression; and modifies the identified emoji based on the extracted facial features of the user; and a network interface that communicates over a communication channel, wherein the network interface transmits the emoji to at least one peer of the user over the communication channel, the at least one peer associated with the gameplay when the gameplay status met the predetermined gameplay milestone. ### Claim 14 (depends on 13) The system of claim 13, wherein the camera captures another image based on one or more of an image capture command or a scheduled time. ### Claim 15 (depends on 13) The system of claim 13, wherein the captured image is among multiple images captured of the user, and wherein the processor executes further instructions to: determine a change in the facial features of the user from the multiple images; and select one of the multiple images as the selected image based on the determined change, wherein the facial features are extracted from the selected image. ### Claim 16 (depends on 13) The system of claim 13, wherein the processor extracts the facial features of the user by determining landmarks in the image based on a facial recognition algorithm. ### Claim 17 (depends on 13) The system of claim 13, wherein the identified emoji is a model emoji, wherein the processor modifies the identified emoji by adjusting portions of the model emoji based on the extracted facial features to form a modified model emoji. ### Claim 18 (depends on 13) The system of claim 13, wherein the processor identifies the emoji corresponding to the identified facial expression by comparing the extracted facial features to one or more facial features mapped to the stored plurality of emoji. ### Claim 19 (depends on 13) The system of claim 13, wherein the processor identifies the emoji corresponding to the identified facial expression by determining an emotion associated with the extracted facial features, wherein the identified emoji corresponds to the determined emotion. ### Claim 20 A tangible, non-transitory, computer-readable storage medium, having instructions encoded thereon, the instructions executable by a processor to perform a method for emoji-based communication, the method comprising: establishing a vector-space based on a plurality of facial expressions, each facial expression associated with a group of position vectors mapped based on facial features that each correspond to a dimension in the vector-space, and wherein establishing the vector-space includes establishing threshold conditions between different facial expressions; storing a plurality of emojis in a lookup table in memory, each emoji corresponding to a different one of the facial expressions; capturing an image of a user with a camera during gameplay, the image captured when a gameplay status meets a predetermined gameplay milestone; extracting facial features of the user from the captured image as captured when the gameplay status met the predetermined gameplay milestone; identifying a facial expression associated with the extracted facial features by comparing a query vector that maps the extracted facial features to a plurality of position vectors associated with the different facial expressions, wherein the facial expression is identified based on proximity of a mapped position vector to the query vector and the threshold conditions; identifying one of the stored emojis in the lookup table as corresponding to the identified facial expression; modifying the identified emoji based on the extracted facial features of the user; and transmitting the emoji to at least one peer of the user over a communication channel, the at least one peer associated with the gameplay when the gameplay status met the predetermined gameplay milestone.