# US 9,437,087 B2 · Method and system for haptic data encoding and streaming using a multiplexed data stream > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,437,087 B2 - Document ID: US9437087B2 - Status: Granted - Kind: B2 - Application number: 14/286,591 - Filed: 2014-05-23 - Published: 2014-11-27 - Granted: 2016-09-06 - Earliest claimed: 2013-05-24 - Assignees: Immersion Corporation - Applicants: Immersion Corporation - Inventors: Loc Phan; Satvir Singh Bhatia; Stephen D. Rank; Christopher J. Ullrich; Jean Francois Dionne; Hugues Antoine Oliver - Web page: https://patentgrep.com/patent/US9437087B2 ## CPC - H04N 21/6379: directed to encoder {, e.g. for requesting a lower encoding rate} - A63F 13/215: comprising means for detecting acoustic signals, e.g. using a microphone - A63F 13/25: Output arrangements for video game devices - A63F 13/285: Generating tactile feedback signals via the game input device, e.g. force feedback - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path} - G08B 6/00: Tactile signalling systems, e.g. tactile personal calling systems - H04L 65/70: Media network packetisation - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/234336: {by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text} - H04N 21/23614: {Multiplexing of additional data and video streams (arrangements for simultaneous broadcast of plural pieces of information H04H20/28)} - H04N 21/4382: {Demodulation or channel decoding, e.g. QPSK demodulation (analog front ends or means for connecting modulators, demodulators or transceivers to a transmission line H04L27/0002)} - H04N 21/4385: Multiplex stream processing, e.g. multiplex stream decrypting - H04N 21/4516: {involving client characteristics, e.g. Set-Top-Box type, software version or amount of memory available} - H04N 21/6336: directed to decoder - H04N 21/6582: {Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number (arrangements where receivers interact with the broadcast H04H20/38)} - H04N 21/8113: {comprising music, e.g. song in MP3 format} - H04N 21/8133: {specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video programme} - H04N 21/816: {involving special video data, e.g 3D video} - H04N 21/845: Structuring of content, e.g. decomposing content into time segments - H04N 21/85406: {involving a specific file format, e.g. MP4 format} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A method includes receiving digital content data including audio data and/or video data, generating haptic data using at least some of the received digital content data, encoding the haptic data for efficient transmission over a communication network, multiplexing the encoded haptic data with the received digital content data, embedding information for decoding the encoded haptic data in metadata of the multiplexed data stream, and sending the multiplexed data stream over the communication network. The method may include analyzing the haptic data to determine at least one characteristic of the haptic data, and the encoding the haptic data may include encoding, based on the determined characteristic, the haptic data to meet a pre-defined criteria. ## Claims ### Claim 1 A method comprising: receiving digital content data including audio data and/or video data; generating haptic data using at least some of the received digital content data; encoding by an encoder the haptic data for efficient transmission over a communication network; multiplexing by a multiplexer the encoded haptic data with the received digital content data; embedding information for decoding the encoded haptic data in metadata of the multiplexed data stream; and sending the multiplexed data stream over the communication network. ### Claim 2 (depends on 1) The method of claim 1, further comprising: analyzing the haptic data to determine at least one characteristic of the haptic data, wherein said encoding the haptic data comprises encoding, based on the determined characteristic, the haptic data to meet a pre-defined criteria. ### Claim 3 (depends on 2) The method of claim 2, wherein the pre-defined criteria includes an output encoded haptic bit stream having the least number of bits. ### Claim 4 (depends on 2) The method of claim 2, further comprising: receiving an endpoint configuration of an end user device from the end user device, wherein the pre-defined criteria includes preserving a range of frequency of the haptic data that correspond to the received endpoint configuration of the end user device. ### Claim 5 (depends on 2) The method of claim 2, wherein said encoding the haptic data comprises selecting the encoder from a list of pre-determined encoders based on the determined characteristic and applying the selected encoder to transform the haptic data. ### Claim 6 (depends on 5) The method of claim 5, wherein the list of pre-determined encoders includes an Advanced Audio Coding encoder and/or a Free Lossless Audio encoder. ### Claim 7 (depends on 2) The method of claim 2, wherein the characteristics of the haptic data comprise one or more types of haptic output devices used to generate haptic effects based on the haptic data, intended use case of the haptic data, magnitude of the haptic data, frequency of the haptic data, and length of the silence in the haptic data. ### Claim 8 (depends on 7) The method of claim 7, wherein the types of haptic output devices include one or more of the group consisting of an eccentric rotating mass actuator, a linear resonant actuator, and a piezoelectric actuator. ### Claim 9 (depends on 7) The method of claim 7, wherein the intended use case of the haptic data is selected from the group consisting of music, movies and games. ### Claim 10 (depends on 2) The method of claim 2, wherein encoding the haptic data comprises: applying each one of a list of pre-determined encoders to the haptic data to generate a corresponding list of encoded haptic data streams; and selecting, using the pre-defined criteria, an encoded haptic data stream from the list of encoded haptic data streams. ### Claim 11 (depends on 1) The method of claim 1, wherein said encoding the haptic data comprises encoding the haptic data for a predetermined list of density factors to generate a plurality of encoded haptic data streams, each encoded haptic data stream corresponding to a density factor of the predetermined list of density factors, and wherein the method further comprises: receiving network bandwidth information from an end user device over a communication network; and selecting one of the encoded haptic data streams based on a corresponding density factor matching a condition of the communication network bandwidth for multiplexing with the received digital content data. ### Claim 12 (depends on 11) The method of claim 11, wherein a first encoded data stream of the plurality of encoded data streams corresponds to a first density factor of the predetermined list of density factors, and a second encoded data stream of the plurality of encoded data streams corresponds to a second density factor of the predetermined list of density factors, and wherein the second encoded data stream comprises more haptic data than the first data stream and the second density factor is greater than the first density factor. ### Claim 13 (depends on 11) The method of claim 11, further comprising: detecting a change in the network bandwidth information; selecting a different one of the encoded haptic data streams based on the corresponding density factor matching the condition of the communication network bandwidth; and transmitting the selected different one of the encoded data streams to the end user device. ### Claim 14 A system comprising: a processor configured to: receive digital content data including audio data and/or video data, generate haptic data using at least some of the received digital content data, encode the haptic data using an encoder for efficient transmission over a communication network, multiplex the encoded haptic data using a multiplexer with the received digital content data, embed information for decoding the encoded haptic data in metadata of the multiplexed data stream; and send the multiplexed data stream over the communication network; and a user device configured to receive the multiplexed encoded haptic data and digital contact data over the communication network, the user device comprising a haptic output device configured to output a haptic effect based on the haptic data. ### Claim 15 (depends on 14) The system of claim 14, wherein the processor is further configured to analyze the haptic data to determine at least one characteristic of the haptic data, and encode the haptic data based on the determined characteristic to meet a pre-defined criteria. ### Claim 16 (depends on 15) The system of claim 15, wherein the pre-defined criteria includes an output encoded haptic bit stream having the least number of bits. ### Claim 17 (depends on 15) The system of claim 15, wherein the processor is further configured to receive an endpoint configuration of the user device from the user device, wherein the pre-defined criteria includes preserving a range of frequency of the haptic data that correspond to the received endpoint configuration of the end user device. ### Claim 18 (depends on 15) The system of claim 15, wherein the processor is further configured to select the encoder from a list of pre-determined encoders based on the determined characteristic and apply the selected encoder to transform the haptic data. ### Claim 19 (depends on 18) The system of claim 18, wherein the list of pre-determined encoders includes an Advanced Audio Coding encoder and/or a Free Lossless Audio encoder. ### Claim 20 (depends on 15) The system of claim 15, wherein the characteristics of the haptic data comprise one or more types of haptic output devices used to generate haptic effects based on the haptic data, intended use case of the haptic data, magnitude of the haptic data, frequency of the haptic data, and length of the silence in the haptic data. ### Claim 21 (depends on 20) The system of claim 20, wherein the types of haptic output devices include one or more of the group consisting of an eccentric rotating mass actuator, a linear resonant actuator, and a piezoelectric actuator. ### Claim 22 (depends on 20) The system of claim 20, wherein the intended use case of the haptic data is selected from the group consisting of music, movies and games. ### Claim 23 (depends on 15) The system of claim 15, wherein the processor is further configured to: apply each one of a list of pre-determined encoders to the haptic data to generate a corresponding list of encoded haptic data streams; and select, using the pre-defined criteria, an encoded haptic data stream from the list of encoded haptic data streams. ### Claim 24 (depends on 14) The system of claim 14, wherein the processor is further configured to encode the haptic data for a predetermined list of density factors to generate a plurality of encoded haptic data streams, each encoded haptic data stream corresponding to a density factor of the predetermined list of density factors; receive network bandwidth information from an end user device over a communication network; and select one of the encoded haptic data streams based on a corresponding density factor matching a condition of the communication network bandwidth to multiplex with the received digital content data. ### Claim 25 (depends on 24) The system of claim 24, wherein a first encoded data stream of the plurality of encoded data streams corresponds to a first density factor of the predetermined list of density factors, and a second encoded data stream of the plurality of encoded data streams corresponds to a second density factor of the predetermined list of density factors, and wherein the second encoded data stream comprises more haptic data than the first data stream and the second density factor is greater than the first density factor. ### Claim 26 (depends on 24) The system of claim 24, wherein the processor is further configured to: detect a change in the network bandwidth information; select a different one of the encoded haptic data streams based on the corresponding density factor matching the condition of the communication network bandwidth; and transmit the selected different one of the encoded data streams to the end user device.