# US 10,313,816 B2 · Augmented performance synchronization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,313,816 B2 - Document ID: US10313816B2 - Status: Granted - Kind: B2 - Application number: 15/708,715 - Filed: 2017-09-19 - Published: 2018-03-22 - Granted: 2019-06-04 - Earliest claimed: 2016-09-19 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Zhipeng Zhang; Brian T. Gleeson; Michael Diu; Addison Cugini - Web page: https://patentgrep.com/patent/US10313816B2 ## CPC - H04S 7/00: Indicating arrangements; Control arrangements, e.g. balance control - G08B 6/00: Tactile signalling systems, e.g. tactile personal calling systems - H04R 3/00: Circuits for transducers (arrangements for producing a reverberation or echo sound G10K15/08; amplifiers H03F) - H04R 1/1008: {Earpieces of the supra-aural or circum-aural type} - H04R 2400/00: Loudspeakers - H04R 2410/00: Microphones - H04R 2460/13: Hearing devices using bone conduction transducers - H04R 2499/11: Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's - H04S 2400/03: Aspects of down-mixing multi-channel audio to configurations with lower numbers of playback channels, e.g. 7.1 -> 5.1 (H04S2400/01 takes precedence) - H04S: STEREOPHONIC SYSTEMS - G08B: SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) ## Abstract Various embodiments of the invention pertain to augmented performance synchronization systems and methods. According to some embodiments of the invention, an audio waveform may be used to generate one or more haptic waveforms for one or more electronic devices. The haptic waveforms may be generated based on any of a number of factors, including features of the audio waveform, capabilities of the haptic actuators performing the haptic waveforms, the number, type and location of devices having haptic actuators, and the like. The haptic waveforms may be synchronized with performance of the audio waveform to provide an augmented listening experience to a user. ## Claims ### Claim 1 A method comprising: receiving an audio waveform at an electronic device that includes one or more speakers and a haptic actuator; generating an actuator control signal from the audio waveform, the actuator control signal comprising discrete haptic actuation taps corresponding to peak intensities or the onset of peak intensities of the audio waveform; generating an audio output using the audio waveform at the one or more speakers; synchronizing transmission of the actuator control signal to the haptic actuator with generation of the audio output at the one or more speakers; and actuating the haptic actuator with the actuator control signal while performing the audio output by the one or more speakers. ### Claim 2 (depends on 1) The method of claim 1, further comprising modifying the actuator control signal based on metadata associated with the audio waveform. ### Claim 3 (depends on 2) The method of claim 2, wherein the metadata associated with the audio waveform is selected from a group consisting of an artist, a genre and an album. ### Claim 4 (depends on 3) The method of claim 3, wherein generating the actuator control signal from the audio waveform comprises filtering the audio waveform using a bandpass filter. ### Claim 5 (depends on 1) The method of claim 1, wherein the electronic device further includes an input element, and wherein the method further comprises: receiving a user input from the input element; and adjusting the actuator control signal based on the user input. ### Claim 6 (depends on 1) The method of claim 1, further comprising: determining a contact location of the electronic device with a user of the electronic device; and modifying the actuator control signal based on the contact location. ### Claim 7 (depends on 1) The method of claim 1, further comprising: identifying one or more different types of instruments contributing to audio waves making up the audio waveform; and modifying the actuator control signal based on the identified type of music. ### Claim 8 (depends on 1) The method of claim 1, wherein the electronic device includes a plurality of haptic actuators. ### Claim 9 (depends on 1) The method of claim 1, wherein generating the actuator control signal includes: extracting a feature from the audio waveform selected from a group consisting of treble, bass, beat, tempo, time signature, rhythmic patterns, loudness range, change of loudness over time, accents, melodic properties, complexity of harmony, prominent pitch classes, melody, chorus, time, verse, number of instruments, types of instruments; applying a haptic element to the actuator control signal based on the extracted feature. ### Claim 10 An electronic device comprising: one or more speakers; a haptic actuator; one or more processors; and a non-transitory computer readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: receiving an audio waveform, wherein the audio waveform is stereophonic; generating an actuator control signal from the audio waveform, the actuator control signal comprising discrete haptic actuation taps corresponding to peak intensities of the audio waveform; and synchronizing transmission of the actuator control signal to the haptic actuator with transmission of the audio waveform to the one or more speakers; and actuating the haptic actuator with the actuator control signal while performing the audio waveform by the one or more speakers. ### Claim 11 (depends on 10) The electronic device of claim 10, wherein the haptic actuator is a linear actuator. ### Claim 12 (depends on 10) The electronic device of claim 10, wherein the electronic device further comprises an input element, and wherein the operations further include: receiving a user input from the input element; and adjusting actuation of the haptic actuator based on the user input. ### Claim 13 (depends on 10) The electronic device of claim 10, wherein the operations further include: determining a contact location of the electronic device with a user of the electronic device; and modifying the actuator control signal based on the contact location. ### Claim 14 (depends on 10) The electronic device of claim 10, wherein the operations further include: identifying one or more different types of instruments contributing to audio waves making up the audio waveform; and modifying the actuator control signal based on the identified type of music. ### Claim 15 (depends on 8) The electronic device of claim 8, wherein generating the actuator control signal includes: extracting a feature from the audio waveform; and applying a haptic element to the feature. ### Claim 16 (depends on 10) The electronic device of claim 10, wherein the electronic device includes a plurality of haptic actuators.