# US 10,503,263 B2 · Systems and methods for Foley-style haptic content creation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,503,263 B2 - Document ID: US10503263B2 - Status: Granted - Kind: B2 - Application number: 16/107,061 - Filed: 2018-08-21 - Published: 2019-01-03 - Granted: 2019-12-10 - Earliest claimed: 2014-06-16 - Assignees: Immersion Corporation - Applicants: Immersion Corporation - Inventors: Doug Billington - Web page: https://patentgrep.com/patent/US10503263B2 ## CPC - G06F 3/016: {Input arrangements with force or tactile feedback as computer generated output to the user} - G11B 27/034: on discs (G11B27/036, G11B27/038 take precedence) - G11B 27/322: {used signal is digitally coded} - H04N 7/0255: {Display systems therefor} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract One illustrative system disclosed herein includes an audio input device configured to detect an audio input and transmit an audio signal associated with the audio input. The illustrative system also includes a haptic trigger device which includes a sensor, wherein the sensor is configured to transmit a sensor signal associated with an event. Further, the illustrative system includes a processor in communication with the audio input device and the haptic trigger device, the processor configured to: receive the audio signal; receive the sensor signal substantially simultaneously to receiving the audio signal; record the audio signal to an audio track; and insert a haptic effect marker into a haptic track based in part on the sensor signal, wherein the haptic track and the audio track are associated with a video. ## Claims ### Claim 1 A computer-implemented method for concurrently modifying an audio track and generating a haptic track for a video file, the computer-implemented method comprising: playing the video file, the video file comprising the audio track; and while the video file is being played: receiving a sound effect from a microphone; substantially simultaneously with receiving the sound effect, detecting an event using a haptic trigger device; based on receiving the sound effect, modifying the audio track by incorporating the sound effect into the audio track such that the sound effect corresponds to one or more frames, in the video file, being played when the sound effect was received; and based on detecting the event, inserting a haptic effect marker into the haptic track such that the haptic effect marker also corresponds to the one or more frames in the video file. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the haptic effect marker comprises timestamp data corresponding to the one or more frames in the video file. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the haptic effect marker comprises haptic effect data that is configured to cause a haptic output device to output a haptic effect during playback of the one or more frames. ### Claim 4 (depends on 3) The computer-implemented method of claim 3, further comprising, prior to inserting the haptic effect marker into the haptic track: receiving user input indicating a selection of a haptic effect from among a plurality of available haptic effects; receiving a customization of a characteristic of the haptic effect; and determining the haptic effect data based on the selected haptic effect and the customization of the characteristic. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the event involves a movement of a prop used to create the sound effect. ### Claim 6 (depends on 5) The computer-implemented method of claim 5, wherein the haptic trigger device is positioned on the prop. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the haptic trigger device includes a gyroscope, an accelerometer, a depth sensor, a camera, a phototransistor, a switch, a button, or a piezoelectric transducer. ### Claim 8 A system for concurrently modifying an audio track and generating a haptic track for a video file, the system comprising: a processor; and a memory on which instructions executable by the processor are stored to cause the processor to: play the video file, the video file comprising the audio track; and while the video file is being played: receive a sound effect from a microphone; substantially simultaneously with receiving the sound effect, detect an event using a haptic trigger device; based on receiving the sound effect, modify the audio track by incorporating the sound effect into the audio track such that the sound effect corresponds to one or more frames, in the video file, being played when the sound effect was received; and based on detecting the event, insert a haptic effect marker into the haptic track such that the haptic effect marker also corresponds to the one or more frames in the video file. ### Claim 9 (depends on 8) The system of claim 8, wherein the haptic effect marker comprises timestamp data corresponding to the one or more frames in the video file. ### Claim 10 (depends on 8) The system of claim 8, wherein the haptic effect marker comprises haptic effect data that is configured to cause a haptic output device to output a haptic effect during playback of the one or more frames. ### Claim 11 (depends on 10) The system of claim 10, wherein the memory further includes instructions that are executable by the processor to cause the processor to, prior to inserting the haptic effect marker into the haptic track: receive user input indicating a selection of a haptic effect from among a plurality of available haptic effects; and determine the haptic effect data based on the selected haptic effect and the sound effect. ### Claim 12 (depends on 8) The system of claim 8, wherein the event involves a movement of a prop used to create the sound effect. ### Claim 13 (depends on 12) The system of claim 12, wherein the haptic trigger device is positioned on the prop. ### Claim 14 (depends on 8) The system of claim 8, wherein the haptic trigger device includes a gyroscope, an accelerometer, a depth sensor, a camera, a phototransistor, a switch, a button, or a piezoelectric transducer. ### Claim 15 A non-transitory computer-readable medium comprising program code that is executable by a processor for causing the processor to: play a video file, the video file comprising an audio track; and while the video file is being played: receive a sound effect from a microphone; substantially simultaneously with receiving the sound effect, detect an event using a haptic trigger device; based on receiving the sound effect, modify the audio track to incorporate the sound effect into the audio track such that the sound effect corresponds to one or more frames, in the video file, being played when the sound effect was received; and based on detecting the event, insert a haptic effect marker into a haptic track such that the haptic effect marker also corresponds to the one or more frames in the video file. ### Claim 16 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the haptic effect marker comprises timestamp data corresponding to the one or more frames in the video file. ### Claim 17 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the haptic effect marker comprises haptic effect data that is configured to cause a haptic output device to output a haptic effect during playback of the one or more frames. ### Claim 18 (depends on 17) The non-transitory computer-readable medium of claim 17, further comprising program code that is executable by the processor to cause the processor to, prior to inserting the haptic effect marker into the haptic track: receive user input indicating a selection of a haptic effect from among a plurality of available haptic effects; and determine the haptic effect data based on the selected haptic effect and the sound effect. ### Claim 19 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the event involves a movement of a prop configured to create the sound effect. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein the haptic trigger device is positioned on the prop. ### Claim 21 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the haptic trigger device includes a gyroscope, an accelerometer, a depth sensor, a camera, a phototransistor, a switch, a button, or a piezoelectric transducer. ### Claim 22 (depends on 15) The non-transitory computer-readable medium of claim 15, wherein the haptic trigger device is separate from the microphone.