# US 10,313,822 B2 · Image and audio based characterization of a human auditory system for personalized audio reproduction > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,313,822 B2 - Document ID: US10313822B2 - Status: Granted - Kind: B2 - Application number: 15/811,295 - Filed: 2017-11-13 - Published: 2018-05-17 - Granted: 2019-06-04 - Earliest claimed: 2016-11-13 - Assignees: EmbodyVR, Inc. - Applicants: EmbodyVR, Inc. - Inventors: Kapil Jain - Web page: https://patentgrep.com/patent/US10313822B2 ## CPC - A61B 5/6803: {Head-worn items, e.g. helmets, masks, headphones or goggles} - A61B 5/12: Audiometering - A61B 5/121: {evaluating hearing capacity} - A61B 5/6815: {Ear} - A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems} - G01B 7/14: for measuring distance or clearance between spaced objects or spaced apertures (G01B7/30 takes precedence) - G06T 1/0007: {Image acquisition} - G06T 7/73: using feature-based methods - G06T 7/80: Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration - G06V 40/10: Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands - H04R 1/005: {using digitally weighted transducing elements} - H04R 1/1008: {Earpieces of the supra-aural or circum-aural type} - H04R 1/1058: {Manufacture or assembly} - H04R 1/22: for obtaining desired frequency characteristic only {(circuit for combining transducers having different responses H04R3/00; for hearing aids H04R25/407)} - H04R 1/32: for obtaining desired directional characteristic only {(specially adapted for hearing aids H04R25/40)} - H04R 1/323: {for loudspeakers (H04R1/34 and H04R1/40 take precedence)} - H04R 3/04: for correcting frequency response - H04R 5/04: Circuit arrangements, {e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments (combinations of amplifiers H03F3/68; stereophonic systems H04S)} - H04S 3/008: {in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02)} - H04S 7/301: {Automatic calibration of stereophonic sound system, e.g. with test microphone} - H04S 7/304: {For headphones} - H04S 7/306: {For headphones} - A61B 5/4005: {for evaluating the sensory system (A61B5/12, A61B5/4058 take precedence)} - A61B 5/7415: {Sound rendering of measured values, e.g. by pitch or volume variation (A61B5/741 takes precedence)} - A61B 6/5217: {extracting a diagnostic or physiological parameter from medical diagnostic data} - G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems - H04R 1/1016: {Earpieces of the intra-aural type} - H04R 5/033: Headphones for stereophonic communication {(details thereof, e.g. relating to batteries, cables or control elements H04R1/10)} - H04R 11/02: Loudspeakers - H04R 2201/029: Manufacturing aspects of enclosures transducers - H04R 2225/77: Design aspects, e.g. CAD, of hearing aid tips, moulds or housings - H04S 7/30: {Control circuits for electronic adaptation of the sound field} - H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence) - H04S 2420/01: Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD] - A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D) - G01B: MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K) - H04S: STEREOPHONIC SYSTEMS - G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA ## Abstract An image sensor in a first earcup captures an image of a pinna. First sound is output by a transducer in a second earcup located at the pinna and respective second sound is detected by each of one or more microphones in the second earcup located at the pinna. Based on the captured image and the respective second audio sound from each of the one or more microphones, a non-linear transfer function is determined which characterizes how sound is transformed by the pinna. A signal is generated indicative of one or more audio cues for spatializing third sound based on the determined non-linear transfer function. ## Claims ### Claim 1 A method comprising: capturing, by an image sensor in a first earcup, a first image of a pinna; outputting, by a first transducer in a second earcup located at the pinna, first sound; detecting, by each of one or more microphones in the second earcup located at the pinna, respective second sound based on the output of the first sound; based on the captured first image and the respective second sound from each of the one or more microphones, determining a first non-linear transfer function which characterizes how sound is transformed by the pinna; generating a signal indicative of one or more audio cues for spatializing third sound based on the first non-linear transfer function; and outputting, by a second transducer in the first or second earcup located on the pinna, the signal indicative of one or more audio cues to facilitate spatial localization of the third sound via the pinna; wherein determining the first non-linear transfer function comprises: comparing characteristics of features of the captured first image to a second image associated with a second non-linear transfer function; defining a first correlation score based on the comparison of the characteristics of the features of the first image to the second image; comparing the detected respective second sound of each of the one or more microphones to a given one or more frequency responses of the second non-linear transfer function; defining a second correlation score based on the comparison of the detected respective second sound to the given one or more frequency responses of the second non-linear transfer function; and based on the first correlation score, the second correlation score, and the second non-linear transfer function, determining the first non-linear transfer function. ### Claim 2 (depends on 1) The method of claim 1, wherein detecting, by each of the one or more microphones in the second earcup located at the pinna, respective second sound based on the output of the first sound comprises detecting, by each microphone of a plurality of microphones in the second earcup, the respective second sound based on the output of the first sound, wherein each microphone is positioned adjacent to a given feature of the pinna. ### Claim 3 (depends on 2) The method of claim 2, wherein the first transducer is surrounded by the plurality of microphones. ### Claim 4 (depends on 1) The method of claim 1, wherein the first and second transducer are the same. ### Claim 5 (depends on 1) The method of claim 1, the first sound comprises a chirp sound in an audible frequency range. ### Claim 6 (depends on 1) The method of claim 1, wherein generating the signal indicative of one or more audio cues comprises modulating a signal indicative of the third sound with the first non-linear transfer function to produce a modulated signal which defines the one or more audio cues for spatializing sound associated with the third sound. ### Claim 7 (depends on 1) The method of claim 1, wherein the one or more audio cues indicates one or more of an elevation, azimuth, distance, and velocity of the spatialized third sound. ### Claim 8 (depends on 1) The method of claim 1, wherein capturing, by the image sensor in the first earcup, the first image of the pinna comprises positioning the first earcup at a given linear distance from the pinna and capturing the first image at the given linear distance from the pinna. ### Claim 9 (depends on 1) The method of claim 1, wherein the pinna is a first pinna; wherein the first earcup and second earcup are coupled to a tensioned headband; and wherein capturing, by the image sensor in the first earcup, the first image of the pinna comprises pulling the first earcup against the tension of the headband to position the first earcup at a given linear distance from the first pinna while the second earcup remains seated on a second pinna; determining that the first earcup is pulled a given linear distance from the pinna; and capturing the first image at the given linear distance from the first pinna when the first earcup is pulled to the given linear distance from the pinna. ### Claim 10 (depends on 9) The method of claim 9, wherein a proximity sensor in the first earcup measures the given linear distance from the pinna. ### Claim 11 (depends on 1) The method of claim 1, wherein the first earcup and second earcup are the same earcup. ### Claim 12 (depends on 1) The method of claim 1, wherein the captured first image is associated with a first individual and the second image is associated with a second individual and the second non-linear transfer function; wherein the detected respective second sound is associated with the first individual and the given one or more frequency responses of the second non-linear transfer function is associated with the second individual. ### Claim 13 One or more non-transitory computer readable media comprising program code stored in memory and executable by a processor, the program code to: capture, by an image sensor, a first image of a pinna; output, by a first transducer in an earcup located at the pinna, first sound; detect, by each of one or more microphones in the earcup located at the pinna, respective second sound based on the output of the first sound; based on the captured first image and the respective second sound from each of the one or more microphones, determine a first non-linear transfer function which characterizes how sound is transformed by the pinna; generate a signal indicative of one or more audio cues for spatializing third sound based on the determined first non-linear transfer function; and output, by a second transducer in the earcup located on the pinna, the signal indicative of one or more audio cues to facilitate spatial localization of the third sound via the pinna; wherein determining the first non-linear transfer function comprises: comparing characteristics of features of the captured first image to a second image associated with a second non-linear transfer function; defining a first correlation score based on the comparison of the characteristics of the features of the captured first image to the second image; comparing the detected respective second sound of each of the one or more microphones to a given one or more frequency responses of the second non-linear transfer function; defining a second correlation score based on the comparison of the detected respective second sound to the given one or more frequency responses of the second non-linear transfer function; and based on the first correlation score, the second correlation score, and the second non-linear transfer function, determining the first non-linear transfer function. ### Claim 14 (depends on 13) The one or more non-transitory computer readable media of claim 13, wherein the one or more audio cues indicates one or more of an elevation, azimuth, distance, and velocity of the spatialized third sound. ### Claim 15 (depends on 13) The one or more non-transitory computer readable media of claim 13, wherein the program code for generating the signal indicative of one or more audio cues comprises modulating a signal indicative of the third sound with the first non-linear transfer function to produce a modulated signal which defines the one or more audio cues for spatializing sound associated with the third sound. ### Claim 16 (depends on 13) The one or more non-transitory computer readable media of claim 13, wherein the captured first image is associated with a first individual and the second image is associated with a second individual and the second non-linear transfer function; wherein the detected respective second sound is associated with the first individual and the given one or more frequency responses of the second non-linear transfer function is associated with the second individual. ### Claim 17 (depends on 13) The one or more non-transitory computer readable media of claim 13, wherein the image sensor is in a mobile phone or another earcup. ### Claim 18 A system comprising: an earcup; and computer instructions stored in memory and executable by a processor to perform the functions of: receiving, from an image sensor, a first image of a pinna; outputting, by a first transducer in the earcup located at the pinna, first sound; receiving, from each of one or more microphones in the earcup located at the pinna, respective second sound based on the output of the first sound; based on the first image and the respective second sound from each of the one or more microphones, determining a first non-linear transfer function which characterizes how sound is transformed by the pinna; generating a signal indicative of one or more audio cues for spatializing third sound based on the determined first non-linear transfer function; and outputting, by a second transducer in the earcup located on the pinna, the signal indicative of one or more audio cues to facilitate spatial localization of the third sound via the pinna; wherein determining the first non-linear transfer function comprises: comparing characteristics of features of the first image to a second image associated with a second non-linear transfer function; defining a first correlation score based on the comparison of the characteristics of the features of the first image to the second image; comparing the detected respective second sound of each of the one or more microphones to a given one or more frequency responses of the second non-linear transfer function; defining a second correlation score based on the comparison of the detected respective second sound to the given one or more frequency responses of the second non-linear transfer function; and based on the first correlation score, the second correlation score, and the second non-linear transfer function, determining the first non-linear transfer function. ### Claim 19 (depends on 18) The system of claim 18, wherein the one or more audio cues indicates one or more of an elevation, azimuth, distance, and velocity of the spatialized third sound. ### Claim 20 (depends on 18) The system of claim 18, wherein the computer instructions stored in memory and executable by the processor for generating the signal indicative of one or more audio cues comprises modulating a signal indicative of the third sound with the first non-linear transfer function to produce a modulated signal which defines the one or more audio cues for spatializing sound associated with the third sound. ### Claim 21 (depends on 18) The system of claim 18, wherein the first image is associated with a first individual and the second image is associated with a second individual and the second non-linear transfer function; wherein the received respective second sound is associated with the first individual and the given one or more frequency responses of the second non-linear transfer function is associated with the second individual. ### Claim 22 (depends on 18) The system of claim 18, wherein the image sensor is in a mobile phone or another earcup.