# US 2014/0150629 A1 · Joint and coordinated visual-sonic metaphors for interactive multi-channel data sonification to accompany data visualization > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0150629 A1 - Document ID: US20140150629A1 - Status: Published application - Kind: A1 - Application number: 14/176,000 - Filed: 2014-02-07 - Published: 2014-06-05 - Earliest claimed: 2010-06-17 - Inventors: Lester F. Ludwig - Web page: https://patentgrep.com/patent/US20140150629A1 ## CPC - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10H 1/06: Circuits for establishing the harmonic content of tones {, or other arrangements for changing the tone colour} - G10H 2210/301: Soundscape or sound field simulation, reproduction or control for musical purposes, e.g. surround or 3D sound; Granular synthesis - G10H 2220/351: Environmental parameters, e.g. temperature, ambient light, atmospheric pressure, humidity, used as input for musical purposes - G10H 2250/471: General musical sound synthesis principles, i.e. sound category-independent synthesis methods - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE ## Abstract Data sonification arrangements for use with data visualization so as to provide parallel perceptual channels for representing complex numerical data to a user seeking to identify correlations within the data are presented. In an implementation, several varying data quantities are represented by time-varying graphics while several other varying data quantities are represented by time-varying sound, both presented to a user to observe correlations between sonic and visual events or trends. Sonification can be used to offload some information-carrying information capacity from a visualization system, while other information can be rendered via both sonification and visualization to provide affirming or orienting redundancy. In an implementation joint and coordinated visual-sonic metaphors are used for this or other purposes. For example, data sonification can include multiple data-modulated sound timbre classes, each rendered within a stereo sound field according to a spatial metaphor that is shared with the visualization. ## Claims ### Claim 1 A joint data sonification and visualization method for representing a first plurality of channels of numerical information via data visualization and a second plurality of channels of numerical information via at least one audio-frequency waveform, the audio-frequency waveform perceived by a user as an audio tone with a plurality of corresponding discernable timbre attributes, further employing spatial sound rendering, the method comprising: receiving multidimensional numerical data; generating plurality of audio-frequency waveforms, each audio-frequency waveform comprising an associated fixed audio-frequency and at least one adjustable timbre attribute responsive to an associated timbre control parameter, each timbre control parameter having an associated adjustable value which can be discernibly varied within a timbre space occupied by the plurality of audio-frequency waveforms; associating aspects of multidimensional numerical data with the at least one timbre control parameter of at least one of the audio-frequency waveforms from the plurality of audio-frequency waveforms; and adjusting the value of each timbre control parameter responsive to an associated first value of the multidimensional numerical data according to a mapping function, the mapping function transforming first values of the multidimensional numerical data to values of associated timbre control parameters, wherein the timbre and the spatial location of the at least one audio-frequency waveform carries information responsive to values of the multidimensional numerical data for presentation as data sonification to a user, and wherein at least a portion of the multidimensional numerical data is provided to data visualization software. ### Claim 2 (depends on 1) The method of claim 1 wherein the at least one audio-frequency waveform is sonically rendered within a stereo sound field, and the spatial location of the sonic rendering of the at least one audio-frequency waveform is responsive to second values of the multidimensional numerical data. ### Claim 3 (depends on 2) The method of claim 2 wherein the spatial location of the sonic rendering of the at least one audio-frequency waveform is made according to another associated mapping function. ### Claim 4 (depends on 1) The method of claim 1 wherein the at least one audio-frequency waveform is sonically rendered within an at least two-dimensional spatial sound field, and the spatial location of the sonic rendering of the at least one audio-frequency waveform is responsive to second values of the multidimensional numerical data. ### Claim 5 (depends on 1) The method of claim 1 wherein the values of the multidimensional numerical data are sequentially selected responsive to a position on a trajectory specified by the user through a subset of the multidimensional numerical data, the trajectory rendered in the data visualization. ### Claim 6 (depends on 1) The method of claim 1 wherein the values of the multidimensional numerical data are sequentially selected responsive to a position of a cursor within the rendered data visualization. ### Claim 7 (depends on 6) The method of claim 6 wherein the position of the cursor and the values of the multidimensional numerical data used in the data sonification are sequentially selected according to a trajectory selected through a subset of the multidimensional numerical data. ### Claim 8 (depends on 1) The method of claim 1 wherein the multidimensional numerical data is retrieved from storage. ### Claim 9 (depends on 1) The method of claim 1 wherein the multidimensional numerical data is provided by a live real-time data stream. ### Claim 10 (depends on 1) The method of claim 1 wherein the multidimensional numerical data is created from live sensor data. ### Claim 11 (depends on 1) The method of claim 1 wherein the mapping comprises pre-sonification operations involving the imposing of indexing on selected data. ### Claim 12 (depends on 1) The method of claim 1 wherein the mapping comprises pre-sonification operations involving data suppression. ### Claim 13 (depends on 1) The method of claim 1 wherein the mapping comprises pre-sonification operations involving data value normalization. ### Claim 14 (depends on 1) The method of claim 1 wherein the mapping comprises pre-sonification operations involving linear transformations. ### Claim 15 (depends on 1) The method of claim 1 wherein the mapping comprises pre-sonification operations involving affine transformations. ### Claim 16 (depends on 1) The method of claim 1 wherein the mapping comprises pre-sonification signal processing filtering operations on selected data. ### Claim 17 (depends on 1) The method of claim 1 wherein the mapping comprises pre-sonification operations involving nonlinear transformations. ### Claim 18 (depends on 1) The method of claim 1 wherein the mapping comprises pre-sonification operations involving linear predictor operations. ### Claim 19 (depends on 1) The method of claim 1 wherein the mapping comprises pre-sonification operations involving interpolation operations. ### Claim 20 (depends on 1) The method of claim 1 wherein at least one audio-frequency waveform comprises a plurality of timbre control parameters, each timbre control parameter affecting the timbre of the audio-frequency waveform; and wherein the mapping element adjusts the value of each of the plurality of timbre control parameters responsive to values of the multidimensional numerical data. ### Claim 21 (depends on 1) The method of claim 1 wherein a selected metaphor is used to automatically generate parameter assignments and graphics rendering operations. ### Claim 22 (depends on 1) The method of claim 1 wherein the first plurality of channels of numerical information and second plurality of channels of numerical information both comprise at least one channel of numerical information in common. ### Claim 23 (depends on 1) The method of claim 1 wherein the data sonification and the data visualization both employ a spatial metaphor. ### Claim 24 (depends on 1) The method of claim 1 wherein the data sonification and the data visualization share at least one data preprocessing operation. ### Claim 25 (depends on 1) The method of claim 1 wherein the data sonification and the data visualization and share a common data preprocessing environment.