# US 12,182,909 B2 · Image processing apparatus and image processing method for generating a sound image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,909 B2
- Document ID: US12182909B2
- Status: Granted
- Kind: B2
- Application number: 17/801,035
- Filed: 2020-02-20
- Published: 2023-07-13
- Granted: 2024-12-31
- Earliest claimed: 2020-02-20
- Assignees: Nissan Motor Co., Ltd.
- Applicants: Nissan Motor Co., Ltd.
- Inventors: Satoru Hirose; Toru Takagi
- Web page: https://patentgrep.com/patent/US12182909B2

## CPC

- G01H 3/08: Analysing frequencies present in complex vibrations, e.g. comparing harmonics present {(acoustic presence detection G01V1/001)}
- G06T 11/26: Drawing of charts or graphs
- G01H 13/00: Measuring resonant frequency
- G06V 10/72: Data preparation, e.g. statistical preprocessing of image or video features
- G10L 21/14: by displaying frequency domain information
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L 25/90: Pitch determination of speech signals
- G01H: MEASUREMENT OF MECHANICAL VIBRATIONS OR ULTRASONIC, SONIC OR INFRASONIC WAVES
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

An image processing apparatus includes a controller. The controller calculates a fundamental frequency component included in sound data and a harmonic component corresponding to the fundamental frequency component, converts the fundamental frequency component and the harmonic component into image data, and generates a sound image where the fundamental frequency component and the harmonic component converted into the image data are arranged adjacent each other.

## Claims

### Claim 1

An image processing apparatus for acquiring sound data from a machine and analyzing the sound data that have been acquired, the image processing apparatus comprising:
a controller configured to:
transform the sound data to a frequency domain;
calculate a fundamental frequency component included in the sound data and a harmonic component corresponding to the fundamental frequency component, wherein the fundamental frequency is calculated based on a difference of frequencies between peaks in the frequency domain and a frequency of a first peak in the frequency domain; and
convert the fundamental frequency component and the harmonic component that have been calculated into image data;
generate a sound image comprising a two-dimensional matrix including the fundamental frequency component, the harmonic component, and other frequency components that have been converted into the image data arranged into predetermined areas, the arrangement based on the fundamental frequency component, and
determine, using a machine learning model generated with labelled sound images having a first fundamental frequency the same as the fundamental frequency, whether an abnormality is present in the machine using the sound image,
wherein in the sound image the fundamental frequency component and the harmonic component that have been converted into the image data are arranged adjacent each other,
wherein the two-dimensional matrix having a predetermined area set for each frequency component.

### Claim 2 (depends on 1)

The image processing apparatus according to claim 1, wherein the fundamental frequency component and the harmonic component have an overtone relationship, a same pitch name, or a relationship with respect to a critical band of human hearing.

### Claim 3 (depends on 1)

The image processing apparatus according to claim 1, wherein the controller arranges the fundamental frequency component and the harmonic component adjacent to each other on a vertical axis or a horizontal axis of the sound image.

### Claim 4 (depends on 1)

The image processing apparatus according to claim 1, wherein the controller generates the sound image by converting the fundamental frequency component and the harmonic component into image data in which a brightness or a color corresponding to an amplitude of the sound data is set.

### Claim 5 (depends on 1)

The image processing apparatus according to claim 1, wherein the controller arranges a plurality of frequency components of the sound data in frequency order on a vertical axis or a horizontal axis of the sound image.

### Claim 6 (depends on 1)

The image processing apparatus according to claim 1, wherein
the controller determines an abnormality in the sound image with the machine learning model by comparing the fundamental frequency component and the harmonic component with the other frequency components.

### Claim 7

A method of determining an abnormality in a machine, comprising:
acquiring sound data of the machine;
transforming the sound data to a frequency domain forming frequency data;
calculating a fundamental frequency component included in the sound data and a harmonic component corresponding to the fundamental frequency component, wherein the fundamental frequency is calculated based on a difference of frequencies between peaks in the frequency data and a frequency of a first peak in the frequency data;
converting the fundamental frequency component and the harmonic component that have been calculated into image data;
generating a sound image comprising a two-dimensional matrix including the fundamental frequency component, the harmonic component, and other frequency components that have been converted into image data arranged into predetermined areas, the arrangement based on the fundamental frequency component, and
determining, using a machine learning model generated with labelled sound images having a first fundamental frequency the same as the fundamental frequency, whether an abnormality is present in the machine using the sound image,
wherein in the sound image the fundamental frequency component and the harmonic component that have been converted into the image data are arranged adjacent each other,
wherein the two-dimensional matrix having a predetermined area set for each frequency component.