Method and device for interactive controlling on adult product based on multimodal model, and adult product

A method for dynamically controlling a physical actuator of an adult product, comprising:

• acquiring perception data of a user interacting with the adult product by at least one microphone and at least one sensor, wherein the perception data of the user comprises one or more modal data of voice data, tactile data, and visual data;
• the at least one sensor comprises at least one pressure sensor;
• processing the perception data of the user by using a processor to run a multimodal neural network model, which comprises:
• conducting feature-extracting on the one or more modal data in the perception data to obtain modal feature vectors corresponding to the one or more modal data respectively;
• applying an attention mechanism to each modal feature vector to obtain modal weights by calculation and generate a weighted cross-modal feature representation that quantifies an emotional state of the user;
• mapping the weighted cross-modal feature representation to a specific emotion class of the user selected from a predefined set of emotion classes;
• conducting a search in a data structure of action parameter vectors to generate a multi-component action parameter vector, A=[a1, a2, . . . , an], wherein the data structure of action parameter vectors stores predefined action parameter vectors for each emotion class in the predefined set of emotion classes as a response to the specific emotion class of the user;
• each component of the action parameter vector represents a specific numerical control value for an independent physical control dimension of the adult product, the physical control dimensions comprising at least vibration frequency and contraction intensity;
• and
• converting the numerical control values of the action parameter vector into drive signals and transmitting the drive signals to the physical actuator, so that the adult product performs physical actions defined by the numerical control values,
• wherein the conducting feature-extracting on the one or more modal data in the perception data to obtain modal feature vectors corresponding to the one or more modal data respectively comprises:
• when the perception data comprises voice data, converting voice data using a pre-trained language model into voice text and extracting semantic representation vectors, extracting fundamental audio features from the voice data using Mel-frequency cepstral coefficients, and extracting acoustic feature vectors from the fundamental audio features using spectrum analysis, wherein the acoustic feature vectors comprise fundamental frequency, energy, and harmonic-to-noise ratio;
• when the perception data comprises tactile data, processing tactile data by using a multilayer perceptron network to obtain tactile features vectors corresponding to tactile data, wherein the tactile feature vectors comprise one or more of touch pattern, touch force, touch rhythm, touch trajectory, touch area, and action frequency;
• and
• when the perception data comprises visual data, extracting facial expression features based on a facial recognition neural network, determining gazing duration and gazing frequency using a pupil tracking algorithm to obtain eye contact feature vectors, and extracting body posture feature vectors using a human pose estimation model.