Send Audio data represent as numpy array from python to Javascript

by

I have a TTS (text-to-speech) system that produces audio in numpy-array form whose data type is np.float32. This system is running in the backend and I want to transfer the data from the backend to the frontend to be played when a certain event happens.

The obvious solution for this problem is to write the audio data on disk as a wav file and then pass the path to the frontend to be played. This worked fine, but I don’t want to do that for administrative reasons. I just want to transfer only the audio data (numpy array) to the frontend.

What I have done till now is the following:

backend

text = "Hello"
wav, sr = tts_model.synthesize(text)
data = {"snd", wav.tolist()}
flask_response = app.response_class(response=flask.json.dumps(data),
                                    status=200,
                                    mimetype='application/json' )
# then return flask_response

frontend

// gets wav from backend
let arrayData = new Float32Array(wav);
let blob = new Blob([ arrayData ]);
let url = URL.createObjectURL(blob);
let snd = new Audio(url);
snd.play()

That what I have done till now, but the JavaScript throws the following error:

Uncaught (in promise) DOMException: Failed to load because no supported source was found.

This is the gist of what I’m trying to do. I’m so sorry, you can’t repreduce the error as you don’t have the TTS system, so this is an audio file generated by it which you can use to see what I’m doing wrong.

Other things I tried:

  • Change the audio datatype to np.int8, np.int16 to be casted in the JavaScript by Int8Array() and int16Array() respectively.
  • tried different types when creating the blob such as {"type": "application/text;charset=utf-8;"} and {"type": "audio/ogg; codecs=opus;"}.

I have been struggling in this issue for so long, so any help is appriciated !!

37 thoughts on “Send Audio data represent as numpy array from python to Javascript”