Send Audio data represent as numpy array from python to Javascript

by

I have a TTS (text-to-speech) system that produces audio in numpy-array form whose data type is np.float32. This system is running in the backend and I want to transfer the data from the backend to the frontend to be played when a certain event happens.

The obvious solution for this problem is to write the audio data on disk as a wav file and then pass the path to the frontend to be played. This worked fine, but I don’t want to do that for administrative reasons. I just want to transfer only the audio data (numpy array) to the frontend.

What I have done till now is the following:

backend

text = "Hello"
wav, sr = tts_model.synthesize(text)
data = {"snd", wav.tolist()}
flask_response = app.response_class(response=flask.json.dumps(data),
                                    status=200,
                                    mimetype='application/json' )
# then return flask_response

frontend

// gets wav from backend
let arrayData = new Float32Array(wav);
let blob = new Blob([ arrayData ]);
let url = URL.createObjectURL(blob);
let snd = new Audio(url);
snd.play()

That what I have done till now, but the JavaScript throws the following error:

Uncaught (in promise) DOMException: Failed to load because no supported source was found.

This is the gist of what I’m trying to do. I’m so sorry, you can’t repreduce the error as you don’t have the TTS system, so this is an audio file generated by it which you can use to see what I’m doing wrong.

Other things I tried:

  • Change the audio datatype to np.int8, np.int16 to be casted in the JavaScript by Int8Array() and int16Array() respectively.
  • tried different types when creating the blob such as {"type": "application/text;charset=utf-8;"} and {"type": "audio/ogg; codecs=opus;"}.

I have been struggling in this issue for so long, so any help is appriciated !!

35 thoughts on “Send Audio data represent as numpy array from python to Javascript”

  1. Convert wav array of values to bytes

    Right after synthesis you can convert numpy array of wav to byte object then encode via base64.

    import io
from scipy.io.wavfile import write

bytes_wav = bytes()
byte_io = io.BytesIO(bytes_wav)
write(byte_io, sr, wav)
wav_bytes = byte_io.read()

audio_data = base64.b64encode(wav_bytes).decode('UTF-8')

    This can be used directly to create html audio tag as source (with flask):

    <audio controls src="data:audio/wav;base64, {{ audio_data }}"></audio>

    So, all you need is to convert wav, sr to audio_data representing raw .wav file. And use as parameter of render_template for your flask app. (Solution without sending)

    Or if you send audio_data, in .js file where you accept response, use audio_data to construct url (would be placed as src attribute like in html):

    // get audio_data from response

let snd = new Audio("data:audio/wav;base64, " + audio_data);
snd.play()

    because:

    Audio(url) Return value:
    A new HTMLAudioElement object, configured to be used for playing back the audio from the file specified by url.The new object’s preload property is set to auto and its src property is set to the specified URL or null if no URL is given. If a URL is specified, the browser begins to asynchronously load the media resource before returning the new object.

    Reply

  2. Your sample as is does not work out of the box. (Does not play)

    However with:

    • StarWars3.wav: OK. retrieved from cs.uic.edu
    • your sample encoded in PCM16 instead of PCM32: OK (check the wav metadata)

    Flask

    from flask import Flask, render_template, json
import base64

app = Flask(__name__)

with open("sample_16.wav", "rb") as binary_file:
    # Read the whole file at once
    data = binary_file.read()
    wav_file = base64.b64encode(data).decode('UTF-8')

@app.route('/wav')
def hello_world():
    data = {"snd": wav_file}
    res = app.response_class(response=json.dumps(data),
        status=200,
        mimetype='application/json')
    return res

@app.route('/')
def stat():
    return render_template('index.html')

if __name__ == '__main__':
    app.run(debug = True)

    js

    
  <audio controls></audio>
  <script>
    ;(async _ => {
      const res = await fetch('/wav')
      let {snd: b64buf} = await res.json()
      document.querySelector('audio').src="data:audio/wav;base64, "+b64buf;
    })()
  </script>

    Original Poster Edit

    So, what I ended up doing before (using this solution) that solved my problem is to:

    • First, change the datatype from np.float32 to np.int16:
    wav = (wav * np.iinfo(np.int16).max).astype(np.int16)
    • Write the numpy array into a temporary wav file using scipy.io.wavfile:
    from scipy.io import wavfile
wavfile.write(".tmp.wav", sr, wav)
    • Read the bytes from the tmp file:
    # read the bytes
with open(".tmp.wav", "rb") as fin:
    wav = fin.read()
    • Delete the temporary file
    import os
os.remove(".tmp.wav")
    Reply

  23. community cast now positive words that start with e , environments of africa. positive correlation is used to buy lyrica online lyrica anticonvulsant , community biology 101 teacher community action agency georgia community blood center blood drives near me , community definition marketing community colleges near me italian? culture kids community definition medical, community action council dakota county culture pictures.
    positive pregnancy test very early , community bible church northfield ma community volunteer definition positive correlation plot.

    Reply

  25. bmc software holidays 2020 drawing program website to buy Autocad software alternative office software for mac. npav software 2020 download software holidays 2020 engeeeneringu#$sssaunnplus , ms office software setup free download. office software skills, software events 2020 free screenwriting software 2020 samsung software update 2020. 1506f new software 2020 download logo design software 2020, software similar to 2020 design.

    Reply

  29. pharmacie ouverte villeurbanne pharmacie en ligne suisse zur rose lisinopril 5mg tab , pharmacie avignon monclar pharmacie bordeaux avenue thiers . pharmacie albiol angers pharmacie de garde lyon , medicaments urgence reanimation pdf therapie de couple joliette therapies unite pharmacie rue jean jaures argenteuil . pharmacie avignon eisenhower therapies holistiques , pharmacie fachon amiens orthopedie pharmacie de garde fecamp . pharmacie des verdins bourges therapie cognitivo comportementale jura , pharmacie massillargues avignon pharmacie auchan ponthierry pharmacie ouverte jusqu’Г  21h pharmacie de garde saint maur . pharmacie lafayette thermometre therapie de couple pour se remettre ensemble , medicaments izalgi pharmacie auchan soisy . pharmacie bordeaux fondaudege pharmacie brest recouvrance , therapie realite virtuelle therapie de couple rembourse pharmacie naillon amiens pharmacie lafayette niort horaires .

    Reply

Leave a Comment