The use case: I'm developing a web app to help students learn to read. The student is recorded while reading a text on a web app. The signal is sent by segment of 200ms to the backend and analysed before the student finishes reading to give live feedback during the reading. The server will send feedback after each segment analysis.
On the web app the code looks like this:
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(stream => {
const mediaRecorder = new MediaRecorder(stream)
mediaRecorder.start(200)
mediaRecorder.ondataavailable = event => {
socket.emit('my_event', Blob([event.data]))
}
})
On chrome, the media type produced is webm. I'm wondering how to handle the data on the backend so that I can analyse the media with numpy before the end of the recording.
For now I couldn't find a better way than something like:
from pydub import AudioSegment
def blobToSignal(blob, is_first_sequence):
webm_header = b'\x1aE...'
fp = tempfile.NamedTemporaryFile()
fp.write(blob) if is_first_sequence else fp.write(webm_header + blob)
fp.seek(0)
samples = AudioSegment.from_file(fp.name, 'webm').get_array_of_samples()
fp.close()
return samples # this is almost a numpy array (analyzable)
I tried to change the front to return a Float32Array instead of a webm:
navigator.mediaDevices.getUserMedia({ audio: true, video: false })
.then(stream => {
audio_context = new AudioContext()
var audioInput = audio_context.createMediaStreamSource(stream)
var recorder = audio_context.createScriptProcessor(8096, 1, 1)
recorder.onaudioprocess = event => {
socket.emit(
'my_event',
Array.from(event.inputBuffer.getChannelData(0))
)
}
audioInput.connect(recorder)
recorder.connect(audio_context.destination)
So that the backend can use the raw signal but this method requires a too high bandwidth (~1Mb/s).
So my questions are:
am I doing something wrong here?
is there a Python librairy to decode a webm coming from a Buffer? (or something similar? I'm not so familiar with Python...)
how would you handle this use case?
Thanks for your help!
Related
I have an HTML page that lets you upload a picture and displays it (very simple). But how do I send that particular loaded image into my flask app so that I could, say, perform some image processing on it using python's openCV or PIL packages?
Disclaimer
I've done similar thing recently, It may not be a 100% solution for you, but using parts of this code will propably solve all of your problems.
Flask Code
This part gets all files uploaded from browser and saves them with their respected extensions.
if request.method=="PUT":
for f in request.files.getlist("Files"): #<---- note 1
f.save(os.path.join(Path_To_Folder, f.filename))
return jsonify({"result": "res"}), 200 #status code is only thing important here
HTML
This just allows you to select file(s) from your device
<input type="file" id="file_loader" multiple/>
JavaScript
This code
const file_loader = document.getElementById("file_loader");
file_loader.onchange = async function(e){
alert("Upload started")
let sending = new FormData(); //creates form data object
//this for loop adds all files you selected to a form object
for (let i = 0; i < file_loader.files.length; i++) {
sending.append("Files", file_loader.files[i]); //<---- note 1
}
//this part just sends all the files to server
const podaci = await fetch(url_filesistem, {
method: "PUT",
body: sending,
})
.then((response) => response.json())
.then((pod) => {return pod;});
//removing all selected files from an input, so every time you want to select files
//its just those files, not ones previously selected
while (file_loader.length > 0) {
file_loader.pop();
}
alert("Finnished uploading")
}
Note 1
String "Files" mentioned in both lines needs to be the same in order for this method to work.
Advice
First save all the files on the server and then do the processing. I don't know what "f" object from for loop in python contains, therefore I don't know whether you can process them immediately.
Feel free to ask me anything regarding my code!
Best regards!
I'm looking to deploy a small data collection app that allows users to record themselves saying some phrases.
Building this with streamlit.
I can get it to work locally, but can't seem to find a solution that works in the Streamlit-Shearing service.
Any ideas?
I do not fully understand your question I think this might help.
import streamlit as st
from bokeh.models.widgets import Button
from bokeh.models import CustomJS
from streamlit_bokeh_events import streamlit_bokeh_events
stt_button = Button(label="Speak", width=100)
stt_button.js_on_event("button_click", CustomJS(code="""
var recognition = new webkitSpeechRecognition();
recognition.continuous = true;
recognition.interimResults = true;
recognition.onresult = function (e) {
var value = "";
for (var i = e.resultIndex; i < e.results.length; ++i) {
if (e.results[i].isFinal) {
value += e.results[i][0].transcript;
}
}
if ( value != "") {
document.dispatchEvent(new CustomEvent("GET_TEXT", {detail: value}));
}
}
recognition.start();
"""))
result = streamlit_bokeh_events(
stt_button,
events="GET_TEXT",
key="listen",
refresh_on_update=False,
override_height=75,
debounce_time=0)
if result:
if "GET_TEXT" in result:
st.write(result.get("GET_TEXT"))`
I have been working on implementing a custom component that enables recording audio from the client’s microphone in apps that are deployed on the web (Streamlit-Audio-Recorder). This version of the component supports downloading the recorded audio but not directly returning audio data to streamlit/Python.
I want to stream the audio of my microphone (that is being recorded via pyaudio) via Flask to any client that connects.
This is where the audio comes from:
def getSound(self):
# Current chunk of audio data
data = self.stream.read(self.CHUNK)
self.frames.append(data)
wave = self.save(list(self.frames))
return data
Here's my flask-code:
#app.route('/audiofeed')
def audiofeed():
def gen(microphone):
while True:
sound = microphone.getSound()
#with open('tmp.wav', 'rb') as myfile:
# yield myfile.read()
yield sound
return Response(stream_with_context(gen(Microphone())))
And this is the client:
<audio controls>
<source src="{{ url_for('audiofeed') }}" type="audio/x-wav;codec=pcm">
Your browser does not support the audio element.
</audio>
It does work sometimes, but most of the times I'm getting "[Errno 32] Broken pipe"
When uncommenting that with open("tmp.wav")-part (the self.save() optionally takes all previous frames and saves them in tmp.wav), I kind of get a stream, but all that comes out of the speakers is a "clicking"-noise.
I'm open for any suggestions. How do I get the input of my microphone live-streamed (no pre-recording!) to a webbrowser?
Thanks!
Try This its worked for me. shell cmd "cat" is working perfect see the code
iam using FLASK
import subprocess
import os
import inspect
from flask import Flask
from flask import Response
#app.route('/playaudio')
def playaudio():
sendFileName=""
def generate():
# get_list_all_files_name this function gives all internal files inside the folder
filesAudios=get_list_all_files_name(currentDir+"/streamingAudios/1")
# audioPath is audio file path in system
for audioPath in filesAudios:
data=subprocess.check_output(['cat',audioPath])
yield data
return Response(generate(), mimetype='audio/mp3')
This question was asked long time ago, but since I spent entire day to figure out how to implement the same, I want to give the answer. Maybe it will be helpful for somebody.
"[Errno 32] Broken pipe" error comes from the fact that client can not play audio and closes this stream.
Audio can not be played due to absence of the header in the data stream. You can easily create the header using genHeader(sampleRate, bitsPerSample, channels, samples) function from the code here . This header has to be attached at least to the first chunck of sent data ( chunck=header+data ). Pay attention, that audio can be played ONLY untill client reaches file size in download that you have to specify in the header. So, workaround would be to set in the header some big files size, e.g. 2Gb.
Instead of datasize = len(samples) * channels * bitsPerSample in the header function write datasize = 2000*10**6.
def gen_audio():
CHUNK = 512
sampleRate = 44100
bitsPerSample = 16
channels = 2
wav_header = genHeader(sampleRate, bitsPerSample, channels)
audio = AudioRead()
data = audio.get_audio_chunck()
chunck = wav_header + data
while True:
yield (chunck)
data = audio.get_audio_chunck()
chunck = data
After lots research and tinkering I finally found the solution.
Basically it came down to serving pyaudio.paFloat32 audio data through WebSockets using Flask's SocketIO implementation and receiving/playing the data in JavaScript using HTML5's AudioContext.
As this is requires quite some code, I think it would not be a good idea to post it all here. Instead, feel free to check out the project I'm using it in: simpleCam
The relevant code is in:
- noise_detector.py (recording)
- server.py (WebSocket transfer)
- static/js/player.js (receiving/playing)
Thanks everyone for the support!
I am attempting to implement a flask application for uploading files. This file could be very large. For example, almost 2G in size.
I have finished the server side process function like this:
#app.route("/upload/<filename>", methods=["POST", "PUT"])
def upload_process(filename):
filename = secure_filename(filename)
fileFullPath = os.path.join(application.config['UPLOAD_FOLDER'], filename)
with open(fileFullPath, "wb") as f:
chunk_size = 4096
while True:
chunk = flask.request.stream.read(chunk_size)
if len(chunk) == 0:
return
f.write(chunk)
return jsonify({'filename': filename})
As for browser side, I should give users a from to submit the file. One file at a time. Show progressbar to indicate the uploading process.
But I have no idea about the browser side code. How can I use javascript code to start the uploading and show it status?
This will be a difficult task for your to figure out on your own. I would suggest a plugin like https://blueimp.github.io/jQuery-File-Upload/
You can see from this projects source code they use a method name which essentially looks at how large the file is and how much data has been transferred thus far and how much remains to show a percentage complete div.
code example from this project
progressall: function (e, data) {
var $this = $(this);
$this.find('.fileupload-progress')
.find('.progress').progressbar(
'option',
'value',
parseInt(data.loaded / data.total * 100, 10)
).end()
.find('.progress-extended').each(function () {
$(this).html(
($this.data('blueimp-fileupload') ||
$this.data('fileupload'))
._renderExtendedProgress(data)
);
});
}
https://github.com/blueimp/jQuery-File-Upload/blob/master/js/jquery.fileupload-jquery-ui.js
So if you do want to come up with your own solution, I would suggest you start by building a UI div rectangle which has a dynamic width which updates according to your percentage calculation based upon the file upload size and data uploaded... or just go with an already established solution.
Im trying to read in image file from a server , with the code below . It keeps going into the exception. I know the correct number of bytes are being sent as I print them out when received. Im sending the image file from python like so
#open the image file and read it into an object
imgfile = open (marked_image, 'rb')
obj = imgfile.read()
#get the no of bytes in the image and convert it to a string
bytes = str(len(obj))
#send the number of bytes
self.conn.send( bytes + '\n')
if self.conn.sendall(obj) == None:
imgfile.flush()
imgfile.close()
print 'Image Sent'
else:
print 'Error'
Here is the android part , this is where I'm having the problem. Any suggestions on the best way to go about receiving the image and writing it to a file ?
//read the number of bytes in the image
String noOfBytes = in.readLine();
Toast.makeText(this, noOfBytes, 5).show();
byte bytes [] = new byte [Integer.parseInt(noOfBytes)];
//create a file to store the retrieved image
File photo = new File(Environment.getExternalStorageDirectory(), "PostKey.jpg");
DataInputStream dis = new DataInputStream(link.getInputStream());
try{
os =new FileOutputStream(photo);
byte buf[]=new byte[1024];
int len;
while((len=dis.read(buf))>0)
os.write(buf,0,len);
Toast.makeText(this, "File recieved", 5).show();
os.close();
dis.close();
}catch(IOException e){
Toast.makeText(this, "An IO Error Occured", 5).show();
}
EDIT: I still cant seem to get it working. I have been at it since and the result of all my efforts have either resulted in a file that is not the full size or else the app crashing. I know the file is not corrupt before sending server side. As far as I can tell its definitely sending too as the send all method in python sends all or throws an exception in the event of an error and so far it has never thrown an exception. So the client side is messed up . I have to send the file from the server so I cant use the suggestion suggested by Brian .
The best way to get a bitmap from a server is to execute the following.
HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet("http://yoururl");
HttpResponse response = client.execute(get);
InputStream is = response.getEntity().getContent();
Bitmap image = BitmapFactory.decodeStream(is);
That will give you your bitmap, to save it to a file do something like the following.
FileOutputStream fos = new FileOutputStream("yourfilename");
image.compress(CompressFormat.PNG, 1, fos);
fos.close();
You can also combine the two and just write straight to disk
HttpClient client = new DefaultHttpClient();
HttpGet get = new HttpGet("http://yoururl");
HttpResponse response = client.execute(get);
InputStream is = response.getEntity().getContent();
FileOutputStream fos = new FileOutputStream("yourfilename");
byte[] buffer = new byte[256];
int read = is.read(buffer);
while(read != -1){
fos.write(buffer, 0, read);
read = is.read(buffer);
}
fos.close();
is.close();
Hope this helps;
I'm not sure I understand your code. You are calling dis.readFully(bytes); to write the content of dis into your byte array. But then you don't do anything with the array, and then try to write the content of dis through a buffer into your FileOutputStream.
Try commenting out the line dis.readFully(bytes);.
As a side note, I would write to the log rather than popping up a toast for things like the number of bytes or when an exception occurs:
...
} catch (IOException e) {
Log.e("MyTagName","Exception caught " + e.toString());
e.printStackTrace();
}
You could look at these links for examples of writing a file to the SD card:
Android download binary file problems
Android write to sd card folder
I solved it with the help of a Ubuntu Forums member. It was the reading of the bytes that was the problem . It was cutting some of the bytes from the image. The solution was to just send the image whole and remove the sending of the bytes from the equation all together