We Keep Coding

How to display progress of downloading a YouTube video using pytube?

I've been on multiple forums and I've read the docs. I still don't understand how to make this work.
How do I show the progress while the video is being downloaded? Do I need to provide parameters? I see a lot of people doing yt = YouTube(url, on_progress) without parenthesis or parameters so I'm very confused.
I don't know what file_handle should be, I also don't know where I can get the bytes_remaining.
Thank you in advance
def on_progress(stream, chunk, file_handle, bytes_remaining):
total_size = stream.filesize
bytes_downloaded = total_size - bytes_remaining
percentage_of_completion = bytes_downloaded / total_size * 100
print(percentage_of_completion)
def main():
chunk_size = 1024
url = "https://www.youtube.com/watch?v=GceNsojnMf0"
yt = YouTube(url)
video = yt.streams.get_highest_resolution()
yt.register_on_progress_callback(on_progress(video, chunk_size, 'D:\\Videos', video.filesize))
print(f"Fetching \"{video.title}\"..")
print(f"Fetching successful\n")
print(f"Information: \n"
f"File size: {round(video.filesize * 0.000001, 2)} MegaBytes\n"
f"Highest Resolution: {video.resolution}\n"
f"Author: {yt.author}")
print("Views: {:,}\n".format(yt.views))
print(f"Downloading \"{video.title}\"..")
video.download('D:\\Videos')

The method register_on_progress_callback() from the YouTube object just need a callback function itself not the result of the function. You also need to update the parameters of your function on_progress : only three parameters are used by the method register_on_progress_callback() (stream, chunk and bytes_remaining). You can update your code like that :
def on_progress(stream, chunk, bytes_remaining):
total_size = stream.filesize
bytes_downloaded = total_size - bytes_remaining
percentage_of_completion = bytes_downloaded / total_size * 100
print(percentage_of_completion)
def main():
chunk_size = 1024
url = "https://www.youtube.com/watch?v=GceNsojnMf0"
yt = YouTube(url)
video = yt.streams.get_highest_resolution()
yt.register_on_progress_callback(on_progress)
print(f"Fetching \"{video.title}\"..")
print(f"Fetching successful\n")
print(f"Information: \n"
f"File size: {round(video.filesize * 0.000001, 2)} MegaBytes\n"
f"Highest Resolution: {video.resolution}\n"
f"Author: {yt.author}")
print("Views: {:,}\n".format(yt.views))
print(f"Downloading \"{video.title}\"..")
video.download('D:\\Videos')
main()

How to count how many data has been downloaded in the last second? (FTP)

I want to know how many data has been downloaded in the last 1 second.
I don't have a code yet but I was wondering when I should start counting this 1 second and how to do it.
Should I start counting before retrbinary() or after? Or am I totally wrong?

First, there are ready-made implementations for transfer progress display, including the transfer speed.
For an example, the progressbar2 module. See Show FTP download progress in Python (ProgressBar).
The progressbar2 by default displays FileTransferSpeed widget, what is an average transfer speed since the download started.
Though note that speed displays usually do not show such speed. They display an average speed over last few seconds. That makes the value more informative. The progressbar2 has AdaptiveTransferSpeed widget for that. But it seems to be broken.
If you want to implement the calculation on your own, and are happy with the simple average transfer speed since the download started, it is easy:
from ftplib import FTP
import time
import sys
import datetime
ftp = FTP(host, user, passwd)
print("Downloading")
total_length = 0
start_time = datetime.datetime.now()
def write(data):
f.write(data)
global total_length
global start_time
total_length += sys.getsizeof(data)
elapsed = (datetime.datetime.now() - start_time)
speed = (total_length / elapsed.total_seconds())
print("\rElapsed: {0} Speed: {1:.2f} kB/s".format(str(elapsed), speed / 1024), end="")
f = open('file.dat', 'wb')
ftp.retrbinary("RETR /file.dat", write)
f.close()
print()
print("done")
It is a way more difficult to calculate the average speed in the last seconds. You have to remember the amount of data transferred at past moments. Stealing (and fixing) the code from AdaptiveTransferSpeed, you will get something like:
sample_times = []
sample_values = []
INTERVAL = datetime.timedelta(milliseconds=100)
last_update_time = None
samples=datetime.timedelta(seconds=2)
total_length = 0
def write(data):
f.write(data)
global total_length
total_length += sys.getsizeof(data)
elapsed = (datetime.datetime.now() - start_time)
if sample_times:
sample_time = sample_times[-1]
else:
sample_time = datetime.datetime.min
t = datetime.datetime.now()
if t - sample_time > INTERVAL:
# Add a sample but limit the size to `num_samples`
sample_times.append(t)
sample_values.append(total_length)
minimum_time = t - samples
minimum_value = sample_values[-1]
while (sample_times[2:] and
minimum_time > sample_times[1] and
minimum_value > sample_values[1]):
sample_times.pop(0)
sample_values.pop(0)
delta_time = sample_times[-1] - sample_times[0]
delta_value = sample_values[-1] - sample_values[0]
if delta_time:
speed = (delta_value / delta_time.total_seconds())
print("\rElapsed: {0} Speed: {1:.2f} kB/s".format(
str(elapsed), speed / 1024), end="")
ftp.retrbinary("RETR /medium.dat", write)

Python Threading: Multiline Progress Report

I'm writing this program that downloads multiple files simultaneously using threads and reports the progress for each file on the screen.
I'm using one different thread to download each file. That is no problem:
for file_url, filename in zip(file_urls, filenames):
th = Thread(target=download_file, args=(file_url, filename))
threads.append(th)
th.start()
for thread in threads:
thread.join()
The problem is I don't know anyways that I can print each file progress report on the screen that will look like this:
"Downloading 'example1.zip': 54366 bytes of 2240799 70.7%"
"Downloading 'example2.zip': 31712 bytes of 1924639 64.7%"
"Downloading 'example3.zip': 21712 bytes of 3224979 34.7%"
The following snippet is for a single line progress report:
def chunk_report(bytes_so_far, total_size, filename):
percent = float(bytes_so_far) / total_size
percent = round(percent * 100, 2)
print "Downloading '{0}': {1} of {2} {3:3.2g}% \r".format(filename,
bytes_so_far, total_size, percent),
and the output would look like this:
"Downloading 'example2.zip': 31712 bytes of 1924639 64.7%"
each time a thread call this function it updates the screen for the file that, the thread is downloading.
So, the question is how do I print multiline progress report like the one that I illustrated above in python?
Thanks in advance.

I would use Queue to report progress to a reporting thread:
Create a Queue
Spawn each downloading thread passing the Queue as an argument
Have each downloading put progress messages to the Queue
Spawn a reporting thread which reads progress messages from the Queue and updates the display
A simulated example:
import threading
import time
import random
import Queue
import sys
# a downloading thread
def worker(path, total, q):
size = 0
while size < total:
dt = random.randint(1,3)
time.sleep(dt)
ds = random.randint(1,5)
size = size + ds
if size > total: size = total
q.put(("update", path, total, size))
q.put(("done", path))
# the reporting thread
def reporter(q, nworkers):
status = {}
while nworkers > 0:
msg = q.get()
if msg[0] == "update":
path, total, size = msg[1:]
status[path] = (total, size)
# update the screen here
show_progress(status)
elif msg[0] == "done":
nworkers = nworkers - 1
print ""
def show_progress(status):
line = ""
for path in status:
(total, size) = status[path]
line = line + "%s: %3d/%d " % (path, size,total)
sys.stdout.write("\r"+line)
sys.stdout.flush()
def main():
q = Queue.Queue()
w1 = threading.Thread(target = worker, args = ("abc", 30, q) )
w2 = threading.Thread(target = worker, args = ("foobar", 25, q))
w3 = threading.Thread(target = worker, args = ("bazquux", 16, q))
r = threading.Thread(target = reporter, args = (q, 3))
for t in [w1,w2,w3,r]: t.start()
for t in [w1,w2,w3,r]: t.join()
main()

Live recognition with Python and Pocketsphinx

I have recently been working with pocket sphinx in python. I have successfully got the
example below to work recognising a recorded wav.
#!/usr/bin/env python
import sys,os
def decodeSpeech(hmmd,lmdir,dictp,wavfile):
"""
Decodes a speech file
"""
try:
import pocketsphinx as ps
import sphinxbase
except:
print """Pocket sphinx and sphixbase is not installed
in your system. Please install it with package manager.
"""
speechRec = ps.Decoder(hmm = hmmd, lm = lmdir, dict = dictp)
wavFile = file(wavfile,'rb')
wavFile.seek(44)
speechRec.decode_raw(wavFile)
result = speechRec.get_hyp()
return result[0]
if __name__ == "__main__":
hmdir = "/home/jaganadhg/Desktop/Docs_New/kgisl/model/hmm/wsj1"
lmd = "/home/jaganadhg/Desktop/Docs_New/kgisl/model/lm/wsj/wlist5o.3e-7.vp.tg.lm.DMP"
dictd = "/home/jaganadhg/Desktop/Docs_New/kgisl/model/lm/wsj/wlist5o.dic"
wavfile = "/home/jaganadhg/Desktop/Docs_New/kgisl/sa1.wav"
recognised = decodeSpeech(hmdir,lmd,dictd,wavfile)
print "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
print recognised
print "%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"
The problem is how can I do real time speech recognition from a microphone? In
a while loop with a if statement so that if a set word is recognised from the microphone
a function can be called?

The code for realtime recognition looks like this:
config = Decoder.default_config()
config.set_string('-hmm', path.join(MODELDIR, 'en-us/en-us'))
config.set_string('-lm', path.join(MODELDIR, 'en-us/en-us.lm.bin'))
config.set_string('-dict', path.join(MODELDIR, 'en-us/cmudict-en-us.dict'))
config.set_string('-logfn', '/dev/null')
decoder = Decoder(config)
import pyaudio
p = pyaudio.PyAudio()
stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True, frames_per_buffer=1024)
stream.start_stream()
in_speech_bf = False
decoder.start_utt()
while True:
buf = stream.read(1024)
if buf:
decoder.process_raw(buf, False, False)
if decoder.get_in_speech() != in_speech_bf:
in_speech_bf = decoder.get_in_speech()
if not in_speech_bf:
decoder.end_utt()
print 'Result:', decoder.hyp().hypstr
decoder.start_utt()
else:
break
decoder.end_utt()
You can also use gstreamer python bindings in pocketsphinx, check livedemo.py

Try this. Pocketsphinx is now a GStreamer plugin.

This is the code I see on the internet and I've modified a few things to really listen to the words very bad and slow
You can help me modify it for good. It is built on ubuntu 16.04 LTS
I do not know much about programming
Looking forward to help
# -*- encoding: utf-8 -*-
#!/usr/bin/env python
from pocketsphinx.pocketsphinx import *
from sphinxbase.sphinxbase import *
import os
import pyaudio
import wave
import audioop
from collections import deque
import time
import math;import Mic
"""
Written by Sophie Li, 2016
http://blog.justsophie.com/python-speech-to-text-with-pocketsphinx/
"""
class SpeechDetector:
def __init__(self):
# Microphone stream config.
self.CHUNK = 1024 # CHUNKS of bytes to read each time from mic
self.FORMAT = pyaudio.paInt16
self.CHANNELS = 1
self.RATE = 16000
self.SILENCE_LIMIT = 1 # Silence limit in seconds. The max ammount of seconds where
# only silence is recorded. When this time passes the
# recording finishes and the file is decoded
self.PREV_AUDIO = 0.5 # Previous audio (in seconds) to prepend. When noise
# is detected, how much of previously recorded audio is
# prepended. This helps to prevent chopping the beginning
# of the phrase.
self.THRESHOLD = 4500
self.num_phrases = -1
# These will need to be modified according to where the pocketsphinx folder is
MODELDIR = "/home/l/Desktop/pocketsphinx/model/en-us"
# Create a decoder with certain model
config = Decoder.default_config()
config.set_string('-hmm', os.path.join(MODELDIR, '/home/l/Desktop/pocketsphinx/model/en-us/en-us/'))
config.set_string('-lm', os.path.join(MODELDIR, '/home/l/Desktop/pocketsphinx/model/en-us/en-us.lm.bin'))
config.set_string('-dict', os.path.join(MODELDIR, '/home/l/Desktop/pocketsphinx/model/en-us/cmudict-en-us.dict'))
config.set_string('-keyphrase', 'no one')
config.set_float('-kws_threshold', 1e+20)
# Creaders decoder object for streaming data.
self.decoder = Decoder(config)
def setup_mic(self, num_samples=50):
""" Gets average audio intensity of your mic sound. You can use it to get
average intensities while you're talking and/or silent. The average
is the avg of the .2 of the largest intensities recorded.
"""
#print "Getting intensity values from mic."
p = pyaudio.PyAudio()
stream = p.open(format=self.FORMAT,
channels=self.CHANNELS,
rate=self.RATE,
input=True,
frames_per_buffer=self.CHUNK)
values = [math.sqrt(abs(audioop.avg(stream.read(self.CHUNK), 4)))
for x in range(num_samples)]
values = sorted(values, reverse=True)
r = sum(values[:int(num_samples * 0.2)]) / int(num_samples * 0.2)
#print " Finished "
#print " Average audio intensity is ", r
stream.close()
p.terminate()
if r < 3000:
self.THRESHOLD = 3500
else:
self.THRESHOLD = r + 100
def save_speech(self, data, p):
"""
Saves mic data to temporary WAV file. Returns filename of saved
file
"""
filename = 'output_'+str(int(time.time()))
# writes data to WAV file
data = ''.join(data)
wf = wave.open(filename + '.wav', 'wb')
wf.setnchannels(1)
wf.setsampwidth(p.get_sample_size(pyaudio.paInt16))
wf.setframerate(16000) # TODO make this value a function parameter?
wf.writeframes(data)
wf.close()
return filename + '.wav'
def decode_phrase(self, wav_file):
self.decoder.start_utt()
stream = open(wav_file, "rb")
while True:
buf = stream.read(1024)
if buf:
self.decoder.process_raw(buf, False, False)
else:
break
self.decoder.end_utt()
words = []
[words.append(seg.word) for seg in self.decoder.seg()]
return words
def run(self):
"""
Listens to Microphone, extracts phrases from it and calls pocketsphinx
to decode the sound
"""
self.setup_mic()
#Open stream
p = pyaudio.PyAudio()
stream = p.open(format=self.FORMAT,
channels=self.CHANNELS,
rate=self.RATE,
input=True,
frames_per_buffer=self.CHUNK)
audio2send = []
cur_data = '' # current chunk of audio data
rel = self.RATE/self.CHUNK
slid_win = deque(maxlen=self.SILENCE_LIMIT * rel)
#Prepend audio from 0.5 seconds before noise was detected
prev_audio = deque(maxlen=self.PREV_AUDIO * rel)
started = False
while True:
cur_data = stream.read(self.CHUNK)
slid_win.append(math.sqrt(abs(audioop.avg(cur_data, 4))))
if sum([x > self.THRESHOLD for x in slid_win]) > 0:
if started == False:
print "Bắt đầu ghi âm"
started = True
audio2send.append(cur_data)
elif started:
print "Hoàn thành ghi âm"
filename = self.save_speech(list(prev_audio) + audio2send, p)
r = self.decode_phrase(filename)
print "RESULT: ", r
# hot word for me " no one" if r.count('one') and r.count("no") > 0 the end programs
if r.count("one") > 0 and r.count("no") > 0:
Mic.playaudiofromAudio().play("/home/l/Desktop/PROJECT/Audio/beep_hi.wav")
os.remove(filename)
return
# Removes temp audio file
os.remove(filename)
# Reset all
started = False
slid_win = deque(maxlen=self.SILENCE_LIMIT * rel)
prev_audio = deque(maxlen= 0.5 * rel)
audio2send = []
print "Chế độ nghe ..."
else:
prev_audio.append(cur_data)
print "* Hoàn thành nghe"
stream.close()
p.terminate()

Download progressbar for Python 3

I need a progress to show during file download for Python 3.
I have seen a few topics on Stackoverflow, but considering that I'm a noob at programming and nobody posted a complete example, just fractions of it, or the one that I can make work on Python 3, none are good for me...
additional info:
ok, so i have this:
from urllib.request import urlopen
import configparser
#checks for files which need to be downloaded
print(' Downloading...')
file = urlopen(file_url)
#progress bar here
output = open('downloaded_file.py','wb')
output.write(file.read())
output.close()
os.system('downloaded_file.py')
script is run through python command line

There is urlretrieve() that downloads an url to a file and allows to specify a reporthook callback to report progess:
#!/usr/bin/env python3
import sys
from urllib.request import urlretrieve
def reporthook(blocknum, blocksize, totalsize):
readsofar = blocknum * blocksize
if totalsize > 0:
percent = readsofar * 1e2 / totalsize
s = "\r%5.1f%% %*d / %d" % (
percent, len(str(totalsize)), readsofar, totalsize)
sys.stderr.write(s)
if readsofar >= totalsize: # near the end
sys.stderr.write("\n")
else: # total size is unknown
sys.stderr.write("read %d\n" % (readsofar,))
urlretrieve(url, 'downloaded_file.py', reporthook)
Here's a GUI progress bar:
import sys
from threading import Event, Thread
from tkinter import Tk, ttk
from urllib.request import urlretrieve
def download(url, filename):
root = progressbar = quit_id = None
ready = Event()
def reporthook(blocknum, blocksize, totalsize):
nonlocal quit_id
if blocknum == 0: # started downloading
def guiloop():
nonlocal root, progressbar
root = Tk()
root.withdraw() # hide
progressbar = ttk.Progressbar(root, length=400)
progressbar.grid()
# show progress bar if the download takes more than .5 seconds
root.after(500, root.deiconify)
ready.set() # gui is ready
root.mainloop()
Thread(target=guiloop).start()
ready.wait(1) # wait until gui is ready
percent = blocknum * blocksize * 1e2 / totalsize # assume totalsize > 0
if quit_id is None:
root.title('%%%.0f %s' % (percent, filename,))
progressbar['value'] = percent # report progress
if percent >= 100: # finishing download
quit_id = root.after(0, root.destroy) # close GUI
return urlretrieve(url, filename, reporthook)
download(url, 'downloaded_file.py')
On Python 3.3 urlretrieve() has different reporthook interface (see issue 16409). To workaround it, you could access the previous interface via FancyURLopener:
from urllib.request import FancyURLopener
urlretrieve = FancyURLopener().retrieve
To update the progress bar within the same thread, you could inline urlretrieve() code:
from tkinter import Tk, ttk
from urllib.request import urlopen
def download2(url, filename):
response = urlopen(url)
totalsize = int(response.headers['Content-Length']) # assume correct header
outputfile = open(filename, 'wb')
def download_chunk(readsofar=0, chunksize=1 << 13):
# report progress
percent = readsofar * 1e2 / totalsize # assume totalsize > 0
root.title('%%%.0f %s' % (percent, filename,))
progressbar['value'] = percent
# download chunk
data = response.read(chunksize)
if not data: # finished downloading
outputfile.close()
root.destroy() # close GUI
else:
outputfile.write(data) # save to filename
# schedule to download the next chunk
root.after(0, download_chunk, readsofar + len(data), chunksize)
# setup GUI to show progress
root = Tk()
root.withdraw() # hide
progressbar = ttk.Progressbar(root, length=400)
progressbar.grid()
# show progress bar if the download takes more than .5 seconds
root.after(500, root.deiconify)
root.after(0, download_chunk)
root.mainloop()
download2(url, 'downloaded_file.py')

I think this piece of code can help you. I'm not quite sure it's exactly what you want. At least it should give you something to work on.
import tkinter
from tkinter import ttk
from urllib.request import urlopen
def download(event):
file = urlopen('http://www.python.org/')
output = open('downloaded_file.txt', 'wb')
lines= file.readlines()
i = len(lines)
for line in lines:
output.write(line)
pbar.step(100/i)
output.close()
file.close()
root = tkinter.Tk()
root.title('Download bar')
pbar = ttk.Progressbar(root, length=300)
pbar.pack(padx=5, pady=5)
btn = tkinter.Button(root, text="Download")
# bind to left mouse button click
btn.bind("<Button-1>", download)
btn.pack(pady=10)
root.mainloop()
This works, I've tried it.