I am trying to make a simple function to download file in python
The code is something like
def download(url , dest):
urllib.urlretrieve(url, dest)
My issue is that if I want to cancel the download process in the middle of downloading how do I approach???
This function runs in the background of app and is triggered by a button. Now I am trying to trigger it off with another button.
The platform is XBMC.
A simple class to do the same as your download function:
import urllib
import threading
class Downloader:
def __init__(self):
self.stop_down = False
self.thread = None
def download(self, url, destination):
self.thread = threading.Thread(target=self.__down, args=(url, destination))
self.thread.start()
def __down(self, url, dest):
_continue = True
handler = urllib.urlopen(url)
self.fp = open(dest, "w")
while not self.stop_down and _continue:
data = handler.read(4096)
self.fp.write(data)
_continue = data
handler.close()
self.fp.close()
def cancel(self):
self.stop_down = True
So, when someone clicks the "Cancel" button you have to call the cancel() method.
Please note that this will not remove the partially downloaded file if you cancel it, but that should not be hard to achieve using os.unlink(), for example.
The following example script shows how to use it, starting the download of a ~20Mb file and cancelling it after 5 seconds:
import time
if __name__ == "__main__":
url = "http://ftp.postgresql.org/pub/source/v9.2.3/postgresql-9.2.3.tar.gz"
down = Downloader()
down.download(url, "file")
print "Download started..."
time.sleep(5)
down.cancel()
print "Download canceled"
If you are canceling by pressing CTRL+C, then you can use this built in exception and proceed with what you think the best move should be.
In this case, if I cancel in the middle of a download, I simply want that partial file to be deleted:
def download(url , dest):
try:
urllib.urlretrieve(url, dest)
except KeyboardInterrupt:
if os.path.exists(dest):
os.remove(dest)
except Exception, e:
raise
Related
I am having an application that listens to changes in a certain folder with watchdog
inside an infinite loop
class MyObserver:
def __init__(self, observer, handler):
self._event_handler = handler
self._event_observer = observer
def watch(self, path):
self.start(path)
try:
while True:
time.sleep(60)
except KeyboardInterrupt:
self.stop()
except Exception as err:
# log
def start(self, path):
self._schedule(path)
self._event_observer.start()
def stop(self):
self._event_observer.stop()
self._event_observer.join()
def _schedule(self, path):
self._event_observer.schedule(self._event_handler, path, recursive=True)
this is my eventHandler
class ImagesEventHandler(RegexMatchingEventHandler):
IMAGES_REGEX = [r"(.*).(jpe?g|bmp)$"]
def __init__(self):
super().__init__(self.IMAGES_REGEX)
def on_created(self, event):
return self._process(event)
#staticmethod
def _process(event):
# log
return event.src_path # i want to test this!
I want to be able to start the application, insert an image manually to the folder
and be able to test the results of capturing the event of creating the image
and also finishing the infinite loop,
I tried using interruptingcow to cut the loop but it's not working on Windows,
and also tried to mock the time.sleep but it didnt work as well
#pytest.mark.integration_test
class TestLoop(TestCase):
#patch("time.sleep", side_effect=InterruptedError)
def test_collect_images(self, mocked_sleep):
try:
main.run()
except InterruptedError:
print("Test passed")
I see this test separated into 3 parts:
starting the loop
inserting images to my folder
interrupting the loop
validating the log messages or the return value of _process somehow
how would you test it?
I have a function that downloads videos from specific URLs and I launch this function through a thread to avoid GUI freezing, but I want a function to stop or pause the download. How to do this?
Here is the code:
def download_videos(self):
ydl1 = youtube_dl.YoutubeDL(self.get_opts())
if self.get_Urls().__len__() > 0:
ydl1.download(self.get_Urls())
def downloadVideoThrd(self):
self.t1 = threading.Thread(target=self.download_videos())
self.t1.start()
You can use these two options. Use multiprocessing library instead threading or raise SystemExit exception in the subthread
Note: When i tested youtube-dl, it is resume downloading from where it is left. That's why when we start to download same url, youtube-dl will resume, but downloaded file need to be in the filesystem
Here is the first option, in this solution, we not use thread, we use subprocess because we can send any signal to the subprocess and handle this signal whatever we want.
import multiprocessing
import time
import ctypes,signal,os
import youtube_dl
class Stop_Download(Exception): # This is our own special Exception class
pass
def usr1_handler(signum,frame): # When signal comes, run this func
raise Stop_Download
def download_videos(resume_event):
signal.signal(signal.SIGUSR1,usr1_handler)
def download(link):
ydl1 = youtube_dl.YoutubeDL()
ydl1.download([link])
try:
download(link)
except Stop_Download: #catch this special exception and do what ever you want
print("Stop_Download exception")
resume_event.wait() # wait for event to resume or kill this process by sys.exit()
print("resuming")
download(link)
def downloadVideoThrd():
resume_event=multiprocessing.Event()
p1 = multiprocessing.Process(target=download_videos,args=(resume_event,))
p1.start()
print("mp start")
time.sleep(5)
os.kill(p1.pid,signal.SIGUSR1) # pause the download
time.sleep(5)
down_event.set() #resume downlaod
time.sleep(5)
os.kill(p1.pid,signal.SIGUSR2) # stop the download
downloadVideoThrd()
Here is the second solution, you can also check this for more detail about killing the thread. We will raise a SystemExit exception in the subthread by main thread. We can both stop or pause thread. To pause thread you can use threading.Event() class. To stop the thread(not process) you can use sys.exit()
import ctypes,time, threading
import youtube_dl,
def download_videos(self):
try:
ydl1 = youtube_dl.YoutubeDL(self.get_opts())
if self.get_Urls().__len__() > 0:
ydl1.download(self.get_Urls())
except SystemExit:
print("stopped")
#do whatever here
def downloadVideoThrd(self):
self.t1 = threading.Thread(target=self.download_videos)
self.t1.start()
time.sleep(4)
"""
raise SystemExit exception in self.t1 thread
"""
ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(self.t1.ident),
ctypes.py_object(SystemExit))
I have below code to download a file inside a loop,
import wget
try:
wget.download(url)
except:
pass
But if the Internet goes down, it doesn't return!
So my whole loop is stuck.
I want to repeat the same download if internet goes down. So I wanna know does any error happen.
How can i mitigate this?
One simple solution is to move your download code to a thread and make it a separate process which can be interrupted.
You can use python Thread and Timer module to achieve it.
from threading import Thread, Timer
from functools import partial
import time
import urllib
def check_connectivity(t):
try:
urllib.request.urlopen("http://google.com", timeout=2)
except Exception as e:
t._Thread__stop()
class Download(Thread):
def run(self):
print("Trying to download file....")
con = partial(check_connectivity, self)
while True:
t = Timer(5, con) # Checks the connectivity every 5 second or less.
t.start()
# your download code....
def main():
down = Download()
down.start()
down.join()
You code move your main download loop inside the thread's run method. And start a timer inside which listens for the network connectivity.
I'm working on an application to download the code of a web page and captures the links.
It works, but if I connect the program to a GUI, it locks the corresponding button until the download is completed.
If I trigger the download via a separate thread, to avoid the button lock, it just freezes and does not complete execution.
Is this normal? Or am I missing something?
Below goes the snippet of code. If I call grab() from a separate thread, nothing happens, neither errors.
The function update_observers() only notifies the observers, not doing else.
The observer is the responsible by making any changes, in this case, redraw the GUI.
def grab(self, url):
try:
self._status = 'Downloading page.'
self.update_observers()
inpu = urllib2.urlopen(url)
except URLError, e:
self._status = 'Error: '+ e.reason
self.update_observers()
return None
resp = []
self._status = 'Parsing links'
self.update_observers()
for line in inpu.readlines():
for reg in self._regexes:
links = reg.findall(line)
for link in links:
resp.append(link)
self._status = 'Ready.'
self.update_observers()
return resp
This code is called here:
def grab(self, widget):
t = Thread(target=self.work)
t.setDaemon(True)
t.start()
def work(self):
print "Working"
self.links = None
self.links = self.grabber.grab(self.txtLink.get_text())
for link in self.links:
self.store.append([link])
print "Ok."
If I move the code from work() to grab, removing the threading stuff, it's all ok.
I just called gtk.gdk.threads_init() before gtk.main() and everything worked perfectly without any changes.
I'm working with threads and I need to download a website with a thread. I also have a thread that sends the petition to the site but doesn't wait for an answer.
The one that doesn't wait is this:
class peticion(Thread):
def __init__(self, url):
Thread.__init__(self)
self.url = url
def run(self):
f = urllib.urlopen(self.url)
f.close()
This one works correctly, however the one that has to wait for the response takes something like a random time to complete, from 5 seconds to 2 minutes, or it may never finish. This is the class:
class playerConn(Thread):
def __init__(self, ev):
Thread.__init__(self)
self.ev = ev
def run(self):
try:
params = urllib.urlencode('''params go here''')
f = urllib.urlopen('''site goes here''')
resp = f.read()
f.close()
finally:
# do something with the response
Whether or not I use the try...finally statement it doesn't work, the code after the urlopen function won't get to execute.
What can I do?
It appear to just be a problem with the URL, the code is fine and it does not appear to do something wrong.
I bet you have some type of problem from the website, maybe a 404 or similar.
Try opening something in localhost, just to test.