I am not sure if I am doing something wrong. I am using progressbar to show how long a task is taking. This is the code I have wrapped around a to_excel command:
dfPub = pd.DataFrame(aPub)
if dfPub.empty:
print("There are no Publications")
else:
with progressbar.ProgressBar(max_value=10) as bar:
for i in range(10):
dfPub.to_excel(writer, 'Publications', columns=cols, index=False)
time.sleep(0.1)
bar.update(i)
It is working, but when testing with and without there is a massive difference in time it is taking to run i.e. without the progressbar, it takes about 2-3 seconds, and with it is taking around 15 seconds.
Am I implementing it incorrectly?
use multi-threading method, with threading module, put your progressbar on a new thread and test it again, you can read more on : https://pymotw.com/2/threading/
Related
I have the following code
def leftdoor():
press('a')
pyautogui.sleep(1)
press('a')
def rightdoor():
press('d')
pyautogui.sleep(1)
press('d')
leftdoor()
rightdoor()
and when I run the code what happens is the letter A is pressed and 1 second is waited and then its pressed again. Then the same happens for the D key. However is there a way for me to be able to press them both down and express that in code by calling both functions and not having to wait for the .sleep of the previous function?
There are two ways to run your code concurrently:
Combine the functions (might not be possible for large functions)
In the case of your code, it would look like this:
def door():
press('a')
press('d')
sleep(1)
press('a')
press('d')
door()
If this isn't what you're looking for, use threading.
Theading
Here is a link to a tutorial on the module, and the code is below.
from threading import Thread # Module import
rdt = Thread(target=rightdoor) # Create two Thread objects
ldt = Thread(target=leftdoor)
rdt.start() # start and join the objects
ldt.start()
rdt.join()
ldt.join()
print("Finished execution") # done!
Note that using this does not absolutely guarantee that a and d will be pressed at the same time (I got a ~10 millisecond delay at max, and it might have been from the program I used to time it), but it should work for all purposes.
I write a lot of code that relies on precise periodic method calls. I've been using Python's futures library to submit calls onto the runtime's thread pool and sleeping between calls in a loop:
executor = ThreadPoolExecutor(max_workers=cpu_count())
def remote_call():
# make a synchronous bunch of HTTP requests
def loop():
while True:
# do work here
executor.submit(remote_call)
time.sleep(60*5)
However, I've noticed that this implementation introduces some drift after a long duration of running (e.g. I've run this code for about 10 hours and noticed about 7 seconds of drift). For my work I need this to run on the exact second, and millisecond would be even better. Some folks have pointed me to asyncio ("Fire and forget" python async/await), but I have not been able to get this working in Python 2.7.
I'm not looking for a hack. What I really want is something akin to Go's time.Tick or Netty's HashedWheelTimer.
Nothing like that comes with Python. You'd need to manually adjust your sleep times to account for time spent working.
You could fold that into an iterator, much like the channel of Go's time.Tick:
import itertools
import time
import timeit
def tick(interval, initial_wait=False):
# time.perf_counter would probably be more appropriate on Python 3
start = timeit.default_timer()
if not initial_wait:
# yield immediately instead of sleeping
yield
for i in itertools.count(1):
time.sleep(start + i*interval - timeit.default_timer())
yield
for _ in tick(300):
# Will execute every 5 minutes, accounting for time spent in the loop body.
do_stuff()
Note that the above ticker starts ticking when you start iterating, rather than when you call tick, which matters if you try to start a ticker and save it for later. Also, it doesn't send the time, and it won't drop ticks if the receiver is slow. You can adjust all that on your own if you want.
There are numerous existing questions regarding the display of progress bars in the terminal while a Python script executes, but every one of them is based on a loop where you perform an operation and then update the progress graphic.
Unfortunately, the function whose progress I want to show--or at least a spinner object to show that it's working--is a black-box that I can't (at least really, really shouldn't) alter. Essentially, what I want to do is:
#pseudocode input
print('Loading')
spinner.begin()
blackbox() #a few thousand operations happen in here
spinner.end()
print('Finished')
#pseudocode output
Loading.
Loading..
Loading...
Loading.
Loading..
Loading...
Finished
Although ideally that would be an animation of the ellipsis instead of printing multiple lines. Before I can even start building silly ascii animations though, there's the main hurdle:
Is there a way to run spinner and blackbox() at the same time? Alternately, is there a hack to pause blackbox(), regardless of its content, every few hundred milliseconds, update the spinner graphic, and then resume where it left off?
I've tried this with the progress module but had no luck... I couldn't even get the example code to work, it just hung up after I started iterating until I Ctrl+C'd out.
I like using alive_progress for this.
from typing import ContextManager, Optional
from alive_progress import alive_bar
def spinner(title: Optional[str] = None) -> ContextManager:
"""
Context manager to display a spinner while a long-running process is running.
Usage:
with spinner("Fetching data..."):
fetch_data()
Args:
title: The title of the spinner. If None, no title will be displayed.
"""
return alive_bar(monitor=None, stats=None, title=title)
To install: pip install alive-progress
Threads is probably the easiest way to make this work. Here is a vastly simplified version that should get the point across. I wasn't sure whether you actually have the spinner function or not, so I made my own.
import threading
import time
def blackbox():
time.sleep(10)
thread = threading.Thread(target=blackbox)
thread.start()
eli_count = 0
while thread.is_alive():
print('Loading', '.'*(eli_count+1), ' '*(2-eli_count), end='\r')
eli_count = (eli_count + 1) % 3
time.sleep(0.1)
thread.join()
print('Done ')
So, while blackbox runs, the loading message is updated periodically. Once it finishes, the thread is joined and the loading message is replaced with a completed message.
You probably want to use threads (import threading). Have spinner.begin() start a thread that prints your messages, then let your blackbox run, and then have spinner.end() send a finish message to the thread using a Queue (from Queue import Queue) or something, join() the thread and keep doing whatever it is you do.
As a design choice, hide the prints somewhere deeper, not in the same block of code as the begin and end calls.
I have a multi-threaded SMTP server. Each thread takes care of one client. I need to set a timeout value of 10 seconds on each server thread to terminate dormant or misbehaving clients.
I have used the time.time(), to find the start time and my checkpoint time and the difference gives the running time. But I believe it gives the system time and not the time this thread was running.
Is there a Thread local timer API in Python ?
import threading
stop = 0
def hello():
stop = 1
t=threading.Timer(10,hello)
t.start()
while stop != 1:
print stop
print "stop changed"
This prints 0 (initial stop) in a loop and does not come out of the while loop.
Python has progressed in the 6 years since this question was asked, and in version 3.3 it's introduced a tool for exactly what was being asked for here:
time.clock_gettime(time.CLOCK_THREAD_CPUTIME_ID)
Python 3.7 additionally introduced an analogous time.clock_gettime_ns.
Detailed docs are exactly where you'd expect but the feature is pretty straightforward straight out of the box.
In the python documentation there is no mention of "thread timing". Either the clocks are process-wide or system-wide. In particular time.clock measures process time while time.time returns the system time.
In python3.3 the timings API was revised and improved but still, I can't see any timer that would return the process time taken by a single thread.
Also note that even if possible it's not at all easy to write such a timer.
Timers are OS specific, so you would have to write a different version of the module for every OS. If you want to profile a specific action, just launch it without threads.
When threaded the timing either it runs as expected, or it is a lot slower because of the OS, in which case you can't do nothing about it(at least, if you don't want to write a patch that "fixes" the GIL or removes it safely).
Python 3.7 has added the time.thread_time() method that seems to do what this question needs. According to the docs, it is thread-specific and excludes time spent sleeping.
The hello function's stop value is local, not the global one.
Add the following:
def hello():
global stop
stop = 1
I am posting a sample code which can measure the running time of the thread, you can modify the code, so as to use with your function.
import time
import threading
def hello():
x = 0
while x < 100000000:
pass
x += 1
start = time.clock()
t = threading.Thread(target = hello, args = ())
t.start()
t.join()
end = time.clock()
print "The time was {}".format(end - start)
On my system, it gave a time of 8.34 seconds.
I have a set of instructions, say {I} and I would like to perform this set {I}
at predefined time for instance each minute.
I'm not asking how to insert a delay of 1 minutes between to successive executions of
the set {I}, I want to start the instructions {I} each minute independently of the time of execution of {I}.
If I inderstand the following code
import time
while True:
{I}
time.sleep(60)
would simply insert a delay of 60 secs between the end of the execution of {I} and the following one. Is it true? Instead I would like that the set of instructions {I} starts each minute (for instance at 9.00 am, 9.01 am, 9.02 am, etc).
Is it possible to perform such a task inside python, or is it preferable to write a script with {I} that I execute each minutes, for instance, with Crontab?
Thank you in advance
Looks like signal.alarm and signal.signal(signal.SIGALRM, handler) should help you.
If you don't need finer resolution than a minute, cron would be the easiest option. Otherwise you'd end up re-writing something like it.
If you need intervals shorter than a minute, you might consider "timeouts" from the glib library. It has Python bindings. The timeout should then probably start the task in a separate process.
Something like APScheduler might meet your needs.
I'm sure there are other similar packages out there as well.
Chances are, you'd have to instantiate separate threads for every instruction to be run concurrently, and simply dispatch them in your delayed while loop.
You could spawn a thread every second using threading.Timer:
import threading
import time
def do_stuff(count):
print(count)
if c < 10: # Let's build in some way to quit
t = threading.Timer(1.0, do_stuff, args=[count+1])
t.start()
t = threading.Timer(0.0, do_stuff, args=[0])
t.start()
t.join()
Using the sched module is another possibility, but note that the sched.scheduler.run method blocks the main process until the event queue is empty. (So if the do_stuff function takes longer than a second, the next event won't run on time.) If you want nonblocking events, use threading.Timer.