I am trying to perform addition in an efficient way in python over large loops . I am trying to loop over a range of 100000000.
from datetime import datetime
start_time = datetime.now()
sum = 0
for i in range(100000000):
sum+=i
end_time = datetime.now()
print('--- %s seconds ---{}'.format(end_time - start_time))
print(sum)
The output from the above code is
--- %s seconds ---0:00:16.662666
4999999950000000
When i try to do it in C, its taking 0.43 seconds
From what i read, python creates new memory everytime when you perform addition to variable. I read some articles and came to know how to perform string concatenation in these situations by avoiding '+' sign . But i dont find anything how to do with integers.
Consider using the sum() function if you can process the list as a whole, which loops entirely in C code and is much faster, and also avoids the creation of new Python objects.
sum(range(100000000))
In my computer, your code takes 07.189210 seconds, while the above statement takes 02.751251 seconds, increasing the processing speed more than 3 times.
Edit: as suggested by mtrw, numpy.sum() can speed up processing even more.
Here is a comparison of three methods: your original way, using sum(range(100000000)) as suggested by Alex Metsai, and using the NumPy numerical library's sum and range functions:
from datetime import datetime
import numpy as np
def orig():
start_time = datetime.now()
sum = 0
for i in range(100000000):
sum+=i
end_time = datetime.now()
print('--- %s seconds ---{}'.format(end_time - start_time))
print(sum)
def pyway():
start_time = datetime.now()
mysum = sum(range(100000000))
end_time = datetime.now()
print('--- %s seconds ---{}'.format(end_time - start_time))
print(mysum)
def npway():
start_time = datetime.now()
sum = np.sum(np.arange(100000000))
end_time = datetime.now()
print('--- %s seconds ---{}'.format(end_time - start_time))
print(sum)
On my computer, I get:
>>> orig()
--- %s seconds ---0:00:09.504018
4999999950000000
>>> pyway()
--- %s seconds ---0:00:02.382020
4999999950000000
>>> npway()
--- %s seconds ---0:00:00.683411
4999999950000000
NumPy is the fastest, if you can use it in your application.
But, as suggested by Ethan in a comment, it's worth pointing out that calculating the answer directly is by far the fastest:
def mathway():
start_time = datetime.now()
mysum = 99999999*(99999999+1)/2
end_time = datetime.now()
print('--- %s seconds ---{}'.format(end_time - start_time))
print(mysum)
>>> mathway()
--- %s seconds ---0:00:00.000013
4999999950000000.0
I assume your actual problem is not so easily solved by pencil and paper :)
I have a function
start_time = time.time()
That is producing some wild numbers
i.e.
start time: 1611368981.2445016
That is causing the following error
ValueError: time data '6.9141387939453125e-06' does not match format '%H/%M/%S'
Why is time.time() producing such wild times and how to get it into normal format?
lines of code in question:
if plant_warning_flag == 0:
start_time = time.time() #start time flag for plant temperature warning
plant_warning_flag = 1
elapsed_time = time.time() - start_time
print("start time: ", start_time)
print('my elapsed time: ', elapsed_time)
newelaptime = time.strptime(str(elapsed_time), "%H/%M/%S")
newmthactime = pd.to_datetime(maxtime_heatac.strip(), format='%H:%M:%S')
if newcread > plant_warning + critical_threshold_ac:
session = requests.Session()
session.post(acserver_url,headers=headers,data=payload)
elif newlaptime > newmthactime:
payload = {'on1':'4000'}
session = requests.Session()
print('peek-a-boo')
how do I retrieve time.time() is regular date format?
Your error seem to be coming from this line:
newelaptime = time.strptime(str(elapsed_time), "%H/%M/%S")
because you trying to convert Unix time (number of secs of the Unix Epoch - counted from 1.01.1970) which is number - into string - and then using strptime trying to convert this into time object. ...which goes into error - because strptime converts time from human-readable-string into time-object.
Unix Time is great for operators and calculating time ranges but not good for reading by human :)
To get human readable value - try something like this:
datetime.datetime.utcfromtimestamp(YOUR_UNIX_TIMESTAMP).strftime('%Y-%m-%dT%H:%M:%SZ')
it will give you a string 'YYYY-mm-dd H:M:Secs'
...shorter version like this:
datetime.datetime.utcfromtimestamp(YOUR_UNIX_TIMESTAMP)
...will give you a time object - which is great when you want to operate with (less mathematic / more calendar) units like months years etc.
you can use strftime and pass the format of the time which you are interested to get from the local time.
from time import localtime, strftime
result = strftime("%H:%M:%S", localtime())
print(result)
How can I convert the execution time to milliseconds.
I already multiplied the start and end time to 1000.
I used time.time()
Execution Time
Result:
('Start time: ', 1596465418538.365)
Remove.IntNonIdUniqueIndex
('End time: ', 1596465418538.399)
('Execution time: ', 3.409385681152344e-05)
time.time() basic unit is second. It's enough to multiply the difference between end and start by 1000 to get the milliseconds.
import time
start = time.time()
time.sleep(1)
end = time.time()
d = end - start
print(f'executed in {d} seconds or {d*1000} milliseconds')
executed in 1.003673791885376 seconds or 1003.673791885376 milliseconds
If you are measuring program execution times, it is best to use time.monotonic() or time.monotonic_ns().
These functions are guaranteed never to go backwards even in the event of system clock updates.
The first returns a value in seconds, the second in nanoseconds.
I'm creating a loop which executes every 5 seconds, starting at the startTime variable and ending at the stopTime variable. However, the code below is disregarding the minutes within my startTime and endTime variables and only executing on the hour. For example, even though my startTime is '1130', the code is executing 11:05, rather than ending the loop. I have the same problem with the endTime variable. If the current time is 12:45, the code still executes even though the endTime variable is '1230'. The code will stop executing at '1300'.
frequency = 5
startTime = '1130'
endTime = '1230'
while True:
now = datetime.now().strftime('%H:%M:%S')
if startTime <= now <= endTime:
print('Loop is working. Time is: ',now)
time.sleep(frequency)
else:
print('Loop is stopped')
break
I live in Central Time, so I tried switching to Eastern timezone by modifying the "now" variable to:
now = datetime.now(timezone('US/Eastern')).strftime('%H:%M:%S.%f %Z')
but I still get the same problem when I substitute eastern times with startTime and endTime when using the eastern datetime.now().
Is executing code at a precise minute possible with strftime()?
EDIT: (this is now the answer to the real question (oops))
If you want to wait till for example 11:30 (which was the real question)
you can calculate the time (in seconds) the program should sleep (and let it sleep for that time):
def wait_till(hour, minute, second=0):
# get system time (and date)
now_time = datetime.datetime.now()
# create time point we are waiting for (this year, this month and this day)
wait_till_time = datetime.datetime(year=now_time.year, month=now_time.month, day=now_time.day, hour=hour, minute=minute, second=second)
# calculate time we want to wait for and convert to seconds
wait_for = (wait_till_time - now_time).total_seconds()
# check if it's going to be tomorrow (if we would sleep for a negative amount of seconds)
if wait_for < 0:
# add one day
wait_till_time = wait_till_time.replace(day=now_time.day+1)
# recalculate (not very beautiful, but i don't know a better way)
wait_for = (wait_till_time - now_time).total_seconds()
# printing this waiting time (in seconds)
print("waiting for",wait_for,"seconds")
# sleeping for that time
time.sleep(wait_for)
# printing the new now time, so we can see how accurate it is
print("its now",datetime.datetime.now())
and say for example:
wait_till(20, 24) # waiting till 20:24 (today)
and get:
waiting for 15.32297 seconds
its now 2019-03-11 20:24:00.003857
which is pretty darn close to what we wanted (20:24:00.000000) and this delay is probably only caused by the calculation lag of formatting the string.
(The old stuff ...)
if it's not important that it takes 100% 5s (but rather 100.04546642303467% --> it will get off a little bit every time) you can just do
import time
frequency = 5 #every 5 seconds
start_time = time.time()
while 1:
elspsed_time = time.time() - start_time
print(elspsed_time)
time.sleep(frequency)
but if you need the 100% you can try this autocorrecting solution:
import time
from threading import Timer
frequency = 5 #every 5 seconds
start_time = time.time()
def what_to_do_after_5s():
elapsed_time = time.time() - start_time
print(elapsed_time)
# next call
Timer(5.0 - (elapsed_time - int(elapsed_time)), what_to_do_after_5s, ()).start()
what_to_do_after_5s()
and we can see that it autocorrects:
0.0
5.000170707702637
10.000272989273071
15.000539064407349
20.001248836517334
25.00046443939209
30.000929355621338
35.00142860412598
40.0007688999176
45.00128436088562
50.00045442581177
55.000683069229126
60.00123882293701
65.00095415115356
70.0015127658844
I know that I can cause a thread to sleep for a specific amount of time with:
time.sleep(NUM)
How can I make a thread sleep until 2AM? Do I have to do math to determine the number of seconds until 2AM? Or is there some library function?
( Yes, I know about cron and equivalent systems in Windows, but I want to sleep my thread in python proper and not rely on external stimulus or process signals.)
Here's a half-ass solution that doesn't account for clock jitter or adjustment of the clock. See comments for ways to get rid of that.
import time
import datetime
# if for some reason this script is still running
# after a year, we'll stop after 365 days
for i in xrange(0,365):
# sleep until 2AM
t = datetime.datetime.today()
future = datetime.datetime(t.year,t.month,t.day,2,0)
if t.hour >= 2:
future += datetime.timedelta(days=1)
time.sleep((future-t).total_seconds())
# do 2AM stuff
You can use the pause package, and specifically the pause.until function, for this:
import pause
from datetime import datetime
pause.until(datetime(2015, 8, 12, 2))
Slightly more generalized solution (based off of Ross Rogers') in case you'd like to add minutes as well.
def sleepUntil(self, hour, minute):
t = datetime.datetime.today()
future = datetime.datetime(t.year, t.month, t.day, hour, minute)
if t.timestamp() > future.timestamp():
future += datetime.timedelta(days=1)
time.sleep((future-t).total_seconds())
Another approach, using sleep, decreasing the timeout logarithmically.
def wait_until(end_datetime):
while True:
diff = (end_datetime - datetime.now()).total_seconds()
if diff < 0: return # In case end_datetime was in past to begin with
time.sleep(diff/2)
if diff <= 0.1: return
Building on the answer of #MZA and the comment of #Mads Y
One possible approach is to sleep for an hour. Every hour, check if the time is in the middle of the night. If so, proceed with your operation. If not, sleep for another hour and continue.
If the user were to change their clock in the middle of the day, this approach would reflect that change. While it requires slightly more resources, it should be negligible.
I tried the "pause" pacakage. It does not work for Python 3.x. From the pause package I extracted the code required to wait until a specific datetime and made the following def.
def wait_until(execute_it_now):
while True:
diff = (execute_it_now - datetime.now()).total_seconds()
if diff <= 0:
return
elif diff <= 0.1:
time.sleep(0.001)
elif diff <= 0.5:
time.sleep(0.01)
elif diff <= 1.5:
time.sleep(0.1)
else:
time.sleep(1)
adapt this:
from datetime import datetime, timedelta
from time import sleep
now = datetime.utcnow
to = (now() + timedelta(days = 1)).replace(hour=1, minute=0, second=0)
sleep((to-now()).seconds)
Slightly beside the point of the original question:
Even if you don't want to muck around with crontabs, if you can schedule python scripts to those hosts, you might be interested to schedule anacron tasks? anacron's major differentiator to cron is that it does not rely the computer to run continuously. Depending on system configuration you may need admin rights even for such user-scheduled tasks.
A similar, more modern tool is upstart provided by the Ubuntu folks: http://upstart.ubuntu.com/
This does not yet even have the required features. But scheduling jobs and replacing anacron is a planned feature. It has quite some traction due to its usage as Ubuntu default initd replacement. (I am not affiliated with the project)
Of course, with the already provided answer, you can code the same functionality into your python script and it might suit you better in your case.
Still, for others, anacron or similar existing systems might be a better solution. anacron is preinstalled on many current linux distributions (there are portability issues for windows users).
Wikipedia provides a pointer page: https://en.wikipedia.org/wiki/Anacron
If you do go for a python version I'd look at the asynchronous aspect, and ensure the script works even if the time is changed (daylight savings, etc) as others have commented already. Instead of waiting til a pre-calculated future, I'd always at maximum wait one hour, then re-check the time. The compute cycles invested should be negligible even on mobile, embedded systems.
Asynchronous version of Omrii's solution
import datetime
import asyncio
async def sleep_until(hour: int, minute: int, second: int):
"""Asynchronous wait until specific hour, minute and second
Args:
hour (int): Hour
minute (int): Minute
second (int): Second
"""
t = datetime.datetime.today()
future = datetime.datetime(t.year, t.month, t.day, hour, minute, second)
if t.timestamp() > future.timestamp():
future += datetime.timedelta(days=1)
await asyncio.sleep((future - t).total_seconds())
I know is way late for this, but I wanted to post an answer (inspired on the marked answer) considering systems that might have - incorrect - desired timezone + include how to do this threaded for people wondering how.
It looks big because I'm commenting every step to explain the logic.
import pytz #timezone lib
import datetime
import time
from threading import Thread
# using this as I am, check list of timezone strings at:
## https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
TIMEZONE = pytz.timezone("America/Sao_Paulo")
# function to return desired seconds, even if it's the next day
## check the bkp_time variable (I use this for a bkp thread)
## to edit what time you want to execute your thread
def get_waiting_time_till_two(TIMEZONE):
# get current time and date as our timezone
## later we remove the timezone info just to be sure no errors
now = datetime.datetime.now(tz=TIMEZONE).replace(tzinfo=None)
curr_time = now.time()
curr_date = now.date()
# Make 23h30 string into datetime, adding the same date as current time above
bkp_time = datetime.datetime.strptime("02:00:00","%H:%M:%S").time()
bkp_datetime = datetime.datetime.combine(curr_date, bkp_time)
# extract the difference from both dates and a day in seconds
bkp_minus_curr_seconds = (bkp_datetime - now).total_seconds()
a_day_in_seconds = 60 * 60 * 24
# if the difference is a negative value, we will subtract (- with + = -)
# it from a day in seconds, otherwise it's just the difference
# this means that if the time is the next day, it will adjust accordingly
wait_time = a_day_in_seconds + bkp_minus_curr_seconds if bkp_minus_curr_seconds < 0 else bkp_minus_curr_seconds
return wait_time
# Here will be the function we will call at threading
def function_we_will_thread():
# this will make it infinite during the threading
while True:
seconds = get_waiting_time_till_two(TIMEZONE)
time.sleep(seconds)
# Do your routine
# Now this is the part where it will be threading
thread_auto_update = Thread(target=function_we_will_thread)
thread_auto_update.start()
It takes only one of the very basic libraries.
import time
sleep_until = 'Mon Dec 25 06:00:00 2020' # String format might be locale dependent.
print("Sleeping until {}...".format(sleep_until))
time.sleep(time.mktime(time.strptime(sleep_until)) - time.time())
time.strptime() parses the time from string -> struct_time tuple. The string can be in different format, if you give strptime() parse-format string as a second argument. E.g.
time.strptime("12/25/2020 02:00AM", "%m/%d/%Y %I:%M%p")
time.mktime() turns the struct_time -> epoch time in seconds.
time.time() gives current epoch time in seconds.
Substract the latter from the former and you get the wanted sleep time in seconds.
sleep() the amount.
If you just want to sleep until whatever happens to be the next 2AM, (might be today or tomorrow), you need an if-statement to check if the time has already passed today. And if it has, set the wake up for the next day instead.
import time
sleep_until = "02:00AM" # Sets the time to sleep until.
sleep_until = time.strftime("%m/%d/%Y " + sleep_until, time.localtime()) # Adds todays date to the string sleep_until.
now_epoch = time.time() #Current time in seconds from the epoch time.
alarm_epoch = time.mktime(time.strptime(sleep_until, "%m/%d/%Y %I:%M%p")) # Sleep_until time in seconds from the epoch time.
if now_epoch > alarm_epoch: #If we are already past the alarm time today.
alarm_epoch = alarm_epoch + 86400 # Adds a day worth of seconds to the alarm_epoch, hence setting it to next day instead.
time.sleep(alarm_epoch - now_epoch) # Sleeps until the next time the time is the set time, whether it's today or tomorrow.
What about this handy and simple solution?
from datetime import datetime
import time
pause_until = datetime.fromisoformat('2023-02-11T00:02:00') # or whatever timestamp you gonna need
time.sleep((pause_until - datetime.now()).total_seconds())
from datetime import datetime
import time, operator
time.sleep([i[0]*3600 + i[1]*60 for i in [[H, M]]][0] - [i[0]*3600 + i[1]*60 for i in [map(int, datetime.now().strftime("%H:%M").split(':'))]][0])
Instead of using the wait() function, you can use a while-loop checking if the specified date has been reached yet:
if datetime.datetime.utcnow() > next_friday_10am:
# run thread or whatever action
next_friday_10am = next_friday_10am()
time.sleep(30)
def next_friday_10am():
for i in range(7):
for j in range(24):
for k in range(60):
if (datetime.datetime.utcnow() + datetime.timedelta(days=i)).weekday() == 4:
if (datetime.datetime.utcnow() + datetime.timedelta(days=i, hours=j)).hour == 8:
if (datetime.datetime.utcnow() + datetime.timedelta(days=i, hours=j, minutes=k)).minute == 0:
return datetime.datetime.utcnow() + datetime.timedelta(days=i, hours=j, minutes=k)
Still has the time-checking thread check the condition every after 30 seconds so there is more computing required than in waiting, but it's a way to make it work.