I have a time series dataset of a Pandas series df that I am trying to add a new value to the bottom of the df and then increment the timestamp which is the df index.
For example the new value I can add to the bottom of the df like this:
testday.loc[len(testday.index)] = testday_predict[0]
print(testday)
Which seems to work but the time stamp is just incremented:
kW
Date
2022-07-29 00:00:00 39.052800
2022-07-29 00:15:00 38.361600
2022-07-29 00:30:00 38.361600
2022-07-29 00:45:00 38.534400
2022-07-29 01:00:00 38.880000
... ...
2022-07-29 23:00:00 36.806400
2022-07-29 23:15:00 36.806400
2022-07-29 23:30:00 36.633600
2022-07-29 23:45:00 36.806400
96 44.482361 <---- my predicted value added at the bottom good except for the time stamp value of 96
Like the value of 96 is just the next value in the length of the df.index hopefully this makes sense.
If I try:
from datetime import timedelta
last_index_stamp = testday.last_valid_index()
print(last_index_stamp)
This returns:
Timestamp('2022-07-29 23:45:00')
And then I can add 15 minutes to this Timestamp (my data is 15 minute data) like this:
new_timestamp = last_index_stamp + timedelta(minutes=15)
print(new_timestamp)
Which returns what I am looking instead of the value of 96:
Timestamp('2022-07-30 00:00:00')
But how do I replace the value of 96 with new_timestampt? If I try:
testday.index[-1:] = new_timestamp
This will error out:
TypeError: Index does not support mutable operations
Any tips greatly appreciated...
This should do the trick:
testday.loc[new_timestamp,:] = testday_predict[0]
I have function which is generating a time stamp of 10 days at interval of 15mins. These values are stored in a numpy array. I want to apply a condition on each that checks when the time is between 6am and 7pm then print "something" and when its above 7pm to next day till 6am print "something else".
my code for time stamp:
import pandas as pd
import numpy as np
timeStamp = pd.date_range(start='1/1/2020', end='1/10/2020', freq='15min')
chargeTime = datetime.time(6,0,0)
dischargeTime = datetime.time(19,0,0)
I don't know how to grab the date from the timeStamp and compare it chargeTime and dischargeTime.
I think what you want is something like this, where you set the charge_time and discharge_time to integers and compare to the .hour attribute of the timestamp:
import pandas as pd
import numpy as np
time_stamps = pd.date_range(start='1/1/2020', end='1/10/2020', freq='15min')
charge_time = 6 # 6am
discharge_time = 19 # 7pm
for t in time_stamps:
if charge_time <= t.hour < discharge_time:
print str(t) + " - something"
else:
print str(t) + " - something else"
which will print the timestamp and then the message like this:
2020-01-09 18:45:00 - something
2020-01-09 19:00:00 - something else
This should work. I'm using dt.hour to extract the hour of the datetime object and then np.where() to check for the conditions if it's between 6 and 19, and "something" else (between 19 and 6) "something else". This avoids using for loops and is pretty efficient:
timeStamp = pd.date_range(start='1/1/2020', end='1/10/2020', freq='15min')
df = timeStamp.to_frame()
df.columns = ['dates']
df['conditions'] = np.where((df['dates'].dt.hour > 6) & (df['dates'].dt.hour < 19),"something","something else")
For example:
print(df.loc['2020-01-01 06:45:00':'2020-01-01 07:30:00'])
Output:
dates conditions
2020-01-01 06:45:00 2020-01-01 06:45:00 something else
2020-01-01 07:00:00 2020-01-01 07:00:00 something
2020-01-01 07:15:00 2020-01-01 07:15:00 something
2020-01-01 07:30:00 2020-01-01 07:30:00 something
Finally, if you wish to provide different values for charge_time and discharge_time you can simply define them and use them in the np.where().
charge_time = 6
discharge_time = 19
df['conditions'] = np.where((df['dates'].dt.hour > charge_time) & (df['dates'].dt.hour < discharge_time),"something","something else")
I have time in hours as shown below
hour:
0
1
2
3
4
5
6
7
8
9
10
11
12
14
15
16
17
18
19
20
21
22
23
I would like to have the time like:
00:00:00
01:00:00
02:00:00
03:00:00 etc
Here's the solution by using pandas and datetime python library. I hope it will help you
Code:
import pandas as pd
from datetime import datetime
hour = 13
a = pd.to_datetime(hour, format='%H')
converted_time = a.strftime("%H:%M:%S")
print(converted_time)
Output
13:00:00
Just use time from datetime:
In [1]: from datetime import time
In [2]: for i in range(4):
...: t = time(i, 0, 0)
...: print(t.strftime('%H:%M:%S'))
...:
00:00:00
01:00:00
02:00:00
03:00:00
You can use datetime with its function strptime. Used like that :
from datetime import datetime
t = datetime.strptime(str(your_int), "%H")
print(t.strftime("%H"))
Using the strptime is useful, because if your int format change, you can still directly format it to a date, using the different codes available here https://docs.python.org/2/library/datetime.html?highlight=datetime#strftime-strptime-behavior
I have a CSV file that contains start-time and end-time for sessions.
I would like to understand how I can do End-time - Start-time to get the duration of a session.
So far I have this and it works
start_time = "2016-11-09 18:06:17"
end_time ="2016-11-09 18:21:07"
start_dt = dt.datetime.strptime(start_time, '%Y-%m-%d %H:%M:%S')
end_dt = dt.datetime.strptime(end_time, '%Y-%m-%d %H:%M:%S')
diff = (end_dt - start_dt)
duration = diff.seconds/60
print (duration)
but I want to do it for the whole column at once.
To import from a csv and then manipulate the date, pandas is the way to go. Since the only info you gave about your data was start and end time, I will show that.
Code:
import pandas as pd
df = pd.read_csv(data, parse_dates=['start_time', 'end_time'],
infer_datetime_format=True)
print(df)
df['time_delta'] = df.end_time.values - df.start_time.values
print(df.time_delta)
Test Data:
from io import StringIO
data = StringIO(u'\n'.join([x.strip() for x in """
start_time,end_time,a_number
2013-09-19 03:00:00,2013-09-19 04:00:00,221.0797
2013-09-19 04:00:00,2013-09-19 05:00:00,220.5083
2013-09-24 03:00:00,2013-09-24 05:00:00,221.7733
2013-09-24 04:00:00,2013-09-24 06:00:00,221.2493
""".split('\n')[1:-1]]))
Results:
start_time end_time a_number
0 2013-09-19 03:00:00 2013-09-19 04:00:00 221.0797
1 2013-09-19 04:00:00 2013-09-19 05:00:00 220.5083
2 2013-09-24 03:00:00 2013-09-24 05:00:00 221.7733
3 2013-09-24 04:00:00 2013-09-24 06:00:00 221.2493
0 01:00:00
1 01:00:00
2 02:00:00
3 02:00:00
Name: time_delta, dtype: timedelta64[ns]
It seems you are trying to run diff against strings, instead of datetime values.
How about something like this?
from datetime import datetime
start_time = datetime(2016, 11, 12, 18, 06, 17)
end_time = datetime(2016, 11, 09, 18, 21, 07)
diff = end_time - start_time
print(diff.seconds / 60)
I think this should work.
Basically, I want my script to pause between 4 and 5 AM. The only way to do this I've come up with so far is this:
seconds_into_day = time.time() % (60*60*24)
if 60*60*4 < seconds_into_day < 60*60*5:
sleep(time_left_till_5am)
Any "proper" way to do this? Aka some built-in function/lib for calculating time; rather than just using seconds all the time?
You want datetime
The datetime module supplies classes for manipulating dates and times in both simple and complex ways
If you use date.hour from datetime.now() you'll get the current hour:
datetimenow = datetime.now();
if datetimenow.hour in range(4, 5)
sleep(time_left_till_5am)
You can calculate time_left_till_5am by taking 60 - datetimenow.minute multiplying by 60 and adding to 60 - datetimenow.second.
Python has a built-in datetime library: http://docs.python.org/library/datetime.html
This should probably get you what you're after:
import datetime as dt
from time import sleep
now = dt.datetime.now()
if now.hour >= 4 andnow.hour < 5:
sleep((60 - now.minute)*60 + (60 - now.second))
OK, the above works, but here's the purer, less error-prone solution (and what I was originally thinking of but suddenly forgot how to do):
import datetime as dt
from time import sleep
now = dt.datetime.now()
pause = dt.datetime(now.year, now.month, now.day, 4)
start = dt.datetime(now.year, now.month, now.day, 5)
if now >= pause and now < start:
sleep((start - now).seconds)
That's where my original "timedelta" comment came from -- what you get from subtracting two datetime objects is a timedelta object (which in this case we pull the 'seconds' attribute from).
The following code covers the more general case where a script needs to pause during any fixed window of less than 24 hours duration. Example: must sleep between 11:00 PM and 01:00 AM.
import datetime as dt
def sleep_duration(sleep_from, sleep_to, now=None):
# sleep_* are datetime.time objects
# now is a datetime.datetime object
if now is None:
now = dt.datetime.now()
duration = 0
lo = dt.datetime.combine(now, sleep_from)
hi = dt.datetime.combine(now, sleep_to)
if lo <= now < hi:
duration = (hi - now).seconds
elif hi < lo:
if now >= lo:
duration = (hi + dt.timedelta(hours=24) - now).seconds
elif now < hi:
duration = (hi - now).seconds
return duration
tests = [
(4, 5, 3, 30),
(4, 5, 4, 0),
(4, 5, 4, 30),
(4, 5, 5, 0),
(4, 5, 5, 30),
(23, 1, 0, 0),
(23, 1, 0, 30),
(23, 1, 0, 59),
(23, 1, 1, 0),
(23, 1, 1, 30),
(23, 1, 22, 30),
(23, 1, 22, 59),
(23, 1, 23, 0),
(23, 1, 23, 1),
(23, 1, 23, 59),
]
for hfrom, hto, hnow, mnow in tests:
sfrom = dt.time(hfrom)
sto = dt.time(hto)
dnow = dt.datetime(2010, 7, 5, hnow, mnow)
print sfrom, sto, dnow, sleep_duration(sfrom, sto, dnow)
and here's the output:
04:00:00 05:00:00 2010-07-05 03:30:00 0
04:00:00 05:00:00 2010-07-05 04:00:00 3600
04:00:00 05:00:00 2010-07-05 04:30:00 1800
04:00:00 05:00:00 2010-07-05 05:00:00 0
04:00:00 05:00:00 2010-07-05 05:30:00 0
23:00:00 01:00:00 2010-07-05 00:00:00 3600
23:00:00 01:00:00 2010-07-05 00:30:00 1800
23:00:00 01:00:00 2010-07-05 00:59:00 60
23:00:00 01:00:00 2010-07-05 01:00:00 0
23:00:00 01:00:00 2010-07-05 01:30:00 0
23:00:00 01:00:00 2010-07-05 22:30:00 0
23:00:00 01:00:00 2010-07-05 22:59:00 0
23:00:00 01:00:00 2010-07-05 23:00:00 7200
23:00:00 01:00:00 2010-07-05 23:01:00 7140
23:00:00 01:00:00 2010-07-05 23:59:00 3660
When dealing with dates and times in Python I still prefer mxDateTime over Python's datetime module as although the built-in one has improved greatly over the years it's still rather awkward and lacking in comparison. So if interested go here: mxDateTime It's free to download and use. Makes life much easier when dealing with datetime math.
import mx.DateTime as dt
from time import sleep
now = dt.now()
if 4 <= now.hour < 5:
stop = dt.RelativeDateTime(hour=5, minute=0, second=0)
secs_remaining = ((now + stop) - now).seconds
sleep(secs_remaining)