I want to distribute the numbers preset in the list in whole month
a) Given a Holiday list, I want to dynamically assign '1' on the holiday date and '0' for working day .
eg.
Holiday_List = ['2020-01-01','2020-01-05','2020-01-12','2020-01-19','2020-01-26']
Start_date = datetime.datetime(year=2020, month =1 , day=1)
end_date = datetime.datetime(year =2020,month =1,day=28 )
Below is the outpput I am looking for in dataframe,where 'Date' and 'Holiday' are columns.
Date Holiday
01-01-2020 1
02-01-2020 0
03-01-2020 0
04-01-2020 0
05-01-2020 1
06-01-2020 0
07-01-2020 0
08-01-2020 0
09-01-2020 0
10-01-2020 0
11-01-2020 0
12-01-2020 1
13-01-2020 0
14-01-2020 0
15-01-2020 0
16-01-2020 0
17-01-2020 0
18-01-2020 0
19-01-2020 1
20-01-2020 0
21-01-2020 0
22-01-2020 0
23-01-2020 0
24-01-2020 0
25-01-2020 0
26-01-2020 1
27-01-2020 0
28-01-2020 0
B) Given a list of nos like [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18].. I want to break into 3 equal part and store it in 3 different list.
a=[1,2,3,4,5,6],b=[7,8,9,10,11,12], c=[13,14,15,16,17,18]..
sequence should be there like first 6 element in a, sec in 'b' and 3rd in 'c'
C) I want to distribute the above lists a,b,c in whole months such that gap between 1 element of a,b and
c should be 8 days only..similarly for others nos. and there is one constraint I cannot assign any no. of holiday.
Below is the final output I am looking for, where list values are assign in column "Values" and Here I have assigning dummy value 'NW' to have gap of 8 days between every list.
Date Holiday Values
01-01-2020 1 Holiday
02-01-2020 0 1
03-01-2020 0 2
04-01-2020 0 3
05-01-2020 1 Holiday
06-01-2020 0 4
07-01-2020 0 5
08-01-2020 0 6
09-01-2020 0 NW
10-01-2020 0 NW
11-01-2020 0 7
12-01-2020 1 Holiday
13-01-2020 0 8
14-01-2020 0 9
15-01-2020 0 10
16-01-2020 0 11
17-01-2020 0 12
18-01-2020 0 NW
19-01-2020 1 Holiday
20-01-2020 0 13
21-01-2020 0 14
22-01-2020 0 15
23-01-2020 0 16
24-01-2020 0 17
25-01-2020 0 18
26-01-2020 1 Holiday
27-01-2020 0 NW
28-01-2020 0 NW
A) You can use date_range to create column with dates
df = pd.DataFrame()
df['Date'] = pd.date_range(start_date, end_date)
Next you can create column Holiday with zeros in all cells
df['Holiday'] = 0
And next you can replace some values
for item in holiday_list:
item = datetime.datetime.strptime(item, '%Y-%m-%d')
df['Holiday'][ df['Date'] == item ] = 1
but maybe this part could be simpler using isin()
mask = df['Date'].dt.strftime('%Y-%m-%d').isin(holiday_list)
df['Holiday'][mask] = 1
or using numpy.where()
import numpy as np
mask = df['Date'].dt.strftime('%Y-%m-%d').isin(holiday_list)
df['Holiday'] = np.where(mask, 1, 0)
or simply keep it as True/False instead of 1/0
df['Holiday'] = df['Date'].dt.strftime('%Y-%m-%d').isin(holiday_list)
import pandas as pd
import datetime
holiday_list = ['2020-01-01','2020-01-05','2020-01-12','2020-01-19','2020-01-26']
start_date = datetime.datetime(year=2020, month=1, day=1)
end_date = datetime.datetime(year=2020,month=1, day=28)
df = pd.DataFrame()
df['Date'] = pd.date_range(start_date, end_date)
df['Holiday'] = 0
mask = df['Date'].dt.strftime('%Y-%m-%d').isin(holiday_list)
df['Holiday'][mask] = 1
print(df)
B) you could use [start:start+size] to split list
numbers = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]
size = len(numbers)//3
print(d[size*0:size*1], d[size*1:size*2], d[size*2:size*3])
or
print(d[:size], d[size:size*2], d[size*2:])
Similar way you can split dataframe (after filtered "Holiday") to work with 8 days [start:star+8] but I wil use it in (C)
C) You can create column Values with NW in all cells
df['Values'] = 'NW'
Next you can use previous mask to assign "Holiday"
mask = df['Date'].dt.strftime('%Y-%m-%d').isin(holiday_list)
df['Values'][ mask ] = 'Holiday'
Using ~ you can negate mask to reverse selection - to select cells withou "Holiday"
selected = df['Values'][ ~mask ]
and now I can try to assing
for a, b in zip(range(0, len(selected), 8), range(0, len(numbers), size)):
selected[a:a+size] = numbers[b:b+size]
df['Values'][ ~mask ] = selected
but maybe it can be done in simpler way. Maybe with groupby() or rolling() ?
import pandas as pd
import datetime
holiday_list = ['2020-01-01','2020-01-05','2020-01-12','2020-01-19','2020-01-26']
start_date = datetime.datetime(year=2020, month=1, day=1)
end_date = datetime.datetime(year=2020,month=1, day=28)
df = pd.DataFrame()
# ---
df['Date'] = pd.date_range(start_date, end_date)
mask = df['Date'].dt.strftime('%Y-%m-%d').isin(holiday_list)
df['Holiday'] = 0
df['Holiday'][mask] = 1
# ---
df['Values'] = 'NW'
df['Values'][ mask ] = 'Holiday'
numbers = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]
size = len(numbers)//3
selected = df['Values'][ ~mask ]
for a, b in zip(range(0, len(selected), 8), range(0, len(numbers), size)):
selected[a:a+size] = numbers[b:b+size]
df['Values'][ ~mask ] = selected
print(df)
EDIT:
I created this code.
Main problem was it sometimes create copy of data and it change values in this copy but not in original dataframe - so I use masks instead of slicings.
It may display warning that it changes values in copy of data (not in original dataframe) but finally it gives me correct result.
Maybe using information from Returning a view versus a cop it could remove this warning
import pandas as pd
import datetime
holiday_list = [
'2020-01-01','2020-01-05',
#'2020-01-10','2020-01-11', # add more to test when there is less then 7 NW
'2020-01-12','2020-01-19','2020-01-26'
]
start_date = datetime.datetime(year=2020, month=1, day=1)
end_date = datetime.datetime(year=2020,month=1, day=28)
df = pd.DataFrame()
# ---
df['Date'] = pd.date_range(start_date, end_date)
mask = df['Date'].dt.strftime('%Y-%m-%d').isin(holiday_list)
df['Holiday'] = 0
df['Holiday'][mask] = 1
# ---
df['Values'] = 'NW'
df['Values'][ mask ] = 'Holiday'
numbers = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]
size = len(numbers)//3
start = 0
for b in range(0, len(numbers), size):
# find first and last NW to replace (needs `start` to keep few NW at the end of previous 8 days gap)
mask = (df['Values'] == 'NW') & (df.index >= start)
# change size if there is less then 7 `NW`
print('NW:', sum(mask)) # sum() counts all `True` in mask
if sum(mask) <= size:
left = size - sum(mask)
size = sum(mask)
print('shorter:', size, left)
# first and last NW to replace
start = df[ mask ].index[0]
end = df[ mask ].index[size-1]
print('start, end:', start, end)
# use new mask to select and replace values
# (using slicing [0:6] doesn't work beacuse it create copy of data
# and it doesn't replace in original dataframe)
mask = mask & (df.index >= start) & (df.index <= end)
df['Values'][ mask ] = numbers[b:b+size]
# create gap 8days
start += 8+1
print(df)
I hope you solved it by now :) anyway this is my approach to solve the problem,
First of all, there are certain assumptions that I consider about when writing the code,
The length of the given array of integers is <= 18 which makes the length of a,b,c arrays <= 8
First, we need to divide the given array into equal three parts,
and if the length of split arrays are < 8 we need to fill them with NW dummy values so the array length becomes 8.
To do that easily, we could use numpy.array, the array needs to split and add string type data NW. to do that we could use object as dtype of the array numpy.chararray here is an application
arr = np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18], dtype=object)
then we need to split the array into three equal parts,
arr = np.split(arr,3)
those created arrays need to fill if their length is < 8, np.insert
for i in range(len(arr[0]), 8):
arr = np.insert(arr, i, dummy, axis=1) # fill remaining slots of arrays with dummy value(NW)
Then we need to consider,
Part- A
We need to get the number of days between two days delta (can put that calculation inside the for statement)
we need to get the dates for that range of days with the help of (datetime — Basic date and time types ) and iteration.
delta = end_date - Start_date
for i in range(delta.days + 1):
day = Start_date + timedelta(days=i)
we can use .strftime() to define the time format we need.
day.strftime("%d-%m-%Y")
Finally, we need to check the current date given from the iteration is in the Holiday_List and print 1 Holiday next to date. If not, we need to print 0 and the elements from arrays next to date and also need to make sure to have a gap of 8 days between every list and empty day slot need to fill with the dummy value NW.
count = 0
for i in range(delta.days + 1):
day = Start_date + timedelta(days=i)
if day.strftime("%Y-%m-%d") in Holiday_List:
print("{}\t{}\t{}".format(day.strftime("%d-%m-%Y"), 1, hDay))
else:
print("{}\t{}\t{}".format(day.strftime("%d-%m-%Y"), 0, arr[count//8][count%8]))
count += 1
here count//8 will decide which array need to use to print its' elements and count%8 choose which element needs to print.
So the program,
import datetime
import numpy as np
from datetime import timedelta
Holiday_List = ['2020-01-01','2020-01-05','2020-01-12','2020-01-19','2020-01-26']
Start_date = datetime.datetime(year=2020, month =1 , day=1)
end_date = datetime.datetime(year =2020,month =1,day=28 )
delta = end_date - Start_date
print(delta)
hDay = "Holiday"
dummy = "NW"
# --- numpy array ---
arr = np.array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18], dtype=object) #Assumed that the array length of is divisible by 3 every time
arr = np.split(arr,3) #spilts the array to three equal parts
for i in range(len(arr[0]), 8):
arr = np.insert(arr, i, dummy, axis=1) # fill remaining slots with dummy value(NW)
print("{}\t{}\t{}".format("Date", "Holiday", "Values"))
count = 0
for i in range(delta.days + 1):
day = Start_date + timedelta(days=i)
if day.strftime("%Y-%m-%d") in Holiday_List:
print("{}\t{}\t{}".format(day.strftime("%d-%m-%Y"), 1, hDay))
else:
print("{}\t{}\t{}".format(day.strftime("%d-%m-%Y"), 0, arr[count//8][count%8]))
count += 1
EDIT:
The above code has an issue in the last part that determines the gap and setting the dummy value NW
"When there are no holidays then you would need 3 NW so I would add 3 NW to every list ('a', 'b', 'c'), and then I would work with every list separately. I would use external for-loop like for data in arr: instead of arr[count//8] and I would count gap to skip last element if gap is 8 and element is 'NW' (BTW: if you add more holidays then you has to create gap bigger then 8). – #furas "
So with the help of #furas able to solve the issue(Thanks to him) :), Excess dummy values NW were neglected by iterating through the list,
import datetime
import numpy as np
from datetime import timedelta
Holiday_List = ['2020-01-01','2020-01-05','2020-01-12','2020-01-19','2020-01-26']
Start_date = datetime.datetime(year=2020, month=1, day=1)
end_date = datetime.datetime(year=2020, month=1, day=28)
delta = end_date - Start_date
print(delta)
hDay = "Holiday"
dummy = "NW"
# --- numpy array ---
arr = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18], dtype=object) # Assumed that the array length of is divisible by 3 every time
arr = np.split(arr, 3) # spilts the array to three equal parts
for i in range(len(arr[0]), 9): # add 3 'NW' instead of 2 'NW'
arr = np.insert(arr, i, dummy, axis=1) # fill remaining slots with dummy value(NW)
print("{}\t{}\t{}".format("Date", "Holiday", "Values"))
# ---
i = 0
for numbers in arr:
gap = 0
numbers_index = 0
numbers_count = len(numbers) - 3 # count numbers without 3 `NW`
while i < delta.days + 1:
day = Start_date + timedelta(days=i)
i += 1
if day.strftime("%Y-%m-%d") in Holiday_List:
print("{}\t{}\t{}".format(day.strftime("%d-%m-%Y"), 1, hDay))
if numbers_index > 0: # don't count Holiday before displaying first number from list `data` (ie. '2020-01-01')
gap += 1
else:
value = numbers[numbers_index]
# always put number (!='NW') or put 'NW' when gap is too small (<9)
if value != 'NW' or gap < 9:
print("{}\t{}\t{}".format(day.strftime("%d-%m-%Y"), 0, value))
numbers_index += 1
gap += 1
# IDEA: maybe it could use `else:` to put `NW` without adding `NW` to `arr`
# exit loop if all numbers are displayed and gap is big enough
if numbers_index >= numbers_count and gap >= 9:
break
Answer provided by the #furas is less messier, you should study that.
Cheers mate, learned a lot actually!
I have an unbalanced panel that I'm trying to aggregate up to a regular, weekly time series. The panel looks as follows:
Group Date value
A 1/1/2000 5
A 1/17/2000 10
B 1/9/2000 3
B 1/23/2000 7
C 1/22/2000 20
To give a better sense of what I'm looking for, I'm including an intermediate step, which I'd love to skip if possible. Basically some data needs to be filled in so that it can be aggregated. As you can see, missing weeks in between observations are interpolated. All other values are set equal to zero.
Group Date value
A 1/1/2000 5
A 1/8/2000 5
A 1/15/2000 10
A 1/22/2000 0
B 1/1/2000 0
B 1/8/2000 3
B 1/15/2000 3
B 1/22/2000 7
C 1/1/2000 0
C 1/8/2000 0
C 1/15/2000 0
C 1/22/2000 20
The final result that I'm looking for is as follows:
Date value
1/1/2000 5 = 5 + 0 + 0
1/8/2000 8 = 5 + 3 + 0
1/15/2000 13 = 10 + 3 + 0
1/22/2000 27 = 0 + 7 + 20
I haven't gotten very far, managed to create a panel:
panel = df.set_index(['Group','week']).to_panel()
Unfortunately, if I try to resample, I get an error
panel.resample('W')
TypeError: Only valid with DatetimeIndex or PeriodIndex
Assume df is your second dataframe with weeks, you can try the following:
df.groupby('week').sum()['value']
The documentation of groupby() and its application is here. It's similar to group-by function in SQL.
To obtain the second dataframe from the first one, try the following:
Firstly, prepare a function to map the day to week
def d2w_map(day):
if day <=7:
return 1
elif day <= 14:
return 2
elif day <= 21:
return 3
else:
return 4
In the method above, days from 29 to 31 are considered in week 4. But you get the idea. You can modify it as needed.
Secondly, take the lists out from the first dataframe, and convert days to weeks
df['Week'] = df['Day'].apply(d2w_map)
del df['Day']
Thirdly, initialize your second dataframe with only columns of 'Group' and 'Week', leaving the 'value' out. Assume now your initialized new dataframe is result, you can now do a join
result = result.join(df, on=['Group', 'Week'])
Last, write a function to fill the Nan up in the 'value' column with the nearby element. The Nan is what you need to interpolate. Since I am not sure how you want the interpolation to work, I will leave it to you.
Here is how you can change d2w_map to convert string of date to integer of week
from datetime import datetime
def d2w_map(day_str):
return datetime.strptime(day_str, '%m/%d/%Y').weekday()
Returned value of 0 means Monday, 1 means Tuesday and so on.
If you have the package dateutil installed, the function can be more robust:
from dateutil.parser import parse
def d2w_map(day_str):
return parse(day_str).weekday()
Sometimes, things you want are already implemented by magic :)
Turns out the key is to resample a groupby object like so:
df_temp = (df.set_index('date')
.groupby('Group')
.resample('W', how='sum', fill_method='ffill'))
ts = (df_temp.reset_index()
.groupby('date')
.sum()[value])
Used this tab delimited test.txt:
Group Date value
A 1/1/2000 5
A 1/17/2000 10
B 1/9/2000 3
B 1/23/2000 7
C 1/22/2000 20
You can skip the intermediate datafile as follows. Don't have time now. Just play around with it to get it right.
import pandas as pd
import datetime
time_format = '%m/%d/%Y'
Y = pd.read_csv('test.txt', sep="\t")
dates = Y['Date']
dates_right_format = map(lambda s: datetime.datetime.strptime(s, time_format), dates)
values = Y['value']
X = pd.DataFrame(values)
X.index = dates_right_format
print X
X = X.sort()
print X
print X.resample('W', how=sum, closed='right', label='right')
Last print
value
2000-01-02 5
2000-01-09 3
2000-01-16 NaN
2000-01-23 37