We Keep Coding

Pandas: Generate date intervals between two dates with yearly reset

I am trying to generate 8 day intervals between two-time periods using pandas.date_range. In addition, when the 8 day interval exceeds the end of year (i.e., 365/366), I would like the range start to reset to the beginning of respective year. Below is the example code for just two years, however, I do plan to use it across several years, e.g., 2014-01-01 to 2021-01-01.
import pandas as pd
print(pd.date_range(start='2018-12-01', end='2019-01-31', freq='8D'))
Results in,
DatetimeIndex(['2018-12-01', '2018-12-09', '2018-12-17', '2018-12-25','2019-01-02', '2019-01-10', '2019-01-18', '2019-01-26'], dtype='datetime64[ns]', freq='8D')
However, I would like the start of the interval in 2019 to reset to the first day, e.g., 2019-01-01

You could loop creating a date_range up to the start of the next year for each year, appending them until you hit the end date.
import pandas as pd
from datetime import date
def date_range_with_resets(start, end, freq):
start = date.fromisoformat(start)
end = date.fromisoformat(end)
result = pd.date_range(start=start, end=start, freq=freq) # initialize result with just start date
next_year_start = start.replace(year=start.year+1, month=1, day=1)
while next_year_start < end:
result = result.append(pd.date_range(start=start, end=next_year_start, freq=freq))
start = next_year_start
next_year_start = next_year_start.replace(year=next_year_start.year+1)
result = result.append(pd.date_range(start=start, end=end, freq=freq))
return result[1:] # remove duplicate start date
start = '2018-12-01'
end = '2019-01-31'
date_range_with_resets(start, end, freq='8D')
Edit:
Here's a simpler way without using datetime. Create a date_range of years between start and end, then loop through those.
def date_range_with_resets(start, end, freq):
years = pd.date_range(start=start, end=end, freq='YS') # YS=year start
if len(years) == 0:
return pd.date_range(start=start, end=end, freq=freq)
result = pd.date_range(start=start, end=years[0], freq=freq)
for i in range(0, len(years) - 1):
result = result.append(pd.date_range(start=years[i], end=years[i+1], freq=freq))
result = result.append(pd.date_range(start=years[-1], end=end, freq=freq))
return result

Dates/months calculation

My below working code calculates date/month ranges, but I am using the Pandas library, which I want to get rid of.
import pandas as pd
dates=pd.date_range("2019-12","2020-02",freq='MS').strftime("%Y%m%d").tolist()
#print dates : ['20191101','20191201','20200101','20200201']
df=(pd.to_datetime(dates,format="%Y%m%d") + MonthEnd(1)).strftime("%Y%m%d").tolist()
#print df : ['20191130','20191231','20200131','20200229']
How can I rewrite this code without using Pandas?
I don't want to use Pandas library as I am triggering my job through Oozie and we don't have Pandas installed on all our nodes.

Pandas offers some nice functionalities when using datetimes which the standard library datetime module does not have (like the frequency or the MonthEnd). You have to reproduce these yourself.
import datetime as DT
def next_first_of_the_month(dt):
"""return a new datetime where the month has been increased by 1 and
the day is always the first
"""
new_month = dt.month + 1
if new_month == 13:
new_year = dt.year + 1
new_month = 1
else:
new_year = dt.year
return DT.datetime(new_year, new_month, day=1)
start, stop = [DT.datetime.strptime(dd, "%Y-%m") for dd in ("2019-11", "2020-02")]
dates = [start]
cd = next_first_of_the_month(start)
while cd <= stop:
dates.append(cd)
cd = next_first_of_the_month(cd)
str_dates = [d.strftime("%Y%m%d") for d in dates]
print(str_dates)
# prints: ['20191101', '20191201', '20200101', '20200201']
end_dates = [next_first_of_the_month(d) - DT.timedelta(days=1) for d in dates]
str_end_dates = [d.strftime("%Y%m%d") for d in end_dates]
print(str_end_dates)
# prints ['20191130', '20191231', '20200131', '20200229']
I used here a function to get a datetime corresponding to the first day of the next month of the input datetime. Sadly, timedelta does not work with months, and adding 30 days of course is not feasible (not all months have 30 days).
Then a while loop to get a sequence of fist days of the month until the stop date.
And to the get the end of the month, again get the next first day of the month fo each datetime in your list and subtract a day.

Pandas select last friday of each month [duplicate]

I've written this function to get the last Thursday of the month
def last_thurs_date(date):
month=date.dt.month
year=date.dt.year
cal = calendar.monthcalendar(year, month)
last_thurs_date = cal[4][4]
if month < 10:
thurday_date = str(year)+'-0'+ str(month)+'-' + str(last_thurs_date)
else:
thurday_date = str(year) + '-' + str(month) + '-' + str(last_thurs_date)
return thurday_date
But its not working with the lambda function.
datelist['Date'].map(lambda x: last_thurs_date(x))
Where datelist is
datelist = pd.DataFrame(pd.date_range(start = pd.to_datetime('01-01-2014',format='%d-%m-%Y')
, end = pd.to_datetime('06-03-2019',format='%d-%m-%Y'),freq='D').tolist()).rename(columns={0:'Date'})
datelist['Date']=pd.to_datetime(datelist['Date'])

Jpp already added the solution, but just to add a slightly more readable formatted string - see this awesome website.
import calendar
def last_thurs_date(date):
year, month = date.year, date.month
cal = calendar.monthcalendar(year, month)
# the last (4th week -> row) thursday (4th day -> column) of the calendar
# except when 0, then take the 3rd week (February exception)
last_thurs_date = cal[4][4] if cal[4][4] > 0 else cal[3][4]
return f'{year}-{month:02d}-{last_thurs_date}'
Also added a bit of logic - e.g. you got 2019-02-0 as February doesn't have 4 full weeks.

Scalar datetime objects don't have a dt accessor, series do: see pd.Series.dt. If you remove this, your function works fine. The key is understanding that pd.Series.apply passes scalars to your custom function via a loop, not an entire series.
def last_thurs_date(date):
month = date.month
year = date.year
cal = calendar.monthcalendar(year, month)
last_thurs_date = cal[4][4]
if month < 10:
thurday_date = str(year)+'-0'+ str(month)+'-' + str(last_thurs_date)
else:
thurday_date = str(year) + '-' + str(month) + '-' + str(last_thurs_date)
return thurday_date
You can rewrite your logic more succinctly via f-strings (Python 3.6+) and a ternary statement:
def last_thurs_date(date):
month = date.month
year = date.year
last_thurs_date = calendar.monthcalendar(year, month)[4][4]
return f'{year}{"-0" if month < 10 else "-"}{month}-{last_thurs_date}'

I know that a lot of time has passed since the date of this post, but I think it would be worth adding another option if someone came across this thread
Even though I use pandas every day at work, in that case my suggestion would be to just use the datetutil library. The solution is a simple one-liner, without unnecessary combinations.
from dateutil.rrule import rrule, MONTHLY, FR, SA
from datetime import datetime as dt
import pandas as pd
# monthly options expiration dates calculated for 2022
monthly_options = list(rrule(MONTHLY, count=12, byweekday=FR, bysetpos=3, dtstart=dt(2022,1,1)))
# last satruday of the month
last_saturday = list(rrule(MONTHLY, count=12, byweekday=SA, bysetpos=-1, dtstart=dt(2022,1,1)))
and then of course:
pd.DataFrame({'LAST_ST':last_saturdays}) #or whatever you need

This question answer Calculate Last Friday of Month in Pandas
This can be modified by selecting the appropriate day of the week, here freq='W-FRI'
I think the easiest way is to create a pandas.DataFrame using pandas.date_range and specifying freq='W-FRI.
W-FRI is Weekly Fridays
pd.date_range(df.Date.min(), df.Date.max(), freq='W-FRI')
Creates all the Fridays in the date range between the min and max of the dates in df
Use a .groupby on year and month, and select .last(), to get the last Friday of every month for every year in the date range.
Because this method finds all the Fridays for every month in the range and then chooses .last() for each month, there's not an issue with trying to figure out which week of the month has the last Friday.
With this, use pandas: Boolean Indexing to find values in the Date column of the dataframe that are in last_fridays_in_daterange.
Use the .isin method to determine containment.
pandas: DateOffset objects
import pandas as pd
# test data: given a dataframe with a datetime column
df = pd.DataFrame({'Date': pd.date_range(start=pd.to_datetime('2014-01-01'), end=pd.to_datetime('2020-08-31'), freq='D')})
# create a dateframe with all Fridays in the daterange for min and max of df.Date
fridays = pd.DataFrame({'datetime': pd.date_range(df.Date.min(), df.Date.max(), freq='W-FRI')})
# use groubpy and last, to get the last Friday of each month into a list
last_fridays_in_daterange = fridays.groupby([fridays.datetime.dt.year, fridays.datetime.dt.month]).last()['datetime'].tolist()
# find the data for the last Friday of the month
df[df.Date.isin(last_fridays_in_daterange)]

How to add business days in date excluding holidays

I have a dataframe (df) with start_date column's and add_days column's (=10). I want to create target_date (=start_date + add_days) excluding week-end and holidays (holidays as dataframe).
I do some research and I try this.
from datetime import date, timedelta
import datetime as dt
df["star_date"] = pd.to_datetime(df["star_date"])
Holidays['Date_holi'] = pd.to_datetime(Holidays['Date_holi'])
def date_by_adding_business_days(from_date, add_days, holidays):
business_days_to_add = add_days
current_date = from_date
while business_days_to_add > 0:
current_date += datetime.timedelta(days=1)
weekday = current_date.weekday()
if weekday >= 5: # sunday = 6
continue
if current_date in holidays:
continue
business_days_to_add -= 1
return current_date
#demo:
base["Target_date"]=date_by_adding_business_days(df["start_date"], 10, Holidays['Date_holi'])
but i get this error:
AttributeError: 'Series' object has no attribute 'weekday'
Thanks you for your help.

The comments by ALollz are very valid; customizing your date during creation to only keep what is defined as business day for your problem would be optimal.
However, I assume that you cannot define the business day beforehand and that you need to solve the problem with the data frame constructed as is.
Here is one possible solution:
import pandas as pd
import numpy as np
from datetime import timedelta
# Goal is to offset a start date by N business days (weekday + not a holiday)
# Here we fake the dataset as it was not provided
num_row = 1000
df = pd.DataFrame()
df['start_date'] = pd.date_range(start='1/1/1979', periods=num_row, freq='D')
df['add_days'] = pd.Series([10]*num_row)
# Define what is a week day
week_day = [0,1,2,3,4] # Monday to Friday
# Define what is a holiday with month and day without year (you can add more)
holidays = ['10-30','12-24']
def add_days_to_business_day(df, week_day, holidays, increment=10):
'''
modify the dataframe to increment only the days that are part of a weekday
and not part of a pre-defined holiday
>>> add_days_to_business_day(df, [0,1,2,3,4], ['10-31','12-31'])
this will increment by 10 the days from Monday to Friday excluding Halloween and new year-eve
'''
# Increment everything that is in a business day
df.loc[df['start_date'].dt.dayofweek.isin(week_day),'target_date'] = df['start_date'] + timedelta(days=increment)
# Remove every increment done on a holiday
df.loc[df['start_date'].dt.strftime('%m-%d').isin(holidays), 'target_date'] = np.datetime64('NaT')
add_days_to_business_day(df, week_day, holidays)
df
To Note: I'm not using the 'add_days' column since its just a repeated value. I am instead using a parameter for my function increment which will increment by N number of days (with a default of N = 10).
Hope it helps!

How do I divide a date range into months in Python?

I have the following date range:
begin: 2018-02-15
end: 2018-04-23
I want to achieve the following:
["2018-02-15 - 2018-02-28", "2018-03-01 - 2018-03-31", "2018-04-01 - 2018-04-23"]
Essentially, I want to divide a given date range into months. I can't think of a way to accomplish this in Python.
I have considered the solution here, however, this splits the date range based on a specified interval. I want to be able to split a date range dynamically.
Hence, given a date range from 15 February 2018 to 23 April 2018, I want to be able to get the individual months in the range, like so:
15 February 2018 to 28 February 2018
01 March 2018 to 31 March 2018
01 April 2018 to 23 April 2018

In a loop; starting at the first day continually add one day till you get to the end date; whenever the month changes save the dates.
import datetime
begin = '2018-02-15'
end = '2018-04-23'
dt_start = datetime.datetime.strptime(begin, '%Y-%m-%d')
dt_end = datetime.datetime.strptime(end, '%Y-%m-%d')
one_day = datetime.timedelta(1)
start_dates = [dt_start]
end_dates = []
today = dt_start
while today <= dt_end:
#print(today)
tomorrow = today + one_day
if tomorrow.month != today.month:
start_dates.append(tomorrow)
end_dates.append(today)
today = tomorrow
end_dates.append(dt_end)
out_fmt = '%d %B %Y'
for start, end in zip(start_dates,end_dates):
print('{} to {}'.format(start.strftime(out_fmt), end.strftime(out_fmt)))
Result:
>>>
15 February 2018 to 28 February 2018
01 March 2018 to 31 March 2018
01 April 2018 to 23 April 2018
>>>
You could probably figure out a way to get a range of months between the start and end dates; create a datetime object for the first day of each of those months store them and the days just prior to them. Dates spanning a change of year might be problematic though.

To work with convenient date objects, always use the standard module datetime. This wraps your string formatted dates, and allows easier calculations as well as tailored output formatting.
Unfortunately, it seems to miss one important piece of information: the last day of each month, given a year (which is necessary for Februari). There is an additional module calendar which returns the last day for a month, but since this is all you need of it and there is a simple datetime based function that does the same thing, I chose the latter.
With that, you can set any begin date and append it to your list, together with its last day of that month, then set begin to the next month's 1st and continue until you pass end.
A caveat/finetuning: I realized it would not work if both begin and end fall inside the same month. That needs an interim check, so I changed my initial while begin < end to while True and moved the check for crossing the end date into a separate line.
Also, to cross a year needs a separate test again, because else the statement month+1 will fail on December.
import datetime
# borrowed from https://stackoverflow.com/a/13565185
# as noted there, the calendar module has a function of its own
def last_day_of_month(any_day):
next_month = any_day.replace(day=28) + datetime.timedelta(days=4) # this will never fail
return next_month - datetime.timedelta(days=next_month.day)
begin = "2018-02-15"
end = "2018-04-23"
def monthlist(begin,end):
begin = datetime.datetime.strptime(begin, "%Y-%m-%d")
end = datetime.datetime.strptime(end, "%Y-%m-%d")
result = []
while True:
if begin.month == 12:
next_month = begin.replace(year=begin.year+1,month=1, day=1)
else:
next_month = begin.replace(month=begin.month+1, day=1)
if next_month > end:
break
result.append ([begin.strftime("%Y-%m-%d"),last_day_of_month(begin).strftime("%Y-%m-%d")])
begin = next_month
result.append ([begin.strftime("%Y-%m-%d"),end.strftime("%Y-%m-%d")])
return result
date_list = monthlist(begin,end)
print (date_list)
results in
[ ['2018-02-15', '2018-02-28'],
['2018-03-01', '2018-03-31'],
['2018-04-01', '2018-04-23'] ]
(slightly formatted for readability only)

If you don't mind using pandas, there's a nice helper date_range that will achieve what you want:
import pandas as pd
start = pd.Timestamp('20180215')
end = pd.Timestamp('20180423')
parts = list(pd.date_range(start, end, freq='M'))
# parts = [Timestamp('2018-02-28 00:00:00', freq='M'), Timestamp('2018-03-31 00:00:00', freq='M')]
if start != parts[0]:
parts.insert(0, start)
if end != parts[-1]:
parts.append(end)
parts[0] -= pd.Timedelta('1d') # we add back one day later
pairs = zip(map(lambda d: d + pd.Timedelta('1d'), parts[:-1]), parts[1:])
pairs_str = list(map(lambda t: t[0].strftime('%Y-%m-%d') + ' - ' + t[1].strftime('%Y-%m-%d'), pairs))
# pairs_str = ['2018-02-15 - 2018-02-28', '2018-03-01 - 2018-03-31', '2018-04-01 - 2018-04-23']

Using python calendar and accounting for change of the year
import calendar
from datetime import datetime
begin = '2018-02-15'
end= '2018-04-23'
begin_year, begin_month, begin_date = [int(i) for i in begin.split("-")]
end_year, end_month, end_date = [int(i) for i in end.split("-")]
years = end_year - begin_year
# if date range contains more than single year, we calculate total months
if years:
months = (12 - begin_month) + end_month + (12 * (years - 1))
else:
months = end_month - begin_month
dates = []
month = begin_month
year = begin_year
def create_datetime_object(y, m, d):
return datetime.strptime('{}-{}-{}'.format(y, m, d), '%Y-%m-%d')
# append the first date
dates.append(create_datetime_object(begin_year, begin_month, begin_date))
for i in range(months+1):
days_in_month = calendar.monthrange(year, month)[-1]
if month == begin_month and year == begin_year:
dates.append(create_datetime_object(begin_year, begin_month, days_in_month))
elif month == end_month and year == end_year:
dates.append(create_datetime_object(end_year, end_month, 1))
else:
dates.append(create_datetime_object(year, month, 1))
dates.append(create_datetime_object(year, month, days_in_month))
if month == 12:
month = 0
year += 1
month += 1
# append the last date
dates.append(create_datetime_object(end_year, end_month, end_date))
And to get a list in the question, we could do something like -
dates = [datetime.strftime(dt, '%Y-%m-%d') for dt in dates]

I had to do a similar manipulation and ended up building this function. I tested it on different use cases (different years, same month...) and it's working well.
It is inspired from S.Lott answer here
Creating a range of dates in Python
import datetime
def get_segments(start_date, end_date):
"""
Divides input date range into associated months periods
Example:
Input: start_date = 2018-02-15
end_date = 2018-04-23
Output:
["2018-02-15 - 2018-02-28",
"2018-03-01 - 2018-03-31",
"2018-04-01 - 2018-04-23"]
"""
curr_date = start_date
curr_month = start_date.strftime("%m")
segments = []
loop = (curr_date!=end_date)
days_increment = 1
while loop:
# Get incremented date with 1 day
curr_date = start_date + datetime.timedelta(days=days_increment)
# Get associated month
prev_month = curr_month
curr_month = curr_date.strftime("%m")
# Add to segments if new month
if prev_month!=curr_month:
# get start of segment
if not segments:
start_segment = start_date
else:
start_segment = segments[-1][1] + datetime.timedelta(days=1)
# get end of segment
end_segment = curr_date - datetime.timedelta(days=1)
# define and add segment
segment = [start_segment, end_segment]
segments.append(segment)
# stop if last day reached
loop = (curr_date!=end_date)
# increment added days
days_increment += 1
if not segments or segments[-1][1]!=end_date:
if not segments:
start_last_segment = start_date
else:
start_last_segment = segments[-1][1] + datetime.timedelta(days=1)
last_segment = [start_last_segment, end_date]
segments.append(last_segment)
for i in range(len(segments)):
segments[i][0] = segments[i][0].strftime("%Y-%m-%d")
segments[i][1] = segments[i][1].strftime("%Y-%m-%d")
return segments
Here is an example:
start_date = datetime.datetime(2020, 5, 27)
end_date = datetime.datetime(2021, 3, 1)
segments = get_segments(start_date, end_date)
for seg in segments:
print(seg)
Output:
['2020-05-27', '2020-05-31']
['2020-06-01', '2020-06-30']
['2020-07-01', '2020-07-31']
['2020-08-01', '2020-08-31']
['2020-09-01', '2020-09-30']
['2020-10-01', '2020-10-31']
['2020-11-01', '2020-11-30']
['2020-12-01', '2020-12-31']
['2021-01-01', '2021-01-31']
['2021-02-01', '2021-02-28']
['2021-03-01', '2021-03-01']

I extend the solution by #wwii
Now you will not have duplicate start and/or end dates
def date_range_split_monthly(begin, end):
dt_start = datetime.strptime(begin, '%Y-%m-%d')
dt_end = datetime.strptime(end, '%Y-%m-%d')
one_day = timedelta(1)
start_dates = [dt_start]
end_dates = []
today = dt_start
while today <= dt_end:
# print(today)
tomorrow = today + one_day
if tomorrow.month != today.month:
if tomorrow <= dt_end:
start_dates.append(tomorrow)
end_dates.append(today)
today = tomorrow
end_dates.append(dt_end)
return start_dates, end_dates

For people using Pendulum :
import pendulum
start = pendulum.now().subtract(months=6)
end = pendulum.today()
period = pendulum.period(start, end)
time_ranges = list(period.range("months"))
arr = []
for index, dt in enumerate(time_ranges):
if index < len(time_ranges) - 1:
start_range = time_ranges[index].format("YYYY-MM-D")
end_range = time_ranges[index + 1].format("YYYY-MM-D")
litt = F"{start_range} - {end_range}"
print(litt)
arr.append(litt)
print(arr)
More about period here

i'm quoting the comment of Kiran Subbaraman, just with addition of the exact keyword (otherwise, whole months will be returned even if ranges fall beyond the start or the end).
#!pip install arrow
from arrow import Arrow
Arrow.span_range('month', start, end, exact=True)