I have a current script that fills out a template word doc using MailMerge. The issue I noticed is that after iterating through 15 rows/documents, the process works correctly, but in my Task Manager, there is 15 open idle Microsoft Word applications open.. This is not ideal as this will be used daily and should shut down the app after the doc has been created. Any idea on what to do? Will the Word app close after being idle for x amount of time or should I try to programmatically fix this?
for i in range(len(exampleData)):
#Open Template
template = r'exampletemplate.docx'
#Create Editable Doc
document = MailMerge(template)
#Create Word Naming Convention - Whatever the files should be named with
InsuredName = exampleData.loc[i, "InsuredName"]
PolicyNumber = exampleData.loc[i, "PolicyNumber"]
#Write the values into the doc
document.merge(
InsuredName = InsuredName,
Address1 = exampleData.loc[i, "Address1"],
Address2 = exampleData.loc[i, "Address2"],
ProducerName = exampleData.loc[i, "ProducerName"],
PolicyNumber = PolicyNumber,
ExpirationDate = datetime.strptime((str(exampleData.loc[i, "ExpirationDate"])), '%Y-%m-%d %H:%M:%S').strftime('%m/%d/%Y'),
DueDate = datetime.strptime((str(exampleData.loc[i, "DueDate"])), '%Y-%m-%d %H:%M:%S').strftime('%m/%d/%Y'),
EffectiveDate = datetime.strptime((str(exampleData.loc[i, "EffectiveDate"])), '%Y-%m-%d %H:%M:%S').strftime('%m/%d/%Y'),
ReportPeriodBegin = datetime.strptime((str(exampleData.loc[i, "ReportPeriodBegin"])), '%Y-%m-%d %H:%M:%S').strftime('%m/%d/%Y'),
ReportPeriodEnd = datetime.strptime((str(exampleData.loc[i, "ReportPeriodEnd"])), '%Y-%m-%d %H:%M:%S').strftime('%m/%d/%Y')
)
document.write(r'\\endpath\\' + PolicyNumber + '_' + InsuredName + '.docx')
I was facing the same problem. You're doing the process but you're not closing the template file.
try this:
document.close()
documentation
I was trying to get the number of insurance claims by Day and what time it occurred. For Example, it is March 22 today and there were claims on 8 am another in 12noon. I was trying to get all the claim count this day and what hour did the claims occured.
I did try using modified_at__hour but it doesn't show what I needed.
Sample code for claims per week and show what day it was claimed
class GetClaimsCompare_week(APIView):
def get_claim_count(self, claims_data, claim_type):
claims_count = claims_data.filter(claim_type = claim_type).count()
return claims_count
def get_claims_type(self, claims_per_day):
return claims_per_day.claim_type
#return claims_data.claim_type
def get(self, request):
today = datetime.now()
claims_data = Claims.objects.filter(modified_at__day = today.day)
claims_per_day = claims_data.annotate(day = TruncDay('modified_at')).values('day').annotate(claim_type=Count('id'))
labels = []
claims_type = list(set(map(self.get_claims_type, claims_data)))
final = {}
for claims in claims_per_day:
labels.append(claims['day'])
#data.append(claims['claims'])
for claim_type in claims_type:
final[claim_type] = self.get_claim_count(claims_data, claim_type)
context = {
'Date(s)': labels,
'Claims on this date': final
}
return Response(context)
For the node 'TransactionDate' i have a logic before updating it for policy"POL000002NGJ".
The logic i am trying to implement is If existing 'TransactionDate' < than today, then add 5 days with current value and parse it to xml.
Transaction Date Format in XML : 2020-03-23T10:56:15.00
Please Note that, If i parsing the DateTime value like below, It works good But i dont want to hardcode the value... I want to Parse it as a string object to handle for any datetime in format ""%Y-%m-%dT%H:%M:%S.%f""...
# <TransactionDate>
today = datetime.now()
TransactionDate = doc.find('TransactionDate')
Date = '2020-03-24T10:56:15.00'
previous_update = datetime.strptime(Date, "%Y-%m-%dT%H:%M:%S.%f")
if previous_update < today:
today = previous_update - timedelta(days=-5)
TransactionDate = today.strftime("%Y-%m-%dT%H:%M:%S.%f")
Below code while parsing it as a DateTime Object, I have an issue.. I got struck here and referenced other answers in stackoverflow and python forums, But still i got struct up here and unable to resolve the issue...
if any help to fix will be a great helpful. Thanks. Below code using lxml and getting help to support below code will helpful. Because i already completed for other nodes. My understanding is Date variable is calling as None.. But struck here to fix.. Please help..
# <TransactionDate>
today = datetime.now()
TransactionDate = doc.find('TransactionDate')
Date = str(TransactionDate)
previous_update = datetime.strptime(Date, "%Y-%m-%dT%H:%M:%S.%f")
if previous_update < today:
today = previous_update - timedelta(days=-5)
TransactionDate = today.strftime("%Y-%m-%dT%H:%M:%S.%f")
Full Code is Below
from lxml import etree
from datetime import datetime, timedelta
import random, string
doc = etree.parse(r'C:\Users\python.xml')
# <PolicyId> - Random generated policy number
Policy_Random_Choice = 'POL' + ''.join(random.choices(string.digits, k=6)) + 'NGJ'
# <TransactionDate>
today = datetime.now()
TransactionDate = doc.find('TransactionDate')
Date = str(TransactionDate)
previous_update = datetime.strptime(Date, "%Y-%m-%dT%H:%M:%S.%f")
if previous_update < today:
today = previous_update - timedelta(days=-5)
TransactionDate = today.strftime("%Y-%m-%dT%H:%M:%S.%f")
#Parsing the Variables
replacements = [Policy_Random_Choice , TransactionDate ]
targets = doc.xpath('//ROW[PolicyId="POL000002NGJ"]')
for target in targets:
target.xpath('./PolicyId')[0].text = replacements[0]
target.xpath('.//TransactionDate')[0].text = replacements[1]
print(etree.tostring(doc).decode())
Sample XML
<TABLE>
<ROW>
<PolicyId>POL000002NGJ</PolicyId>
<BusinessCoverageCode>COV00002D3X1</BusinessCoverageCode>
<TransactionDate>2020-03-23T10:56:15.00</TransactionDate>
</ROW>
<ROW>
<PolicyId>POL111111NGJ</PolicyId>
<BusinessCoverageCode>COV00002D3X4</BusinessCoverageCode>
<TransactionDate>2020-03-23T10:56:15.00</TransactionDate>
</ROW>
</TABLE>
Maybe the find method is wrong. Try this one
# <TransactionDate>
today = datetime.now()
TransactionDate = doc.xpath('//ROW/TransactionDate') # Change find to xpath
Date = str(TransactionDate[0].text) # Use the first one
previous_update = datetime.strptime(Date, "%Y-%m-%dT%H:%M:%S.%f")
I have an exsting script which is used to extract data for T and T-1 day. I tried to modify it extract two weeks data, but the script is not able to search the dates other than current date
Check the code section :
def parse_gov():
reject_patterns = generate_reject_patterns()
today_str = date.today().strftime('%Y.%m.%d')
yesterday = date.today() - timedelta(days=14)
yesterday_str = yesterday.strftime('%Y.%m.%d')
#query_date = date.today()
#query_date = yesterday
query_last = '''select last sym, last source, last lastTimeStamp, last objectName...
query_all = '''select objectName, IONrecType, sym, source, lastTimeStamp, objectName, ....
def query(q, query_date):
if query_date = date.today() - timedelta(days=14):
date_clause = "date <= {date}, ".format(date = query_date)
kdbport = '1000' ( historical database)
else:
date_clause = ""
kdbport = '1001' (current database)
Your else part is not triggering because the comparison is not happening over there.
you need to change
query_date = date.today() - timedelta(days=14):
to
query_date == (date.today() - timedelta(days=14)):
You always use == for comparison, = will assign the value to the variable.
I have a class assignment to write a python program to download end-of-day data last 25 years the major global stock market indices from Yahoo Finance:
Dow Jones Index (USA)
S&P 500 (USA)
NASDAQ (USA)
DAX (Germany)
FTSE (UK)
HANGSENG (Hong Kong)
KOSPI (Korea)
CNX NIFTY (India)
Unfortunately, when I run the program an error occurs.
File "C:\ProgramData\Anaconda2\lib\site-packages\yahoofinancials__init__.py", line 91, in format_date
form_date = datetime.datetime.fromtimestamp(int(in_date)).strftime('%Y-%m-%d')
ValueError: timestamp out of range for platform localtime()/gmtime() function
If you see below, you can see the code that I have written. I'm trying to debug my mistakes. Can you help me out please? Thanks
from yahoofinancials import YahooFinancials
import pandas as pd
# Select Tickers and stock history dates
index1 = '^DJI'
index2 = '^GSPC'
index3 = '^IXIC'
index4 = '^GDAXI'
index5 = '^FTSE'
index6 = '^HSI'
index7 = '^KS11'
index8 = '^NSEI'
freq = 'daily'
start_date = '1993-06-30'
end_date = '2018-06-30'
# Function to clean data extracts
def clean_stock_data(stock_data_list):
new_list = []
for rec in stock_data_list:
if 'type' not in rec.keys():
new_list.append(rec)
return new_list
# Construct yahoo financials objects for data extraction
dji_financials = YahooFinancials(index1)
gspc_financials = YahooFinancials(index2)
ixic_financials = YahooFinancials(index3)
gdaxi_financials = YahooFinancials(index4)
ftse_financials = YahooFinancials(index5)
hsi_financials = YahooFinancials(index6)
ks11_financials = YahooFinancials(index7)
nsei_financials = YahooFinancials(index8)
# Clean returned stock history data and remove dividend events from price history
daily_dji_data = clean_stock_data(dji_financials
.get_historical_stock_data(start_date, end_date, freq)[index1]['prices'])
daily_gspc_data = clean_stock_data(gspc_financials
.get_historical_stock_data(start_date, end_date, freq)[index2]['prices'])
daily_ixic_data = clean_stock_data(ixic_financials
.get_historical_stock_data(start_date, end_date, freq)[index3]['prices'])
daily_gdaxi_data = clean_stock_data(gdaxi_financials
.get_historical_stock_data(start_date, end_date, freq)[index4]['prices'])
daily_ftse_data = clean_stock_data(ftse_financials
.get_historical_stock_data(start_date, end_date, freq)[index5]['prices'])
daily_hsi_data = clean_stock_data(hsi_financials
.get_historical_stock_data(start_date, end_date, freq)[index6]['prices'])
daily_ks11_data = clean_stock_data(ks11_financials
.get_historical_stock_data(start_date, end_date, freq)[index7]['prices'])
daily_nsei_data = clean_stock_data(nsei_financials
.get_historical_stock_data(start_date, end_date, freq)[index8]['prices'])
stock_hist_data_list = [{'^DJI': daily_dji_data}, {'^GSPC': daily_gspc_data}, {'^IXIC': daily_ixic_data},
{'^GDAXI': daily_gdaxi_data}, {'^FTSE': daily_ftse_data}, {'^HSI': daily_hsi_data},
{'^KS11': daily_ks11_data}, {'^NSEI': daily_nsei_data}]
# Function to construct data frame based on a stock and it's market index
def build_data_frame(data_list1, data_list2, data_list3, data_list4, data_list5, data_list6, data_list7, data_list8):
data_dict = {}
i = 0
for list_item in data_list2:
if 'type' not in list_item.keys():
data_dict.update({list_item['formatted_date']: {'^DJI': data_list1[i]['close'], '^GSPC': list_item['close'],
'^IXIC': data_list3[i]['close'], '^GDAXI': data_list4[i]['close'],
'^FTSE': data_list5[i]['close'], '^HSI': data_list6[i]['close'],
'^KS11': data_list7[i]['close'], '^NSEI': data_list8[i]['close']}})
i += 1
tseries = pd.to_datetime(list(data_dict.keys()))
df = pd.DataFrame(data=list(data_dict.values()), index=tseries,
columns=['^DJI', '^GSPC', '^IXIC', '^GDAXI', '^FTSE', '^HSI', '^KS11', '^NSEI']).sort_index()
return df
Your problem is your datetime stamps are in the wrong format. If you look at the error code it clugely tells you:
datetime.datetime.fromtimestamp(int(in_date)).strftime('%Y-%m-%d')
Notice the int(in_date) part?
It wants the unix timestamp. There are several ways to get this, out of the time module or the calendar module, or using Arrow.
import datetime
import calendar
date = datetime.datetime.strptime("1993-06-30", "%Y-%m-%d")
start_date = calendar.timegm(date.utctimetuple())
* UPDATED *
OK so I fixed up to the dataframes portion. Here is my current code:
# Select Tickers and stock history dates
index = {'DJI' : YahooFinancials('^DJI'),
'GSPC' : YahooFinancials('^GSPC'),
'IXIC':YahooFinancials('^IXIC'),
'GDAXI':YahooFinancials('^GDAXI'),
'FTSE':YahooFinancials('^FTSE'),
'HSI':YahooFinancials('^HSI'),
'KS11':YahooFinancials('^KS11'),
'NSEI':YahooFinancials('^NSEI')}
freq = 'daily'
start_date = '1993-06-30'
end_date = '2018-06-30'
# Clean returned stock history data and remove dividend events from price history
daily = {}
for k in index:
tmp = index[k].get_historical_stock_data(start_date, end_date, freq)
if tmp:
daily[k] = tmp['^{}'.format(k)]['prices'] if 'prices' in tmp['^{}'.format(k)] else []
Unfortunately I had to fix a couple things in the yahoo module. For the class YahooFinanceETL:
#staticmethod
def format_date(in_date, convert_type):
try:
x = int(in_date)
convert_type = 'standard'
except:
convert_type = 'unixstamp'
if convert_type == 'standard':
if in_date < 0:
form_date = datetime.datetime(1970, 1, 1) + datetime.timedelta(seconds=in_date)
else:
form_date = datetime.datetime.fromtimestamp(int(in_date)).strftime('%Y-%m-%d')
else:
split_date = in_date.split('-')
d = date(int(split_date[0]), int(split_date[1]), int(split_date[2]))
form_date = int(time.mktime(d.timetuple()))
return form_date
AND:
# private static method to scrap data from yahoo finance
#staticmethod
def _scrape_data(url, tech_type, statement_type):
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")
script = soup.find("script", text=re.compile("root.App.main")).text
data = loads(re.search("root.App.main\s+=\s+(\{.*\})", script).group(1))
if tech_type == '' and statement_type != 'history':
stores = data["context"]["dispatcher"]["stores"]["QuoteSummaryStore"]
elif tech_type != '' and statement_type != 'history':
stores = data["context"]["dispatcher"]["stores"]["QuoteSummaryStore"][tech_type]
else:
if "HistoricalPriceStore" in data["context"]["dispatcher"]["stores"] :
stores = data["context"]["dispatcher"]["stores"]["HistoricalPriceStore"]
else:
stores = data["context"]["dispatcher"]["stores"]["QuoteSummaryStore"]
return stores
You will want to look at the daily dict, and rewrite your build_data_frame function, which it should be a lot simpler now since you are working with a dictionary already.
I am actually the maintainer and author of YahooFinancials. I just saw this post and wanted to personally apologize for the inconvenience and let you all know I will be working on fixing the module this evening.
Could you please open an issue on the module's Github page detailing this?
It would also be very helpful to know which version of python you were running when you encountered these issues.
https://github.com/JECSand/yahoofinancials/issues
I am at work right now, however as soon as I get home in ~7 hours or so I will attempt to code a fix and release it. I'll also work on the exception handling. I try my best to maintain this module, but my day (and often night time) job is rather demanding. I will report back with the final results of these fixes and publish to pypi when it is done and stable.
Also if anyone else has any feedback or personal fixes made you can offer, it would be a huge huge help in fixing this. Proper credit will be given of course. I am also in desperate need of contributers, so if anyone is interested in that as well let me know. I am really wanting to take YahooFinancials to the next level and have this project become a stable and reliable alternative for free financial data for python projects.
Thank you for your patience and for using YahooFinancials.