We Keep Coding

How to fine tune the redundant code in python

I am trying to create two set of databases every time inside my python script for the same of which I have written the below set of code which looks redundant to me since I am initializing the variable ext 2 times and hence if anyone can suggest some better alternatives, that would be really helpful.
def create_datasets(database, ext):
try:
dataset = "bq --location=US mk -d " + database + ext
try:
return_cd, out, err = run_sys_command(dataset)
except Exception as e:
print(e)
except Exception as e:
print(e)
raise
ext = ''
create_datasets(database, ext)
ext = '_stg'
create_datasets(database, ext)

Use a loop?
for ext in ['', '_stg']:
create_datasets(database, ext)
About your function:
def create_datasets(database, ext):
try:
dataset = f"bq --location=US mk -d {database}{ext}"
return_cd, out, err = run_sys_command(dataset)
except Exception as e: # <- you should catch sub exception!
print(e)

Any exception Exception raised in your function is caught and handled by the inner try block. The outer therefore seems redundant.
def create_datasets(database, ext):
try:
dataset = "bq --location=US mk -d " + database + ext
return_cd, out, err = run_sys_command(dataset)
except Exception as e:
print(e)

How to use both default arguments and asterisk (*) in a function definition

I'm trying to log some message into a text file, and may also log an error if there was one.
I want to log to multiple files, but only print once, and using the same function.
Current Implementation:
def log_and_sysout(log_file, message, e=''):
'''
Writes logs and outputs to terminal
Params:
log_file: File to log to
message: -
e: Error, if there is one
'''
sys.stdout.write(message)
with open(log_file, "a") as f:
f.write(message + str(e))
try:
# Logic
success_message = 'Translations: Success \n'
log_and_sysout(log_path_1, success_message)
log_and_sysout(log_path_2, success_message)
except Exception as e:
fail_message = 'Translations: Fail \n'
log_and_sysout(log_path_1, fail_message, e)
log_and_sysout(log_path_2, fail_message, e)
I want something like this:
def log_and_sysout(*log_files, message, e=''):
sys.stdout.write(message)
for file in log_files:
with open(file, "a") as f:
f.write(message + str(e))
try:
# Logic
success_message = 'Translations: Success \n'
log_and_sysout(log_path_1, log_path_2, success_message)
except Exception as e:
fail_message = 'Translations: Fail \n'
log_and_sysout(log_path_1, log_path_2, fail_message, e)

Your are almost there:
success_message = 'Translations: Success \n'
log_and_sysout(log_path_1, log_path_2, message=success_message)
However, I would suggest you to define your function like that to avoid the parameter assignment:
def log_and_sysout(message, e=‘’, *list_file)
Last but not least, you might consider the library ‘logging’ for what you are trying to do.

Edit:
It seems that the correct format is
def log_and_sysout(message, *log_files, e=''):
sys.stdout.write(message)
for file in log_files:
with open(file, "a") as f:
f.write(message + str(e))
log_and_sysout(message, log_path_1, log_path_2)

While calling function with pool, It shows the variable is not defined which is there in main (global)

May be my question looks simple (or the Bug might be minor) but I could not be able find the bug, really I struggled a lot to figure out the issue.
I've created a Framework to Extract the Data from Salesforce with Simple Salesforce package, but I've encountered with the bug when I'm using multiprocessing.
My code is pretty much straight forward but tedious. I don't want to paste entire code here, So here is my Code from my GitHub.
Issue:
When I'm calling this Extract Data function with Pool, the variable which is there in the __name__ == '__main__' is not working.
In Other words I'm getting, NameError: name 'SFAPI' is not defined - But It's there in the main as Global Variable and It's working without pool (A single call).
Execution Example:
python "E:\Documents\myPy\SF Project\sf_extraction.py" -pr data_extraction -tn Opportunity Account
Small Snippet from my code, where I'm getting issues:
def ExtractData(table_name):
logging.info('Extract Data for Table Name: ' + table_name + ' at ' + getCurrDatetime())
try:
rec_count = getRecordCount(table_name)
print(rec_count)
if int(rec_count) == 0:
logging.info('There is no data to Extract for {}'.format(table_name))
else:
soql = SFAPI.CreateSOQL(table_name)
data = SFAPI.ExecuteSOQL(soql, is_count=0)
extract_file_nm = table_name + '_' + db_name + '_' + sc_name + '_' + curr_datetime + '.csv'
print(data)
print(type(data))
extract_file = os.path.expanduser(os.path.join(script_path,extract_file_nm))
data.to_csv(extract_file, index=False)
logging.info('Data has been extrcated as {} at {}'.format(extract_file, getCurrDatetime()))
except Exception as e:
logging.info('Error in Extraction')
err_msg = "FATAL_ERROR: In the ExtractData Function : {0}\n\n{1}".format(e, traceback.format_exc())
raise Exception(str(err_msg))
Place or Snippet from where I'm calling this:
if __name__ == '__main__':
try:
SFAPI = SalesforceAPICall(username=config['username'],
password=config['password'],
security_token=config['sf_token'],
)
if len(table_name) != 0 and 'data_extraction' in process_nm:
try:
if len(table_name) == 1:
print(table_name[0])
ExtractData(table_name[0])
if type(table_name) == list and len(table_name) > 1:
#p = Pool(processes=int(processes))
print('Calling Pool : ' + str(os.cpu_count()))
#out = p.map(ExtractData, table_name)
#p.close()
#p.join()
p = Pool()
print(table_name)
x = p.map(ExtractData, table_name)
x.get()
p.close()
p.join()
except Exception as e:
if len(table_name) > 1:
p.terminate()
p.join()
logging.error("Process Failed - " + str(e))
except Exception as e:
chk_err('FATAL_ERROR: ' + " from main exception : {0}\n\n{1}".format(e, traceback.format_exc()))
You can very well refer my code in GitHub if it looks clumsy or if you feel this is not enough amount of information to fix.
Again it might be a small Bug, Hope You Understand what I'm trying to convey !!! Thanks in
Advance !!!
Regards,
Parvathirajan Natarajan

Python custom 404 response error

I wrote a hiscore checker for a game that I play, basically you enter a list of usernames into the .txt file & it outputs the results in found.txt.
However if the page responds a 404 it throws an error instead of returning output as " 0 " & continuing with the list.
Example of script,
#!/usr/bin/python
import urllib2
def get_total(username):
try:
req = urllib2.Request('http://services.runescape.com/m=hiscore/index_lite.ws?player=' + username)
res = urllib2.urlopen(req).read()
parts = res.split(',')
return parts[1]
except urllib2.HTTPError, e:
if e.code == 404:
return "0"
except:
return "err"
filename = "check.txt"
accs = []
handler = open(filename)
for entry in handler.read().split('\n'):
if "No Displayname" not in entry:
accs.append(entry)
handler.close()
for account in accs:
display_name = account.split(':')[len(account.split(':')) - 1]
total = get_total(display_name)
if "err" not in total:
rStr = account + ' - ' + total
handler = open('tried.txt', 'a')
handler.write(rStr + '\n')
handler.close()
if total != "0" and total != "49":
handler = open('found.txt', 'a')
handler.write(rStr + '\n')
handler.close()
print rStr
else:
print "Error searching"
accs.append(account)
print "Done"
HTTPERROR exception that doesn't seem to be working,
except urllib2.HTTPError, e:
if e.code == 404:
return "0"
except:
return "err"
Error response shown below.
Now I understand the error shown doesn't seem to be related to a response of 404, however this only occurs with users that return a 404 response from the request, any other request works fine. So I can assume the issue is within the 404 response exception.
I believe the issue may lay in the fact that the 404 is a custom page which you get redirected too?
so the original page is " example.com/index.php " but the 404 is " example.com/error.php "?
Not sure how to fix.
For testing purposes, format to use is,
ID:USER:DISPLAY
which is placed into check.txt

It seems that total can end up being None. In that case you can't check that it has 'err' in it. To fix the crash, try changing that line to:
if total is not None and "err" not in total:
To be more specific, get_total is returning None, which means that either
parts[1] is None or
except urllib2.HTTPError, e: is executed but e.code is not 404.
In the latter case None is returned as the exception is caught but you're only dealing with the very specific 404 case and ignoring other cases.

How to get python default exception message in exception handling

When I handle an exception in python
try:
a = dict()
a[1]
except Exception as e:
print str(e)
It prints
1
I expect it to print
KeyError: 1
Is there a way to retrieve the default error message ?

Instead of this:
print str(e)
do this:
print(type(e).__name__ + ": " + str(e))
or just this:
print(type(e).__name__, e)

If you replace str(e) with repr(e) Python 2 will produce KeyError(1,) and Python 3 will produce KeyError(1)
This doesn't quite produce your desired output, but it may be close enough?