I have a simple for loop which iterates through all objects in a local database. For each object, I reference a presalesEngineer and pass that ID to an API call to retrieve a JSON response. However, there are records in the database for which there is no value for presalesEngineer. When this is the case, the empty string throws a HttpError when a bad URL is passed to the API call. How can I handle when presalesEngineer does not exist, so the API is not passed an empty value?
views.py
objects = Opportunity.objects.all()
for object in objects:
try:
ps_e = object.presalesEngineer
if ps_e:
presales_engineers = [cwObj.get_member_by_id(ps_e) for object in objects]
else:
presales_engineers = 'None'
except NameError:
presales_engineers = 'None'
This codeblock should try to grab the presalesEngineer of your object or return None (note that the string 'None' does not equal the pytohon object None)
for object in objects:
try:
ps_e = object.presalesEngineer
# Do stuff with an object you know for sure will not trigger an exception
# Something like:
# if ps_e != '': < the object is not an empty string
# or
# if ps_e: < the object is not None
# after you pass whatever checks you deem necessary, you launch your API call.
except AttributeError:
# You can either pass here or return a None object/Empty list
ps_e = None
Possible implementation below:
# Empty list of whatever you are searching for
engineers = []
for my_object in objects:
try:
ps_e = my_object.presalesEngineer
# This is here to avoid none values in your API call
if ps_e:
# Just in case your API call falls
# It will fail silently in this try codeblock
try:
# Assuming cwObj is your driver/API endpoint builder
# And that you only get one single string as response
# And that string is not some data structure that you need to split
my_desired_id = cwObj.get_member_by_id(ps_e)
engineers.append(my_desired_id)
# Using bare except statements is not a good idea
# Use HttpError here if you don't want to pass on any exception
except:
pass
except AttributeError:
# You can either pass here or return a None object/Empty list
pass
print engineers
Related
I have a number of constants,variable in which i keep names.
ATTR_ITEM_NAME = 'pro'
I check if the attribute is attached to an objects:
if hasattr(obj1, ATTR_ITEM_NAME):
then if exist I want the attribute value to be passed to an attribute of an object, something like this:
obj2.fm = obj1.ATTR_ITEM_NAME
ATTR_ITEM_NAME being a string and not an attribute is an error, I need something that works;
Python also has getattr which works like hasattr but returns the value:
obj2.fm = getattr(obj1, ATTR_ITEM_NAME)
If you are not sure the attribute exists you could:
assign a default value (e.g. None)
DEFAULT = None
obj2.fm = getattr(obj1, ATTR_ITEM_NAME, DEFAULT)
or catch the exception using
try:
obj2.fm = getattr(obj1, ATTR_ITEM_NAME)
except AttributeError:
pass # or do something else...
I have a dictionary list of size ~250k in python (i.e 250k dictionaries in a list), which I try to process as shown below. The aim is to clean up the dictionary and return an iterable at the end. So, I have something like this:
def check_qs(dict_list_in):
try:
del_id=[]
for i in dict_list_in:
tmp=i["get_url"][0]
if i["from"][0]=="var0":
try:
URLValidator()(tmp)
except:
del_id.append( i["id"] )
elif i["from"][0]=="var1":
try:
URLValidator()( tmp.split("\"")[1] )
except:
del_id.append( i["id"] )
elif i["from"][0]=="var2":
try:
URLValidator()( tmp.split("\'")[1] )
except:
del_id.append( i["id"] )
else:
del_id.append( i["id"] )
gc.collect()
result = filter(lambda x: x['id'] not in del_id,dict_list_in)
return result
except:
return dict_list_in
What I am doing above, is checking each dictionary in ths list for some condition, and if this fails, I get the id and then use filter to delete those dictionaries specific from the list.
At the moment, this takes a long time to run - and I was wondering if there were any obvious optimizations I am missing out on. I think at the moment the above code is too naive.
I made a couple changes. I put the validation instance out of the loop so that you don't have to initialize it every time. If it's required to instantiate every time, just move it into the try accept block. I also changed from deleting items in the original list, to appending the items to a new list that you want, removing the need for a filter. I also moved the validation out of the if statements so that if you hit the else statement you don't have to run the validation. Look at the logic of the if statements, it is the same as yours. It appears that you are using django, but if you aren't change the except to except Exception.
from django.core.exceptions import ValidationError
def check_qs(dict_list_in):
new_dict_list = []
validate = URLValidator()
for i in dict_list_in:
test_url = i["get_url"][0]
if i["from"][0] == "var0":
pass
elif i["from"][0] == "var1":
test_url = test_url.split("\"")[1]
elif i["from"][0] == "var2":
test_url = test_url.split("\'")[1]
else:
continue
try:
validate(test_url)
# If you aren't using django you can change this to 'Exception'
except ValidationError:
continue
new_dict_list.append(i)
return new_dict_list
in fucntion getLink(urls), I have return (cloud,parent,children)
in main function, I have (cloud,parent,children) = getLink(urls) and I got error of this line: TypeError: 'NoneType' object is not iterable
parent and children are all list of http links. since, it is not able to paste them here, parent is a list contains about 30 links; children is a list contains about 30 items, each item is about 10-100 links which is divide by ",".
cloud is a list contain about 100 words, like that: ['official store', 'Java Applets Centre', 'About Google', 'Web History'.....]
I didnot know why I get an error. Is there anything wrong in passing parameter? Or because the list take too much space?
#crawler url: read webpage and return a list of url and a list of its name
def crawler(url):
try:
m = urllib.request.urlopen(url)
msg = m.read()
....
return (list(set(list(links))),list(set(list(titles))) )
except Exception:
print("url wrong!")
#this is the function has gone wrong: it throw an exception here, also the error I mentioned, also it will end while before len(parent) reach 100.
def getLink(urls):
try:
newUrl=[]
parent = []
children =[]
cloud =[]
i=0
while len(parent)<=100:
url = urls[i]
if url in parent:
i += 1
continue
(links, titles) = crawler(url)
parent.append(url)
children.append(",".join(links))
cloud = cloud + titles
newUrl= newUrl+links
print ("links: ",links)
i += 1
if i == len(urls):
urls = list(set(newUrl))
newUrl = []
i = 0
return (cloud,parent,children)
except Exception:
print("can not get links")
def readfile(file):
#not related, this function will return a list of url
def main():
file='sampleinput.txt'
urls=readfile(file)
(cloud,parent,children) = getLink(urls)
if __name__=='__main__':
main()
There might be a way that your function ends without reaching the explicit return statement.
Look at the following example code.
def get_values(x):
if x:
return 'foo', 'bar'
x, y = get_values(1)
x, y = get_values(0)
When the function is called with 0 as parameter the return is skipped and the function will return None.
You could add an explicit return as the last line of your function. In the example given in this answer it would look like this.
def get_values(x):
if x:
return 'foo', 'bar'
return None, None
Update after seing the code
When the exception is triggered in get_link you just print something and return from the function. You have no return statement, so Python will return None. The calling function now tries to expand None into three values and that fails.
Change your exception handling to return a tuple with three values like you do it when everything is fine. Using None for each value is a good idea for it shows you, that something went wrong. Additionally I wouldn't print anything in the function. Don't mix business logic and input/output.
except Exception:
return None, None, None
Then in your main function use the following:
cloud, parent, children = getLink(urls)
if cloud is None:
print("can not get links")
else:
# do some more work
I would like to ask some guidelines on a small task that I am trying to solve.
I am experimenting with a small app that uses JSON data to save entities.
I know that you can easily convert a dict to an entity by just creating the model but, I am trying to build a more generic approach that would convert any dict to an entity.
My steps are:
Get the dict.
Validate that the dict keys correspond to an entitys model definitions by reading the class.dict of the model.
Try to unpack the validated properties in the model class contructor (create the model instance)
return it.
So far I am ok but lack of my python knowledge, is either constraining me, or confusing me.
Maybe I am as well forgetting or unaware of more simple way to do it.
So here is it:
#classmethod
def entity_from_dict(cls, parent_key, dict):
valid_properties = {}
logging.info(cls.__dict__)
for property,value in dict.iteritems():
if property in cls.__dict__: # should not iterate over functions, classmethods, and #property
logging.info(cls.__dict__[property]) # this outputs eg: StringProperty('title', required=True)
logging.info(type(cls.__dict__[property])) #this is more interesting <class 'google.appengine.ext.ndb.model.StringProperty'>
valid_properties.update({property: value})
# Update the id from the dict
if 'id' in dict: # if not creating a new entity
valid_properties['id'] = dict['id']
# Add the parent
valid_properties['parent'] = parent_key
#logging.info(valid_properties)
try:
entity = cls(**valid_properties)
except Exception as e:
logging.exception('Could not create entity \n' + repr(e))
return False
return entity
My problem is that I want only to validate ndb. Properties and not #classmethods, #property as well because this causes a conflict.
I am also using expando classes, so any property in the dict that is extra gets stored.
How can I check against these specific types?
Solved it as #Tim Hoffman proposed using the ._properties of the Ndb model.
A thing I didn't know is that via the ._properties I could get the model definition properties and I thought that it would only return the instance properties :-).
Also I did not use populate because I find that it does the same as passing the valid dict unpacked in the model's contructor ;-)
So here it is:
#classmethod
def entity_from_dict(cls, parent_key, data_dict):
valid_properties = {}
for cls_property in cls._properties:
if cls_property in data_dict:
valid_properties.update({cls_property: data_dict[cls_property]})
#logging.info(valid_properties)
# Update the id from the data_dict
if 'id' in data_dict: # if creating a new entity
valid_properties['id'] = data_dict['id']
# Add the parent
valid_properties['parent'] = parent_key
try:
entity = cls(**valid_properties)
except Exception as e:
logging.exception('Could not create entity \n' + repr(e))
return False
return entity
The JSON dump method in python which we using during the converting models to JSON for export converts non-strings into strings. Therefore Jimmy Kane methods throw the error due to model incompatibility. To avoid this problem I updated his method and added a method named prop_literal just for converting non-string characters which capsuled in the string into their literal type.
I also added the entity.put() to add the entity to datastore because the aim was that :)
def prop_literal(prop_type,prop_val):
"""
Convert non-string encapsulated in the string into literal type
"""
if "Integer" in prop_type:
return int(prop_val)
elif "Float" in prop_type:
return float(prop_val)
elif "DateTime" in prop_type:
# bos gecsin neticede locale
return None
elif ("String" in prop_type) or ("Text" in prop_type):
return prop_val
elif "Bool" in prop_type:
return True if prop_val == True else False
else:
return prop_val
def entity_from_dict(cls, parent_key, data_dict):
valid_properties = {}
for cls_property in cls._properties:
if cls_property in data_dict:
prop_type = str(cls._properties[cls_property])
# logging.info(prop_type)
real_val = prop_literal(prop_type,data_dict[cls_property])
try:
valid_properties.update({cls_property: real_val})
except Exception as ex:
# logging.info("Veri aktariminda hata:"+str(ex))
else:
# logging.info("prop skipped")
#logging.info(valid_properties)
# Update the id from the data_dict
if 'id' in data_dict: # if creating a new entity
valid_properties['id'] = data_dict['id']
# Add the parent
valid_properties['parent'] = parent_key
try:
entity = cls(**valid_properties)
logging.info(entity)
entity.put()
except Exception as e:
logging.exception('Could not create entity \n' + repr(e))
return False
return entity
I'm making my first attempts at Python.
I need to loop over a log, parse log entries and then update an object, which includes nested objects for machines listed in the log.
This is what I have:
import re
format_pat= re.compile(
r"(?P<host>(?:[\d\.]|[\da-fA-F:])+)\s"
r"(?P<identity>\S*)\s"
r"(?P<user>\S*)\s"
r"\[(?P<time>.*?)\]\s"
r'"(?P<request>.*?)"\s'
r"(?P<status>\d+)\s"
r"(?P<bytes>\S*)\s"
r'"(?P<referer>.*?)"\s'
r'"(?P<user_agent>.*?)"\s*'
)
from json import JSONEncoder
class MyEncoder(JSONEncoder):
def default(self, o):
return o.__dict__
# JSON response object
class ResponseObject(object):
def __init__(self, dict):
self.__dict__ = dict
# check for JSON response object
try:
obj
except NameError:
obj = ResponseObject({})
test = ['2001:470:1f14:169:15f3:824f:8a61:7b59 - SOFTINST [14/Nov/2012:09:32:31 +0100] "POST /setComputer HTTP/1.1" 200 4 "-" "-" 102356']
# log loop
for line in test:
try:
# try to create object from log entry
m = format_pat.match(line)
if m:
res = m.groupdict()
res["status"] = int(res["status"])
# register machine if not done
if not hasattr(obj, res["user"]):
setattr(obj, res["user"], {"downtime":"0","flag":"false","downstart":"0","init":res["time"],"last":"","uptime":"","downtime":"","totaltime":""})
machine = getattr(obj, res["user"])
flag = machine["flag"]
start = machine["downstart"]
down = machine["downtime"]
last = machine["last"]
print "done"
# set last
last = res["time"]
# PROBLEM this does not work
setattr(machine, last, res["time"])
print machine
else:
print "nope"
except:
print "nope base"
print MyEncoder().encode(obj)
The error I'm getting when trying to setattr() is
AttributeError: 'dict' object has no attribute ''
but I was afraid it was not as easy as this...
Question:
How do I update the last value in my nested object using 'setattr'? Or is there another way to update nested object attributes?
I think you need to do this:
setattr(machine, 'last', res["time"])
As setattr needs a string of the name of the attribute to be set
Do not use setattr. Just assign a value to the "last" key for each machine dictionary.
(actually you answered your own question!)
I don't understand why, but I can set the value of last like this:
print machine
print machine["last"]
print res["time"]
# this works
machine["last"] = res["time"]
print machine
If someone can explain, would be nice :-)