AttributeError: 'AsyncResult' object has no attribute 'head'
In my views.py file the error throwing at data = df.head(50) when i'm running and retrive the data as asynchronous background task.
Could anybody help me here?
task.py
#shared_task
def eda_flow_task(path, mode):
sleep(30)
try:
with adls_client.open(path, mode) as f:
df = pd.read_csv(f, low_memory=False)
return 'data load success'
except Exception as e:
response_dict.update({'error': str(e)})
view.py
def eda_flow(request):
path = '/data/satyajit/us_amz.csv'
mode = 'rb'
df = eda_flow_task.delay(path, mode)
data = df.head(50)
json_records = data.reset_index().to_json(orient ='records')
data = []
data = json.loads(json_records)
context = {'data': data}
return render(request, "home/tables-simple.html", context)
The following statement is wrong: df = eda_flow_task.delay(path, mode)
delay() returns AsyncResult, not Pandas data-frame... I am not even sure you can return data frame as it probably is not serializable. I certainly never tried it so I do not know. In the case it is as I suspect (not serializable), you should return an object of type that is serializable, like a list of objects for an example.
Your task actually returns a string data load success or None in case of an exception.
In order to get result of Celery task you need to do the following:
res = eda_flow_task.delay(path, mode).get() # NOTE: this blocks!
As mentioned above, res will most likely end up being a hard-coded string.
Related
I am attempting to pull data from an api into an empty list and convert it to a dataframe. I have used a try and except for cases where the api would return a null value as well, but still keep getting this error. This is my code: (I have written a function called get_data to pull in my required datapoints from the API:
d = []
base_url = "api url inserted here"
for i in range(len(df)):
try:
address = base_url + df.loc[i,"column_for_api_call"]
d.append(get_data(address))
except (IndexError, TypeError) as error:
print('ERROR at index {}: {!r}'.format(i, address))
d.append([" "])
lat = pd.DataFrame(d)
But keep getting the error TypeError: object of type 'NoneType' has no len() after the entire code runs. What am I missing?
df is None so your for-loop never gets entered. You should specify what should happen in that case:
if df is None:
pass # FIXME: what should happen here?
else:
for i in range(len(df):
address = base_url + df.loc[i,"column_for_api_call"]
d.append(get_data(address))
I'm fairly new to Python so bear with me please.
I have a function that takes two parameters, an api response and an output object, i need to assign some values from the api response to the output object:
def map_data(output, response):
try:
output['car']['name'] = response['name']
output['car']['color'] = response['color']
output['car']['date'] = response['date']
#other mapping
.
.
.
.
#other mapping
except KeyError as e:
logging.error("Key Missing in api Response: %s", str(e))
pass
return output
Now sometimes, the api response is missing some keys i'm using to generate my output object, so i used the KeyError exception to handle this case.
Now my question is, in a case where the 'color' key is missing from the api response, how can i catch the exception and continue to the line after it output['car']['date'] = response['date'] and the rest of the instructions.
i tried the pass instruction but it didn't have any affect.
Ps: i know i can check the existence of the key using:
if response.get('color') is not None:
output['car']['color'] = response['color']
and then assign the values but seeing that i have about 30 values i need to map, is there any other way i can implement ? Thank you
A few immediate ideas
(FYI - I'm not going to explain everything in detail - you can check out the python docs for more info, examples etc - that will help you learn more, rather than trying to explain everything here)
Google 'python handling dict missing keys' for a million methods/ideas/approaches - it's a common use case!
Convert your response dict to a defaultdict. In that case you can have a default value returned (eg None, '', 'N/A' ...whatever you like) if there is no actual value returned.
In this case you could do away with the try and every line would be executed.
from collections import defaultdict
resp=defaultdict(lambda: 'NA', response)
output['car']['date'] = response['date'] # will have value 'NA' if 'date' isnt in response
Use the in syntax, perhaps in combination with a ternary else
output['car']['color'] = response['color'] if 'color' in response
output['car']['date'] = response['date'] if 'date' in response else 'NA'
Again you can do away with the try block and every line will execute.
Use the dictionary get function, which allows you to specify a default if there is no value for that key:
output['car']['color'] = response.get('car', 'no car specified')
You can create a utility function that gets the value from the response and if the value is not found, it returns an empty string. See example below:
def get_value_from_response_or_null(response, key):
try:
value = response[key]
return value
except KeyError as e:
logging.error("Key Missing in api Response: %s", str(e))
return ""
def map_data(output, response):
output['car']['name'] = get_value_from_response_or_null(response, 'name')
output['car']['color'] = get_value_from_response_or_null(response, 'color')
output['car']['date'] = get_value_from_response_or_null(response, 'date')
# other mapping
# other mapping
return output
I wrote little script on python 3.7 to receive actual browser version
Here is it:
import json
def json_open():
file_json = open('/Users/user/PycharmProjects/Test/configuration.json')
return json.load(file_json)
def get_last_version(browser_name):
f = json_open()
res = (f['global']['link_to_latest_browser_version'])
last_version = repr(res.json()['latest']['client'][browser_name]['version'])
#print(last_version[1:-1])
return last_version[1:-1]
Also, json file exists, but it does not matter now.
Received:
AttributeError: 'str' object has no attribute 'json'.
In row
last_version = repr(res.json()['latest']['client'][browser_name]['version'])
Please, tell me what is my mistake?
If you are trying to convert res as a json object try json.loads(res) instead of res.json()
Try this:
import json
FILEJSON = '/Users/user/PycharmProjects/Test/configuration.json'
def get_last_version(browser_name):
with open(FILEJSON, 'r') as fson:
res = json.load(fson)
last_version = res['global']['link_to_latest_browser_version']\
['latest']['client'][browser_name]['version'][1:-1]
return last_version
I think that the json_open function is unnecessary. Also take into account that the behavior of the json.load() method depends on the type of file you are reading.
Ok, the problem is here:
last_version = repr(res.json()['latest']['client'][browser_name]['version'])
A JSON object is basically a dictionary. So when you do json['key'] it returns the content, not a json object.
Here res is a string, not a json object and thus does not have the .json() attribute.
Edit:
If you want a string to be return in your situation:
res = json.loads(f['global']['link_to_latest_browser_version'])
last_version = res['latest']['client'][browser_name]['version']
return last_version
Your "res" variable is of type string.
Strings do not have an attribute called json.
So res.json() is invalid.
I have the following function:
def request( url, type, headers, simulate = False, data = {}):
I want to be able to load the parameters from a text file and pass them to the function, I tried using evil eval below:
if execute_recovery:
for command in content:
logger.debug("Executing: "+command)
try:
result = eval(utilities.request("{0}").format(command))
if not result["Success"]:
continue_recovery = utilities.query_yes_no("Warning: Previous recovery command failed, attempt to continue recovery?\n")
if not continue_recovery:
break
else:
logger.debug("Command executed successfully...")
except Exception, e:
logger.debug( "Recovery: Eval Error, %s" % str(e) )
Where command would be a line in a text file like:
"http://192.168.1.1/accounts/1/users/1",delete,headers,simulate=False,data={}
This throws me the following error:
'request() takes at least 3 arguments (1 given)'
So presumably this means that it is interpreting the command as a single string instead of different parameters.
Does anybody know how to solve this?
I can't understand what you are trying to do there with eval or format. For one thing, you've put eval around the call to request itself, so it will evaluate the return value rather than call it with some dynamic value.
But you don't need eval at all. You just need to pass the arguments using the * and ** operators:
args = []
kwargs = {}
for arg in command.split(','):
if '=' in arg:
k, v = arg.split('=')
kwargs[k] = ast.literal_eval(v)
else:
args.append(arg)
result = utilities.request(*args, **kwargs)
Using #BurhanKhalid's suggestion, I decided to store the parameters as a json object and load them at run time like so:
Store parameters here:
def request( url, type, headers, simulate = False, data = {}):
if simulate:
recovery_command = {"url":url, "type" : type, "data" : data}
recovery.add_command(json.dumps(recovery_command))
...
Load parameters here:
def recovery():
...
if execute_recovery:
for command in content:
logger.debug("Executing: "+command)
try:
recovery_command = json.loads(command)
result = utilities.request(url = recovery_command["url"], type = recovery_command["type"], headers = headers, simulate = False, data = recovery_command["data"])
if not result["Success"]:
continue_recovery = utilities.query_yes_no("Warning: Previous recovery command failed, attempt to continue recovery?\n")
if not continue_recovery:
break
else:
logger.debug("Command executed successfully...")
except Exception, e:
logger.debug( "Recovery: Eval Error, %s" % str(e) )
How to send a class object in the payload of a task in python? I want to send an object in the parameters of a task.
When I use simplejson, I get the error: Object is not serializable.
When I use pickle, I get KeyValue Error.
How to do this ?
This is the class which I want to serialize
class Matrix2D_icfg:
name = ""
indices = []
value = {}
def __init__(self,s):
self.name = s
self.indices = []
def __getitem__(self,i):
self.indices.append(i)
if len(self.indices)==2:
(m,n) = self.indices
self.indices = []
if self.value.has_key(m*4276+n) == True :
value = self.value[m*4276+n]
else :
value = 0
return value
else: return self
def __setitem__(self,i,value):
self.indices.append(i)
if len(self.indices)==2:
(m,n) = self.indices
if value != 0 : self.value[m*4276+n] = value
self.indices = []
return self
icfg = Matrix2D_icfg("icfg") #declaring object
icfg_compress = pickle.dumps(icfg) #to pickle
icfg = pickle.loads(icfg_compress) # to unload
I get the following error when i pass the pickled object as payload and unload it later
File "/Users/praveensekar/myFYP/gaecode/pknots4d.2.3/pknots.py", line 439, in post
icfg = pickle.loads(icfg_compress)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 1374, in loads
return Unpickler(file).load()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 858, in load
dispatch[key](self)
KeyError: '\x00'
The problem was with the type of data that was unloaded. I casted it to type str and everything seemed to work properly.
I just changed it to
icfg = Matrix2D_icfg("icfg") #declaring object
icfg_compress = pickle.dumps(icfg) #to pickle
icfg = pickle.loads(str(icfg_compress)) # to unload
Have you looked at the deferred library? It's designed for exactly this, and takes care of serialization and deserialization for you.
This is a part of my task queueing service. It just posts a list to another task to break the project up into manageable parts. It's just part of it but you should get most of the idea for what you need to do.
To save it:
from django.utils import simplejson as json
.... stuff happens
index = 0
current_list = []
while index < len(item_list):
if index+500 < len(item_list):
for item in item_list[index:index+500]:
current_list.append(item.item_number)
jsondump = json.dumps(current_list)
taskqueue.add(url = '/queuer',
headers = {'Content-Type':'application/json'},
payload = jsondump)
To load it:
from django.utils import simplejson as json
class TaskQueuer(webapp.RequestHandler):
def post(self):
request = self.request.body
task_list = json.loads(request)