How to send a class object in the payload of a task in python? I want to send an object in the parameters of a task.
When I use simplejson, I get the error: Object is not serializable.
When I use pickle, I get KeyValue Error.
How to do this ?
This is the class which I want to serialize
class Matrix2D_icfg:
name = ""
indices = []
value = {}
def __init__(self,s):
self.name = s
self.indices = []
def __getitem__(self,i):
self.indices.append(i)
if len(self.indices)==2:
(m,n) = self.indices
self.indices = []
if self.value.has_key(m*4276+n) == True :
value = self.value[m*4276+n]
else :
value = 0
return value
else: return self
def __setitem__(self,i,value):
self.indices.append(i)
if len(self.indices)==2:
(m,n) = self.indices
if value != 0 : self.value[m*4276+n] = value
self.indices = []
return self
icfg = Matrix2D_icfg("icfg") #declaring object
icfg_compress = pickle.dumps(icfg) #to pickle
icfg = pickle.loads(icfg_compress) # to unload
I get the following error when i pass the pickled object as payload and unload it later
File "/Users/praveensekar/myFYP/gaecode/pknots4d.2.3/pknots.py", line 439, in post
icfg = pickle.loads(icfg_compress)
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 1374, in loads
return Unpickler(file).load()
File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/pickle.py", line 858, in load
dispatch[key](self)
KeyError: '\x00'
The problem was with the type of data that was unloaded. I casted it to type str and everything seemed to work properly.
I just changed it to
icfg = Matrix2D_icfg("icfg") #declaring object
icfg_compress = pickle.dumps(icfg) #to pickle
icfg = pickle.loads(str(icfg_compress)) # to unload
Have you looked at the deferred library? It's designed for exactly this, and takes care of serialization and deserialization for you.
This is a part of my task queueing service. It just posts a list to another task to break the project up into manageable parts. It's just part of it but you should get most of the idea for what you need to do.
To save it:
from django.utils import simplejson as json
.... stuff happens
index = 0
current_list = []
while index < len(item_list):
if index+500 < len(item_list):
for item in item_list[index:index+500]:
current_list.append(item.item_number)
jsondump = json.dumps(current_list)
taskqueue.add(url = '/queuer',
headers = {'Content-Type':'application/json'},
payload = jsondump)
To load it:
from django.utils import simplejson as json
class TaskQueuer(webapp.RequestHandler):
def post(self):
request = self.request.body
task_list = json.loads(request)
Related
AttributeError: 'AsyncResult' object has no attribute 'head'
In my views.py file the error throwing at data = df.head(50) when i'm running and retrive the data as asynchronous background task.
Could anybody help me here?
task.py
#shared_task
def eda_flow_task(path, mode):
sleep(30)
try:
with adls_client.open(path, mode) as f:
df = pd.read_csv(f, low_memory=False)
return 'data load success'
except Exception as e:
response_dict.update({'error': str(e)})
view.py
def eda_flow(request):
path = '/data/satyajit/us_amz.csv'
mode = 'rb'
df = eda_flow_task.delay(path, mode)
data = df.head(50)
json_records = data.reset_index().to_json(orient ='records')
data = []
data = json.loads(json_records)
context = {'data': data}
return render(request, "home/tables-simple.html", context)
The following statement is wrong: df = eda_flow_task.delay(path, mode)
delay() returns AsyncResult, not Pandas data-frame... I am not even sure you can return data frame as it probably is not serializable. I certainly never tried it so I do not know. In the case it is as I suspect (not serializable), you should return an object of type that is serializable, like a list of objects for an example.
Your task actually returns a string data load success or None in case of an exception.
In order to get result of Celery task you need to do the following:
res = eda_flow_task.delay(path, mode).get() # NOTE: this blocks!
As mentioned above, res will most likely end up being a hard-coded string.
I am currently having an issue where I am trying to store data in a list (using dataclasses). When I print the data inside the list in the function (PullIncursionData()) it responded with a certain amount of numbers (never the same, not possible due to it's nature). When printing it after it being called to store it's return in a Var it somehow prints only the same number.
I cannot share the numbers, as they update with EVE Online's API, so the only way is to run it locally and read the first list yourself.
The repository is Here: https://github.com/AtherActive/EVEAPI-Demo
Heads up! Inside the main.py (the file with issues) (a snippet of code is down below) are more functions. All functions from line 90 and forward are important, the rest can be ignored for this question, as they do not interact with the other functions.
def PullIncursionData():
#Pulls data from URL and converts it into JSON
url = 'https://esi.evetech.net/latest/incursions/?datasource=tranquility'
data = rq.get(url)
jsData = data.json()
#Init var to store incursions
incursions = []
#Set lenght for loop. yay
length = len(jsData)
# Every loop incursion data will be read by __parseIncursionData(). It then gets added to var Incursions.
for i in range(length):
# Add data to var Incursion.
incursions.append(__parseIncursionData(jsData, i))
# If Dev mode, print some debug. Can be toggled in settings.py
if settings.developerMode == 1:
print(incursions[i].constellation_id)
return incursions
# Basically parses the input data in a decent manner. No comments needed really.
def __parseIncursionData(jsData, i):
icstruct = stru.Incursion
icstruct.constellation_id = jsData[i]['constellation_id']
icstruct.constellation_name = 'none'
icstruct.staging = jsData[i]['staging_solar_system_id']
icstruct.region_name = ResolveSystemNames(icstruct.constellation_id, 'con-reg')
icstruct.status = jsData[i]['state']
icstruct.systems_id = jsData[i]['infested_solar_systems']
icstruct.systems_names = ResolveSystemNames(jsData[i]['infested_solar_systems'], 'system')
return icstruct
# Resolves names for systems, regions and constellations. Still WIP.
def ResolveSystemNames(id, mode='constellation'):
#init value
output_name = 'none'
# If constellation, pull data and find region name.
if mode == 'con-reg':
url = 'https://www.fuzzwork.co.uk/api/mapdata.php?constellationid={}&format=json'.format(id)
data = rq.get(url)
jsData = data.json()
output_name = jsData[0]['regionname']
# Pulls system name form Fuzzwork.co.uk.
elif mode == 'system':
#Convert output to a list.
output_name = []
lenght = len(id)
# Pulls system name from Fuzzwork. Not that hard.
for i in range(lenght):
url = 'https://www.fuzzwork.co.uk/api/mapdata.php?solarsystemid={}&format=json'.format(id[i])
data = rq.get(url)
jsData = data.json()
output_name.append(jsData[i]['solarsystemname'])
return output_name
icdata = PullIncursionData()
print('external data check:')
length = len(icdata)
for i in range(length):
print(icdata[i].constellation_id)
structures.py (custom file)
#dataclass
class Incursion:
constellation_id = int
constellation_name = str
staging = int
staging_name = str
systems_id = list
systems_names = list
region_name = str
status = str
def ___init___(self):
self.constellation_id = -1
self.constellation_name = 'undefined'
self.staging = -1
self.staging_name = 'undefined'
self.systems_id = []
self.systems_names = []
self.region_name = 'undefined'
self.status = 'unknown'
The purpose of this form is to let users enter a lot of places (comma separated) and it'll retrieve the phone, name, website. Have it working in a python IDE, no problem, but having issues putting it into my webapp.
I'm getting the error Exception Value: Can't pickle local object 'GetNums.<locals>.get_data' at the line where a is assigned. I checked the type of inputText and verified that it is indeed a list. So, I'm not sure why it won't pickle.
def GetNums(request):
form = GetNumsForm(request.POST or None)
if form.is_valid():
inputText = form.cleaned_data.get('getnums')
# all experimental
inputText = inputText.split(',')
def get_data(i):
#DON'T FORGET TO MOVE THE PRIMARY KEY LATER TO SETTINGS
r1 = requests.get('https://maps.googleapis.com/maps/api/place/textsearch/json?query=' + i + '&key=GET_YOUR_OWN')
a = r1.json()
pid = a['results'][0]['place_id']
r2 = requests.get('https://maps.googleapis.com/maps/api/place/details/json?placeid=' + pid + '&key=GET_YOUR_OWN')
b = r2.json()
phone = b['result']['formatted_phone_number']
name = b['result']['name']
try:
website = b['result']['website']
except:
website ='No website found'
return ' '.join((phone, name, website))
v = str(type(inputText))
with Pool(5) as p:
a = (p.map(get_data, inputText))
# for line in p.map(get_data, inputText):
# print(line)
#code assist by http://stackoverflow.com/a/34512870/5037442
#end experimental
return render(request, 'about.html', {'v': a})
It's actually barfing when trying to pickle get_data, which is a nested function/closure.
Move get_data out of GetNums (and agh rename it to snake_case please) and it should work.
I have some Data that I need to write to a JSON string.
I have it working with dict items but want to encompass it all in classes to help ensure the correct data.
The following code is a comparison between the dict items and the class item output. They don't match, and I can't figure out what I am missing.
I get a "bound method Event.encode of Event..." in my JSON string.
from collections import namedtuple
import json
class Event(namedtuple('Event', 'itemName, itemID')):
def encode(self):
obj = {}
obj['itemName'] = str(self.itemName)
obj['itemID'] = int(self.itemID)
return json.dumps(obj)
curEv = Event('MyName', 5)
print 'ClassEv : ', curEv.encode()
curEv2 = {'itemName':'MyName', 'itemID':5}
print 'DictEv : ', json.dumps(curEv2)
class Packet(namedtuple('Packet', 'pID, itemType, itemData')):
def encode(self):
obj = {}
obj['pID'] = int(self.pID)
obj['itemType'] = int(self.itemType)
obj['itemData'] = str(self.itemData.encode)
return json.dumps(obj)
packet = Packet(11, 0, curEv)
print 'ClassPacket: ', packet.encode()
packet2 = {'pID':11, 'itemType':0}
packet2['itemData'] = curEv2
print 'DictPacket : ', json.dumps(packet2)
You are failing to call the itemData.encode() function. Instead you are simply returning a reference to it.
Try:
obj['itemData'] = str(self.itemData.encode())
Note the extra () at the end.
I'm trying to use memcache to cache data retrevied from the datastore. Storing stings works fine. But can't one store an object? I get an error "TypeError: 'str' object is not callable" when trying to store with this:
pageData = StandardPage(page)
memcache.add(memcacheid, pageData, 60)
I've read in the documentation that it requires "The value type can be any value supported by the Python pickle module for serializing values." But don't really understand what that is. Or how to convert pageData to it.
Any ideas?
..fredrik
EDIT:
I was a bit unclear. PageType is an class that amongst other thing get data from the datastore and manipulate it. The class looks like this:
class PageType():
def __init__(self, page):
self.pageData = PageData(data.Modules.gql('WHERE page = :page', page = page.key()).fetch(100))
self.modules = []
def renderEditPage(self):
self.addModules()
return self.modules
class StandardPage(PageTypes.PageType):
templateName = 'Altan StandardPage'
templateFile = 'templates/page.html'
def __init__(self, page):
self.pageData = PageTypes.PageData(data.Modules.gql('WHERE page = :page', page = page.key()).fetch(100))
self.modules = []
self.childModules = []
for child in page.childPages:
self.childModules.append(PageTypes.PageData(data.Modules.gql('WHERE page = :page', page = child.key()).fetch(100)))
def addModules(self):
self.modules.append(PageTypes.getStandardHeading(self, 'MainHeading'))
self.modules.append(PageTypes.getStandardTextBox(self, 'FirstTextBox'))
self.modules.append(PageTypes.getStandardTextBox(self, 'SecondTextBox'))
self.modules.append(PageTypes.getStandardHeading(self, 'ListHeading'))
self.modules.append(PageTypes.getStandardTextBox(self, 'ListTextBox'))
self.modules.append(PageTypes.getDynamicModules(self))
You can use db.model_to_protobuf to turn your object into something that can be stored in memcache. Similarly, db.model_from_protobuf will get your object back.
Resource:
Datastore Functions