In my construct below, I am trying to pass a JSON object through my web service. As a new requirement I have to pass a dictionary object which is sent in the code below. Can you please guide me how to add the dictionary to JSON object?
if plain_text is not None:
blob = TextBlob(plain_text)
sentiment = TextBlob(plain_text)
sent = {}
for sentence in blob.sentences:
sent[sentence] =sentence.sentiment.polarity
print sent
return json.dumps(
{'input' : plain_text,
'Polarity': sentiment.polarity,
#'sent': json.dumps(sent) # this is where I am stuck as this 'sent' is a dict
},
indent=4)
If I uncomment the line I get the below error:
Exception:
TypeError('keys must be a string',)
Traceback:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\bottle-0.12.7-py2.7.egg\bottle.py", line 862, in _handle
return route.call(**args)
File "C:\Python27\lib\site-packages\bottle-0.12.7-py2.7.egg\bottle.py", line 1729, in wrapper
rv = callback(*a, **ka)
File "C:\Users\hp\Desktop\RealPy\WebServices\bottle\server_bckup.py", line 53, in sentimentEngine
'sent': json.dumps(sent),
File "C:\Python27\lib\json\__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "C:\Python27\lib\json\encoder.py", line 201, in encode
chunks = self.iterencode(o, _one_shot=True)
File "C:\Python27\lib\json\encoder.py", line 264, in iterencode
return _iterencode(o, 0)
TypeError: keys must be a string
In JSON, dictionary keys must be strings. You have a Python dictionary sent, that you want to serialize into JSON. This fails, as your dictionary sent has keys that are not strings, but textblob.blob.Sentence instances.
If it makes sense, you can change your code to read:
for sentence in blob.sentences:
sent[str(sentence)] = sentence.sentiment.polarity
Or, you can customize the Python JSON encoder to know how to serialize TextBlob Sentences.
Related
I have a Celery app (4.4.6) that uses dataclasses. Since JSON can't serialize/deserialize dataclasses, I have forced the use of pickle throughout (I'm aware of the risk, but I think it's mitigated in the way the app is deployed). However, I am getting errors from within the app from kombu, saying TypeError: Object of type ResourceGroup is not JSON serializable. Everything else is working, so in general, it has to be using pickle OK, but in this one case it isn't. However nothing in the stacktrace that comes with the exception mentions my code. The structure of the software is tasks that create other tasks dynamically as they perform discovery, using delay(). Almost all the tasks are passed this ResourceGroup object, and they are running fine, except one (I think, judging by the frequency of these errors, and the logging I am getting for completing tasks).
This is how I'm configuring Celery in my worker. Is there some other setting I need to set to really really make it use pickle in all situations? (alternatively, is there a JSON serializer/deserializer that can reconstitute dataclasses?)
class CeleryConfig:
task_serializer = 'pickle'
result_serializer = 'pickle'
event_serializer = 'pickle'
accept_content = ['pickle']
result_accept_content = ['pickle']
app = Celery('collector',
backend='redis://' + Config.REDIS_HOST,
broker='redis://' + Config.REDIS_HOST,
include=['tasks'])
app.config_from_object(CeleryConfig)
Update:
Here's a full example of one of the exceptions:
[2020-07-23 05:02:10,621: DEBUG/MainProcess] pidbox received method active() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'cde2e89b-bb81-3c19-8491-b57b072e5f29'} ticket:6c2cc493-0e2d-4a85-821b-350bdc4bceeb]
[2020-07-23 05:02:10,621: ERROR/MainProcess] Control command error: EncodeError(TypeError('Object of type ResourceGroup is not JSON serializable'))
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/usr/local/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/usr/local/lib/python3.8/site-packages/kombu/utils/json.py", line 69, in dumps
return _dumps(s, cls=cls or _default_encoder,
File "/usr/local/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.8/site-packages/kombu/utils/json.py", line 59, in default
return super(JSONEncoder, self).default(o)
File "/usr/local/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type ResourceGroup is not JSON serializable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/celery/worker/pidbox.py", line 46, in on_message
self.node.handle_message(body, message)
File "/usr/local/lib/python3.8/site-packages/kombu/pidbox.py", line 145, in handle_message
return self.dispatch(**body)
File "/usr/local/lib/python3.8/site-packages/kombu/pidbox.py", line 112, in dispatch
self.reply({self.hostname: reply},
File "/usr/local/lib/python3.8/site-packages/kombu/pidbox.py", line 149, in reply
self.mailbox._publish_reply(data, exchange, routing_key, ticket,
File "/usr/local/lib/python3.8/site-packages/kombu/pidbox.py", line 280, in _publish_reply
producer.publish(
File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 167, in publish
body, content_type, content_encoding = self._prepare(
File "/usr/local/lib/python3.8/site-packages/kombu/messaging.py", line 252, in _prepare
body) = dumps(body, serializer=serializer)
File "/usr/local/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.8/site-packages/kombu/serialization.py", line 54, in _reraise_errors
reraise(wrapper, wrapper(exc), sys.exc_info()[2])
File "/usr/local/lib/python3.8/site-packages/vine/five.py", line 194, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/kombu/serialization.py", line 50, in _reraise_errors
yield
File "/usr/local/lib/python3.8/site-packages/kombu/serialization.py", line 221, in dumps
payload = encoder(data)
File "/usr/local/lib/python3.8/site-packages/kombu/utils/json.py", line 69, in dumps
return _dumps(s, cls=cls or _default_encoder,
File "/usr/local/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.8/site-packages/kombu/utils/json.py", line 59, in default
return super(JSONEncoder, self).default(o)
File "/usr/local/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
kombu.exceptions.EncodeError: Object of type ResourceGroup is not JSON serializable
I agree that ResourceGroup isn't JSON serializable. I just don't understand why it's trying! :-)
update2:
I got the celery worker running with concurrency=1 in the debugger, and now with a breakpoint in the JSON serializer (which I should never hit) I get a stop. From reading back inside kombu, the first mention of serializer in the stack trace seems to be from Node.reply() in pidbox.py where self.mailbox.serializer is None (and defaults to JSON because of that). I don't see where that Mailbox object is originally created though.
update3:
The Control object appears to always create a Mailbox that will accept and send JSON, regardless of any config:
"""Worker remote control client."""
Mailbox = Mailbox
def __init__(self, app=None):
self.app = app
self.mailbox = self.Mailbox(
app.conf.control_exchange,
type='fanout',
accept=['json'],
producer_pool=lazy(lambda: self.app.amqp.producer_pool),
queue_ttl=app.conf.control_queue_ttl,
reply_queue_ttl=app.conf.control_queue_ttl,
queue_expires=app.conf.control_queue_expires,
reply_queue_expires=app.conf.control_queue_expires,
)
Im trying to use Python requests module to call a service which returns a Python dict that contains a datetime object...
I get this following errror,
File "/usr/local/lib/python2.7/dist-packages/bottle.py", line 862, in _handle
return route.call(**args)
File "/usr/local/lib/python2.7/dist-packages/bottle.py", line 1733, in wrapper
json_response = dumps(rv)
File "/usr/lib/python2.7/dist-packages/simplejson/__init__.py", line 286, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 226, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 296, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python2.7/dist-packages/simplejson/encoder.py", line 202, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: datetime.date(2014, 4, 16) is not JSON serializable
Call stmt:
r = requests.get('http://localhost:8089/allfeeds',verify=False)
print r.status_code
print r.text
Is your web app written using bottle? How do I know that?
Try to make the same call from command line, e.g.:
$ curl http://localhost:8089/allfeeds
I think, that the output of such request will be exactly the same.
the line print r.text is just printing the response - not breaking.
In short: the requests call works perfectly well, but your web service is returning a string, which looks like a problem. But the problem is in your web app, not on client side.
The stacktrace is from the server. The stdlib-only code:
from urllib2 import urlopen
r = urlopen('http://localhost:8089/allfeeds')
print r.code
print r.read().decode(r.info().getparam('charset') or 'utf-8')
would produce the same result. r.code should be 5xx, meaning "Server Error".
datetime.date is not JSON serializable; you should modify your server to convert dates into strings e.g., some_date.isoformat() or numbers (POSIX timestamp) e.g., calendar.timegm(some_date.timetuple()) assuming some_date is in UTC, see Converting datetime.date to UTC timestamp in Python.
I'm trying to implement a simple Django view that accept a file attachment and some other parameters and proxy the request on a remote API call.
Please note: the proxy is not the point of my question :)
This is how I implemented the view:
def image_upload(request):
token = request.POST['token']
image_file = request.FILES.values()[0]
files = {'file': ('myupload.txt', image_file.read())}
client_id = request.POST['client_id']
folder_id = request.POST['folder_id']
advert_id = request.POST['advert_id']
image_type = request.POST['image_type']
crop_image = request.POST['crop_image']
api_base_url = settings.API_BASE_URL
file_post_data = {'client_id': client_id, 'folder_id': folder_id, 'advert_id': advert_id,
'image_type': image_type, 'crop_image': crop_image}
auth_header = {'Authorization': 'Token ' + token}
r = requests.post(api_base_url + 'assets/image/upload/',
data = json.dumps(file_post_data),
headers = auth_header,
files = files)
return r.json()
The problem is that when I test this view (I use Django Test Client to do it) I get an error on the line "files = files)" that says "ValueError: cannot encode objects that are not 2-tuples".
The complete trace log is this one:
======================================================================
ERROR: test_image_upload (fbx.tests.FbxTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/andrea/Documents/src/fbxapp/onboard/fbx/tests.py", line 18, in test_image_upload
'image_type': 'A', 'crop_image': False, 'attachment': fp})
File "/usr/local/lib/python2.7/dist-packages/django/test/client.py", line 449, in post
response = super(Client, self).post(path, data=data, content_type=content_type, **extra)
File "/usr/local/lib/python2.7/dist-packages/django/test/client.py", line 262, in post
return self.request(**r)
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/home/andrea/Documents/src/fbxapp/onboard/fbx/views.py", line 42, in image_upload
files = files)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 324, in request
prep = req.prepare()
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 225, in prepare
p.prepare_body(self.data, self.files)
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 385, in prepare_body
(body, content_type) = self._encode_files(files, data)
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 99, in _encode_files
fields = to_key_val_list(data or {})
File "/usr/local/lib/python2.7/dist-packages/requests/utils.py", line 136, in to_key_val_list
raise ValueError('cannot encode objects that are not 2-tuples')
ValueError: cannot encode objects that are not 2-tuples
I've also tried a quick test using this to read a file: files = {'file': ('myupload.txt', open('/tmp/mytmp.txt', 'rb'))}
but it doesn't work. Do tou have any idea about how to fix this?
Thanks!
You cannot simultaneously post JSON data and multipart/form-data which is in essence what you're trying to do here. json.dumps returns a string so you're sending a string that looks like
'{"client_id": 1, "folder_id": 2, "advert_id": 3, "image_type": "jpeg", "crop_image": true}'
And then telling requests you want to use that in combination with a multipart/form-data request by sending something in via the files parameter. That is impossible and could possibly raise a better exception.
Either add the file to the JSON data after reading it into memory and send that with the appropriate Content-Type header or send the entire thing as a multipart/form-data request without using json.dumps and simply passing the dictionary you're creating to data. Use one or the other but not both.
That aside, you say that your exception comes from the line that only has files=files) on it, but it is not that line alone that causes the exception. The exception is raised by the function which happens to end on that line. The fact that your exception arises from that too is mere coincidence. This is an wart in Python that may possibly be fixed in Python 3.4. You should upgrade, because 3.4 will be awesome and newer versions of Django support Python 3.x.
Here is the sample string i am receiving from one of the web services,
body=%7B%22type%22%3A%22change%22%2C%22url%22%3A%22http%3A%2F%2Fapi.pachube.com%2Fv2%2Ftriggers%2F4100%22%2C%22environment%22%3A%7B%22feed%22%3A%22http%3A%2F%2Fapi.pachube.com%2Fv2%2Ffeeds%2F36133%22%2C%22title%22%3A%22Current+Cost+Bridge%22%2C%22description%22%3Anull%2C%22id%22%3A36133%7D%2C%22threshold_value%22%3Anull%2C%22timestamp%22%3A%222012-01-05T09%3A27%3A01Z%22%2C%22triggering_datastream%22%3A%7B%22url%22%3A%22http%3A%2F%2Fapi.pachube.com%2Fv2%2Ffeeds%2F36133%2Fdatastreams%2F1%22%2C%22value%22%3A%7B%22value%22%3A%22523%22%2C%22max_value%22%3A1269.0%2C%22min_value%22%3A0.0%7D%2C%22id%22%3A%221%22%2C%22units%22%3A%7B%22symbol%22%3A%22W%22%2C%22type%22%3A%22derivedUnits%22%2C%22label%22%3A%22watts%22%7D%7D%2C%22id%22%3A4100%7D
Here is the code,
class Feeds():
def GET(self):
print "Get request is accepted."
return render.index(None)
def POST(self):
print "Post request is accepted."
print (web.data())
Now when that web-service posts the above given data, how will i convert it to readable format? Then, i need to convert it to JSON object and use further. So, how will i convert it?
When i try this code,
json_data = json.loads(web.data())
print json_data['body']
return render.index(json_data['body'])
It gives me an error,
enter code Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/web/application.py", line 237, in process
return self.handle()
File "/usr/local/lib/python2.6/dist-packages/web/application.py", line 228, in handle
return self._delegate(fn, self.fvars, args)
File "/usr/local/lib/python2.6/dist-packages/web/application.py", line 409, in _delegate
return handle_class(cls)
File "/usr/local/lib/python2.6/dist-packages/web/application.py", line 385, in handle_class
return tocall(*args)
File "/home/ubuntu/pachubeConsumer/src/controllers/feeds.py", line 17, in POST
json_data = json.loads(web.data())
File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.6/json/decoder.py", line 338, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded here
Where am i going wrong ??
Thanks in advance.
First you need to unquote the body
import urllib
body="%7B%22type%22%3A%22change%22%2C%22url%22%3A%22http%3A%2F%2Fapi.pachube.com%2Fv2%2Ftriggers%2F4100%22%2C%22environment%22%3A%7B%22feed%22%3A%22http%3A%2F%2Fapi.pachube.com%2Fv2%2Ffeeds%2F36133%22%2C%22title%22%3A%22Current+Cost+Bridge%22%2C%22description%22%3Anull%2C%22id%22%3A36133%7D%2C%22threshold_value%22%3Anull%2C%22timestamp%22%3A%222012-01-05T09%3A27%3A01Z%22%2C%22triggering_datastream%22%3A%7B%22url%22%3A%22http%3A%2F%2Fapi.pachube.com%2Fv2%2Ffeeds%2F36133%2Fdatastreams%2F1%22%2C%22value%22%3A%7B%22value%22%3A%22523%22%2C%22max_value%22%3A1269.0%2C%22min_value%22%3A0.0%7D%2C%22id%22%3A%221%22%2C%22units%22%3A%7B%22symbol%22%3A%22W%22%2C%22type%22%3A%22derivedUnits%22%2C%22label%22%3A%22watts%22%7D%7D%2C%22id%22%3A4100%7D"
unquoted = urllib.unquote(body)
Then just load the JSON like normal
import json
pythonDict = json.loads(unquoted)
you need to unescape the query string first using urllib.unquote()
import urllib
unescaped = urllib.unquote(web.data())
then you may use json.loads to convert it into json.
json_data = json.loads(unescaped)
Python 3 urllib unquote move to parse:
from urllib import parse
parse.unquote(web.data())
How do I dump the data in a queryset of Mongo documents to a .json file using python?
I've tried to use django serializer, but didn't work as the fields are not accessed in django the same way as they are accessed in mongo.
for model in models:
json_serializer.serialize(model.objects.all(), indent=2, stream=output_file_mongo)
I also tried to use python JSON encode/decoder,
import json
for model in mongo_models:
output_file_mongo.write(json.dumps(model.objects.all()))
I get an exception
File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python2.7/json/encoder.py", line 178, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: [<MongoModelA: MongoModelA object>] is not JSON serializable
The django serializer does not know how to handle mongoengine objects, You'll most likely have to write your own json encoder to map them to a simple dictionary:
class MyEncoder(json.JSONEncoder):
def encode_object(self, obj):
return { 'id':unicode(obj.id), 'other_property': obj.other_property }
def default(self, obj):
if hasattr(obj, '__iter__'):
return [ self.encode_object(x) for x in obj ]
else:
return self.encode_object(obj)
then call it like so:
import json
import MyEncoder
json_string = json.dumps(model.objects.all(), cls=MyEncoder)