I need to post data to a Django server. I'd like to use pickle. (There're no security requirements -> small intranet app.)
First, I pickle the data on the client and sending it with urllib2
def dispatch(self, func_name, *args, **kwargs):
dispatch_url = urljoin(self.base_url, "api/dispatch")
pickled_args = cPickle.dumps(args, 2)
pickled_kwargs = cPickle.dumps(kwargs, 2)
data = urllib.urlencode({'func_name' : func_name,
'args' : pickled_args,
'kwargs': pickled_kwargs})
resp = self.opener.open(dispatch_url, data)
Recieving the data at the server works, too:
def dispatch(request):
func_name = request.POST["func_name"]
pickled_args = request.POST["args"]
pickled_kwargs = request.POST["kwargs"]
But unpickling raises an error:
cPickle.loads(pickled_args)
Traceback (most recent call last):
File "<string>", line 1, in <fragment>
TypeError: must be string, not unicode
Obviously the urllib.urlencode has created a unicode string. But how can I convert it back to be able to unpickling (laods) again?
By the way, using pickle format 0 (ascii) works. I can convert to string before unpickling, but I'd rather use format 2.
Also, recommendations about how to get binary data to a Django view are highly appreciated.
Obviously the urllib.urlencode has created a unicode string.
urllib.urlencode doesn't return Unicode string.
It might be a feature of the web framework you use that request.POST contains Unicode strings.
Don't use pickle to communicate between services it is not secure, brittle, not portable, hard to debug and it makes your components too coupled.
Related
I'm following the Databricks example for uploading a file to DBFS (in my case .csv):
import json
import requests
import base64
DOMAIN = '<databricks-instance>'
TOKEN = '<your-token>'
BASE_URL = 'https://%s/api/2.0/dbfs/' % (DOMAIN)
def dbfs_rpc(action, body):
""" A helper function to make the DBFS API request, request/response is encoded/decoded as JSON """
response = requests.post(
BASE_URL + action,
headers={'Authorization': 'Bearer %s' % TOKEN },
json=body
)
return response.json()
# Create a handle that will be used to add blocks
handle = dbfs_rpc("create", {"path": "/temp/upload_large_file", "overwrite": "true"})['handle']
with open('/a/local/file') as f:
while True:
# A block can be at most 1MB
block = f.read(1 << 20)
if not block:
break
data = base64.standard_b64encode(block)
dbfs_rpc("add-block", {"handle": handle, "data": data})
# close the handle to finish uploading
dbfs_rpc("close", {"handle": handle})
When using the tutorial as is, I get an error:
Traceback (most recent call last):
File "db_api.py", line 65, in <module>
data = base64.standard_b64encode(block)
File "C:\Miniconda3\envs\dash_p36\lib\base64.py", line 95, in standard_b64encode
return b64encode(s)
File "C:\Miniconda3\envs\dash_p36\lib\base64.py", line 58, in b64encode
encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'str'
I tried doing with open('./sample.csv', 'rb') as f: before passing the blocks to base64.standard_b64encode but then getting another error:
TypeError: Object of type 'bytes' is not JSON serializable
This happens when the encoded block data is being sent into the API call.
I tried skipping encoding entirely and just passing the blocks into the post call. In this case the file gets created in the DBFS but has 0 bytes size.
At this point I'm trying to make sense of it all. It doesn't want a string but it doesn't want bytes either. What am I doing wrong? Appreciate any help.
In Python we have strings and bytes, which are two different entities note that there is no implicit conversion between them, so you need to know when to use which and how to convert when necessary. This answer provides nice explanation.
With the code snippet I see two issues:
This you already got - open by default reads the file as text. So your block is a string, while standard_b64encode expects bytes and returns bytes. To read bytes from file it needs to be opened in binary mode:
with open('/a/local/file', 'rb') as f:
Only strings can be encoded as JSON. There's no source code available for dbfs_rpc (or I can't find it), but apparently it expects a string, which it internally encodes. Since your data is bytes, you need to convert it to string explicitly and that's done using decode:
dbfs_rpc("add-block", {"handle": handle, "data": data.decode('utf8')})
Usually request.json is used to access json objects send to a flask view. I would like to assign something to request.json
from flask import request
print("request.json = ", request.json)
print("request.form['json'] = ", request.form['json'])
request.json = jsonify(request.form['json'])
leads to
request.json = None
request.form['json'] = {
"test": "test"
}
Traceback (most recent call last):
...
File "/ajax_handlers.py", line 952, in X
request.json = jsonify(request.form['json'])
File "/opt/anaconda3/lib/python3.7/site-packages/werkzeug/local.py", line 365, in <lambda>
__setattr__ = lambda x, n, v: setattr(x._get_current_object(), n, v)
AttributeError: can't set attribute
Any idea how I can assign a json object to request.json?
Presumably for safety, you cannot replace the values on a request object; they come in from the client, and they remain as-is. When you think about it, that's good practice; if various pieces of code try to modify your request object's data in various ways, it would lead to inconsistent and hard-to-test code.
As an alternative, you can assign meaningful attributes that aren't part of the default request object. For example, perhaps you're trying to remember, for elsewhere in your stack, who the user is authenticated as:
# Don't do this!
flask.request.json['authenticated_user'] = authenticated_user
That's messing with the original request data; don't do that. It'll make debugging (not to mention security) a nightmare. A better alternative:
# Do this instead
flask.request.authenticated_user = authenticated_user
You can add to the existing request with new attributes, but you can't go replacing existing properties on the request. Nor should you try!
I'm trying out Falcon for a small api project. Unfortunate i'm stuck on the json parsing stuff and code from the documentation examples does not work.
I have tried so many things i've found on Stack and Google but no changes.
I've tried the following codes that results in the errors below
import json
import falcon
class JSON_Middleware(object):
def process_request(self, req, resp):
raw_json = json.loads(req.stream.read().decode('UTF-8'))
"""Exception: AttributeError: 'str' object has no attribute 'read'"""
raw_json = json.loads(req.stream.read(), 'UTF-8')
"""Exception: TypeError: the JSON object must be str, not 'bytes'"""
raw_json = json.loads(req.stream, 'UTF-8')
"""TypeError: the JSON object must be str, not 'Body'"""
I'm on the way of giving up, but if somebody can tell me why this is happening and how to parse JSON in Falcon i would be extremely thankful.
Thanks
Environment:
OSX Sierra
Python 3.5.2
Falcon and other is the latest version from Pip
your code should work if other pieces of code are in place . a quick test(filename app.py):
import falcon
import json
class JSON_Middleware(object):
def process_request(self, req, resp):
raw_json = json.loads(req.stream.read())
print raw_json
class Test:
def on_post(self,req,resp):
pass
app = application = falcon.API(middleware=JSON_Middleware())
t = Test()
app.add_route('/test',t)
run with: gunicorn app
$ curl -XPOST 'localhost:8000' -d '{"Hello":"wold"}'
You have to invoke encode() on the bytes returned by read() with something like req.stream.read().encode('utf-8').
This way the bytes are converted to a str as expected by json.loads().
The other way not to bother with all this boring and error prone encode/decode and bytes/str stuff (which BTW differs in Py2 and Py3), is to use simplejson as a replacement for json. It is API compatible, so the only change is to replace import json with import simplejson as json in your code.
In addition, it simplifies the code since reading the body can be done with json.load(req.bounded_stream), which is much shorter and more readable than json.loads(req.bounded_stream.read().encode('utf-8')).
I now do it this way, and don't use the standard json module any more.
I'm beginning to use CGI with Python.
After running the following piece of code:
#!c:\python34\python.exe
import cgi
print("Content-type: text/html\n\n") #important
def getData():
formData = cgi.FieldStorage()
InputUN = formData.getvalue('username')
InputPC = formData.getvalue('passcode')
TF = open("TempFile.txt", "w")
TF.write(InputUN)
TF.write(InputPC)
TF.close()
if __name__ =="__main__":
LoginInput = getData()
print("cgi worked")
The following error occurs:
Traceback (most recent call last):
File "C:\xampp\htdocs\actual\loginvalues.cgi", line 21, in <module>
LoginInput = getData()
File "C:\xampp\htdocs\actual\loginvalues.cgi", line 16, in getData
TF.write(InputUN)
TypeError: must be str, not None
>>>
I'm trying to write the values, inputted in html, to a text file.
Any help would be appreciated :)
Your calls to getValue() are returning None, meaning the form either didn't contain them, had them set to an empty string, or had them set by name only. Python's CGI module ignores inputs that aren't set to a non-null string.
Works for Python CGI:
mysite.com/loginvalues.cgi?username=myname&pass=mypass
Doesn't work for Python CGI:
mysite.com/loginvalues.cgi?username=&pass= (null value(s))
mysite.com/loginvalues.cgi?username&pass (Python requires the = part.)
To account for this, introduce a default value for when a form element is missing, or handle the None case manually:
TF.write('anonymous' if InputUN is None else InputUN)
TF.write('password' if InputPC is None else InputUN)
As a note, passwords and other private login credentials should never be used in a URL. URLs are not encrypted. Even in HTTPS, the URL is sent in plain text that anyone on the network(s) between you and your users can read.
The only time a URL is ever encrypted is over a tunneled SSH port or an encrypted VPN, but you can't control that, so never bank on it.
There have been a number of other questions on AttributeErrors here, but I've read through them and am still not sure what's causing the type mismatch in my specific case.
Thanks in advance for any thoughts on this.
My model:
class Object(db.Model):
notes = db.StringProperty(multiline=False)
other_item = db.ReferenceProperty(Other)
time = db.DateTimeProperty(auto_now_add=True)
new_files = blobstore.BlobReferenceProperty(required=True)
email = db.EmailProperty()
is_purple = db.BooleanProperty()
My BlobstoreUploadHandler:
class FormUploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
try:
note = self.request.get('notes')
email_addr = self.request.get('email')
o = self.request.get('other')
upload_file = self.get_uploads()[0]
# Save the object record
new_object = Object(notes=note,
other=o,
email=email_addr,
is_purple=False,
new_files=upload_file.key())
db.put(new_object)
# Redirect to let user know everything's peachy.
self.redirect('/upload_success.html')
except:
self.redirect('/upload_failure.html')
And every time I submit the form that uploads the file, it throws the following exception:
ERROR 2010-10-30 21:31:01,045 __init__.py:391] 'unicode' object has no attribute 'has_key'
Traceback (most recent call last):
File "/home/user/Public/dir/google_appengine/google/appengine/ext/webapp/__init__.py", line 513, in __call__
handler.post(*groups)
File "/home/user/Public/dir/myapp/myapp.py", line 187, in post
new_files=upload_file.key())
File "/home/user/Public/dir/google_appengine/google/appengine/ext/db/__init__.py", line 813, in __init__
prop.__set__(self, value)
File "/home/user/Public/dir/google_appengine/google/appengine/ext/db/__init__.py", line 3216, in __set__
value = self.validate(value)
File "/home/user/Public/dir/google_appengine/google/appengine/ext/db/__init__.py", line 3246, in validate
if value is not None and not value.has_key():
AttributeError: 'unicode' object has no attribute 'has_key'
What perplexes me most is that this code is nearly straight out of the documentation, and jives with other examples of blob upload handler's I've found online in tutorials as well.
I've run --clear-datastore to ensure that any changes I've made to the DB schema aren't causing problems, and have tried casting upload_file as all sorts of things to see if it would appease Python - any ideas on what I've screwed up?
Edit: I've found a workaround, but it's suboptimal.
Altering the UploadHandler to this instead resolves the issue:
...
# Save the object record
new_object = Object()
new_object.notes = note
new_object.other = o
new_object.email = email.addr
new_object.is_purple = False
new_object.new_files = upload_file.key()
db.put(new_object)
...
I made this switch after noticing that commenting out the files line threw the same issues for the other line, and so on. This isn't an optimal solution, though, as I can't enforce validation this way (in the model, if I set anything as required, I can't declare an empty entity like above without throwing an exception).
Any thoughts on why I can't declare the entity and populate it at the same time?
You're passing in o as the value of other_item (in your sample code, you call it other, but I presume that's a typo). o is a string fetched from the request, though, and the model definition specifies that it's a ReferenceProperty, so it should either be an instance of the Other class, or a db.Key object.
If o is supposed to be a stringified key, pass in db.Key(o) instead, to deserialize it.
Object is a really terrible name for a datastore class (or any class, really), by the way - the Python base object is called object, and that's only one capitalized letter away - very easy to mistake.
has_key error is due to the ReferenceProperty other_items. You are most likely passing in '' for other_items when appengine's api expects a dict. In order to get around this, you need to convert other_items to hash.
[caveat lector: I know zilch about "google_app_engine"]
The message indicates that it is expecting a dict (the only known object that has a has_key attribute) or a work-alike object, not the unicode object that you supplied. Perhaps you should be passing upload_file, not upload_file.key() ...