Google app engine key value error - python

I am writing a google app engine app and I have this key value error upon requests coming in
from the backtrace I just access and cause the key error
self.request.headers
entire code snippet is here, I just forward the headers unmodified
response = fetch( "%s%s?%s" % (
self.getApiServer() ,
self.request.path.replace("/twitter/", ""),
self.request.query_string
),
self.request.body,
method,
self.request.headers,
)
and get method handling the request calling proxy()
# handle http get
def get(self, *args):
parameters = self.convertParameters(self.request.query_string)
# self.prepareHeader("GET", parameters)
self.request.query_string = "&".join("%s=%s" % (quote(key) , quote(value)) for key, value in parameters.items())
self.proxy(GET, *args)
def convertParameters(self, source):
parameters = {}
for pairs in source.split("&"):
item = pairs.split("=")
if len(item) == 2:
parameters[item[0]] = unquote(item[1])
return parameters
the error back trace:
'CONTENT_TYPE'
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 513, in __call__
handler.post(*groups)
File "/base/data/home/apps/waytosing/1.342850593213842824/com/blogspot/zizon/twitter/RestApiProxy.py", line 67, in post
self.proxy(POST, *args)
File "/base/data/home/apps/waytosing/1.342850593213842824/com/blogspot/zizon/twitter/RestApiProxy.py", line 47, in proxy
self.request.headers,
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/urlfetch.py", line 240, in fetch
allow_truncated, follow_redirects)
File "/base/python_runtime/python_lib/versions/1/google/appengine/api/urlfetch.py", line 280, in make_fetch_call
for key, value in headers.iteritems():
File "/base/python_runtime/python_dist/lib/python2.5/UserDict.py", line 106, in iteritems
yield (k, self[k])
File "/base/python_runtime/python_lib/versions/1/webob/datastruct.py", line 40, in __getitem__
return self.environ[self._trans_name(item)]
KeyError: 'CONTENT_TYPE'
Any idea why it happens or is this a known bug?

This looks weird. The docs mention that response "Headers objects do not raise an error when you try to get or delete a key that isn't in the wrapped header list. Getting a nonexistent header just returns None". It's not clear from the request documentation if request.headers are also objects of this class, but even they were regular dictionaries, iteritems seems to be misbehaving. So this might be a bug.
It might be worth inspecting self.request.headers, before calling fetch, and see 1) its actual type, 2) its keys, and 3) if trying to get self.request.headers['CONTENT_TYPE'] raises an error then.
But, if you simply want to solve your problem and move forward, you can try to bypass it like:
if 'CONTENT_TYPE' not in self.request.headers:
self.request.headers['CONTENT_TYPE'] = None
(I'm suggesting setting it to None, because that's what a response Header object should return on non-existing keys)

Here's my observation about this problem:
When the content-type is application/x-www-form-urlencoded and POST data is empty (e.g. jquery.ajax GET, twitter's favorite and retweet API...), the content-type is dropped by Google appengine.
You can add:
self.request.headers.update({'content-type':'application/x-www-form-urlencoded'})
before urlfetch.

Edit: indeed, looking at the error more carefully, it doesn't seem to be related to convertParameters, as the OP points out in the comments. I'm retiring this answer.
I'm not entirely sure what you mean by "just forward the headers unmodified", but have you taken a look at self.request.query_string before and after you call convertParameters? More to the point, you're leaving out any (valid) GET parameters of the form "key=" (that is, keys with empty values).
Maybe your original query_string had a value like "CONTENT_TYPE=", and your convertParameters is stripping it out.

Known issue http://code.google.com/p/googleappengine/issues/detail?id=3427 and potential workarounds here http://code.google.com/p/googleappengine/issues/detail?id=2040

Related

I want to make a playlist from all my liked songs on spotify with spotipy library

Since my liked songs aren't public I want spotipy to get a list from all the songs and add them to my playlist, but when I try to do that with a loop it says that the uri is incorrect, I don't know if I should use another method.
client_credentials_manager = SpotifyClientCredentials(client_id=cid, client_secret=secret)
scope = 'user-library-read playlist-modify-public'
sp = spotipy.Spotify(client_credentials_manager=client_credentials_manager,auth_manager=SpotifyOAuth(scope=scope))
def show_tracks(results):
for item in results['items']:
track = item['track']
#print("%32.32s %s" % (track['artists'][0]['name'], track['name']))
sp.playlist_add_items(playlist_id, track['uri'])
results = sp.current_user_saved_tracks()
show_tracks(results)
while results['next']:
results = sp.next(results)
show_tracks(results)
The error is
HTTP Error for POST to https://api.spotify.com/v1/playlists/5ZzsovDgANZfiXgRrwq5fw/tracks returned 400 due to Invalid track uri: spotify:track:s
Traceback (most recent call last):
File "C:\Users\ferch\AppData\Local\Programs\Python\Python37\lib\site-packages\spotipy\client.py", line 245, in _internal_call
response.raise_for_status()
File "C:\Users\ferch\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.spotify.com/v1/playlists/5ZzsovDgANZfiXgRrwq5fw/tracks
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "make_playlists.py", line 23, in <module>
show_tracks(results)
File "make_playlists.py", line 20, in show_tracks
sp.playlist_add_items(playlist_id, track['uri'])
File "C:\Users\ferch\AppData\Local\Programs\Python\Python37\lib\site-packages\spotipy\client.py", line 1025, in playlist_add_items
position=position,
File "C:\Users\ferch\AppData\Local\Programs\Python\Python37\lib\site-packages\spotipy\client.py", line 296, in _post
return self._internal_call("POST", url, payload, kwargs)
File "C:\Users\ferch\AppData\Local\Programs\Python\Python37\lib\site-packages\spotipy\client.py", line 266, in _internal_call
headers=response.headers,
spotipy.exceptions.SpotifyException: http status: 400, code:-1 - https://api.spotify.com/v1/playlists/5ZzsovDgANZfiXgRrwq5fw/tracks:
Invalid track uri: spotify:track:s, reason: None
I think this problem is because of the type of variable of track['uri']
playlist_add_items is expecting a list of URIs, URLs, or IDs to add to the playlist, but right now you're passing a single URI, which is a string like this: spotify:track:2t7rS8BHF5TmnBR5PmnnSU. The code for the spotipy library is likely doing a loop for item in items..., so when you pass it a string, it considers each character in the string as a different item. So it encounters the first character, s and tries to make a URI out of it resulting in spotify:track:s. This isn't a valid URI, so the request fails.
You can try wrapping the uri in a list like so:
for item in results['items']:
track = item['track']
# Note brackets around track['uri']
sp.playlist_add_items(playlist_id, [track['uri']])
This will handle the issue you're getting now, but you may have issues down the line making one request per track you want to add to the playlist. You could run into rate limiting issues, so I recommend trying to build a list of 100 URIs at a time, which is the max that can be sent in one request.
Keeping this in mind, we could try something like this:
def show_tracks(results):
for idx in range(0, len(results['items']), 100):
uris = [item['track']['uri'] for item in results['items'][idx:idx+100]]
sp.playlist_add_items(playlist_id, uris)
Another way to do this would be to create a list with all the uris/ids of the tracks you want to add, and then pass that list into the sp.playlist_add_items() function. This could be useful if you need the list of uris again further down the line. Like so :
uris = []
for item in results['items']:
track = item['track']
uris.append(track['uri'])
sp.playlist_add_items(playlist_id, uris)
Bear in mind, sp.playlist_add_items only lets you add <= 100 tracks at a time. i created this loop to handle adding a list of tracks no matter the size: (where songIDs is a list of song ids / uris)
i = 0
increment = 99
while i < len(songIDS)+increment:
try:
sp.playlist_add_items(playlistID, songIDS[i: i+increment])
except spotipy.exceptions.SpotifyException:
pass
i += increment
Hope this helps, i've only been using spotipy for a week myself

How to call an Odoo model's method with no parameter(except self) on a specific record through xmlrpc in Odoo 13?

I am developing a script to create a record in a model of an Odoo. I need to run this model's methods on specific records. In my case the method which I need to run on a specific record doesn't have any parameter (just has self). I want to know how can I run the method on a specific record of the model through xmlrpc call from client to Odoo server. Below is the way I tried to call the method and pass the id of a specific record regarding this question.
xmlrpc_object.execute('test_db', user, 'admin', 'test.test', 'action_check_constraint', [record_id])
action_check_constraint checks some constraints on each record of the model and if all the constraints passed, changes the state of the record or raise validation errors. But the above method call with xmlrpc raise below error:
xmlrpc.client.Fault: <Fault cannot marshal None unless allow_none is enabled: 'Traceback (most recent call last):\n File "/home/ibrahim/workspace/odoo13/odoo/odoo/addons/base/controllers/rpc.py", line 60, in xmlrpc_1\n response = self._xmlrpc(service)\n File "/home/ibrahim/workspace/odoo13/odoo/odoo/addons/base/controllers/rpc.py", line 50, in _xmlrpc\n return dumps((result,), methodresponse=1, allow_none=False)\n File "/usr/local/lib/python3.8/xmlrpc/client.py", line 968, in dumps\n data = m.dumps(params)\n File "/usr/local/lib/python3.8/xmlrpc/client.py", line 501, in dumps\n dump(v, write)\n File "/usr/local/lib/python3.8/xmlrpc/client.py", line 523, in __dump\n f(self, value, write)\n File "/usr/local/lib/python3.8/xmlrpc/client.py", line 527, in dump_nil\n raise TypeError("cannot marshal None unless allow_none is enabled")\nTypeError: cannot marshal None unless allow_none is enabled\n'>
> /home/ibrahim/workspace/scripts/automate/automate_record_creation.py(328)create_record()
Can anyone help with the correct and best way of calling a model's method (with no parameter except self) on a specific record through xmlrpc client to Odoo server?
That error is raised, because the xmlrpc library is not allowing None as return value as default. But you should change that behaviour by just allowing it.
Following line is from Odoo's external API documentation, extended to allow None as return value:
models = xmlrpc.client.ServerProxy(
'{}/xmlrpc/2/object'.format(url), allow_none=True)
For more information about xmlrpc ServerProxy look into the python documentation
You can get the error if action_check_constraint does not return anything (by default None).
Try to run the server with the log-level option set to debug_rpc_answer to get more details.
After lost of search and try first I used this fix to solve the error but I think this fix is not a best practice. So, I found OdooRPC which does the same job but it handled the above case and there's no such error for model methods which return None. Using OdooRPC solved my problem and I done what I needed to do with xmlrpc in Odoo.

Assign a json object to request.json in flask

Usually request.json is used to access json objects send to a flask view. I would like to assign something to request.json
from flask import request
print("request.json = ", request.json)
print("request.form['json'] = ", request.form['json'])
request.json = jsonify(request.form['json'])
leads to
request.json = None
request.form['json'] = {
"test": "test"
}
Traceback (most recent call last):
...
File "/ajax_handlers.py", line 952, in X
request.json = jsonify(request.form['json'])
File "/opt/anaconda3/lib/python3.7/site-packages/werkzeug/local.py", line 365, in <lambda>
__setattr__ = lambda x, n, v: setattr(x._get_current_object(), n, v)
AttributeError: can't set attribute
Any idea how I can assign a json object to request.json?
Presumably for safety, you cannot replace the values on a request object; they come in from the client, and they remain as-is. When you think about it, that's good practice; if various pieces of code try to modify your request object's data in various ways, it would lead to inconsistent and hard-to-test code.
As an alternative, you can assign meaningful attributes that aren't part of the default request object. For example, perhaps you're trying to remember, for elsewhere in your stack, who the user is authenticated as:
# Don't do this!
flask.request.json['authenticated_user'] = authenticated_user
That's messing with the original request data; don't do that. It'll make debugging (not to mention security) a nightmare. A better alternative:
# Do this instead
flask.request.authenticated_user = authenticated_user
You can add to the existing request with new attributes, but you can't go replacing existing properties on the request. Nor should you try!

Upload file to GCS from Appengine (Endpoints). AttributeTypeError:

I'm trying to upload a file to GCS from Appengine Endpoints. I'm using Python. When the file ends to upload, shows an error " AttributeError: 'str' object has no attribute 'ToMessage' ".
So, if I go to GCS File Explorer, in the browser, I see the recently filename uploaded but its size is 0K.
This is my model:
class File(EndpointsModel):
_message_fields_schema = ('blob', 'url')
blob = ndb.BlobKeyProperty() #stored in GCS
url = ndb.StringProperty()
enable = ndb.BooleanProperty(default=True)
def create_file(filename):
file_info = blobstore.FileInfo(filename)
filename = '/gs'+ str(file_info.filename.blob)
gcs.open(secrets.BUCKET_NAME +'/' + filename, 'w').close()
return blobstore.create_gs_key(filename)
So, what I need to do to upload correctly a file to GCS from Appengine Endpoints.
Traceback:
ERROR 2014-11-25 20:35:22,654 service.py:191] Encountered unexpected error from ProtoRPC method implementation: AttributeError ('str' object has no attribute 'ToMessage')
Traceback (most recent call last):
File "/home/alpocr/workspace/google_appengine/lib/protorpc-1.0/protorpc/wsgi/service.py", line 181, in protorpc_service_app
response = method(instance, request)
File "/home/alpocr/workspace/google_appengine/lib/endpoints-1.0/endpoints/api_config.py", line 1332, in invoke_remote
return remote_method(service_instance, request)
File "/home/alpocr/workspace/google_appengine/lib/protorpc-1.0/protorpc/remote.py", line 412, in invoke_remote_method
response = method(service_instance, request)
File "/home/alpocr/workspace/mall4g-backend/libs/endpoints_proto_datastore/ndb/model.py", line 1429, in EntityToRequestMethod
response = response.ToMessage(fields=response_fields)
AttributeError: 'str' object has no attribute 'ToMessage'
It sounds like you have defined the return type correctly for your endpoints method, and it's expecting to turn the result into a Message object, but the endpoints method code is actually returning a string. Can you post the endpoints method that is called when this error occurs?
Either that or endpoints proto model is acting weird when you (somewhere in your code) assign a string value to one of its properties. When it tries to convert it to a Message (and thus recursively to turn its properties into Messages), it finds the String and bugs out. It's hard to tell without seeing the affected endpoint method's code.
UPDATE: Also, checking the source of endpoints_proto_datastore, we see the following comment above the line that bugs:
# If developers using a custom request message class with
# response_fields to create a response message class for them, it is
# up to them to return an instance of the current EndpointsModel
# class. If not, their API users will receive a 503 from an uncaught
# exception.
Could this apply to you?

AppEngine -> "AttributeError: 'unicode' object has no attribute 'has_key'" when using blobstore

There have been a number of other questions on AttributeErrors here, but I've read through them and am still not sure what's causing the type mismatch in my specific case.
Thanks in advance for any thoughts on this.
My model:
class Object(db.Model):
notes = db.StringProperty(multiline=False)
other_item = db.ReferenceProperty(Other)
time = db.DateTimeProperty(auto_now_add=True)
new_files = blobstore.BlobReferenceProperty(required=True)
email = db.EmailProperty()
is_purple = db.BooleanProperty()
My BlobstoreUploadHandler:
class FormUploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
try:
note = self.request.get('notes')
email_addr = self.request.get('email')
o = self.request.get('other')
upload_file = self.get_uploads()[0]
# Save the object record
new_object = Object(notes=note,
other=o,
email=email_addr,
is_purple=False,
new_files=upload_file.key())
db.put(new_object)
# Redirect to let user know everything's peachy.
self.redirect('/upload_success.html')
except:
self.redirect('/upload_failure.html')
And every time I submit the form that uploads the file, it throws the following exception:
ERROR 2010-10-30 21:31:01,045 __init__.py:391] 'unicode' object has no attribute 'has_key'
Traceback (most recent call last):
File "/home/user/Public/dir/google_appengine/google/appengine/ext/webapp/__init__.py", line 513, in __call__
handler.post(*groups)
File "/home/user/Public/dir/myapp/myapp.py", line 187, in post
new_files=upload_file.key())
File "/home/user/Public/dir/google_appengine/google/appengine/ext/db/__init__.py", line 813, in __init__
prop.__set__(self, value)
File "/home/user/Public/dir/google_appengine/google/appengine/ext/db/__init__.py", line 3216, in __set__
value = self.validate(value)
File "/home/user/Public/dir/google_appengine/google/appengine/ext/db/__init__.py", line 3246, in validate
if value is not None and not value.has_key():
AttributeError: 'unicode' object has no attribute 'has_key'
What perplexes me most is that this code is nearly straight out of the documentation, and jives with other examples of blob upload handler's I've found online in tutorials as well.
I've run --clear-datastore to ensure that any changes I've made to the DB schema aren't causing problems, and have tried casting upload_file as all sorts of things to see if it would appease Python - any ideas on what I've screwed up?
Edit: I've found a workaround, but it's suboptimal.
Altering the UploadHandler to this instead resolves the issue:
...
# Save the object record
new_object = Object()
new_object.notes = note
new_object.other = o
new_object.email = email.addr
new_object.is_purple = False
new_object.new_files = upload_file.key()
db.put(new_object)
...
I made this switch after noticing that commenting out the files line threw the same issues for the other line, and so on. This isn't an optimal solution, though, as I can't enforce validation this way (in the model, if I set anything as required, I can't declare an empty entity like above without throwing an exception).
Any thoughts on why I can't declare the entity and populate it at the same time?
You're passing in o as the value of other_item (in your sample code, you call it other, but I presume that's a typo). o is a string fetched from the request, though, and the model definition specifies that it's a ReferenceProperty, so it should either be an instance of the Other class, or a db.Key object.
If o is supposed to be a stringified key, pass in db.Key(o) instead, to deserialize it.
Object is a really terrible name for a datastore class (or any class, really), by the way - the Python base object is called object, and that's only one capitalized letter away - very easy to mistake.
has_key error is due to the ReferenceProperty other_items. You are most likely passing in '' for other_items when appengine's api expects a dict. In order to get around this, you need to convert other_items to hash.
[caveat lector: I know zilch about "google_app_engine"]
The message indicates that it is expecting a dict (the only known object that has a has_key attribute) or a work-alike object, not the unicode object that you supplied. Perhaps you should be passing upload_file, not upload_file.key() ...

Categories