We Keep Coding

Requests dict from cookiejar issue with escaped chars

I'm running into some issues getting a cookie into a dictionary with python. It seems to be all escaped somehow even after running the command provided by requests.
resp = requests.get(geturl, cookies=cookies)
cookies = requests.utils.dict_from_cookiejar(resp.cookies)
and this is what cookies looks like
{'P-fa9d887b1fe1a997d543493080644610': '"\\050dp1\\012S\'variant\'\\012p2\\012S\'corrected\'\\012p3\\012sS\'pid\'\\012p4\\012VNTA2NjU0OTU4MDc5MTgwOA\\075\\075\\012p5\\012sS\'format\'\\012p6\\012S\'m3u8\'\\012p7\\012sS\'mode\'\\012p8\\012Vlive\\012p9\\012sS\'type\'\\012p10\\012S\'video/mp2t\'\\012p11\\012s."'}
Is there any way to make the characters unescaped in the value section of P-fa9d887b1fe1a997d543493080644610 become escaped and part of the dict itself?
Edit:
I would like the dictionary to look something like:
{'format': 'm3u8', 'variant': 'corrected', 'mode': u'live', 'pid': u'NTA2NjU0OTU4MDc5MTgwOA==', 'type': 'video/mp2t'}

You are dealing with the Python Pickle format for data serialisation. Once you have evaluated the expression, so escaped characters are unescaped, you need to load the pickle from a string using the pickle.loads function.
>>> import pickle
>>> import ast
>>> pickle.loads(ast.literal_eval("'''" + cookies.values()[0] + "'''")[1:-1])
{'pid': u'NTA2NjU0OTU4MDc5MTgwOA==', 'type': 'video/mp2t', 'variant': 'corrected', 'mode': u'live', 'format': 'm3u8'}

Does boto.dynamodb2 support storing a dict of dicts?

Solution: Updating to the brand-new boto 2.35.2 fixed the problem.
How can I store a dict of dicts in DynamoDB using boto?
The straightforward approach that I've been trying doesn't seem to work. Trying to save an item defined this way:
data = {
'id': '123456',
'foo': {'hello': 'world'}
}
item = Item(my_table, data=data)
item.save(overwrite=True)
generates this exception:
TypeError: Unsupported type "<type 'dict'>" for value "{'hello': 'world'}"
I've seen conflicting information on the web about whether this is supported. I can't get it to work; am using boto 2.35.1.
Here's a complete example that demonstrates the problem:
import boto.dynamodb2
from boto.dynamodb2.fields import HashKey
from boto.dynamodb2.table import Table
from boto.dynamodb2.items import Item
conn = boto.dynamodb2.connect_to_region('us-east-1')
my_table = Table.create('my_table',
connection=conn,
schema=[
HashKey('id')
])
my_table = Table('my_table')
data = {
'id': '123456',
'foo': {'hello': 'world'}
}
item = Item(my_table, data=data)
item.save(overwrite=True)

DynamoDB API now supports map and list object:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html#HowItWorks.DataTypes
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DataFormat.html
And seams that last boto version (yesterday at the time I answered) added this too:
http://boto.readthedocs.org/en/latest/releasenotes/v2.35.2.html
but personally I didn't play with yet.

Confused by Python returning JSON as string instead of literal

I've done some coding in RoR, and in Rails, when I return a JSON object via an API call, it returns as
{ "id" : "1", "name" : "Dan" }.
However in Python (with Flask and Flask-SQLAlchemy), when I return a JSON object via json.dumps or jsonpickle.encode it is returned as
"{ \"id\" : \"1\", \"name\": \"Dan\" }" which seems very unwieldily as it can't easily be parsed on the other end (by an iOS app in this case - Obj-C).
What am I missing here, and what should I do to return it as a JSON literal, rather than a JSON string?
This is what my code looks like:
people = models.UserRelationships.query.filter_by(user_id=user_id, active=ACTIVE_RECORD)
friends = people.filter_by(friends=YES)
json_object = jsonpickle.encode(friends.first().as_dict(), unpicklable=False, keys=True)
print(json_object) # this prints here, i.e. { "id" : "1", "name" : "Dan" }
return json_object # this returns "{ \"id\" : \"1\", \"name\": \"Dan\" }" to the browser

What is missing in your understanding here is that when you use the JSON modules in Python, you're not working with a JSON object. JSON is by definition just a string that matches a certain standard.
Lets say you have the string:
friends = '{"name": "Fred", "id": 1}'
If you want to work with this data in python, you will want to load it into a python object:
import json
friends_obj = json.loads(friends)
At this point friends_obj is a python dictionary.
If you want to convert it (or any other python dictionary or list) then this is where json.dumps comes in handy:
friends_str = json.dumps(friends_obj)
print friends_str
'{"name": "Fred", "id": 1}'
However if we attempt to "dump" the original friends string you'll see you get a different result:
dumped_str = json.dumps(friends)
print dumped_str
'"{\\"name\\": \\"Fred\\", \\"id\\": 1}"'
This is because you're basically attempting to encode an ordinary string as JSON and it is escaping the characters. I hope this helps make sense of things!
Cheers

Looks like you are using Django here, in which case do something like
from django.utils import simplejson as json
...
return HttpResponse(json.dumps(friends.first().as_dict()))

This is almost always a sign that you're double-encoding your data somewhere. For example:
>>> obj = { "id" : "1", "name" : "Dan" }
>>> j = json.dumps(obj)
>>> jj = json.dumps(j)
>>> print(obj)
{'id': '1', 'name': 'Dan'}
>>> print(j)
{"id": "1", "name": "Dan"}
>>> print(jj)
"{\"id\": \"1\", \"name\": \"Dan\"}"
Here, jj is a perfectly valid JSON string representation—but it's not a representation of obj, it's a representation of the string j, which is useless.
Normally you don't do this directly; instead, either you started with a JSON string rather than an object in the first place (e.g., you got it from a client request or from a text file), or you called some function in a library like requests or jsonpickle that implicitly calls json.dumps with an already-encoded string. But either way, it's the same problem, with the same solution: Just don't double-encode.

You should be using flask.jsonify, which will not only encode correctly, but also set the content-type headers accordingly.
people = models.UserRelationships.query.filter_by(user_id=user_id, active=ACTIVE_RECORD)
friends = people.filter_by(friends=YES)
return jsonify(friends.first().as_dict())

How to url encode the binary contents of a video in Python?

I wanted to port video to Tumblr using the API using the Tumblpy library.
My code is this:
import requests
r = requests.get(video-url)
f = {'data':r.content}
dat = urllib.urlencode(f)
t.post('post', blog_url='http://tumblrname.tumblr.com/',params={'type':'video',
'title':post.title, 'slug': post.slug,'date':post.date,'data':dat,'tags':post.tagscsv,
'caption': post.body_html}) #t is TumblPy instance
Well, I am not being successful in this. I do think I am missing out on how to encode the binary contents to make the post successful, though I am not sure.

Presumably it's going to be similar to how you post a photo, in which case the library wants a file(like) object. A requests response can act as a file-like object just fine:
import requests
r = requests.get(video_url)
t.post('post', blog_url='http://tumblrname.tumblr.com/',
params={'type': 'video', 'title': post.title, 'slug': post.slug,
'date': post.date, 'data': r.raw, 'tags': post.tagscsv,
'caption': post.body_html})
where r.raw gives you a file-like object that, when read, yields the video data read from video_url.

jquery: post with json will actually post array

I have a python as CGI and the POST from jquery will transform json object to array, so when I see the POST from jquery, I actually see:
login_user[username]=dfdsfdsf&login_user[password]=dsfsdf
(the [ and ] already escaped)
My question is how I can convert this string back to JSON in python? Or, how can I convert this string to python array/dict structure so that I can process it easier?
[edit]
My jquery is posting:
{'login_user': {'username':username, 'password':password}}

If what you want to accomplish is to send structured data from the browser and then unpack it in your Python backend and keep the same structure, I suggest the following:
Create JavaScript objects in the browser to hold your data:
var d = {}
d['login_user'] = { 'username': 'foo', 'password': 'bar' }
Serialize to JSON, with https://github.com/douglascrockford/JSON-js
POST to your backend doing something like this:
$.post(url, {'data': encoded_json_data}, ...)
In your Python code, parse the JSON, POST in my example is where you get your POST data in your CGI script:
data = json.loads(POST['data'])
data['login_user']

import re
thestring = "login_user[username]=dfdsfdsf&login_user[password]=dafef"
pattern = re.compile(r'^login_user\[username\]=(.*)&login_user\[password\]=(.*)')
match = pattern.search(thestring)
print match.groups()
Output:
>>> ('dfdsfdsf', 'dafef')
Thus,
lp = match.groups()
print "{'login_user': {'username':"+lp[0]+", 'password':"+lp[1]+"}}"
shall bear: >>> {'login_user': {'username':dfdsfdsf, 'password':dafef}}

>>> import json
>>> data = {'login_user':{'username':'dfdsfdsf', 'password':'dsfsdf'}}
>>> json.dumps(data)
'{"login_user": {"username": "dfdsfdsf", "password": "dsfsdf"}}'
I suspect that data would already be contained in a GET var if that's coming from the URL...