How to MySQL to store multi-params for HTTP POST? - python

HTTP Post may have multiple params such as:
http://example.com/controller/ (while params=({'country':'US'},{'city':'NYC'})
I am developing a web spider with Python, I face a problem how to track difference with same url with different params. Now I can load the content, but I have no idea how to store the post params in a field of SQLite3 table. It is easy to store the params in database like MySQL for system developer, but since the params of different sites are various. I prefer to store the post params in single field, rather than one-on-one relationship mapping in a table.
HTTP GET
>>> import urllib
>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)
>>> print f.read()
HTTP POST
>>> import urllib
>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
>>> print f.read()
My project configuration:
Python + SQLite3
Store http url and post params and tracking the changes.
The post params contains multiple key-value pairs
The stored params should be decoded back to params.
The encode issues should be covered.
I saw multiple solutions like JSON, XML and YAML. I guess this format actually stored as string (CHAR) type in SQLite, in UTF-8. But I have no idea if there is any handy way to convert them back to Python tuple type? Or, can I encode the post params into get params with + and & symbal, and decode it back to post params?
Sorry, I am just a newbie for Python.

You can convert to and from json easily like this:
>>> import json
>>> json.dumps({'spam': 1, 'eggs': 2, 'bacon': 0})
'{"eggs": 2, "bacon": 0, "spam": 1}'
>>> json.loads('{"eggs": 2, "bacon": 0, "spam": 1}')
{u'eggs': 2, u'bacon': 0, u'spam': 1}
>>> json.dumps((1,2,3,4))
'[1, 2, 3, 4]'
>>> json.loads('[1, 2, 3, 4]')
[1, 2, 3, 4]
>>>
Better use it because it is more versatile than home made & separated encoding, it supports any nesting complexity.

I would probably go with Frost's suggestion - JSON encoding is far more robust. However, there have been situations in the past where I've been forced to go a simpler route:
>>> d = {'spam': 1, 'eggs': 2, 'bacon': 0}
>>> l = [(a +":"+str(b)) for a,b in d.items()]
>>> ','.join(l)
'eggs:2,bacon:0,spam:1'
Obviously your delimiters (, and : in this case) need to be carefully chosen, but this works in a pinch.

Related

Python - Extracting data from text to use

probably a basic question to most.
Though I am looking at getting some specific values that would follow certain words within a mass amount of text like so - (https://character-service.dndbeyond.com/character/v5/character/00000001)
In this instance if I wanted to take the value after baseHitPoints and also the value after removedHitPoints how would I go about this in the simplest way? The aim is to take these values and any others, even potentially text after a certain word, and store them as strings/integers to use in calculations.
I am also using BeautifulSoup to get this information if that effects anything in your answers.
Thanks in advance!
Tried a few different approaches.
Learning JSON document structure
First of all, you don't actually need to use BeautifulSoup here. The URL is targeting to JSON document, which is quite easy to navigate and get values from if you know the structure of JSON document.
To examine the structure of JSON document I can recommend using browser extensions for prettifying JSON, like JSON lite for Opera. But this is not best approach if JSON document is so large, your browser will be struggling render so much text, and extension will be struggling trying to format the JSON document. If document is large - use second approach.
You can download JSON document, and use Text Editors with plugins for JSON prettifying (like powerful Notepad++, plugin JSFormat).
Also, you can examine the structure of JSON document dirrectly in python using library for pretty print output called pprint (this is also not the best option with large documents).
import pprint
json = {"data": {"actions": {"items": [1, 2, 3], "colors": ['red', 'green', 'blue'], "items1": [1, 2, 3], "items2": {"numbers": [1, 2, 3], "characters": ['a', 'b', 'c']}}, "some_other_data": ['one', 'two', 'three']}, "new_data": [11, 22, 33, 44, 55]}
pprint.pprint(json)
Outputs same structure but more structured, and you can easily check how to retrieve certain value from JSON:
{'data': {'actions': {'colors': ['red', 'green', 'blue'],
'items': [1, 2, 3],
'items1': [1, 2, 3],
'items2': {'characters': ['a', 'b', 'c'],
'numbers': [1, 2, 3]}},
'some_other_data': ['one', 'two', 'three']},
'new_data': [11, 22, 33, 44, 55]}
Retrieving data from your URL
As we can see from the JSON document in your URL, JSON-path to baseHitPoints is: data -> baseHitPoints. JSON-path to removedHitPoints is: data -> removedHitPoints.
Knowing that, we can now easily retrieve those values using library requests:
import requests
URL = "https://character-service.dndbeyond.com/character/v5/character/00000001"
req = requests.get(URL)
json_data = req.json()
baseHitPoints = json_data['data']['baseHitPoints']
removedHitPoints = json_data['data']['removedHitPoints']
print(f"{baseHitPoints = }\n{removedHitPoints = }")
Output:
baseHitPoints = 8
removedHitPoints = 2
What you have there is JSON, not really a standard string. Because that's JSON you can get the data out of there easily by doing something like this:
import json
myData = json.load(myJSON) #myJSON would be the contents of that link(https://character-service.dndbeyond.com/character/v5/character/00000001) you provided
baseHitPoints = myData["data"]["baseHitPoints"]
removedHitPoints = myData["data"]["removedHitPoints"]
The values of those variables in my example using your JSON:
baseHitPoints = 8
baseHitPoints = 2
To make your life a little easier going forward I'd look for a tutorial for how to parse through JSON.

DataError: Invalid input of type: 'dict'. Convert to a byte, string or number first. | Google Cloud Memory Store (Managed Redis) [duplicate]

# I have the dictionary my_dict
my_dict = {
'var1' : 5
'var2' : 9
}
r = redis.StrictRedis()
How would I store my_dict and retrieve it with redis. For example, the following code does not work.
#Code that doesn't work
r.set('this_dict', my_dict) # to store my_dict in this_dict
r.get('this_dict') # to retrieve my_dict
You can do it by hmset (multiple keys can be set using hmset).
hmset("RedisKey", dictionaryToSet)
import redis
conn = redis.Redis('localhost')
user = {"Name":"Pradeep", "Company":"SCTL", "Address":"Mumbai", "Location":"RCP"}
conn.hmset("pythonDict", user)
conn.hgetall("pythonDict")
{'Company': 'SCTL', 'Address': 'Mumbai', 'Location': 'RCP', 'Name': 'Pradeep'}
you can pickle your dict and save as string.
import pickle
import redis
r = redis.StrictRedis('localhost')
mydict = {1:2,2:3,3:4}
p_mydict = pickle.dumps(mydict)
r.set('mydict',p_mydict)
read_dict = r.get('mydict')
yourdict = pickle.loads(read_dict)
As the basic answer has already give by other people, I would like to add some to it.
Following are the commands in REDIS to perform basic operations with HashMap/Dictionary/Mapping type values.
HGET => Returns value for single key passed
HSET => set/updates value for the single key
HMGET => Returns value for single/multiple keys passed
HMSET => set/updates values for the multiple key
HGETALL => Returns all the (key, value) pairs in the mapping.
Following are their respective methods in redis-py library :-
HGET => hget
HSET => hset
HMGET => hmget
HMSET => hmset
HGETALL => hgetall
All of the above setter methods creates the mapping, if it doesn't exists.
All of the above getter methods doesn't raise error/exceptions, if mapping/key in mapping doesn't exists.
Example:
=======
In [98]: import redis
In [99]: conn = redis.Redis('localhost')
In [100]: user = {"Name":"Pradeep", "Company":"SCTL", "Address":"Mumbai", "Location":"RCP"}
In [101]: con.hmset("pythonDict", {"Location": "Ahmedabad"})
Out[101]: True
In [102]: con.hgetall("pythonDict")
Out[102]:
{b'Address': b'Mumbai',
b'Company': b'SCTL',
b'Last Name': b'Rajpurohit',
b'Location': b'Ahmedabad',
b'Name': b'Mangu Singh'}
In [103]: con.hmset("pythonDict", {"Location": "Ahmedabad", "Company": ["A/C Pri
...: sm", "ECW", "Musikaar"]})
Out[103]: True
In [104]: con.hgetall("pythonDict")
Out[104]:
{b'Address': b'Mumbai',
b'Company': b"['A/C Prism', 'ECW', 'Musikaar']",
b'Last Name': b'Rajpurohit',
b'Location': b'Ahmedabad',
b'Name': b'Mangu Singh'}
In [105]: con.hget("pythonDict", "Name")
Out[105]: b'Mangu Singh'
In [106]: con.hmget("pythonDict", "Name", "Location")
Out[106]: [b'Mangu Singh', b'Ahmedabad']
I hope, it makes things more clear.
If you want to store a python dict in redis, it is better to store it as json string.
import json
import redis
r = redis.StrictRedis(host='localhost', port=6379, db=0)
mydict = { 'var1' : 5, 'var2' : 9, 'var3': [1, 5, 9] }
rval = json.dumps(mydict)
r.set('key1', rval)
While retrieving de-serialize it using json.loads
data = r.get('key1')
result = json.loads(data)
arr = result['var3']
What about types (eg.bytes) that are not serialized by json functions ?
You can write encoder/decoder functions for types that cannot be serialized by json functions. eg. writing base64/ascii encoder/decoder function for byte array.
Another way: you can use RedisWorks library.
pip install redisworks
>>> from redisworks import Root
>>> root = Root()
>>> root.something = {1:"a", "b": {2: 2}} # saves it as Hash type in Redis
...
>>> print(root.something) # loads it from Redis
{'b': {2: 2}, 1: 'a'}
>>> root.something['b'][2]
2
It converts python types to Redis types and vice-versa.
>>> root.sides = [10, [1, 2]] # saves it as list in Redis.
>>> print(root.sides) # loads it from Redis
[10, [1, 2]]
>>> type(root.sides[1])
<class 'list'>
Disclaimer: I wrote the library. Here is the code: https://github.com/seperman/redisworks
HMSET is deprecated per the Redis docs. You can now use HSET with a dictionary as follows:
import redis
r = redis.Redis('localhost')
key = "hashexample"
entry = {
"version":"1.2.3",
"tag":"main",
"status":"CREATED",
"timeout":"30"
}
r.hset(key, mapping=entry)
Caution: very unintuitively, hset won't accept a dictionary (raising an error suggesting it does not accept dictionaries, see [1]) if it is simply passed to the 2nd positional (unnamed) argument. You need to pass the dictionary to a named argument mapping=.
[1] *** redis.exceptions.DataError: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.
One might consider using MessagePack which is endorsed by redis.
import msgpack
data = {
'one': 'one',
'two': 2,
'three': [1, 2, 3]
}
await redis.set('my-key', msgpack.packb(data))
val = await redis.get('my-key')
print(msgpack.unpackb(val))
# {'one': 'one', 'two': 2, 'three': [1, 2, 3]}
Using msgpack-python and aioredis
The redis SET command stores a string, not arbitrary data. You could try using the redis HSET command to store the dict as a redis hash with something like
for k,v in my_dict.iteritems():
r.hset('my_dict', k, v)
but the redis datatypes and python datatypes don't quite line up. Python dicts can be arbitrarily nested, but a redis hash is going to require that your value is a string. Another approach you can take is to convert your python data to string and store that in redis, something like
r.set('this_dict', str(my_dict))
and then when you get the string out you will need to parse it to recreate the python object.
An other way you can approach the matter:
import redis
conn = redis.Redis('localhost')
v={'class':'user','grants': 0, 'nome': 'Roberto', 'cognome': 'Brunialti'}
y=str(v)
print(y['nome']) #<=== this return an error as y is actually a string
conn.set('test',y)
z=eval(conn.get('test'))
print(z['nome']) #<=== this really works!
I did not test it for efficiency/speed.
If you don't know exactly how to organize data in Redis, I did some performance tests, including the results parsing.
The dictonary I used (d) had 437.084 keys (md5 format), and the values of this form:
{"path": "G:\tests\2687.3575.json",
"info": {"f": "foo", "b": "bar"},
"score": 2.5}
First Test (inserting data into a redis key-value mapping):
conn.hmset('my_dict', d) # 437.084 keys added in 8.98s
conn.info()['used_memory_human'] # 166.94 Mb
for key in d:
json.loads(conn.hget('my_dict', key).decode('utf-8').replace("'", '"'))
# 41.1 s
import ast
for key in d:
ast.literal_eval(conn.hget('my_dict', key).decode('utf-8'))
# 1min 3s
conn.delete('my_dict') # 526 ms
Second Test (inserting data directly into Redis keys):
for key in d:
conn.hmset(key, d[key]) # 437.084 keys added in 1min 20s
conn.info()['used_memory_human'] # 326.22 Mb
for key in d:
json.loads(conn.hgetall(key)[b'info'].decode('utf-8').replace("'", '"'))
# 1min 11s
for key in d:
conn.delete(key)
# 37.3s
As you can see, in the second test, only 'info' values have to be parsed, because the hgetall(key) already returns a dict, but not a nested one.
And of course, the best example of using Redis as python's dicts, is the First Test
DeprecationWarning: Redis.hmset() is deprecated. Use Redis.hset() instead.
Since HMSET is deprecated you can use HSET:
import redis
r = redis.Redis(host='localhost', port=6379, decode_responses=True)
r.hset('user:23', mapping={'id': 23, 'name': 'ip'})
r.hgetall('user:23')
Try rejson-py which is relatively new since 2017. Look at this introduction.
from rejson import Client, Path
rj = Client(host='localhost', port=6379)
# Set the key `obj` to some object
obj = {
'answer': 42,
'arr': [None, True, 3.14],
'truth': {
'coord': 'out there'
}
}
rj.jsonset('obj', Path.rootPath(), obj)
# Get something
print 'Is there anybody... {}?'.format(
rj.jsonget('obj', Path('.truth.coord'))
)
# Delete something (or perhaps nothing), append something and pop it
rj.jsondel('obj', Path('.arr[0]'))
rj.jsonarrappend('obj', Path('.arr'), 'something')
print '{} popped!'.format(rj.jsonarrpop('obj', Path('.arr')))
# Update something else
rj.jsonset('obj', Path('.answer'), 2.17)
In the context of Nameko (Python microservices framework which frequently uses a redis backend) you can use hmset as follows:
import uuid
from nameko.rpc import rpc
from nameko_redis import Redis
class AirportsService:
name = "trips_service"
redis = Redis('development')
#rpc
def get(self, trip_id):
trip = self.redis.get(trip_id)
return trip
#rpc
def create(self, airport_from_id, airport_to_id):
trip_id = uuid.uuid4().hex
pyDict = {"from":airport_from_id, "to":airport_to_id}
self.redis.hmset(trip_id, pyDict)
return trip_id

How to maintain dictionary element order in JSON dump in Python

Due to a very helpful person on this site, I have been able to put together a script which takes user names (inputted by user) and then use a loop to append each of those user names to a specific JSON structure. However, I have noticed that the order within my JSON dump is not maintained. For example, here is what my loop and JSON dump looks like:
list_users = []
for user in users:
list_users.append({"name": user,
"member": 123,
"comment": "this is a comment"})
json_user_list_encoding = json.dumps(list_users, indent=2)
My print looks like this:
({"member": 123,
"comment": "this is a comment"
"name": user
})
I was wondering if it was possible to maintain the same order in "list_users" when I use the JSON dump. I have looked at this site a bit and from what I have read, I will have to assign keys in order to maintain a specific order. However, I am not sure how to do this being that I only have one JSON object. I hope this make sense. Thanks for the help.
If you just want them ordered reliably/repeatably, but don't care about the order itself, then json dumps accepts a sort_keys kwarg that can help:
>>> json.dumps({'z': 3, 'b': 2, 'a': 1})
'{"a": 1, "z": 3, "b": 2}'
>>> json.dumps({'z': 3, 'b': 2, 'a': 1}, sort_keys=True)
'{"a": 1, "b": 2, "z": 3}'
Otherwise, put the users in an OrderedDict instead of a plain dict.
from collections import OrderedDict
list_users.append(OrderedDict([
("name", user),
("member", 123),
("comment", "this is a comment"),
]))
If you need them to deserialize to the same order too, use the object_pairs_hook when loading.

How to store and retrieve a dictionary with redis

# I have the dictionary my_dict
my_dict = {
'var1' : 5
'var2' : 9
}
r = redis.StrictRedis()
How would I store my_dict and retrieve it with redis. For example, the following code does not work.
#Code that doesn't work
r.set('this_dict', my_dict) # to store my_dict in this_dict
r.get('this_dict') # to retrieve my_dict
You can do it by hmset (multiple keys can be set using hmset).
hmset("RedisKey", dictionaryToSet)
import redis
conn = redis.Redis('localhost')
user = {"Name":"Pradeep", "Company":"SCTL", "Address":"Mumbai", "Location":"RCP"}
conn.hmset("pythonDict", user)
conn.hgetall("pythonDict")
{'Company': 'SCTL', 'Address': 'Mumbai', 'Location': 'RCP', 'Name': 'Pradeep'}
you can pickle your dict and save as string.
import pickle
import redis
r = redis.StrictRedis('localhost')
mydict = {1:2,2:3,3:4}
p_mydict = pickle.dumps(mydict)
r.set('mydict',p_mydict)
read_dict = r.get('mydict')
yourdict = pickle.loads(read_dict)
As the basic answer has already give by other people, I would like to add some to it.
Following are the commands in REDIS to perform basic operations with HashMap/Dictionary/Mapping type values.
HGET => Returns value for single key passed
HSET => set/updates value for the single key
HMGET => Returns value for single/multiple keys passed
HMSET => set/updates values for the multiple key
HGETALL => Returns all the (key, value) pairs in the mapping.
Following are their respective methods in redis-py library :-
HGET => hget
HSET => hset
HMGET => hmget
HMSET => hmset
HGETALL => hgetall
All of the above setter methods creates the mapping, if it doesn't exists.
All of the above getter methods doesn't raise error/exceptions, if mapping/key in mapping doesn't exists.
Example:
=======
In [98]: import redis
In [99]: conn = redis.Redis('localhost')
In [100]: user = {"Name":"Pradeep", "Company":"SCTL", "Address":"Mumbai", "Location":"RCP"}
In [101]: con.hmset("pythonDict", {"Location": "Ahmedabad"})
Out[101]: True
In [102]: con.hgetall("pythonDict")
Out[102]:
{b'Address': b'Mumbai',
b'Company': b'SCTL',
b'Last Name': b'Rajpurohit',
b'Location': b'Ahmedabad',
b'Name': b'Mangu Singh'}
In [103]: con.hmset("pythonDict", {"Location": "Ahmedabad", "Company": ["A/C Pri
...: sm", "ECW", "Musikaar"]})
Out[103]: True
In [104]: con.hgetall("pythonDict")
Out[104]:
{b'Address': b'Mumbai',
b'Company': b"['A/C Prism', 'ECW', 'Musikaar']",
b'Last Name': b'Rajpurohit',
b'Location': b'Ahmedabad',
b'Name': b'Mangu Singh'}
In [105]: con.hget("pythonDict", "Name")
Out[105]: b'Mangu Singh'
In [106]: con.hmget("pythonDict", "Name", "Location")
Out[106]: [b'Mangu Singh', b'Ahmedabad']
I hope, it makes things more clear.
If you want to store a python dict in redis, it is better to store it as json string.
import json
import redis
r = redis.StrictRedis(host='localhost', port=6379, db=0)
mydict = { 'var1' : 5, 'var2' : 9, 'var3': [1, 5, 9] }
rval = json.dumps(mydict)
r.set('key1', rval)
While retrieving de-serialize it using json.loads
data = r.get('key1')
result = json.loads(data)
arr = result['var3']
What about types (eg.bytes) that are not serialized by json functions ?
You can write encoder/decoder functions for types that cannot be serialized by json functions. eg. writing base64/ascii encoder/decoder function for byte array.
Another way: you can use RedisWorks library.
pip install redisworks
>>> from redisworks import Root
>>> root = Root()
>>> root.something = {1:"a", "b": {2: 2}} # saves it as Hash type in Redis
...
>>> print(root.something) # loads it from Redis
{'b': {2: 2}, 1: 'a'}
>>> root.something['b'][2]
2
It converts python types to Redis types and vice-versa.
>>> root.sides = [10, [1, 2]] # saves it as list in Redis.
>>> print(root.sides) # loads it from Redis
[10, [1, 2]]
>>> type(root.sides[1])
<class 'list'>
Disclaimer: I wrote the library. Here is the code: https://github.com/seperman/redisworks
HMSET is deprecated per the Redis docs. You can now use HSET with a dictionary as follows:
import redis
r = redis.Redis('localhost')
key = "hashexample"
entry = {
"version":"1.2.3",
"tag":"main",
"status":"CREATED",
"timeout":"30"
}
r.hset(key, mapping=entry)
Caution: very unintuitively, hset won't accept a dictionary (raising an error suggesting it does not accept dictionaries, see [1]) if it is simply passed to the 2nd positional (unnamed) argument. You need to pass the dictionary to a named argument mapping=.
[1] *** redis.exceptions.DataError: Invalid input of type: 'dict'. Convert to a bytes, string, int or float first.
One might consider using MessagePack which is endorsed by redis.
import msgpack
data = {
'one': 'one',
'two': 2,
'three': [1, 2, 3]
}
await redis.set('my-key', msgpack.packb(data))
val = await redis.get('my-key')
print(msgpack.unpackb(val))
# {'one': 'one', 'two': 2, 'three': [1, 2, 3]}
Using msgpack-python and aioredis
The redis SET command stores a string, not arbitrary data. You could try using the redis HSET command to store the dict as a redis hash with something like
for k,v in my_dict.iteritems():
r.hset('my_dict', k, v)
but the redis datatypes and python datatypes don't quite line up. Python dicts can be arbitrarily nested, but a redis hash is going to require that your value is a string. Another approach you can take is to convert your python data to string and store that in redis, something like
r.set('this_dict', str(my_dict))
and then when you get the string out you will need to parse it to recreate the python object.
An other way you can approach the matter:
import redis
conn = redis.Redis('localhost')
v={'class':'user','grants': 0, 'nome': 'Roberto', 'cognome': 'Brunialti'}
y=str(v)
print(y['nome']) #<=== this return an error as y is actually a string
conn.set('test',y)
z=eval(conn.get('test'))
print(z['nome']) #<=== this really works!
I did not test it for efficiency/speed.
If you don't know exactly how to organize data in Redis, I did some performance tests, including the results parsing.
The dictonary I used (d) had 437.084 keys (md5 format), and the values of this form:
{"path": "G:\tests\2687.3575.json",
"info": {"f": "foo", "b": "bar"},
"score": 2.5}
First Test (inserting data into a redis key-value mapping):
conn.hmset('my_dict', d) # 437.084 keys added in 8.98s
conn.info()['used_memory_human'] # 166.94 Mb
for key in d:
json.loads(conn.hget('my_dict', key).decode('utf-8').replace("'", '"'))
# 41.1 s
import ast
for key in d:
ast.literal_eval(conn.hget('my_dict', key).decode('utf-8'))
# 1min 3s
conn.delete('my_dict') # 526 ms
Second Test (inserting data directly into Redis keys):
for key in d:
conn.hmset(key, d[key]) # 437.084 keys added in 1min 20s
conn.info()['used_memory_human'] # 326.22 Mb
for key in d:
json.loads(conn.hgetall(key)[b'info'].decode('utf-8').replace("'", '"'))
# 1min 11s
for key in d:
conn.delete(key)
# 37.3s
As you can see, in the second test, only 'info' values have to be parsed, because the hgetall(key) already returns a dict, but not a nested one.
And of course, the best example of using Redis as python's dicts, is the First Test
DeprecationWarning: Redis.hmset() is deprecated. Use Redis.hset() instead.
Since HMSET is deprecated you can use HSET:
import redis
r = redis.Redis(host='localhost', port=6379, decode_responses=True)
r.hset('user:23', mapping={'id': 23, 'name': 'ip'})
r.hgetall('user:23')
Try rejson-py which is relatively new since 2017. Look at this introduction.
from rejson import Client, Path
rj = Client(host='localhost', port=6379)
# Set the key `obj` to some object
obj = {
'answer': 42,
'arr': [None, True, 3.14],
'truth': {
'coord': 'out there'
}
}
rj.jsonset('obj', Path.rootPath(), obj)
# Get something
print 'Is there anybody... {}?'.format(
rj.jsonget('obj', Path('.truth.coord'))
)
# Delete something (or perhaps nothing), append something and pop it
rj.jsondel('obj', Path('.arr[0]'))
rj.jsonarrappend('obj', Path('.arr'), 'something')
print '{} popped!'.format(rj.jsonarrpop('obj', Path('.arr')))
# Update something else
rj.jsonset('obj', Path('.answer'), 2.17)
In the context of Nameko (Python microservices framework which frequently uses a redis backend) you can use hmset as follows:
import uuid
from nameko.rpc import rpc
from nameko_redis import Redis
class AirportsService:
name = "trips_service"
redis = Redis('development')
#rpc
def get(self, trip_id):
trip = self.redis.get(trip_id)
return trip
#rpc
def create(self, airport_from_id, airport_to_id):
trip_id = uuid.uuid4().hex
pyDict = {"from":airport_from_id, "to":airport_to_id}
self.redis.hmset(trip_id, pyDict)
return trip_id

How does JSON work in my python program?

So I use the Java Debugger JSON in my python program because a few months ago I was told that this was the best way of opening a text file and making it into a dictionary and also saving the dictionary to a text file. However I am not sure how it works.
Below is how I am using it within my program:
with open ("totals.txt", 'r') as f30:
totaldict = json.load(f30)
and
with open ("totals.txt", 'w') as f29:
json.dump(totaldict, f29)
I need to explain how it works for my project so could anyone explain for me how exactly json works when loading a text file into dictionary format and when dumping contents into the text file?
Thanks.
Edit: please don't just post links to other articles as I have tried to look at these and they have offered me not much help as they are not in my context of using JSON for dictionaries and a bit overwhelming as I am only a beginner.
JSON is J ava S cript O bject N otation. It works in Python like it does anywhere else, by giving you a syntax for describing arbitrary things as objects.
Most JSON is primarily composed of JavaScript arrays, which look like this:
[1, 2, 3, 4, 5]
Or lists of key-value pairs describing an object, which look like this:
{"key1": "value1", "key2": "value2"}
These can also be nested in either direction:
[{"object1": "data1"}, {"object2": "data2"}]
{"object1": ["list", "of", "data"]}
Naturally, Python can very easily treat these types as lists and dicts, which is exactly what the json module tries to do.
>>> import json
>>> json.loads('[{"object1": "data1"}, {"object2": "data2"}]')
[{'object1': 'data1'}, {'object2': 'data2'}]
>>> json.dumps(_)
'[{"object1": "data1"}, {"object2": "data2"}]'
Try this: Python Module of the Week
The json module provides an API similar to pickle for converting in-memory Python objects to a serialized representation known as JavaScript Object Notation (JSON). Unlike pickle, JSON has the benefit of having implementations in many languages (especially JavaScript)
Encoding and Decoding Simple Data Types
The encoder understands Python’s native types by default (string, unicode, int, float, list, tuple, dict).
import json
data = [ { 'a':'A', 'b':(2, 4), 'c':3.0 } ]
print 'DATA:', repr(data)
data_string = json.dumps(data)
print 'JSON:', data_string
Values are encoded in a manner very similar to Python’s repr() output.
$ python json_simple_types.py
DATA: [{'a': 'A', 'c': 3.0, 'b': (2, 4)}]
JSON: [{"a": "A", "c": 3.0, "b": [2, 4]}]
Encoding, then re-decoding may not give exactly the same type of object.
import json
data = [ { 'a':'A', 'b':(2, 4), 'c':3.0 } ]
data_string = json.dumps(data)
print 'ENCODED:', data_string
decoded = json.loads(data_string)
print 'DECODED:', decoded
print 'ORIGINAL:', type(data[0]['b'])
print 'DECODED :', type(decoded[0]['b'])
In particular, strings are converted to unicode and tuples become lists.
$ python json_simple_types_decode.py
ENCODED: [{"a": "A", "c": 3.0, "b": [2, 4]}]
DECODED: [{u'a': u'A', u'c': 3.0, u'b': [2, 4]}]
ORIGINAL: <type 'tuple'>
DECODED : <type 'list'>

Categories