We Keep Coding

How to filter out data from a print out message

I have this one problem, where I print out a message response from a website(JSON response), and the response I get is this.
Here is my model with fake data:
{"token": "MTAxOTAwNjM4NjEyMzg0OTkwMQ.8hkyLV.n0ir2UA4qFE5pXen9YnPtFzgn4xP8tHmVmmkrl", "user_settings": {"locale": "en-US", "theme": "dark"}, "user_id": "101900638614857883"}
And, if I only want the value of "token" data which are this (MTAxOTAwNjM4NjEyMzg0OTkwMQ.8hkyLV.n0ir2UA4qFE5pXen9YnPtFzgn4xP8tHmVmmkrl) and I want to store it into a txt file, is there any good way to do it?
Thank you, guys!
I tried print(r.text('token')) but it did not work, since it only works on printing the category of the data's (like : Category : {"token" : 'daefafa', "user-id" : 'er121231231', more})

In python, JSON is treated as a dictionary.
To filter it use dictionary comprehension
tokenData = {key: val for key,val in data_json.items() if key == 'token'}
Full Code Snippet :
from urllib.request import urlopen
import json
url = "enter-your-url"
response = urlopen(url)
data_json = json.loads(response.read())
print(type(data_json)) # <class 'dict'>
#use dict comprehension
jsonToken = {key: val for key,val in data_json.items() if key == 'result'}
strToken = json.dumps(jsonToken)
# Only string json can be written to files
with open('data.txt','w') as file:
file.write(strToken)
file.close()

You need to parse the JSON into a dictionary using json.loads(). Like this:
import json
# ...
# request-getting code
# ...
data = json.loads(r.text)
print(data['token'])

Python: Convert dictionary to bytes

I'm trying to convert a dictionary to bytes but facing issues in converting it to a correct format.
First, I'm trying to map an dictionary with an custom schema. Schema is defined as follows -
class User:
def __init__(self, name=None, code=None):
self.name = name
self.code = code
class UserSchema:
name = fields.Str()
code = fields.Str()
#post_load
def create_userself, data):
return User(**data)
My Dictionary structure is as follows-
user_dict = {'name': 'dinesh', 'code': 'dr-01'}
I'm trying to map the dictionary to User schema with the below code
schema = UserSchema(partial=True)
user = schema.loads(user_dict).data
While doing, schema.loads expects the input to be str, bytes or bytearray. Below are the steps that I followed to convert dictionary to Bytes
import json
user_encode_data = json.dumps(user_dict).encode('utf-8')
print(user_encode_data)
Output:
b'{"name ": "dinesh", "code ": "dr-01"}
If I try to map with the schema I'm not getting the required schema object. But, if I have the output in the format given below I can able to get the correct schema object.
b'{\n "name": "dinesh",\n "code": "dr-01"}\n'
Any suggestions how can I convert a dictionary to Bytes?

You can use indent option in json.dumps() to obtain \n symbols:
import json
user_dict = {'name': 'dinesh', 'code': 'dr-01'}
user_encode_data = json.dumps(user_dict, indent=2).encode('utf-8')
print(user_encode_data)
Output:
b'{\n "name": "dinesh",\n "code": "dr-01"\n}'

You can use Base64 library to convert string dictionary to bytes, and although you can convert bytes result to a dictionary using json library. Try this below sample code.
import base64
import json
input_dict = {'var1' : 0, 'var2' : 'some string', 'var1' : ['listitem1','listitem2',5]}
message = str(input_dict)
ascii_message = message.encode('ascii')
output_byte = base64.b64encode(ascii_message)
msg_bytes = base64.b64decode(output_byte)
ascii_msg = msg_bytes.decode('ascii')
# Json library convert stirng dictionary to real dictionary type.
# Double quotes is standard format for json
ascii_msg = ascii_msg.replace("'", "\"")
output_dict = json.loads(ascii_msg) # convert string dictionary to dict format
# Show the input and output
print("input_dict:", input_dict, type(input_dict))
print()
print("base64:", output_byte, type(output_byte))
print()
print("output_dict:", output_dict, type(output_dict))
Output:
>>> print("input_dict:", input_dict, type(input_dict))
input_dict: {'var1': ['listitem1', 'listitem2', 5], 'var2': 'some string'} <class 'dict'>
>>> print()
>>> print("base64:", output_byte, type(output_byte))
base64: b'eyd2YXIxJzogWydsaXN0aXRlbTEnLCAnbGlzdGl0ZW0yJywgNV0sICd2YXIyJzogJ3NvbWUgc3RyaW5nJ30=' <class 'bytes'>
>>> print()
>>> print("output_dict:", output_dict, type(output_dict))
output_dict: {'var1': ['listitem1', 'listitem2', 5], 'var2': 'some string'} <class 'dict'>

Best way to add dictionary entry and append to JSON file in Python

I have a need to add entries to a dictionary with the following keys:
name
element
type
I want each entry to append to a JSON file, where I will access them for another piece of the project.
What I have below technically works, but there are couple things(at least) wrong with this.
First, it doesn't prevent duplicates being entered. For example I can have 'xyz', '4444' and 'test2' appear as JSON entries multiple times. Is there a way to correct this?
Is there a cleaner way to write the actual data entry piece so when I am entering these values into the dictionary it's not directly there in the parentheses?
Finally, is there a better place to put the JSON piece? Should it be inside the function?
Just trying to clean this up a bit. Thanks
import json
element_dict = {}
def add_entry(name, element, type):
element_dict["name"] = name
element_dict["element"] = element
element_dict["type"] = type
return element_dict
#add entry
entry = add_entry('xyz', '4444', 'test2')
#export to JSON
with open('elements.json', 'a', encoding="utf-8") as file:
x = json.dumps(element_dict, indent=4)
file.write(x + '\n')

There are several questions here. The main points worth mentioning:
Use can use a list to hold your arguments and use *args to unpack when you supply them to add_entry.
To check / avoid duplicates, you can use set to track items already added.
For writing to JSON, now you have a list, you can simply iterate your list and write in one function at the end.
Putting these aspects together:
import json
res = []
seen = set()
def add_entry(res, name, element, type):
# check if in seen set
if (name, element, type) in seen:
return res
# add to seen set
seen.add(tuple([name, element, type]))
# append to results list
res.append({'name': name, 'element': element, 'type': type})
return res
args = ['xyz', '4444', 'test2']
res = add_entry(res, *args) # add entry - SUCCESS
res = add_entry(res, *args) # try to add again - FAIL
args2 = ['wxy', '3241', 'test3']
res = add_entry(res, *args2) # add another - SUCCESS
Result:
print(res)
[{'name': 'xyz', 'element': '4444', 'type': 'test2'},
{'name': 'wxy', 'element': '3241', 'type': 'test3'}]
Writing to JSON via a function:
def write_to_json(lst, fn):
with open(fn, 'a', encoding='utf-8') as file:
for item in lst:
x = json.dumps(item, indent=4)
file.write(x + '\n')
#export to JSON
write_to_json(res, 'elements.json')

you can try this way
import json
import hashlib
def add_entry(name, element, type):
return {hashlib.md5(name+element+type).hexdigest(): {"name": name, "element": element, "type": type}}
#add entry
entry = add_entry('xyz', '4444', 'test2')
#Update to JSON
with open('my_file.json', 'r') as f:
json_data = json.load(f)
print json_data.values() # View Previous entries
json_data.update(entry)
with open('elements.json', 'w') as f:
f.write(json.dumps(json_data))

(Python) merge new and existing JSON with deduplication

I'm querying an API with Python, This API sends JSON of the last X events and I want to keep a history of what it sent me.
So this is what the API sends, and I have the same type of elements in my flat history file (but with many more of the same objects).
The API and my final file doesn't have a key on which to setup a dictionary.
[{
"Item1": "01234",
"Item2": "Company",
"Item3": "XXXXXXXXX",
"Item4": "",
"Item5": "2015-12-17T12:00:01.553",
"Item6": "2015-12-18T12:00:00"
},
{
"Item1": "01234",
"Item2": "Company2",
"Item3": "XXXXXXX",
"Item4": null,
"Item5": "2015-12-17T16:49:23.76",
"Item6": "2015-12-18T11:00:00",
}]
How do I add up elements of the API only if they are not in the original file?
I have a skeleton of opening/closing file but have not many ideas about the processing.
main_file=open("History.json","r")
new_items=[]
api_data=requests.get(#here lies the api address and the header)
#here should be the deplucation/processing process
for item in api_data
if item not in main_file
new_items.append(item)
main_file.close()
try:
file_updated = open("History.json",'w')
file_updated.write(new_items + main_file)
file_updated.close()
print("File updated")
except :
print("Error writing file")
EDIT : I used the json to object method to do this :
from collections import namedtuple
Event = namedtuple('Event', 'Item1, Item2, Item3, Item4, Item5, Item6')
def parse_json_events(text):
events = [ Event(**k) for k in json.loads(text) ]
return events
if path.exists('Mainfile.json'):
with open('Mainfile.json') as data_file:
local_data = json.load(data_file)
print(local_data.text) #debug purposes
events_local=parse_json_events(local_data.text)
else:
events_local=[]
events_api=parse_json_events(api_request.text)
inserted_events=0
for e in events_api[::-1]:
if e not in events_local:
events_local.insert(0, e)
inserted_events=inserted_events+1
print("inserted elements %d" % inserted_events)
print(events_local) # this is OK, gives me a list of events
print(json.dump(events_local)) # this ... well... I want the list of object to be serialized but I get this error :
TypeError: dump() missing 1 required positional argument: 'fp'

Normally you solve this kind of problems by defining a schema with/without a third party tool (like Avro, Thrift, etc.). Basically, every record you get from the API needs to be translated to an entity in the programming language you are using.
Let's take as an example this JSON object:
{
"Item1": "01234",
"Item2": "Company",
"Item3": "XXXXXXXXX",
"Item4": "",
"Item5": "2015-12-17T12:00:01.553",
"Item6": "2015-12-18T12:00:00"
},
If you have a schema like
Company(object):
company_number = ...
name = ...
# other fields
Then, all you need to do is to serialize and deserialize the raw data.
Ideally, you'd read the JSON response from the API and then you could simply split each json object as a schema object (with or without a tool). In pseudocode:
api_client = client(http://..., )
response = api_client.get("/resources")
json = response.json
companies = parse_json_companies(json) # list of Company objects
At this point, it's really easy to handle the data you got from the api. You should do the same for the files you have stored on the filesystem. Load your files and deserialize the records (to Company objects). Then, it will be easy to compare the objects, as they will be like "normal" Python objects, so that you can perform comparisons, etc etc.
For example:
from collections import namedtuple
import json
Company = namedtuple('Company', 'Item1, Item2, Item3, Item4, Item5, Item6')
def parse_json_companies(text):
companies = [Company(**k) for k in json.loads(text)]
return companies
>>> companies = parse_json_companies(response.json)
>>> companies
[Company(Item1='01234', Item2='Company', Item3='XXXXXXXXX', Item4=u'', Item5='2015-12-17T12:00:01.553', Item6='2015-12-18T12:00:00'), Company(Item1='01234', Item2='Company2', Item3='XXXXXXX', Item4=None, Item5='2015-12-17T16:49:23.76', Item6='2015-12-18T11:00:00')]
Update after error on .dump(obj, fp) .
If you get the error with json.dump, refer to the documentation please. It clearly states that obj and fp are required arguments.
Serialize obj as a JSON formatted stream to fp (a .write()-supporting file-like object) using this conversion table.
So, you need to pass an object that supports .write (e.g., a file opened in write mode).

I think the best way of solving this would be to think about your data structure. It seems like you're using the same data structure as the api at this moment.
Is there an Id among these item fields? If so use that field for deduplication. But for this example I'll use company name.
with open('history.json') as f:
historic_data = json.load(f)
api_data = requests.get()
for item in api_data:
historic_data[item['Item2']] = item
f.write(json.dumps(historic_data))
Every time the name in this case already exists in the dictionary it will be overwritten. If the name isn't existing it will be added.

Parsing a list of dictionaries passed as a POST parameter

I have a list of python dictionaries that look like this:
sandwiches = [
{'bread':'wheat', 'topping':'tomatoes', 'meat':'bacon'},
{'bread':'white', 'topping':'peanut butter', 'meat':'bacon'},
{'bread':'sourdough', 'topping':'cheese', 'meat':'bacon'}
]
I want to pass this as a POST parameter to another Django app. What does the client app need to do to iterate through the list?
I want to do something like:
for sandwich in request.POST['sandwiches']:
print "%s on %s with %s is yummy!" % (sandwich['meat'], sandwich['bread'], sandwich['topping'])
But I don't seem to have a list of dicts when my data arrives at my client app.

You don't say how you're POSTing to the app. My advice would be to serialize the dictionaries as JSON, then simply POST that.
import json, urllib, urllib2
data = json.dumps(sandwiches)
urllib2.urlopen(myurl, urllib.urlencode({'data': data}))
... and on the receiver:
data = request.POST['data']
sandwiches = json.loads(data)

I would use the JSON libraries to serialize your array of dicts into a single string. Send this string as a POST param, and parse the string back into the python datatypes in the Django app using the same library.
http://docs.python.org/library/json.html

firstly make string of the list
import ast, requests
# request send code
dev_list = [{"1": 2}, {"2": 4}]
str_list = str(dev_list) # '[{"1": 2}, {"2": 4}]'
data = {"a" : "mynema",
'str_list': str_list
}
requests.post(url, data )
#request receive side
data = request.data
dev_list = data.get('str_list')
dev_list = ast.literal_eval(dev_list)
#dev_list --> [{"1": 2}, {"2": 4}]
Now you receive list of dictionary.