I am extremely confused after trying a few possible solutions and getting various errors that just lead me in circles. I have a function that will grab a tweet, put it in a dictionary, then write that dictionary to a file using dumps like so:
jsonFile = {}
jsonFile["tweet"] = tweet
jsonFile["language"] = language
with open('jsonOutputfile.txt', 'w') as f:
json.dump(jsonFile, f)
I then have another python file that has a function that will return the value of this jsonOutputfile.txt if I want to use it elsewhere. I do that like so:
with open('jsonOutputfile.txt') as f:
jsonObject = json.load(f)
return jsonObject
This function sits on my localhost. The above two functions that have to do with saving and retrieving the JSON file are separate from the rest of my functions below, as I want them to be.
I have another function that will retrieve the values of the returned status using python requests, like so:
def grab_tweet():
return requests.post("http://gateway:8080/function/twittersend")
and then after grabbing the tweet I want to manipulate it, and I want to do so using the JSON that I should have received from this request.
r = grab_tweet()
data = json.dumps(r.text)
return data.get('tweet')
I want this function above to return just the value that is associated with the tweet key in the JSON that I received from when I saved and loaded it. However, I keep on getting the following error: AttributeError: 'str' object has no attribute 'get' which I am confused about because from my understanding using json.dumps() should create a JSON valid string that I can call get on. Is there an encoding error when I am transferring this to and from a file, or maybe when I am receiving my request?
Any help is appreciated.
EDIT:
Here is a sample of a response from my requests.post when I use r.text, it also looks like there is some Unicode in the response so I put an example at the end of the tweet section. (This also doesn't look like a JSON which is what my question is centered around. There should at least be double quotes and no U's right?):
{u'tweet': u'RT THIS IS THE TWEET BLAH BLAH\u2026', u'language': u'en'}
Use .json() in requests module to get response as JSON
Ex:
data = r.json()
return data.get('tweet')
Note: json.dumps convert your response to a string object
Edit as per comment - Try using the ast module.
Ex:
import ast
data = ast.literal_eval(r.text)
You will need to use the .json() method. See requests' documentation: JSON Response Content
Also, for future reference, rather than do
f.write(json.dumps(jsonFile))
You could simply use:
json.dump(jsonFile, f)
Same with using load instead of loads:
jsonObject = json.load(f)
Related
working on a larger project utilizing an API and I am having issues writing / reading the JSON response as a dictionary in a .txt file. I believe I have gotten to the point where the error seems to be driven by the formatting of the dictionary, which writes values in the dictionary such as None and datetime.date(2020, 7, 11).
What am I doing in writing or reading this incorrectly?
I read in by:
with open('./testing/txns.txt', 'r') as f:
txns = f.read()
txns = txns.replace("'", "\"") # to solve for single quotes
Sample of what might be in 'txns.txt':
{'account_owner': None,
'amount': 6.33,
'authorized_date': datetime.date(2020, 12, 23)}
Error from reading:
json.decoder.JSONDecodeError: Expecting value: line 1 column 19 (char 18)
If you need more context - I am happy to provide that. I am trying to work with Plaid's API, specifically the Transactions endpoint and client.transactions_get(request). I successfully acquired transactions through this line, and I would now like to debug, do discovery, and develop with this sample data. I cannot write it and read it correctly and I think this is why :)
Update1:
This is my code now
response = client.transactions_get(request)
with open('txns.txt', 'w') as f:
f.write(json.dumps(response.to_dict()))
Error:TypeError: Object of type date is not JSON serializable
Update2 - SOLVED
With the help of Alex & How to overcome "datetime.datetime not JSON serializable"? I figured it out:
f.write(json.dumps(response.to_dict(), default=str))
Yep, the commenters are correct that the Plaid Python client library doesn't return this value as JSON. There's a section about this in the README at https://github.com/plaid/plaid-python#converting-the-response-to-a-json
Converting the response to a JSON
As this is a common question, we've
included this in the README. plaid-python uses models like
TransactionsGetResponse to encapsulate API responses. If you want to
convert this to a JSON, do something like this:
import json
...
response = ... # type TransactionsGetResponse
# to_dict makes it first a python dictionary, and then we turn it into a
# string JSON.
json_string = json.dumps(response.to_dict())
I am using Google Cloud Document AI's Form Parser API. After i do the request to the API , I get a response with type google.cloud.documentai.v1beta2.types.document.Document. I tried to write it to JSON using json.dumps() but it gives JSONDecodeError because JSON.dumps() dont know how to serialize object of type google.cloud.documentai.v1beta2.types.document.Document.
I am confused how to convert this to JSON
Any Help Appreciated!
I just found out that the google.cloud.documentai.v1beta2.types.document.Document object inherits from proto.Message, which itself inherits from proto.MessageMeta. You can use the proto.MessageMeta.to_json function to convert the Document object to a json string, like so:
import json
from google.cloud.documentai_v1beta3 import Document
json_string = Document.to_json(document)
dict_obj = json.loads(json_string)
with open("document.json", mode='w') as my_file:
json.dump(dict_obj, my_file)
The source code for proto.Message was a little hard to find, so here it is: https://github.com/googleapis/proto-plus-python/blob/cfd5b6caca3fa9add89d8c69ea620505dd90dd7c/proto/message.py#L330
I solved my problem.
basically you have to write a function that explores the Document object, and then assemble entire JSON yourself by code.
Here is my code and does anyone have any ideas what is wrong? I open my JSON content directly by browser and it works,
data = requests.get('http://ws.audioscrobbler.com/2.0/?method=library.getartists&api_key=4c22bd45cf5aa6e408e02b3fc1bff690&user=joanofarctan&format=json').text
data = json.load(data)
print type(data)
return data
thanks in advance,
Lin
This error raised because the data is a unicode/str variable, change the second line of your code to resolve your error:
data = json.loads(data)
json.load get a file object in first parameter position and call the read method of this.
Also you can call the json method of the response to fetch data directly:
response = requests.get('http://ws.audioscrobbler.com/2.0/?method=library.getartists&api_key=4c22bd45cf5aa6e408e02b3fc1bff690&user=joanofarctan&format=json')
data = response.json()
requests.get(…).text returns the content as a single (unicode) string. The json.load() function however requires a file-like argument.
The solution is rather simple: Just use loads instead of load:
data = json.loads(data)
An even better solution though is to simply call json() on the response object directly. So don’t use .text but .json():
data = requests.get(…).json()
While this uses json.loads itself internally, it hides that implementation detail, so you can just focus on getting the JSON response.
So the process I'm performing seems to make logical sense to me but I keep getting an error. So I have this binary file I'm trying to send to a server (Shapeways to be exact. It's a binary 3d model file) so I go through this process to make it acceptable in a URL
theFile = open(fileloc,'rb')
contents = theFile.read()
b64 = base64.urlsafe_b64encode(contents)
url = urllib.urlencode(b64) # error
The problem is the last line always throws the error
TypeError: not a valid non-string sequence or mapping object
Which doesn't make sense to me as the data is suppose to be encoded for URLs. Is it possible it simply contains other characters that weren't encoded or something like that?
urllib.urlencode takes a sequence of two-element tuples or dictionary into a URL query string (it is basicly excerpt from docstring), but you are passing as a argument just a string.
You can try something like that:
theFile = open(fileloc,'rb')
contents = theFile.read()
b64 = base64.urlsafe_b64encode(contents)
url = urllib.urlencode({'shape': b64})
but all you get inside url variable is encoded params, so you still need actual url. If you don't need low level operations it is better to use requests library:
import requests
import base64
url = 'http://example.com'
r = requests.post(
url=url,
data={'shape':base64.urlsafe_b64encode(open(fileloc, 'rb').read())}
)
If you're just trying to send a file to your server, you shouldn't need to urlencode it. Send it using a POST request.
You can use urllib2 or you could use the requests lib which can simplify things a bit.
This SO thread may help you as well.
I reviewed a handful of the questions related to mine and found this slightly unique. I'm using Python 2.7.1 on OS X 10.7. One more note: I'm more of a hacker than developer.
I snagged the syntax below from Python documentation to try to do a "Pretty Print:"
date = {}
data = urllib2.urlopen(url)
s = json.dumps(data.read(), sort_keys=True, indent=4)
print '\n'.join([l.rstrip() for l in s.splitlines()])
I expected using the rstrip / splitlines commands would expand out the calls like in the example.
Also, not sure if it's relevant, but when tring to pipe the output to python -mjson.tool the reply is No JSON object could be decoded
Here's a snippet of the cURL output I'm trying to parse:
{"data":[{"name":"Site Member","created_at":"2012-07-24T11:22:04-07:00","activity_id":"500ee7cbbaf02xxx8e011e2e",
And so on.
The main objective is to make this mess of data more legible so I can learn from it and start structuring some automatic scraping of data based on arguments. Any guidance to get me from green to successful is a huge help.
Thanks,
mjb
The output of the urllib2.urlopen().read() is a string and needs to be converted to an object first before you can call json.dumps() on it.
Modified code:
date = {}
data = urllib2.urlopen(url)
data_obj = json.loads(data.read())
s = json.dumps(data_obj, sort_keys=True, indent=4)
print s