Here is my code and does anyone have any ideas what is wrong? I open my JSON content directly by browser and it works,
data = requests.get('http://ws.audioscrobbler.com/2.0/?method=library.getartists&api_key=4c22bd45cf5aa6e408e02b3fc1bff690&user=joanofarctan&format=json').text
data = json.load(data)
print type(data)
return data
thanks in advance,
Lin
This error raised because the data is a unicode/str variable, change the second line of your code to resolve your error:
data = json.loads(data)
json.load get a file object in first parameter position and call the read method of this.
Also you can call the json method of the response to fetch data directly:
response = requests.get('http://ws.audioscrobbler.com/2.0/?method=library.getartists&api_key=4c22bd45cf5aa6e408e02b3fc1bff690&user=joanofarctan&format=json')
data = response.json()
requests.get(…).text returns the content as a single (unicode) string. The json.load() function however requires a file-like argument.
The solution is rather simple: Just use loads instead of load:
data = json.loads(data)
An even better solution though is to simply call json() on the response object directly. So don’t use .text but .json():
data = requests.get(…).json()
While this uses json.loads itself internally, it hides that implementation detail, so you can just focus on getting the JSON response.
Related
I'm trying to retrieve pickle data I have uploaded to an openstack object storage using openstacksdk's connection.get_object(container,object), I get a response from it, however the file body is a string, I can even save it to file with the outfile option without issues. However I would like to be able to work with it directly without having to resort to save it to file first and then loading it into pickle.
Simply using pickle's load and loads doesn't work as neither takes string objects. Is there another way to retrieve the data so I can work with the pickled data directly or is there some way to parse to string/set a config parameter on get_object()?
If you are using Python 3 - pickle expects a bytes-like object. The load method takes a file path, and relies on the file type to handle the providing of bytes back into pickle. When you use the loads method you need to provide it a bytes-like object, not a string, so you will need to convert the string to bytes.
Best way to convert string to bytes in Python 3?
EDIT:
I found the solution, for pickled objects or any other files retrieved from openstack with openstacksdk, there are a few ways of dealing with the data without resorting to disk.
First my implemented solution was to use openstack's connection method get_object_raw:
conn = connection(foo,bar, arg**)
pickle.loads(conn.get_object_raw('containerName', 'ObjectName').content)
.get_object_raw returns a response request object with the attribute content which is the binary file content which is the pickle content one can load with pickle.
You could also create a temporary in-memory file with io.BytesIO, and using it as the outfile argument in get_object from the connection object.
I am using Google Cloud Document AI's Form Parser API. After i do the request to the API , I get a response with type google.cloud.documentai.v1beta2.types.document.Document. I tried to write it to JSON using json.dumps() but it gives JSONDecodeError because JSON.dumps() dont know how to serialize object of type google.cloud.documentai.v1beta2.types.document.Document.
I am confused how to convert this to JSON
Any Help Appreciated!
I just found out that the google.cloud.documentai.v1beta2.types.document.Document object inherits from proto.Message, which itself inherits from proto.MessageMeta. You can use the proto.MessageMeta.to_json function to convert the Document object to a json string, like so:
import json
from google.cloud.documentai_v1beta3 import Document
json_string = Document.to_json(document)
dict_obj = json.loads(json_string)
with open("document.json", mode='w') as my_file:
json.dump(dict_obj, my_file)
The source code for proto.Message was a little hard to find, so here it is: https://github.com/googleapis/proto-plus-python/blob/cfd5b6caca3fa9add89d8c69ea620505dd90dd7c/proto/message.py#L330
I solved my problem.
basically you have to write a function that explores the Document object, and then assemble entire JSON yourself by code.
I am currently working on getting a basis on how to write a password generator for a final project in my programming class. The area of code that's currently giving me problems looks like:
`if not isfile('words.txt'):
print('Downloading words.txt ...')
url=str('https://raw.githubusercontent.com/dwyl/english-words/master/words.txt')
with open('words.txt', 'w') as f:
f.write(urlopen(url)).read()`
With the help of a friend, we decided to 'brute force' url to have str(). The error message I get implies that I need to put write() into str() but in doing that I get an error about,
'io.TextIOWrapper' object has no attribute 'str'
. This was originally written in Python 2, however, I managed to get everything working in Python 3.8.0 excluding the block of code above. Thank you in advance for the assistance.
urlopen returns a httpresponse object, not a string. You can only write strings or bytes to a file- not an arbitrary object. You use that object like so (read() returns bytes- if you want a string, have to decode it)
with urlopen(url) as resp:
print(resp.read().decode('utf-8'))
So, to get your code working do-
with open('words.txt', 'w') as f:
with urlopen(url) as resp:
f.write(resp.read().decode('utf-8'))
Using
f.write(urlopen(url)).read()
You are trying to read the file f
Try using
f.write(urlopen(url).read())
Apologies in advance since I'm new to Django (and I've also had to freshen up my Python skills). I'm trying to make a simple example of uploading a file through a form and then printing the rows in my terminal (as a test before doing some actual processing). My views.py contains the following:
def upload_csv(request):
if "GET" == request.method:
return render(request, "portal/csvupload.html")
csv_file = request.FILES["csv_file"]
handle_files(csv_file)
return HttpResponseRedirect(reverse("portal:csvupload"))
def handle_files(csvfile):
csv_reader = csv.reader(csvfile)
for line in csv_reader:
print(line)
Now this returns an error message saying "expected str, bytes or os.PathLike object, not InMemoryUploadedFile", and I'm unsure what's wrong with the code based on the error message? From a Python perspective it looks fine I think, but perhaps it's something to do with the re-direct? Apperciate all answers
request.FILES["csv_file"] is returning an InMemoryUploadedFile object and csv.reader does not know how to handle such an object. I believe you need to call the object's read method: handle_files(csv_file.read()). Note the warning in the documentation: "Be careful with this method: if the uploaded file is huge it can overwhelm your system if you try to read it into memory. You’ll probably want to use chunks() instead; see below."
I am extremely confused after trying a few possible solutions and getting various errors that just lead me in circles. I have a function that will grab a tweet, put it in a dictionary, then write that dictionary to a file using dumps like so:
jsonFile = {}
jsonFile["tweet"] = tweet
jsonFile["language"] = language
with open('jsonOutputfile.txt', 'w') as f:
json.dump(jsonFile, f)
I then have another python file that has a function that will return the value of this jsonOutputfile.txt if I want to use it elsewhere. I do that like so:
with open('jsonOutputfile.txt') as f:
jsonObject = json.load(f)
return jsonObject
This function sits on my localhost. The above two functions that have to do with saving and retrieving the JSON file are separate from the rest of my functions below, as I want them to be.
I have another function that will retrieve the values of the returned status using python requests, like so:
def grab_tweet():
return requests.post("http://gateway:8080/function/twittersend")
and then after grabbing the tweet I want to manipulate it, and I want to do so using the JSON that I should have received from this request.
r = grab_tweet()
data = json.dumps(r.text)
return data.get('tweet')
I want this function above to return just the value that is associated with the tweet key in the JSON that I received from when I saved and loaded it. However, I keep on getting the following error: AttributeError: 'str' object has no attribute 'get' which I am confused about because from my understanding using json.dumps() should create a JSON valid string that I can call get on. Is there an encoding error when I am transferring this to and from a file, or maybe when I am receiving my request?
Any help is appreciated.
EDIT:
Here is a sample of a response from my requests.post when I use r.text, it also looks like there is some Unicode in the response so I put an example at the end of the tweet section. (This also doesn't look like a JSON which is what my question is centered around. There should at least be double quotes and no U's right?):
{u'tweet': u'RT THIS IS THE TWEET BLAH BLAH\u2026', u'language': u'en'}
Use .json() in requests module to get response as JSON
Ex:
data = r.json()
return data.get('tweet')
Note: json.dumps convert your response to a string object
Edit as per comment - Try using the ast module.
Ex:
import ast
data = ast.literal_eval(r.text)
You will need to use the .json() method. See requests' documentation: JSON Response Content
Also, for future reference, rather than do
f.write(json.dumps(jsonFile))
You could simply use:
json.dump(jsonFile, f)
Same with using load instead of loads:
jsonObject = json.load(f)